eagle0/docs/SPARKLE_DELTA_UPDATES.md

Sparkle Delta Updates Implementation Plan

Overview

This document outlines the implementation plan for adding delta update support to the Eagle0 macOS auto-update system using Sparkle's BinaryDelta feature.

Current State

• Full DMG downloads (~200MB) for every update
• mac_build_handler.go creates DMG, signs it, uploads to S3, updates appcast.xml
• Keeps last 10 versions in appcast, deletes older DMGs
• Users must download full app even for small changes

Goals

• Reduce update download size from ~200MB to ~10-30MB (85% reduction)
• Maintain backward compatibility with full DMG downloads
• Automatic fallback for users who are many versions behind

Sparkle Delta Update Architecture

Sparkle supports binary delta updates through the <sparkle:deltas> element in the appcast. When a user updates, Sparkle:

1. Checks if a delta patch exists from their current version to the new version
2. If found, downloads the smaller delta patch instead of the full DMG
3. Applies the patch locally to create the new app version
4. Falls back to full DMG if no matching delta exists

Appcast XML Structure with Deltas

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:sparkle="http://www.andymatuschak.org/xml-namespaces/sparkle">
  <channel>
    <title>Eagle0</title>
    <link>https://assets.eagle0.net/mac/appcast.xml</link>
    <description>Eagle0 game updates</description>
    <language>en</language>
    <item>
      <title>Version 1.0.9615</title>
      <pubDate>Sun, 19 Jan 2026 12:00:00 -0800</pubDate>
      <sparkle:version>9615</sparkle:version>
      <sparkle:shortVersionString>1.0.9615</sparkle:shortVersionString>
      <enclosure
        url="https://assets.eagle0.net/mac/builds/eagle0-1.0.9615.dmg"
        length="200000000"
        type="application/octet-stream"
        sparkle:edSignature="..." />
      <sparkle:deltas>
        <enclosure
          url="https://assets.eagle0.net/mac/deltas/9614-9615.delta"
          sparkle:deltaFrom="9614"
          length="15000000"
          type="application/octet-stream"
          sparkle:edSignature="..." />
        <enclosure
          url="https://assets.eagle0.net/mac/deltas/9613-9615.delta"
          sparkle:deltaFrom="9613"
          length="18000000"
          type="application/octet-stream"
          sparkle:edSignature="..." />
        <enclosure
          url="https://assets.eagle0.net/mac/deltas/9612-9615.delta"
          sparkle:deltaFrom="9612"
          length="22000000"
          type="application/octet-stream"
          sparkle:edSignature="..." />
      </sparkle:deltas>
    </item>
    <!-- older versions... -->
  </channel>
</rss>

Implementation Plan

Phase 1: Add S3 Utility Functions

File: src/main/go/net/eagle0/util/aws/bucket_basics.go

Add two new functions to support delta generation:

// ListObjectsWithPrefix returns all object keys matching the given prefix
func (bb BucketBasics) ListObjectsWithPrefix(bucket, prefix string) ([]string, error) {
    var keys []string
    paginator := s3.NewListObjectsV2Paginator(bb.S3Client, &s3.ListObjectsV2Input{
        Bucket: aws.String(bucket),
        Prefix: aws.String(prefix),
    })

    for paginator.HasMorePages() {
        page, err := paginator.NextPage(context.TODO())
        if err != nil {
            return nil, err
        }
        for _, obj := range page.Contents {
            keys = append(keys, *obj.Key)
        }
    }
    return keys, nil
}

// DownloadFile downloads an object to a local file path
func (bb BucketBasics) DownloadFile(bucket, key, localPath string) error {
    result, err := bb.S3Client.GetObject(context.TODO(), &s3.GetObjectInput{
        Bucket: aws.String(bucket),
        Key:    aws.String(key),
    })
    if err != nil {
        return err
    }
    defer result.Body.Close()

    file, err := os.Create(localPath)
    if err != nil {
        return err
    }
    defer file.Close()

    _, err = io.Copy(file, result.Body)
    return err
}

Phase 2: Store App Bundles for Delta Generation

File: src/main/go/net/eagle0/build/mac_build_handler/mac_build_handler.go

Add storage paths:

var appsRoot = "mac/apps/"      // Zipped app bundles for delta generation
var deltasRoot = "mac/deltas/"  // Delta patches

After DMG creation, upload the zipped app bundle:

func uploadAppBundle(bb aws.BucketBasics, appPath string, buildNumber string) error {
    appZipPath := filepath.Join("/tmp", fmt.Sprintf("eagle0-%s.app.zip", buildNumber))

    // Create zip of app bundle using ditto (preserves metadata)
    cmd := exec.Command("ditto", "-c", "-k", "--keepParent", appPath, appZipPath)
    if output, err := cmd.CombinedOutput(); err != nil {
        return fmt.Errorf("failed to zip app: %s: %w", string(output), err)
    }
    defer os.Remove(appZipPath)

    // Upload to S3
    remotePath := appsRoot + fmt.Sprintf("eagle0-%s.app.zip", buildNumber)
    log.Printf("Uploading app bundle to S3: %s", remotePath)
    return bb.UploadFilePublic(bucketName, remotePath, appZipPath)
}

Phase 3: Add Delta XML Structures

File: src/main/go/net/eagle0/build/mac_build_handler/mac_build_handler.go

Add new structs for delta representation:

// Delta represents a delta patch from a previous version
type Delta struct {
    XMLName   xml.Name `xml:"enclosure"`
    URL       string   `xml:"url,attr"`
    DeltaFrom string   `xml:"sparkle:deltaFrom,attr"`
    Length    int64    `xml:"length,attr"`
    Type      string   `xml:"type,attr"`
    EdSig     string   `xml:"sparkle:edSignature,attr"`
}

// Deltas wraps the sparkle:deltas element
type Deltas struct {
    XMLName xml.Name `xml:"sparkle:deltas"`
    Items   []Delta  `xml:"enclosure"`
}

// Update Item struct to include Deltas
type Item struct {
    Title               string     `xml:"title"`
    PubDate             string     `xml:"pubDate"`
    SparkleVersion      string     `xml:"sparkle:version"`
    SparkleShortVersion string     `xml:"sparkle:shortVersionString"`
    Description         string     `xml:"description,omitempty"`
    Enclosure           Enclosure  `xml:"enclosure"`
    Deltas              *Deltas    `xml:"sparkle:deltas,omitempty"`
}

Phase 4: Generate Delta Patches

File: src/main/go/net/eagle0/build/mac_build_handler/mac_build_handler.go

// Maximum number of versions to generate deltas from
const maxDeltaVersions = 5

// generateDeltas creates delta patches from previous versions to the new version
func generateDeltas(bb aws.BucketBasics, newBuildNumber string, newAppPath string, privateKeyPath string) ([]Delta, error) {
    var deltas []Delta

    // Ensure BinaryDelta tool is available
    binaryDeltaPath, err := ensureBinaryDelta()
    if err != nil {
        return nil, fmt.Errorf("failed to get BinaryDelta: %w", err)
    }

    // List available app bundles
    appKeys, err := bb.ListObjectsWithPrefix(bucketName, appsRoot+"eagle0-")
    if err != nil {
        log.Printf("Warning: failed to list app bundles: %v", err)
        return deltas, nil // Continue without deltas
    }

    // Parse build numbers from keys and sort descending
    var buildNumbers []string
    for _, key := range appKeys {
        // Extract build number from "mac/apps/eagle0-9614.app.zip"
        base := filepath.Base(key)
        if strings.HasPrefix(base, "eagle0-") && strings.HasSuffix(base, ".app.zip") {
            bn := strings.TrimSuffix(strings.TrimPrefix(base, "eagle0-"), ".app.zip")
            if bn != newBuildNumber {
                buildNumbers = append(buildNumbers, bn)
            }
        }
    }

    // Sort descending (most recent first) and limit to maxDeltaVersions
    sort.Sort(sort.Reverse(sort.StringSlice(buildNumbers)))
    if len(buildNumbers) > maxDeltaVersions {
        buildNumbers = buildNumbers[:maxDeltaVersions]
    }

    // Generate delta from each previous version
    for _, oldBuild := range buildNumbers {
        delta, err := generateSingleDelta(bb, binaryDeltaPath, oldBuild, newBuildNumber, newAppPath, privateKeyPath)
        if err != nil {
            log.Printf("Warning: failed to generate delta from %s: %v", oldBuild, err)
            continue // Skip this delta but continue with others
        }
        deltas = append(deltas, delta)
    }

    return deltas, nil
}

func generateSingleDelta(bb aws.BucketBasics, binaryDeltaPath, oldBuild, newBuild, newAppPath, privateKeyPath string) (Delta, error) {
    // Download old app bundle
    oldAppZipKey := appsRoot + fmt.Sprintf("eagle0-%s.app.zip", oldBuild)
    oldAppZipLocal := filepath.Join("/tmp", fmt.Sprintf("eagle0-%s.app.zip", oldBuild))
    defer os.Remove(oldAppZipLocal)

    if err := bb.DownloadFile(bucketName, oldAppZipKey, oldAppZipLocal); err != nil {
        return Delta{}, fmt.Errorf("failed to download old app: %w", err)
    }

    // Unzip old app
    oldAppDir := filepath.Join("/tmp", fmt.Sprintf("old-app-%s", oldBuild))
    defer os.RemoveAll(oldAppDir)

    cmd := exec.Command("ditto", "-x", "-k", oldAppZipLocal, oldAppDir)
    if output, err := cmd.CombinedOutput(); err != nil {
        return Delta{}, fmt.Errorf("failed to unzip old app: %s: %w", string(output), err)
    }

    oldAppPath := filepath.Join(oldAppDir, "eagle0.app")

    // Generate delta
    deltaPath := filepath.Join("/tmp", fmt.Sprintf("%s-%s.delta", oldBuild, newBuild))
    defer os.Remove(deltaPath)

    cmd = exec.Command(binaryDeltaPath, "create", oldAppPath, newAppPath, deltaPath)
    if output, err := cmd.CombinedOutput(); err != nil {
        return Delta{}, fmt.Errorf("failed to create delta: %s: %w", string(output), err)
    }

    // Get delta size
    deltaSize, err := getFileSize(deltaPath)
    if err != nil {
        return Delta{}, fmt.Errorf("failed to get delta size: %w", err)
    }
    log.Printf("Delta %s->%s size: %d bytes (%.1f MB)", oldBuild, newBuild, deltaSize, float64(deltaSize)/1024/1024)

    // Sign delta
    signature, err := signWithSparkle(deltaPath, privateKeyPath)
    if err != nil {
        return Delta{}, fmt.Errorf("failed to sign delta: %w", err)
    }

    // Upload delta
    deltaKey := deltasRoot + fmt.Sprintf("%s-%s.delta", oldBuild, newBuild)
    if err := bb.UploadFilePublic(bucketName, deltaKey, deltaPath); err != nil {
        return Delta{}, fmt.Errorf("failed to upload delta: %w", err)
    }

    deltaURL := fmt.Sprintf("https://assets.eagle0.net/%s", deltaKey)
    return Delta{
        URL:       deltaURL,
        DeltaFrom: oldBuild,
        Length:    deltaSize,
        Type:      "application/octet-stream",
        EdSig:     signature,
    }, nil
}

func ensureBinaryDelta() (string, error) {
    binaryDeltaPath := "/tmp/sparkle-cache/Sparkle-2.6.4/bin/BinaryDelta"

    if _, err := os.Stat(binaryDeltaPath); os.IsNotExist(err) {
        log.Println("Sparkle BinaryDelta not found, downloading...")
        cmd := exec.Command("bash", "-c", `
            mkdir -p /tmp/sparkle-cache
            curl -sL https://github.com/sparkle-project/Sparkle/releases/download/2.6.4/Sparkle-2.6.4.tar.xz | tar -xJ -C /tmp/sparkle-cache
        `)
        if output, err := cmd.CombinedOutput(); err != nil {
            return "", fmt.Errorf("failed to download Sparkle: %s: %w", string(output), err)
        }
    }

    return binaryDeltaPath, nil
}

Phase 5: Update Main Deploy Flow

File: src/main/go/net/eagle0/build/mac_build_handler/mac_build_handler.go

Modify main() to integrate delta generation:

func main() {
    // ... existing argument parsing ...

    // Create DMG (existing)
    if err := createDMG(appPath, dmgPath, "Eagle0"); err != nil {
        log.Fatalf("Failed to create DMG: %v", err)
    }

    // ... existing DMG upload ...

    if privateKeyPath != "" {
        // Upload app bundle for future delta generation (NEW)
        log.Println("Uploading app bundle for delta generation...")
        if err := uploadAppBundle(bb, appPath, buildNumber); err != nil {
            log.Printf("Warning: failed to upload app bundle: %v", err)
            // Continue - delta generation is optional
        }

        // Generate deltas from previous versions (NEW)
        log.Println("Generating delta patches...")
        deltas, err := generateDeltas(bb, buildNumber, appPath, privateKeyPath)
        if err != nil {
            log.Printf("Warning: failed to generate deltas: %v", err)
        } else {
            log.Printf("Generated %d delta patches", len(deltas))
        }

        // Update appcast with deltas
        log.Println("Updating appcast.xml...")
        appcast, err := fetchAppcast(bb)
        if err != nil {
            log.Fatalf("Failed to fetch appcast: %v", err)
        }

        // Create new item with deltas
        newItem := Item{
            Title:               fmt.Sprintf("Version %s", version),
            PubDate:             time.Now().Format(time.RFC1123Z),
            SparkleVersion:      buildNumber,
            SparkleShortVersion: version,
            Description:         "",
            Enclosure: Enclosure{
                URL:    downloadURL,
                Length: fileSize,
                Type:   "application/octet-stream",
                EdSig:  signature,
            },
        }

        // Add deltas if any were generated
        if len(deltas) > 0 {
            newItem.Deltas = &Deltas{Items: deltas}
        }

        // ... rest of appcast handling ...
    }
}

Phase 6: Cleanup Old Artifacts

File: src/main/go/net/eagle0/build/mac_build_handler/mac_build_handler.go

When pruning old versions from appcast, also delete associated artifacts:

// In the appcast pruning section, after removing old items:
if len(appcast.Channel.Items) > 10 {
    oldItems := appcast.Channel.Items[10:]
    for _, item := range oldItems {
        oldBuild := item.SparkleVersion

        // Delete old DMG (existing)
        dmgKey := strings.TrimPrefix(item.Enclosure.URL, "https://assets.eagle0.net/")
        log.Printf("Deleting old build: %s", dmgKey)
        bb.DeleteObject(bucketName, dmgKey)

        // Delete old app bundle (NEW)
        appKey := appsRoot + fmt.Sprintf("eagle0-%s.app.zip", oldBuild)
        log.Printf("Deleting old app bundle: %s", appKey)
        bb.DeleteObject(bucketName, appKey)

        // Delete deltas TO this version (NEW)
        deltaKeys, _ := bb.ListObjectsWithPrefix(bucketName, deltasRoot)
        for _, key := range deltaKeys {
            if strings.HasSuffix(key, fmt.Sprintf("-%s.delta", oldBuild)) {
                log.Printf("Deleting old delta: %s", key)
                bb.DeleteObject(bucketName, key)
            }
        }
    }
    appcast.Channel.Items = appcast.Channel.Items[:10]
}

S3 Storage Structure

After implementation, the S3 bucket will have this structure:

eagle0-windows/
├── mac/
│   ├── appcast.xml                    # Update feed with delta info
│   ├── builds/                        # Full DMG downloads
│   │   ├── eagle0-1.0.9620.dmg
│   │   ├── eagle0-1.0.9619.dmg
│   │   ├── ...
│   │   └── eagle0-latest.dmg          # Symlink to latest
│   ├── apps/                          # Zipped app bundles (NEW)
│   │   ├── eagle0-9620.app.zip
│   │   ├── eagle0-9619.app.zip
│   │   ├── eagle0-9618.app.zip
│   │   ├── eagle0-9617.app.zip
│   │   └── eagle0-9616.app.zip        # Keep last 5 for delta gen
│   └── deltas/                        # Delta patches (NEW)
│       ├── 9619-9620.delta
│       ├── 9618-9620.delta
│       ├── 9617-9620.delta
│       ├── 9616-9620.delta
│       ├── 9615-9620.delta
│       ├── 9618-9619.delta
│       ├── 9617-9619.delta
│       └── ...

Storage Impact Analysis

Current Storage (without deltas)

• 10 DMGs × 200MB = ~2GB

Estimated Storage (with deltas)

• 10 DMGs × 200MB = 2GB
• 5 app bundles × 150MB = 0.75GB (zip compression)
• ~25 delta files × 20MB avg = 0.5GB
• Total: ~3.25GB

Trade-offs

• +1.25GB storage (~60% increase)
• -170MB per user update (~85% bandwidth savings)
• Break-even: ~8 user updates to recoup storage cost

Bandwidth Savings

Scenario	Without Deltas	With Deltas	Savings
1 version behind	200MB	~15MB	92%
2 versions behind	200MB	~20MB	90%
3 versions behind	200MB	~25MB	87%
5 versions behind	200MB	~35MB	82%
6+ versions behind	200MB	200MB (full)	0%

Migration Strategy

The implementation is backward-compatible and requires no changes to existing clients:

1. First deploy after implementation:

   • Stores app bundle for the first time
   • No deltas generated (no previous app bundles exist)
   • Appcast has no <sparkle:deltas> element

2. Second deploy:

   • Generates delta from previous version
   • Appcast now has <sparkle:deltas> with one entry
   • Users on previous version get delta update

3. Subsequent deploys:

   • Generate deltas from last 5 versions
   • Users within 5 versions get delta updates
   • Users more than 5 versions behind get full DMG

4. Client behavior:

   • Sparkle automatically checks for matching delta
   • Falls back to full DMG if no delta matches
   • No client code changes required

Error Handling

The implementation handles failures gracefully:

1. S3 list/download fails: Skip delta generation, use full DMG
2. BinaryDelta fails for one version: Log warning, continue with other versions
3. Signing fails: Skip that delta, continue with others
4. Upload fails: Skip that delta, continue with others

The deploy never fails due to delta issues - deltas are optional enhancements.

Verification Plan

Manual Testing

1. Deploy version N:

   • Verify app bundle uploaded to mac/apps/eagle0-N.app.zip
   • Verify appcast has no deltas (first deploy)

2. Deploy version N+1:

   • Verify delta generated at mac/deltas/N-(N+1).delta
   • Verify appcast contains <sparkle:deltas> element
   • Verify delta signature is valid

3. Test update from N to N+1:

   • Install version N manually
   • Check for updates
   • Monitor download size in Player.log (should be ~15-30MB, not 200MB)
   • Verify app updated successfully

4. Test fresh install:

   • Download latest DMG directly
   • Verify installation works normally

5. Test fallback scenario:

   • Install a version more than 5 versions behind
   • Update should download full DMG

Automated Verification

Add to CI workflow (optional):

- name: Verify delta generation
  run: |
    # Check app bundle exists
    aws s3 ls s3://eagle0-windows/mac/apps/ | grep eagle0-${BUILD_NUMBER}.app.zip

    # Check deltas exist (after second deploy)
    aws s3 ls s3://eagle0-windows/mac/deltas/ | head -5

    # Verify appcast has deltas
    curl -s https://assets.eagle0.net/mac/appcast.xml | grep "sparkle:deltas"

Security Considerations

1. All deltas are EdDSA signed: Same signature verification as full DMG
2. BinaryDelta is Sparkle's official tool: Well-audited, production-ready
3. App bundles in S3 are public: Same as DMGs, no additional exposure
4. Cleanup removes old artifacts: No indefinite storage of old versions

Future Enhancements

1. Parallel delta generation: Generate multiple deltas concurrently
2. Delta size threshold: Skip uploading deltas larger than X% of full DMG
3. Delta metrics: Track delta download rates vs full DMG
4. Configurable delta count: Allow adjusting how many versions to keep