eagle0/docs/SPARKLE_DELTA_UPDATES.md

# Sparkle Delta Updates Implementation Plan

## Overview

This document outlines the implementation plan for adding delta update support to the Eagle0 macOS auto-update system using Sparkle's BinaryDelta feature.

### Current State
- Full DMG downloads (~200MB) for every update
- `mac_build_handler.go` creates DMG, signs it, uploads to S3, updates appcast.xml
- Keeps last 10 versions in appcast, deletes older DMGs
- Users must download full app even for small changes

### Goals
- Reduce update download size from ~200MB to ~10-30MB (85% reduction)
- Maintain backward compatibility with full DMG downloads
- Automatic fallback for users who are many versions behind

## Sparkle Delta Update Architecture

Sparkle supports binary delta updates through the `<sparkle:deltas>` element in the appcast. When a user updates, Sparkle:
1. Checks if a delta patch exists from their current version to the new version
2. If found, downloads the smaller delta patch instead of the full DMG
3. Applies the patch locally to create the new app version
4. Falls back to full DMG if no matching delta exists

### Appcast XML Structure with Deltas

```xml
<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:sparkle="http://www.andymatuschak.org/xml-namespaces/sparkle">
  <channel>
    <title>Eagle0</title>
    <link>https://assets.eagle0.net/mac/appcast.xml</link>
    <description>Eagle0 game updates</description>
    <language>en</language>
    <item>
      <title>Version 1.0.9615</title>
      <pubDate>Sun, 19 Jan 2026 12:00:00 -0800</pubDate>
      <sparkle:version>9615</sparkle:version>
      <sparkle:shortVersionString>1.0.9615</sparkle:shortVersionString>
      <enclosure
        url="https://assets.eagle0.net/mac/builds/eagle0-1.0.9615.dmg"
        length="200000000"
        type="application/octet-stream"
        sparkle:edSignature="..." />
      <sparkle:deltas>
        <enclosure
          url="https://assets.eagle0.net/mac/deltas/9614-9615.delta"
          sparkle:deltaFrom="9614"
          length="15000000"
          type="application/octet-stream"
          sparkle:edSignature="..." />
        <enclosure
          url="https://assets.eagle0.net/mac/deltas/9613-9615.delta"
          sparkle:deltaFrom="9613"
          length="18000000"
          type="application/octet-stream"
          sparkle:edSignature="..." />
        <enclosure
          url="https://assets.eagle0.net/mac/deltas/9612-9615.delta"
          sparkle:deltaFrom="9612"
          length="22000000"
          type="application/octet-stream"
          sparkle:edSignature="..." />
      </sparkle:deltas>
    </item>
    <!-- older versions... -->
  </channel>
</rss>
```

## Implementation Plan

### Phase 1: Add S3 Utility Functions

**File:** `src/main/go/net/eagle0/util/aws/bucket_basics.go`

Add two new functions to support delta generation:

```go
// ListObjectsWithPrefix returns all object keys matching the given prefix
func (bb BucketBasics) ListObjectsWithPrefix(bucket, prefix string) ([]string, error) {
    var keys []string
    paginator := s3.NewListObjectsV2Paginator(bb.S3Client, &s3.ListObjectsV2Input{
        Bucket: aws.String(bucket),
        Prefix: aws.String(prefix),
    })

    for paginator.HasMorePages() {
        page, err := paginator.NextPage(context.TODO())
        if err != nil {
            return nil, err
        }
        for _, obj := range page.Contents {
            keys = append(keys, *obj.Key)
        }
    }
    return keys, nil
}

// DownloadFile downloads an object to a local file path
func (bb BucketBasics) DownloadFile(bucket, key, localPath string) error {
    result, err := bb.S3Client.GetObject(context.TODO(), &s3.GetObjectInput{
        Bucket: aws.String(bucket),
        Key:    aws.String(key),
    })
    if err != nil {
        return err
    }
    defer result.Body.Close()

    file, err := os.Create(localPath)
    if err != nil {
        return err
    }
    defer file.Close()

    _, err = io.Copy(file, result.Body)
    return err
}
```

### Phase 2: Store App Bundles for Delta Generation

**File:** `src/main/go/net/eagle0/build/mac_build_handler/mac_build_handler.go`

Add storage paths:
```go
var appsRoot = "mac/apps/"      // Zipped app bundles for delta generation
var deltasRoot = "mac/deltas/"  // Delta patches
```

After DMG creation, upload the zipped app bundle:
```go
func uploadAppBundle(bb aws.BucketBasics, appPath string, buildNumber string) error {
    appZipPath := filepath.Join("/tmp", fmt.Sprintf("eagle0-%s.app.zip", buildNumber))

    // Create zip of app bundle using ditto (preserves metadata)
    cmd := exec.Command("ditto", "-c", "-k", "--keepParent", appPath, appZipPath)
    if output, err := cmd.CombinedOutput(); err != nil {
        return fmt.Errorf("failed to zip app: %s: %w", string(output), err)
    }
    defer os.Remove(appZipPath)

    // Upload to S3
    remotePath := appsRoot + fmt.Sprintf("eagle0-%s.app.zip", buildNumber)
    log.Printf("Uploading app bundle to S3: %s", remotePath)
    return bb.UploadFilePublic(bucketName, remotePath, appZipPath)
}
```

### Phase 3: Add Delta XML Structures

**File:** `src/main/go/net/eagle0/build/mac_build_handler/mac_build_handler.go`

Add new structs for delta representation:
```go
// Delta represents a delta patch from a previous version
type Delta struct {
    XMLName   xml.Name `xml:"enclosure"`
    URL       string   `xml:"url,attr"`
    DeltaFrom string   `xml:"sparkle:deltaFrom,attr"`
    Length    int64    `xml:"length,attr"`
    Type      string   `xml:"type,attr"`
    EdSig     string   `xml:"sparkle:edSignature,attr"`
}

// Deltas wraps the sparkle:deltas element
type Deltas struct {
    XMLName xml.Name `xml:"sparkle:deltas"`
    Items   []Delta  `xml:"enclosure"`
}

// Update Item struct to include Deltas
type Item struct {
    Title               string     `xml:"title"`
    PubDate             string     `xml:"pubDate"`
    SparkleVersion      string     `xml:"sparkle:version"`
    SparkleShortVersion string     `xml:"sparkle:shortVersionString"`
    Description         string     `xml:"description,omitempty"`
    Enclosure           Enclosure  `xml:"enclosure"`
    Deltas              *Deltas    `xml:"sparkle:deltas,omitempty"`
}
```

### Phase 4: Generate Delta Patches

**File:** `src/main/go/net/eagle0/build/mac_build_handler/mac_build_handler.go`

```go
// Maximum number of versions to generate deltas from
const maxDeltaVersions = 5

// generateDeltas creates delta patches from previous versions to the new version
func generateDeltas(bb aws.BucketBasics, newBuildNumber string, newAppPath string, privateKeyPath string) ([]Delta, error) {
    var deltas []Delta

    // Ensure BinaryDelta tool is available
    binaryDeltaPath, err := ensureBinaryDelta()
    if err != nil {
        return nil, fmt.Errorf("failed to get BinaryDelta: %w", err)
    }

    // List available app bundles
    appKeys, err := bb.ListObjectsWithPrefix(bucketName, appsRoot+"eagle0-")
    if err != nil {
        log.Printf("Warning: failed to list app bundles: %v", err)
        return deltas, nil // Continue without deltas
    }

    // Parse build numbers from keys and sort descending
    var buildNumbers []string
    for _, key := range appKeys {
        // Extract build number from "mac/apps/eagle0-9614.app.zip"
        base := filepath.Base(key)
        if strings.HasPrefix(base, "eagle0-") && strings.HasSuffix(base, ".app.zip") {
            bn := strings.TrimSuffix(strings.TrimPrefix(base, "eagle0-"), ".app.zip")
            if bn != newBuildNumber {
                buildNumbers = append(buildNumbers, bn)
            }
        }
    }

    // Sort descending (most recent first) and limit to maxDeltaVersions
    sort.Sort(sort.Reverse(sort.StringSlice(buildNumbers)))
    if len(buildNumbers) > maxDeltaVersions {
        buildNumbers = buildNumbers[:maxDeltaVersions]
    }

    // Generate delta from each previous version
    for _, oldBuild := range buildNumbers {
        delta, err := generateSingleDelta(bb, binaryDeltaPath, oldBuild, newBuildNumber, newAppPath, privateKeyPath)
        if err != nil {
            log.Printf("Warning: failed to generate delta from %s: %v", oldBuild, err)
            continue // Skip this delta but continue with others
        }
        deltas = append(deltas, delta)
    }

    return deltas, nil
}

func generateSingleDelta(bb aws.BucketBasics, binaryDeltaPath, oldBuild, newBuild, newAppPath, privateKeyPath string) (Delta, error) {
    // Download old app bundle
    oldAppZipKey := appsRoot + fmt.Sprintf("eagle0-%s.app.zip", oldBuild)
    oldAppZipLocal := filepath.Join("/tmp", fmt.Sprintf("eagle0-%s.app.zip", oldBuild))
    defer os.Remove(oldAppZipLocal)

    if err := bb.DownloadFile(bucketName, oldAppZipKey, oldAppZipLocal); err != nil {
        return Delta{}, fmt.Errorf("failed to download old app: %w", err)
    }

    // Unzip old app
    oldAppDir := filepath.Join("/tmp", fmt.Sprintf("old-app-%s", oldBuild))
    defer os.RemoveAll(oldAppDir)

    cmd := exec.Command("ditto", "-x", "-k", oldAppZipLocal, oldAppDir)
    if output, err := cmd.CombinedOutput(); err != nil {
        return Delta{}, fmt.Errorf("failed to unzip old app: %s: %w", string(output), err)
    }

    oldAppPath := filepath.Join(oldAppDir, "eagle0.app")

    // Generate delta
    deltaPath := filepath.Join("/tmp", fmt.Sprintf("%s-%s.delta", oldBuild, newBuild))
    defer os.Remove(deltaPath)

    cmd = exec.Command(binaryDeltaPath, "create", oldAppPath, newAppPath, deltaPath)
    if output, err := cmd.CombinedOutput(); err != nil {
        return Delta{}, fmt.Errorf("failed to create delta: %s: %w", string(output), err)
    }

    // Get delta size
    deltaSize, err := getFileSize(deltaPath)
    if err != nil {
        return Delta{}, fmt.Errorf("failed to get delta size: %w", err)
    }
    log.Printf("Delta %s->%s size: %d bytes (%.1f MB)", oldBuild, newBuild, deltaSize, float64(deltaSize)/1024/1024)

    // Sign delta
    signature, err := signWithSparkle(deltaPath, privateKeyPath)
    if err != nil {
        return Delta{}, fmt.Errorf("failed to sign delta: %w", err)
    }

    // Upload delta
    deltaKey := deltasRoot + fmt.Sprintf("%s-%s.delta", oldBuild, newBuild)
    if err := bb.UploadFilePublic(bucketName, deltaKey, deltaPath); err != nil {
        return Delta{}, fmt.Errorf("failed to upload delta: %w", err)
    }

    deltaURL := fmt.Sprintf("https://assets.eagle0.net/%s", deltaKey)
    return Delta{
        URL:       deltaURL,
        DeltaFrom: oldBuild,
        Length:    deltaSize,
        Type:      "application/octet-stream",
        EdSig:     signature,
    }, nil
}

func ensureBinaryDelta() (string, error) {
    binaryDeltaPath := "/tmp/sparkle-cache/Sparkle-2.6.4/bin/BinaryDelta"

    if _, err := os.Stat(binaryDeltaPath); os.IsNotExist(err) {
        log.Println("Sparkle BinaryDelta not found, downloading...")
        cmd := exec.Command("bash", "-c", `
            mkdir -p /tmp/sparkle-cache
            curl -sL https://github.com/sparkle-project/Sparkle/releases/download/2.6.4/Sparkle-2.6.4.tar.xz | tar -xJ -C /tmp/sparkle-cache
        `)
        if output, err := cmd.CombinedOutput(); err != nil {
            return "", fmt.Errorf("failed to download Sparkle: %s: %w", string(output), err)
        }
    }

    return binaryDeltaPath, nil
}
```

### Phase 5: Update Main Deploy Flow

**File:** `src/main/go/net/eagle0/build/mac_build_handler/mac_build_handler.go`

Modify `main()` to integrate delta generation:

```go
func main() {
    // ... existing argument parsing ...

    // Create DMG (existing)
    if err := createDMG(appPath, dmgPath, "Eagle0"); err != nil {
        log.Fatalf("Failed to create DMG: %v", err)
    }

    // ... existing DMG upload ...

    if privateKeyPath != "" {
        // Upload app bundle for future delta generation (NEW)
        log.Println("Uploading app bundle for delta generation...")
        if err := uploadAppBundle(bb, appPath, buildNumber); err != nil {
            log.Printf("Warning: failed to upload app bundle: %v", err)
            // Continue - delta generation is optional
        }

        // Generate deltas from previous versions (NEW)
        log.Println("Generating delta patches...")
        deltas, err := generateDeltas(bb, buildNumber, appPath, privateKeyPath)
        if err != nil {
            log.Printf("Warning: failed to generate deltas: %v", err)
        } else {
            log.Printf("Generated %d delta patches", len(deltas))
        }

        // Update appcast with deltas
        log.Println("Updating appcast.xml...")
        appcast, err := fetchAppcast(bb)
        if err != nil {
            log.Fatalf("Failed to fetch appcast: %v", err)
        }

        // Create new item with deltas
        newItem := Item{
            Title:               fmt.Sprintf("Version %s", version),
            PubDate:             time.Now().Format(time.RFC1123Z),
            SparkleVersion:      buildNumber,
            SparkleShortVersion: version,
            Description:         "",
            Enclosure: Enclosure{
                URL:    downloadURL,
                Length: fileSize,
                Type:   "application/octet-stream",
                EdSig:  signature,
            },
        }

        // Add deltas if any were generated
        if len(deltas) > 0 {
            newItem.Deltas = &Deltas{Items: deltas}
        }

        // ... rest of appcast handling ...
    }
}
```

### Phase 6: Cleanup Old Artifacts

**File:** `src/main/go/net/eagle0/build/mac_build_handler/mac_build_handler.go`

When pruning old versions from appcast, also delete associated artifacts:

```go
// In the appcast pruning section, after removing old items:
if len(appcast.Channel.Items) > 10 {
    oldItems := appcast.Channel.Items[10:]
    for _, item := range oldItems {
        oldBuild := item.SparkleVersion

        // Delete old DMG (existing)
        dmgKey := strings.TrimPrefix(item.Enclosure.URL, "https://assets.eagle0.net/")
        log.Printf("Deleting old build: %s", dmgKey)
        bb.DeleteObject(bucketName, dmgKey)

        // Delete old app bundle (NEW)
        appKey := appsRoot + fmt.Sprintf("eagle0-%s.app.zip", oldBuild)
        log.Printf("Deleting old app bundle: %s", appKey)
        bb.DeleteObject(bucketName, appKey)

        // Delete deltas TO this version (NEW)
        deltaKeys, _ := bb.ListObjectsWithPrefix(bucketName, deltasRoot)
        for _, key := range deltaKeys {
            if strings.HasSuffix(key, fmt.Sprintf("-%s.delta", oldBuild)) {
                log.Printf("Deleting old delta: %s", key)
                bb.DeleteObject(bucketName, key)
            }
        }
    }
    appcast.Channel.Items = appcast.Channel.Items[:10]
}
```

## S3 Storage Structure

After implementation, the S3 bucket will have this structure:

```
eagle0-windows/
├── mac/
│   ├── appcast.xml                    # Update feed with delta info
│   ├── builds/                        # Full DMG downloads
│   │   ├── eagle0-1.0.9620.dmg
│   │   ├── eagle0-1.0.9619.dmg
│   │   ├── ...
│   │   └── eagle0-latest.dmg          # Symlink to latest
│   ├── apps/                          # Zipped app bundles (NEW)
│   │   ├── eagle0-9620.app.zip
│   │   ├── eagle0-9619.app.zip
│   │   ├── eagle0-9618.app.zip
│   │   ├── eagle0-9617.app.zip
│   │   └── eagle0-9616.app.zip        # Keep last 5 for delta gen
│   └── deltas/                        # Delta patches (NEW)
│       ├── 9619-9620.delta
│       ├── 9618-9620.delta
│       ├── 9617-9620.delta
│       ├── 9616-9620.delta
│       ├── 9615-9620.delta
│       ├── 9618-9619.delta
│       ├── 9617-9619.delta
│       └── ...
```

## Storage Impact Analysis

### Current Storage (without deltas)
- 10 DMGs × 200MB = **~2GB**

### Estimated Storage (with deltas)
- 10 DMGs × 200MB = 2GB
- 5 app bundles × 150MB = 0.75GB (zip compression)
- ~25 delta files × 20MB avg = 0.5GB
- **Total: ~3.25GB**

### Trade-offs
- **+1.25GB storage** (~60% increase)
- **-170MB per user update** (~85% bandwidth savings)
- Break-even: ~8 user updates to recoup storage cost

## Bandwidth Savings

| Scenario | Without Deltas | With Deltas | Savings |
|----------|---------------|-------------|---------|
| 1 version behind | 200MB | ~15MB | 92% |
| 2 versions behind | 200MB | ~20MB | 90% |
| 3 versions behind | 200MB | ~25MB | 87% |
| 5 versions behind | 200MB | ~35MB | 82% |
| 6+ versions behind | 200MB | 200MB (full) | 0% |

## Migration Strategy

The implementation is backward-compatible and requires no changes to existing clients:

1. **First deploy after implementation:**
   - Stores app bundle for the first time
   - No deltas generated (no previous app bundles exist)
   - Appcast has no `<sparkle:deltas>` element

2. **Second deploy:**
   - Generates delta from previous version
   - Appcast now has `<sparkle:deltas>` with one entry
   - Users on previous version get delta update

3. **Subsequent deploys:**
   - Generate deltas from last 5 versions
   - Users within 5 versions get delta updates
   - Users more than 5 versions behind get full DMG

4. **Client behavior:**
   - Sparkle automatically checks for matching delta
   - Falls back to full DMG if no delta matches
   - No client code changes required

## Error Handling

The implementation handles failures gracefully:

1. **S3 list/download fails:** Skip delta generation, use full DMG
2. **BinaryDelta fails for one version:** Log warning, continue with other versions
3. **Signing fails:** Skip that delta, continue with others
4. **Upload fails:** Skip that delta, continue with others

The deploy never fails due to delta issues - deltas are optional enhancements.

## Verification Plan

### Manual Testing

1. **Deploy version N:**
   - Verify app bundle uploaded to `mac/apps/eagle0-N.app.zip`
   - Verify appcast has no deltas (first deploy)

2. **Deploy version N+1:**
   - Verify delta generated at `mac/deltas/N-(N+1).delta`
   - Verify appcast contains `<sparkle:deltas>` element
   - Verify delta signature is valid

3. **Test update from N to N+1:**
   - Install version N manually
   - Check for updates
   - Monitor download size in Player.log (should be ~15-30MB, not 200MB)
   - Verify app updated successfully

4. **Test fresh install:**
   - Download latest DMG directly
   - Verify installation works normally

5. **Test fallback scenario:**
   - Install a version more than 5 versions behind
   - Update should download full DMG

### Automated Verification

Add to CI workflow (optional):
```yaml
- name: Verify delta generation
  run: |
    # Check app bundle exists
    aws s3 ls s3://eagle0-windows/mac/apps/ | grep eagle0-${BUILD_NUMBER}.app.zip

    # Check deltas exist (after second deploy)
    aws s3 ls s3://eagle0-windows/mac/deltas/ | head -5

    # Verify appcast has deltas
    curl -s https://assets.eagle0.net/mac/appcast.xml | grep "sparkle:deltas"
```

## Security Considerations

1. **All deltas are EdDSA signed:** Same signature verification as full DMG
2. **BinaryDelta is Sparkle's official tool:** Well-audited, production-ready
3. **App bundles in S3 are public:** Same as DMGs, no additional exposure
4. **Cleanup removes old artifacts:** No indefinite storage of old versions

## Future Enhancements

1. **Parallel delta generation:** Generate multiple deltas concurrently
2. **Delta size threshold:** Skip uploading deltas larger than X% of full DMG
3. **Delta metrics:** Track delta download rates vs full DMG
4. **Configurable delta count:** Allow adjusting how many versions to keep