# Sparkle Delta Updates Implementation Plan ## Overview This document outlines the implementation plan for adding delta update support to the Eagle0 macOS auto-update system using Sparkle's BinaryDelta feature. ### Current State - Full DMG downloads (~200MB) for every update - `mac_build_handler.go` creates DMG, signs it, uploads to S3, updates appcast.xml - Keeps last 10 versions in appcast, deletes older DMGs - Users must download full app even for small changes ### Goals - Reduce update download size from ~200MB to ~10-30MB (85% reduction) - Maintain backward compatibility with full DMG downloads - Automatic fallback for users who are many versions behind ## Sparkle Delta Update Architecture Sparkle supports binary delta updates through the `` element in the appcast. When a user updates, Sparkle: 1. Checks if a delta patch exists from their current version to the new version 2. If found, downloads the smaller delta patch instead of the full DMG 3. Applies the patch locally to create the new app version 4. Falls back to full DMG if no matching delta exists ### Appcast XML Structure with Deltas ```xml Eagle0 https://assets.eagle0.net/mac/appcast.xml Eagle0 game updates en Version 1.0.9615 Sun, 19 Jan 2026 12:00:00 -0800 9615 1.0.9615 ``` ## Implementation Plan ### Phase 1: Add S3 Utility Functions **File:** `src/main/go/net/eagle0/util/aws/bucket_basics.go` Add two new functions to support delta generation: ```go // ListObjectsWithPrefix returns all object keys matching the given prefix func (bb BucketBasics) ListObjectsWithPrefix(bucket, prefix string) ([]string, error) { var keys []string paginator := s3.NewListObjectsV2Paginator(bb.S3Client, &s3.ListObjectsV2Input{ Bucket: aws.String(bucket), Prefix: aws.String(prefix), }) for paginator.HasMorePages() { page, err := paginator.NextPage(context.TODO()) if err != nil { return nil, err } for _, obj := range page.Contents { keys = append(keys, *obj.Key) } } return keys, nil } // DownloadFile downloads an object to a local file path func (bb BucketBasics) DownloadFile(bucket, key, localPath string) error { result, err := bb.S3Client.GetObject(context.TODO(), &s3.GetObjectInput{ Bucket: aws.String(bucket), Key: aws.String(key), }) if err != nil { return err } defer result.Body.Close() file, err := os.Create(localPath) if err != nil { return err } defer file.Close() _, err = io.Copy(file, result.Body) return err } ``` ### Phase 2: Store App Bundles for Delta Generation **File:** `src/main/go/net/eagle0/build/mac_build_handler/mac_build_handler.go` Add storage paths: ```go var appsRoot = "mac/apps/" // Zipped app bundles for delta generation var deltasRoot = "mac/deltas/" // Delta patches ``` After DMG creation, upload the zipped app bundle: ```go func uploadAppBundle(bb aws.BucketBasics, appPath string, buildNumber string) error { appZipPath := filepath.Join("/tmp", fmt.Sprintf("eagle0-%s.app.zip", buildNumber)) // Create zip of app bundle using ditto (preserves metadata) cmd := exec.Command("ditto", "-c", "-k", "--keepParent", appPath, appZipPath) if output, err := cmd.CombinedOutput(); err != nil { return fmt.Errorf("failed to zip app: %s: %w", string(output), err) } defer os.Remove(appZipPath) // Upload to S3 remotePath := appsRoot + fmt.Sprintf("eagle0-%s.app.zip", buildNumber) log.Printf("Uploading app bundle to S3: %s", remotePath) return bb.UploadFilePublic(bucketName, remotePath, appZipPath) } ``` ### Phase 3: Add Delta XML Structures **File:** `src/main/go/net/eagle0/build/mac_build_handler/mac_build_handler.go` Add new structs for delta representation: ```go // Delta represents a delta patch from a previous version type Delta struct { XMLName xml.Name `xml:"enclosure"` URL string `xml:"url,attr"` DeltaFrom string `xml:"sparkle:deltaFrom,attr"` Length int64 `xml:"length,attr"` Type string `xml:"type,attr"` EdSig string `xml:"sparkle:edSignature,attr"` } // Deltas wraps the sparkle:deltas element type Deltas struct { XMLName xml.Name `xml:"sparkle:deltas"` Items []Delta `xml:"enclosure"` } // Update Item struct to include Deltas type Item struct { Title string `xml:"title"` PubDate string `xml:"pubDate"` SparkleVersion string `xml:"sparkle:version"` SparkleShortVersion string `xml:"sparkle:shortVersionString"` Description string `xml:"description,omitempty"` Enclosure Enclosure `xml:"enclosure"` Deltas *Deltas `xml:"sparkle:deltas,omitempty"` } ``` ### Phase 4: Generate Delta Patches **File:** `src/main/go/net/eagle0/build/mac_build_handler/mac_build_handler.go` ```go // Maximum number of versions to generate deltas from const maxDeltaVersions = 5 // generateDeltas creates delta patches from previous versions to the new version func generateDeltas(bb aws.BucketBasics, newBuildNumber string, newAppPath string, privateKeyPath string) ([]Delta, error) { var deltas []Delta // Ensure BinaryDelta tool is available binaryDeltaPath, err := ensureBinaryDelta() if err != nil { return nil, fmt.Errorf("failed to get BinaryDelta: %w", err) } // List available app bundles appKeys, err := bb.ListObjectsWithPrefix(bucketName, appsRoot+"eagle0-") if err != nil { log.Printf("Warning: failed to list app bundles: %v", err) return deltas, nil // Continue without deltas } // Parse build numbers from keys and sort descending var buildNumbers []string for _, key := range appKeys { // Extract build number from "mac/apps/eagle0-9614.app.zip" base := filepath.Base(key) if strings.HasPrefix(base, "eagle0-") && strings.HasSuffix(base, ".app.zip") { bn := strings.TrimSuffix(strings.TrimPrefix(base, "eagle0-"), ".app.zip") if bn != newBuildNumber { buildNumbers = append(buildNumbers, bn) } } } // Sort descending (most recent first) and limit to maxDeltaVersions sort.Sort(sort.Reverse(sort.StringSlice(buildNumbers))) if len(buildNumbers) > maxDeltaVersions { buildNumbers = buildNumbers[:maxDeltaVersions] } // Generate delta from each previous version for _, oldBuild := range buildNumbers { delta, err := generateSingleDelta(bb, binaryDeltaPath, oldBuild, newBuildNumber, newAppPath, privateKeyPath) if err != nil { log.Printf("Warning: failed to generate delta from %s: %v", oldBuild, err) continue // Skip this delta but continue with others } deltas = append(deltas, delta) } return deltas, nil } func generateSingleDelta(bb aws.BucketBasics, binaryDeltaPath, oldBuild, newBuild, newAppPath, privateKeyPath string) (Delta, error) { // Download old app bundle oldAppZipKey := appsRoot + fmt.Sprintf("eagle0-%s.app.zip", oldBuild) oldAppZipLocal := filepath.Join("/tmp", fmt.Sprintf("eagle0-%s.app.zip", oldBuild)) defer os.Remove(oldAppZipLocal) if err := bb.DownloadFile(bucketName, oldAppZipKey, oldAppZipLocal); err != nil { return Delta{}, fmt.Errorf("failed to download old app: %w", err) } // Unzip old app oldAppDir := filepath.Join("/tmp", fmt.Sprintf("old-app-%s", oldBuild)) defer os.RemoveAll(oldAppDir) cmd := exec.Command("ditto", "-x", "-k", oldAppZipLocal, oldAppDir) if output, err := cmd.CombinedOutput(); err != nil { return Delta{}, fmt.Errorf("failed to unzip old app: %s: %w", string(output), err) } oldAppPath := filepath.Join(oldAppDir, "eagle0.app") // Generate delta deltaPath := filepath.Join("/tmp", fmt.Sprintf("%s-%s.delta", oldBuild, newBuild)) defer os.Remove(deltaPath) cmd = exec.Command(binaryDeltaPath, "create", oldAppPath, newAppPath, deltaPath) if output, err := cmd.CombinedOutput(); err != nil { return Delta{}, fmt.Errorf("failed to create delta: %s: %w", string(output), err) } // Get delta size deltaSize, err := getFileSize(deltaPath) if err != nil { return Delta{}, fmt.Errorf("failed to get delta size: %w", err) } log.Printf("Delta %s->%s size: %d bytes (%.1f MB)", oldBuild, newBuild, deltaSize, float64(deltaSize)/1024/1024) // Sign delta signature, err := signWithSparkle(deltaPath, privateKeyPath) if err != nil { return Delta{}, fmt.Errorf("failed to sign delta: %w", err) } // Upload delta deltaKey := deltasRoot + fmt.Sprintf("%s-%s.delta", oldBuild, newBuild) if err := bb.UploadFilePublic(bucketName, deltaKey, deltaPath); err != nil { return Delta{}, fmt.Errorf("failed to upload delta: %w", err) } deltaURL := fmt.Sprintf("https://assets.eagle0.net/%s", deltaKey) return Delta{ URL: deltaURL, DeltaFrom: oldBuild, Length: deltaSize, Type: "application/octet-stream", EdSig: signature, }, nil } func ensureBinaryDelta() (string, error) { binaryDeltaPath := "/tmp/sparkle-cache/Sparkle-2.6.4/bin/BinaryDelta" if _, err := os.Stat(binaryDeltaPath); os.IsNotExist(err) { log.Println("Sparkle BinaryDelta not found, downloading...") cmd := exec.Command("bash", "-c", ` mkdir -p /tmp/sparkle-cache curl -sL https://github.com/sparkle-project/Sparkle/releases/download/2.6.4/Sparkle-2.6.4.tar.xz | tar -xJ -C /tmp/sparkle-cache `) if output, err := cmd.CombinedOutput(); err != nil { return "", fmt.Errorf("failed to download Sparkle: %s: %w", string(output), err) } } return binaryDeltaPath, nil } ``` ### Phase 5: Update Main Deploy Flow **File:** `src/main/go/net/eagle0/build/mac_build_handler/mac_build_handler.go` Modify `main()` to integrate delta generation: ```go func main() { // ... existing argument parsing ... // Create DMG (existing) if err := createDMG(appPath, dmgPath, "Eagle0"); err != nil { log.Fatalf("Failed to create DMG: %v", err) } // ... existing DMG upload ... if privateKeyPath != "" { // Upload app bundle for future delta generation (NEW) log.Println("Uploading app bundle for delta generation...") if err := uploadAppBundle(bb, appPath, buildNumber); err != nil { log.Printf("Warning: failed to upload app bundle: %v", err) // Continue - delta generation is optional } // Generate deltas from previous versions (NEW) log.Println("Generating delta patches...") deltas, err := generateDeltas(bb, buildNumber, appPath, privateKeyPath) if err != nil { log.Printf("Warning: failed to generate deltas: %v", err) } else { log.Printf("Generated %d delta patches", len(deltas)) } // Update appcast with deltas log.Println("Updating appcast.xml...") appcast, err := fetchAppcast(bb) if err != nil { log.Fatalf("Failed to fetch appcast: %v", err) } // Create new item with deltas newItem := Item{ Title: fmt.Sprintf("Version %s", version), PubDate: time.Now().Format(time.RFC1123Z), SparkleVersion: buildNumber, SparkleShortVersion: version, Description: "", Enclosure: Enclosure{ URL: downloadURL, Length: fileSize, Type: "application/octet-stream", EdSig: signature, }, } // Add deltas if any were generated if len(deltas) > 0 { newItem.Deltas = &Deltas{Items: deltas} } // ... rest of appcast handling ... } } ``` ### Phase 6: Cleanup Old Artifacts **File:** `src/main/go/net/eagle0/build/mac_build_handler/mac_build_handler.go` When pruning old versions from appcast, also delete associated artifacts: ```go // In the appcast pruning section, after removing old items: if len(appcast.Channel.Items) > 10 { oldItems := appcast.Channel.Items[10:] for _, item := range oldItems { oldBuild := item.SparkleVersion // Delete old DMG (existing) dmgKey := strings.TrimPrefix(item.Enclosure.URL, "https://assets.eagle0.net/") log.Printf("Deleting old build: %s", dmgKey) bb.DeleteObject(bucketName, dmgKey) // Delete old app bundle (NEW) appKey := appsRoot + fmt.Sprintf("eagle0-%s.app.zip", oldBuild) log.Printf("Deleting old app bundle: %s", appKey) bb.DeleteObject(bucketName, appKey) // Delete deltas TO this version (NEW) deltaKeys, _ := bb.ListObjectsWithPrefix(bucketName, deltasRoot) for _, key := range deltaKeys { if strings.HasSuffix(key, fmt.Sprintf("-%s.delta", oldBuild)) { log.Printf("Deleting old delta: %s", key) bb.DeleteObject(bucketName, key) } } } appcast.Channel.Items = appcast.Channel.Items[:10] } ``` ## S3 Storage Structure After implementation, the S3 bucket will have this structure: ``` eagle0-windows/ ├── mac/ │ ├── appcast.xml # Update feed with delta info │ ├── builds/ # Full DMG downloads │ │ ├── eagle0-1.0.9620.dmg │ │ ├── eagle0-1.0.9619.dmg │ │ ├── ... │ │ └── eagle0-latest.dmg # Symlink to latest │ ├── apps/ # Zipped app bundles (NEW) │ │ ├── eagle0-9620.app.zip │ │ ├── eagle0-9619.app.zip │ │ ├── eagle0-9618.app.zip │ │ ├── eagle0-9617.app.zip │ │ └── eagle0-9616.app.zip # Keep last 5 for delta gen │ └── deltas/ # Delta patches (NEW) │ ├── 9619-9620.delta │ ├── 9618-9620.delta │ ├── 9617-9620.delta │ ├── 9616-9620.delta │ ├── 9615-9620.delta │ ├── 9618-9619.delta │ ├── 9617-9619.delta │ └── ... ``` ## Storage Impact Analysis ### Current Storage (without deltas) - 10 DMGs × 200MB = **~2GB** ### Estimated Storage (with deltas) - 10 DMGs × 200MB = 2GB - 5 app bundles × 150MB = 0.75GB (zip compression) - ~25 delta files × 20MB avg = 0.5GB - **Total: ~3.25GB** ### Trade-offs - **+1.25GB storage** (~60% increase) - **-170MB per user update** (~85% bandwidth savings) - Break-even: ~8 user updates to recoup storage cost ## Bandwidth Savings | Scenario | Without Deltas | With Deltas | Savings | |----------|---------------|-------------|---------| | 1 version behind | 200MB | ~15MB | 92% | | 2 versions behind | 200MB | ~20MB | 90% | | 3 versions behind | 200MB | ~25MB | 87% | | 5 versions behind | 200MB | ~35MB | 82% | | 6+ versions behind | 200MB | 200MB (full) | 0% | ## Migration Strategy The implementation is backward-compatible and requires no changes to existing clients: 1. **First deploy after implementation:** - Stores app bundle for the first time - No deltas generated (no previous app bundles exist) - Appcast has no `` element 2. **Second deploy:** - Generates delta from previous version - Appcast now has `` with one entry - Users on previous version get delta update 3. **Subsequent deploys:** - Generate deltas from last 5 versions - Users within 5 versions get delta updates - Users more than 5 versions behind get full DMG 4. **Client behavior:** - Sparkle automatically checks for matching delta - Falls back to full DMG if no delta matches - No client code changes required ## Error Handling The implementation handles failures gracefully: 1. **S3 list/download fails:** Skip delta generation, use full DMG 2. **BinaryDelta fails for one version:** Log warning, continue with other versions 3. **Signing fails:** Skip that delta, continue with others 4. **Upload fails:** Skip that delta, continue with others The deploy never fails due to delta issues - deltas are optional enhancements. ## Verification Plan ### Manual Testing 1. **Deploy version N:** - Verify app bundle uploaded to `mac/apps/eagle0-N.app.zip` - Verify appcast has no deltas (first deploy) 2. **Deploy version N+1:** - Verify delta generated at `mac/deltas/N-(N+1).delta` - Verify appcast contains `` element - Verify delta signature is valid 3. **Test update from N to N+1:** - Install version N manually - Check for updates - Monitor download size in Player.log (should be ~15-30MB, not 200MB) - Verify app updated successfully 4. **Test fresh install:** - Download latest DMG directly - Verify installation works normally 5. **Test fallback scenario:** - Install a version more than 5 versions behind - Update should download full DMG ### Automated Verification Add to CI workflow (optional): ```yaml - name: Verify delta generation run: | # Check app bundle exists aws s3 ls s3://eagle0-windows/mac/apps/ | grep eagle0-${BUILD_NUMBER}.app.zip # Check deltas exist (after second deploy) aws s3 ls s3://eagle0-windows/mac/deltas/ | head -5 # Verify appcast has deltas curl -s https://assets.eagle0.net/mac/appcast.xml | grep "sparkle:deltas" ``` ## Security Considerations 1. **All deltas are EdDSA signed:** Same signature verification as full DMG 2. **BinaryDelta is Sparkle's official tool:** Well-audited, production-ready 3. **App bundles in S3 are public:** Same as DMGs, no additional exposure 4. **Cleanup removes old artifacts:** No indefinite storage of old versions ## Future Enhancements 1. **Parallel delta generation:** Generate multiple deltas concurrently 2. **Delta size threshold:** Skip uploading deltas larger than X% of full DMG 3. **Delta metrics:** Track delta download rates vs full DMG 4. **Configurable delta count:** Allow adjusting how many versions to keep