admin/eagle0
1
0
Fork 0
mirror of https://github.com/nolen777/eagle0.git synced 2026-07-28 20:55:44 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add Sparkle delta updates implementation plan (#5469)

Browse Source 
Documents the design for implementing delta patches in the Mac auto-update
system to reduce download sizes from ~200MB to ~10-30MB per update.

Covers:
- Appcast XML structure with sparkle:deltas
- S3 storage layout for app bundles and deltas
- BinaryDelta tool usage
- Migration strategy and error handling
- Storage impact analysis (~3.25GB total for 85% bandwidth savings)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
admin
2026-01-19 22:35:53 -08:00
committed by GitHub
co-authored by Claude Opus 4.5
parent 75f08812db
commit 8975514e7c
1 changed files with 568 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/SPARKLE_DELTA_UPDATES.md
+568
View File
@@ -0,0 +1,568 @@
# Sparkle Delta Updates Implementation Plan
## Overview
This document outlines the implementation plan for adding delta update support to the Eagle0 macOS auto-update system using Sparkle's BinaryDelta feature.
### Current State
- Full DMG downloads (~200MB) for every update
- `mac_build_handler.go` creates DMG, signs it, uploads to S3, updates appcast.xml
- Keeps last 10 versions in appcast, deletes older DMGs
- Users must download full app even for small changes
### Goals
- Reduce update download size from ~200MB to ~10-30MB (85% reduction)
- Maintain backward compatibility with full DMG downloads
- Automatic fallback for users who are many versions behind
## Sparkle Delta Update Architecture
Sparkle supports binary delta updates through the `<sparkle:deltas>` element in the appcast. When a user updates, Sparkle:
1. Checks if a delta patch exists from their current version to the new version
2. If found, downloads the smaller delta patch instead of the full DMG
3. Applies the patch locally to create the new app version
4. Falls back to full DMG if no matching delta exists
### Appcast XML Structure with Deltas
```xml
<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:sparkle="http://www.andymatuschak.org/xml-namespaces/sparkle">
<channel>
<title>Eagle0</title>
<link>https://assets.eagle0.net/mac/appcast.xml</link>
<description>Eagle0 game updates</description>
<language>en</language>
<item>
<title>Version 1.0.9615</title>
<pubDate>Sun, 19 Jan 2026 12:00:00 -0800</pubDate>
<sparkle:version>9615</sparkle:version>
<sparkle:shortVersionString>1.0.9615</sparkle:shortVersionString>
<enclosure
url="https://assets.eagle0.net/mac/builds/eagle0-1.0.9615.dmg"
length="200000000"
type="application/octet-stream"
sparkle:edSignature="..." />
<sparkle:deltas>
<enclosure
url="https://assets.eagle0.net/mac/deltas/9614-9615.delta"
sparkle:deltaFrom="9614"
length="15000000"
type="application/octet-stream"
sparkle:edSignature="..." />
<enclosure
url="https://assets.eagle0.net/mac/deltas/9613-9615.delta"
sparkle:deltaFrom="9613"
length="18000000"
type="application/octet-stream"
sparkle:edSignature="..." />
<enclosure
url="https://assets.eagle0.net/mac/deltas/9612-9615.delta"
sparkle:deltaFrom="9612"
length="22000000"
type="application/octet-stream"
sparkle:edSignature="..." />
</sparkle:deltas>
</item>
<!-- older versions... -->
</channel>
</rss>
```
## Implementation Plan
### Phase 1: Add S3 Utility Functions
**File:** `src/main/go/net/eagle0/util/aws/bucket_basics.go`
Add two new functions to support delta generation:
```go
// ListObjectsWithPrefix returns all object keys matching the given prefix
func (bb BucketBasics) ListObjectsWithPrefix(bucket, prefix string) ([]string, error) {
var keys []string
paginator := s3.NewListObjectsV2Paginator(bb.S3Client, &s3.ListObjectsV2Input{
Bucket: aws.String(bucket),
Prefix: aws.String(prefix),
})
for paginator.HasMorePages() {
page, err := paginator.NextPage(context.TODO())
if err != nil {
return nil, err
}
for _, obj := range page.Contents {
keys = append(keys, *obj.Key)
}
}
return keys, nil
}
// DownloadFile downloads an object to a local file path
func (bb BucketBasics) DownloadFile(bucket, key, localPath string) error {
result, err := bb.S3Client.GetObject(context.TODO(), &s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
})
if err != nil {
return err
}
defer result.Body.Close()
file, err := os.Create(localPath)
if err != nil {
return err
}
defer file.Close()
_, err = io.Copy(file, result.Body)
return err
}
```
### Phase 2: Store App Bundles for Delta Generation
**File:** `src/main/go/net/eagle0/build/mac_build_handler/mac_build_handler.go`
Add storage paths:
```go
var appsRoot = "mac/apps/" // Zipped app bundles for delta generation
var deltasRoot = "mac/deltas/" // Delta patches
```
After DMG creation, upload the zipped app bundle:
```go
func uploadAppBundle(bb aws.BucketBasics, appPath string, buildNumber string) error {
appZipPath := filepath.Join("/tmp", fmt.Sprintf("eagle0-%s.app.zip", buildNumber))
// Create zip of app bundle using ditto (preserves metadata)
cmd := exec.Command("ditto", "-c", "-k", "--keepParent", appPath, appZipPath)
if output, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("failed to zip app: %s: %w", string(output), err)
}
defer os.Remove(appZipPath)
// Upload to S3
remotePath := appsRoot + fmt.Sprintf("eagle0-%s.app.zip", buildNumber)
log.Printf("Uploading app bundle to S3: %s", remotePath)
return bb.UploadFilePublic(bucketName, remotePath, appZipPath)
}
```
### Phase 3: Add Delta XML Structures
**File:** `src/main/go/net/eagle0/build/mac_build_handler/mac_build_handler.go`
Add new structs for delta representation:
```go
// Delta represents a delta patch from a previous version
type Delta struct {
XMLName xml.Name `xml:"enclosure"`
URL string `xml:"url,attr"`
DeltaFrom string `xml:"sparkle:deltaFrom,attr"`
Length int64 `xml:"length,attr"`
Type string `xml:"type,attr"`
EdSig string `xml:"sparkle:edSignature,attr"`
}
// Deltas wraps the sparkle:deltas element
type Deltas struct {
XMLName xml.Name `xml:"sparkle:deltas"`
Items []Delta `xml:"enclosure"`
}
// Update Item struct to include Deltas
type Item struct {
Title string `xml:"title"`
PubDate string `xml:"pubDate"`
SparkleVersion string `xml:"sparkle:version"`
SparkleShortVersion string `xml:"sparkle:shortVersionString"`
Description string `xml:"description,omitempty"`
Enclosure Enclosure `xml:"enclosure"`
Deltas *Deltas `xml:"sparkle:deltas,omitempty"`
}
```
### Phase 4: Generate Delta Patches
**File:** `src/main/go/net/eagle0/build/mac_build_handler/mac_build_handler.go`
```go
// Maximum number of versions to generate deltas from
const maxDeltaVersions = 5
// generateDeltas creates delta patches from previous versions to the new version
func generateDeltas(bb aws.BucketBasics, newBuildNumber string, newAppPath string, privateKeyPath string) ([]Delta, error) {
var deltas []Delta
// Ensure BinaryDelta tool is available
binaryDeltaPath, err := ensureBinaryDelta()
if err != nil {
return nil, fmt.Errorf("failed to get BinaryDelta: %w", err)
}
// List available app bundles
appKeys, err := bb.ListObjectsWithPrefix(bucketName, appsRoot+"eagle0-")
if err != nil {
log.Printf("Warning: failed to list app bundles: %v", err)
return deltas, nil // Continue without deltas
}
// Parse build numbers from keys and sort descending
var buildNumbers []string
for _, key := range appKeys {
// Extract build number from "mac/apps/eagle0-9614.app.zip"
base := filepath.Base(key)
if strings.HasPrefix(base, "eagle0-") && strings.HasSuffix(base, ".app.zip") {
bn := strings.TrimSuffix(strings.TrimPrefix(base, "eagle0-"), ".app.zip")
if bn != newBuildNumber {
buildNumbers = append(buildNumbers, bn)
}
}
}
// Sort descending (most recent first) and limit to maxDeltaVersions
sort.Sort(sort.Reverse(sort.StringSlice(buildNumbers)))
if len(buildNumbers) > maxDeltaVersions {
buildNumbers = buildNumbers[:maxDeltaVersions]
}
// Generate delta from each previous version
for _, oldBuild := range buildNumbers {
delta, err := generateSingleDelta(bb, binaryDeltaPath, oldBuild, newBuildNumber, newAppPath, privateKeyPath)
if err != nil {
log.Printf("Warning: failed to generate delta from %s: %v", oldBuild, err)
continue // Skip this delta but continue with others
}
deltas = append(deltas, delta)
}
return deltas, nil
}
func generateSingleDelta(bb aws.BucketBasics, binaryDeltaPath, oldBuild, newBuild, newAppPath, privateKeyPath string) (Delta, error) {
// Download old app bundle
oldAppZipKey := appsRoot + fmt.Sprintf("eagle0-%s.app.zip", oldBuild)
oldAppZipLocal := filepath.Join("/tmp", fmt.Sprintf("eagle0-%s.app.zip", oldBuild))
defer os.Remove(oldAppZipLocal)
if err := bb.DownloadFile(bucketName, oldAppZipKey, oldAppZipLocal); err != nil {
return Delta{}, fmt.Errorf("failed to download old app: %w", err)
}
// Unzip old app
oldAppDir := filepath.Join("/tmp", fmt.Sprintf("old-app-%s", oldBuild))
defer os.RemoveAll(oldAppDir)
cmd := exec.Command("ditto", "-x", "-k", oldAppZipLocal, oldAppDir)
if output, err := cmd.CombinedOutput(); err != nil {
return Delta{}, fmt.Errorf("failed to unzip old app: %s: %w", string(output), err)
}
oldAppPath := filepath.Join(oldAppDir, "eagle0.app")
// Generate delta
deltaPath := filepath.Join("/tmp", fmt.Sprintf("%s-%s.delta", oldBuild, newBuild))
defer os.Remove(deltaPath)
cmd = exec.Command(binaryDeltaPath, "create", oldAppPath, newAppPath, deltaPath)
if output, err := cmd.CombinedOutput(); err != nil {
return Delta{}, fmt.Errorf("failed to create delta: %s: %w", string(output), err)
}
// Get delta size
deltaSize, err := getFileSize(deltaPath)
if err != nil {
return Delta{}, fmt.Errorf("failed to get delta size: %w", err)
}
log.Printf("Delta %s->%s size: %d bytes (%.1f MB)", oldBuild, newBuild, deltaSize, float64(deltaSize)/1024/1024)
// Sign delta
signature, err := signWithSparkle(deltaPath, privateKeyPath)
if err != nil {
return Delta{}, fmt.Errorf("failed to sign delta: %w", err)
}
// Upload delta
deltaKey := deltasRoot + fmt.Sprintf("%s-%s.delta", oldBuild, newBuild)
if err := bb.UploadFilePublic(bucketName, deltaKey, deltaPath); err != nil {
return Delta{}, fmt.Errorf("failed to upload delta: %w", err)
}
deltaURL := fmt.Sprintf("https://assets.eagle0.net/%s", deltaKey)
return Delta{
URL: deltaURL,
DeltaFrom: oldBuild,
Length: deltaSize,
Type: "application/octet-stream",
EdSig: signature,
}, nil
}
func ensureBinaryDelta() (string, error) {
binaryDeltaPath := "/tmp/sparkle-cache/Sparkle-2.6.4/bin/BinaryDelta"
if _, err := os.Stat(binaryDeltaPath); os.IsNotExist(err) {
log.Println("Sparkle BinaryDelta not found, downloading...")
cmd := exec.Command("bash", "-c", `
mkdir -p /tmp/sparkle-cache
curl -sL https://github.com/sparkle-project/Sparkle/releases/download/2.6.4/Sparkle-2.6.4.tar.xz | tar -xJ -C /tmp/sparkle-cache
`)
if output, err := cmd.CombinedOutput(); err != nil {
return "", fmt.Errorf("failed to download Sparkle: %s: %w", string(output), err)
}
}
return binaryDeltaPath, nil
}
```
### Phase 5: Update Main Deploy Flow
**File:** `src/main/go/net/eagle0/build/mac_build_handler/mac_build_handler.go`
Modify `main()` to integrate delta generation:
```go
func main() {
// ... existing argument parsing ...
// Create DMG (existing)
if err := createDMG(appPath, dmgPath, "Eagle0"); err != nil {
log.Fatalf("Failed to create DMG: %v", err)
}
// ... existing DMG upload ...
if privateKeyPath != "" {
// Upload app bundle for future delta generation (NEW)
log.Println("Uploading app bundle for delta generation...")
if err := uploadAppBundle(bb, appPath, buildNumber); err != nil {
log.Printf("Warning: failed to upload app bundle: %v", err)
// Continue - delta generation is optional
}
// Generate deltas from previous versions (NEW)
log.Println("Generating delta patches...")
deltas, err := generateDeltas(bb, buildNumber, appPath, privateKeyPath)
if err != nil {
log.Printf("Warning: failed to generate deltas: %v", err)
} else {
log.Printf("Generated %d delta patches", len(deltas))
}
// Update appcast with deltas
log.Println("Updating appcast.xml...")
appcast, err := fetchAppcast(bb)
if err != nil {
log.Fatalf("Failed to fetch appcast: %v", err)
}
// Create new item with deltas
newItem := Item{
Title: fmt.Sprintf("Version %s", version),
PubDate: time.Now().Format(time.RFC1123Z),
SparkleVersion: buildNumber,
SparkleShortVersion: version,
Description: "",
Enclosure: Enclosure{
URL: downloadURL,
Length: fileSize,
Type: "application/octet-stream",
EdSig: signature,
},
}
// Add deltas if any were generated
if len(deltas) > 0 {
newItem.Deltas = &Deltas{Items: deltas}
}
// ... rest of appcast handling ...
}
}
```
### Phase 6: Cleanup Old Artifacts
**File:** `src/main/go/net/eagle0/build/mac_build_handler/mac_build_handler.go`
When pruning old versions from appcast, also delete associated artifacts:
```go
// In the appcast pruning section, after removing old items:
if len(appcast.Channel.Items) > 10 {
oldItems := appcast.Channel.Items[10:]
for _, item := range oldItems {
oldBuild := item.SparkleVersion
// Delete old DMG (existing)
dmgKey := strings.TrimPrefix(item.Enclosure.URL, "https://assets.eagle0.net/")
log.Printf("Deleting old build: %s", dmgKey)
bb.DeleteObject(bucketName, dmgKey)
// Delete old app bundle (NEW)
appKey := appsRoot + fmt.Sprintf("eagle0-%s.app.zip", oldBuild)
log.Printf("Deleting old app bundle: %s", appKey)
bb.DeleteObject(bucketName, appKey)
// Delete deltas TO this version (NEW)
deltaKeys, _ := bb.ListObjectsWithPrefix(bucketName, deltasRoot)
for _, key := range deltaKeys {
if strings.HasSuffix(key, fmt.Sprintf("-%s.delta", oldBuild)) {
log.Printf("Deleting old delta: %s", key)
bb.DeleteObject(bucketName, key)
}
}
}
appcast.Channel.Items = appcast.Channel.Items[:10]
}
```
## S3 Storage Structure
After implementation, the S3 bucket will have this structure:
```
eagle0-windows/
├── mac/
│ ├── appcast.xml # Update feed with delta info
│ ├── builds/ # Full DMG downloads
│ │ ├── eagle0-1.0.9620.dmg
│ │ ├── eagle0-1.0.9619.dmg
│ │ ├── ...
│ │ └── eagle0-latest.dmg # Symlink to latest
│ ├── apps/ # Zipped app bundles (NEW)
│ │ ├── eagle0-9620.app.zip
│ │ ├── eagle0-9619.app.zip
│ │ ├── eagle0-9618.app.zip
│ │ ├── eagle0-9617.app.zip
│ │ └── eagle0-9616.app.zip # Keep last 5 for delta gen
│ └── deltas/ # Delta patches (NEW)
│ ├── 9619-9620.delta
│ ├── 9618-9620.delta
│ ├── 9617-9620.delta
│ ├── 9616-9620.delta
│ ├── 9615-9620.delta
│ ├── 9618-9619.delta
│ ├── 9617-9619.delta
│ └── ...
```
## Storage Impact Analysis
### Current Storage (without deltas)
- 10 DMGs × 200MB = **~2GB**
### Estimated Storage (with deltas)
- 10 DMGs × 200MB = 2GB
- 5 app bundles × 150MB = 0.75GB (zip compression)
- ~25 delta files × 20MB avg = 0.5GB
- **Total: ~3.25GB**
### Trade-offs
- **+1.25GB storage** (~60% increase)
- **-170MB per user update** (~85% bandwidth savings)
- Break-even: ~8 user updates to recoup storage cost
## Bandwidth Savings
| Scenario | Without Deltas | With Deltas | Savings |
|----------|---------------|-------------|---------|
| 1 version behind | 200MB | ~15MB | 92% |
| 2 versions behind | 200MB | ~20MB | 90% |
| 3 versions behind | 200MB | ~25MB | 87% |
| 5 versions behind | 200MB | ~35MB | 82% |
| 6+ versions behind | 200MB | 200MB (full) | 0% |
## Migration Strategy
The implementation is backward-compatible and requires no changes to existing clients:
1. **First deploy after implementation:**
- Stores app bundle for the first time
- No deltas generated (no previous app bundles exist)
- Appcast has no `<sparkle:deltas>` element
2. **Second deploy:**
- Generates delta from previous version
- Appcast now has `<sparkle:deltas>` with one entry
- Users on previous version get delta update
3. **Subsequent deploys:**
- Generate deltas from last 5 versions
- Users within 5 versions get delta updates
- Users more than 5 versions behind get full DMG
4. **Client behavior:**
- Sparkle automatically checks for matching delta
- Falls back to full DMG if no delta matches
- No client code changes required
## Error Handling
The implementation handles failures gracefully:
1. **S3 list/download fails:** Skip delta generation, use full DMG
2. **BinaryDelta fails for one version:** Log warning, continue with other versions
3. **Signing fails:** Skip that delta, continue with others
4. **Upload fails:** Skip that delta, continue with others
The deploy never fails due to delta issues - deltas are optional enhancements.
## Verification Plan
### Manual Testing
1. **Deploy version N:**
- Verify app bundle uploaded to `mac/apps/eagle0-N.app.zip`
- Verify appcast has no deltas (first deploy)
2. **Deploy version N+1:**
- Verify delta generated at `mac/deltas/N-(N+1).delta`
- Verify appcast contains `<sparkle:deltas>` element
- Verify delta signature is valid
3. **Test update from N to N+1:**
- Install version N manually
- Check for updates
- Monitor download size in Player.log (should be ~15-30MB, not 200MB)
- Verify app updated successfully
4. **Test fresh install:**
- Download latest DMG directly
- Verify installation works normally
5. **Test fallback scenario:**
- Install a version more than 5 versions behind
- Update should download full DMG
### Automated Verification
Add to CI workflow (optional):
```yaml
- name: Verify delta generation
run: |
# Check app bundle exists
aws s3 ls s3://eagle0-windows/mac/apps/ | grep eagle0-${BUILD_NUMBER}.app.zip
# Check deltas exist (after second deploy)
aws s3 ls s3://eagle0-windows/mac/deltas/ | head -5
# Verify appcast has deltas
curl -s https://assets.eagle0.net/mac/appcast.xml | grep "sparkle:deltas"
```
## Security Considerations
1. **All deltas are EdDSA signed:** Same signature verification as full DMG
2. **BinaryDelta is Sparkle's official tool:** Well-audited, production-ready
3. **App bundles in S3 are public:** Same as DMGs, no additional exposure
4. **Cleanup removes old artifacts:** No indefinite storage of old versions
## Future Enhancements
1. **Parallel delta generation:** Generate multiple deltas concurrently
2. **Delta size threshold:** Skip uploading deltas larger than X% of full DMG
3. **Delta metrics:** Track delta download rates vs full DMG
4. **Configurable delta count:** Allow adjusting how many versions to keep