File Deduplication Strategies for Cloud Storage

File deduplication cuts cloud storage 20-90% depending on workload, using one of three techniques: file-level (store identical files once, keyed by SHA-256 hash), block-level (chunk files into 4-128 KB blocks and dedupe per block), or variable-length chunking via content-defined chunking (CDC) with Rabin fingerprints. VM backups see 10:1 reductions; general office files see 2:1; media libraries see almost nothing. Pick the technique that matches your data — running block dedup on a library of already-unique .mp4 files wastes CPU for no gain.

File-Level Dedup: The Simplest Win

File-level dedup compares whole-file hashes. Two files with the same SHA-256 are identical; keep one, point the other at it. Implementation takes a weekend:

1. Inventory bucket contents (S3 Inventory, Azure Inventory, GCS bucket list)
2. Compute SHA-256 for each object (or use provider-supplied ETags, with caveats)
3. Group by hash, pick a canonical key per group, update references, delete duplicates

Caveats: S3 ETags match SHA-256 only for single-part uploads under 5 GB. Multipart uploads use a different formula (hash of hashes). For reliable dedup, compute your own hash with aws s3 cp s3://bucket/key - | sha256sum or compute at upload time and store in metadata.

File-level dedup shines when users routinely upload the same files — vendor PDFs, company templates, shared images. Expect 10-30% savings on typical office workloads.

Block-Level Dedup: The Big Multiplier

Block-level splits each file into fixed-size chunks (4 KB, 16 KB, 64 KB) and hashes each chunk. Two files sharing 80% of their chunks store only the unique 20% plus one copy of the shared blocks. Backup products (Veeam, Rubrik, Commvault), filesystems (ZFS with dedup=on, Btrfs), and some backup clouds (Backblaze B2 with client-side dedup) use this.

Advantages: massive compression on VM images, database backups, and log archives where files share large ranges. Disadvantages: high memory use (the dedup index lives in RAM), CPU cost at write time, and catastrophic amplification if the index corrupts.

For cloud object storage, block dedup usually happens inside a backup product rather than as a native feature. S3 itself doesn't dedupe; Backblaze B2 dedupes at upload time when the client sends block hashes first (b2_start_large_file with pre-hashed parts).

Content-Defined Chunking (CDC)

Fixed-size chunking breaks when a byte is inserted at the beginning of a file — every subsequent block hashes differently. Content-Defined Chunking uses a rolling hash (Rabin-Karp fingerprint) to define chunk boundaries based on content patterns. Insert a byte and only the immediate chunk changes.

CDC is the basis for restic, BorgBackup, Duplicacy, and Kopia. These open-source tools dedupe client-side with variable chunks averaging 1-4 MB. For backing up 500 GB of files that change incrementally, CDC-based backups often use under 50 GB of unique storage.

If you're building a backup or sync system, CDC via a library like fastcdc-rs or chunky is the modern choice. Don't roll your own rolling hash — the edge cases are subtle.

Inline vs Post-Process Dedup

Inline dedup runs at write time — before data hits disk, the system checks if the block already exists. If yes, write a reference; if no, write the block. Used by ZFS, most backup appliances, and some cloud storage tiers.

Post-process dedup writes first, then runs a background job to find duplicates and reclaim space. Used by Windows Server's Data Deduplication, NetApp's SnapVault, and most user-space tools. Post-process has lower write latency but needs more peak storage (the duplicates briefly exist before reclamation).

For cloud workloads, inline dedup is usually unavailable — S3 doesn't offer it. Post-process with a scheduled job (daily inventory, daily dedup run) is the practical pattern.

Where Dedup Doesn't Help

Already-compressed or encrypted data dedupes poorly. Two different .mp4 files, even of similar content, share almost no bytes. Two encrypted .zip files of the same plaintext share zero bytes after encryption — that's the whole point of encryption.

This means end-to-end encrypted file storage can't dedupe across users. Convergent encryption (hash the plaintext, use the hash as the key) was an attempt to enable dedup across E2EE but has security issues — it enables confirmation-of-file attacks. For E2EE file services, accept that dedup happens within a user's own files, not across users.

The Security Side of Dedup

Cross-user dedup in non-E2EE systems creates a side channel: if a file you upload deduplicates to an existing block, the server learns someone else already had that file. Dropbox had this exposed in 2011; some other services have too. For shared business accounts it's fine, but for services claiming privacy, it's a leak.

If confidentiality matters, dedup only within a user's own data (salted with a user-specific key), not across the whole system. Or accept that E2EE means no dedup and size the storage accordingly. Tools that prioritize privacy — HexaTransfer for encrypted ad-hoc transfers, for example — trade dedup efficiency for the guarantee that nobody else (including the service) can tell whether two users have the same file.

Measuring Dedup Effectiveness

Don't just enable dedup and hope. Measure the ratio:

dedup_ratio = logical_bytes / physical_bytes

A ratio of 2:1 means you're storing 2 logical bytes for every 1 physical byte. Report tools: zpool get dedupratio on ZFS, Get-DedupStatus on Windows Server, per-repo stats on restic/Borg.

Healthy ratios by workload:

• VM images: 8-20:1
• Database backups: 10-30:1
• File server (office docs): 1.5-3:1
• Email archives: 2-5:1
• Media libraries: 1.0-1.1:1 (don't bother)
• Encrypted archives: 1.0:1 (impossible)

If a workload ratio is under 1.5:1, turn dedup off — the CPU and memory aren't paying for themselves.

Backup Product Integration

Most enterprises don't implement dedup from scratch — they use a backup product that does it. Comparison points when selecting:

• Veeam: Inline block dedup, default 512 KB chunks, compression after dedup
• Rubrik: Content-defined variable chunks, sub-block dedup
• Commvault: Client-side dedup with per-client and global pools
• restic/Borg/Kopia: Open-source CDC, client-side, S3/B2/Azure backends
• BackupPC: Hardlink-based file-level, simple but dated

For a small business backing up 2 TB to S3 Glacier, restic to Glacier Instant Retrieval costs about $15/month with 5:1 dedup. For a 500 TB enterprise data estate, a proper backup platform with dedup pays for itself in the first year of storage savings.

When to Add Compression on Top

Dedup removes duplicate bytes; compression removes redundancy within unique bytes. They stack. After dedup, apply zstd compression for another 1.5-2x reduction on text-heavy data. BorgBackup supports --compression zstd; restic has --compression max; AWS EFS has transparent compression for OneZone-IA.

Order matters: dedup first (to expose duplicate blocks), then compress the unique blocks. Compressing first usually defeats dedup because small input changes cascade through compressed output. All serious backup products handle this correctly — you just pick the knobs.

Dedup is boring, unglamorous, and the single biggest lever on storage cost for mixed workloads. Inventory the data, pick the technique that matches the content, measure the ratio, and reclaim the budget.

Try it at hexatransfer.com — free, no account, 10 GB max.