Chunk-Based Upload Architecture: Design Guide
Design a robust chunk-based upload system. Learn about chunking strategies, resumable uploads, and parallel transfer optimization.
A chunk-based upload architecture splits a file into fixed- or variable-size pieces and sends them independently, so a network hiccup doesn't force a 10 GB restart from zero. The two dominant open standards are AWS S3 Multipart Upload (5 MB minimum parts, 10,000 parts max) and the tus.io resumable protocol (RFC draft, widely implemented). Done right, chunked uploads deliver 5 to 10x higher throughput on high-latency links, tolerate brief disconnects, and enable parallel decryption on the receive side. Here's how to design one that actually holds up in production.
Why Monolithic Uploads Fail at Scale
A single HTTP PUT of a 5 GB file runs into half a dozen ways to fail. TCP congestion windows take a long time to ramp up, capping throughput well below the link capacity on high-latency paths. Browsers limit concurrent connections per origin to 6, leaving most bandwidth idle. Server-side request timeouts (nginx default 60 seconds, CloudFront 30 seconds for non-streaming) kill long uploads. Mobile networks switching between cell towers drop the connection every few minutes. And a single bit flip forces the whole transfer to restart. Chunked uploads solve all of these by making the unit of work small, independently retryable, and parallel-friendly.
Choosing a Chunk Size
Chunk size is a tradeoff. Smaller chunks recover faster from failures and provide finer progress granularity but add more HTTP overhead. Larger chunks amortize handshake and TLS costs but waste bandwidth when a chunk fails and must retransmit. Typical ranges: 1 MB to 5 MB for mobile uploads with lossy networks, 5 MB to 16 MB for desktop web uploads, 16 MB to 64 MB for server-to-server transfers on reliable links, and 100 MB or more for S3 Multipart where the 10,000-part ceiling forces larger chunks on 1 TB files. Some systems adapt dynamically, starting small and scaling up as the connection proves reliable.
Fixed-Size vs. Content-Defined Chunking
Fixed-size chunking (every 8 MB, say) is trivial to implement, parallel-friendly, and supports exact resume offsets. Content-defined chunking, used in rsync and restic, picks boundaries based on a rolling hash like Rabin fingerprinting so that insertions in the middle of a file don't shift all subsequent chunk boundaries. CDC is fantastic for deduplication in backup tools but adds complexity and CPU cost unrewarded for pure file transfer. For upload architectures, fixed-size wins on simplicity and maps cleanly to S3 multipart parts or tus.io offsets.
Resumable Upload Protocols
The tus.io protocol, implemented in tusd (Go), tus-js-client, Uppy, and many server frameworks, uses HTTP PATCH with an Upload-Offset header to append chunks. A HEAD request returns the current offset on the server, so the client knows where to resume after a network interruption. S3 Multipart Upload uses a different model: initiate upload to get an UploadId, upload each part (1-indexed), then send a CompleteMultipartUpload request with the list of ETags. Clients can query ListParts to see what was uploaded. Both protocols preserve state across client restarts and survive connection drops gracefully.
Parallel Upload Concurrency
Uploading chunks in parallel boosts throughput dramatically on high-latency links. HTTP/1.1 caps at 6 concurrent connections per origin in browsers; HTTP/2 multiplexes many streams over a single connection but still subject to flow control windows. A typical upload scheduler queues chunks and dispatches 4 to 8 in parallel, with backpressure when the server signals 429 or 503. Too much parallelism triggers ISP shaping and middlebox connection limits; too little leaves bandwidth unused. Empirically, 4 parallel streams on home broadband and 8 to 16 on gigabit fiber hit the sweet spot for most workloads.
Client-Side State and Resume Metadata
Resumable uploads require the client to remember enough state to resume after a browser crash or a laptop shutdown. IndexedDB, part of the Web Storage standard, stores upload manifests with the file's hash, the chunk count, and which chunks succeeded. Key the manifest by the file's SHA-256 hash so re-adding the same file picks up where it left off. Clean up stale manifests older than 7 days to avoid bloat. On mobile, WKWebView (iOS) and Chrome Custom Tabs (Android) may evict IndexedDB under memory pressure, so persist critical state to native storage if possible.
Server-Side Chunk Handling
The server needs to reassemble chunks into a complete file or, with S3 Multipart, delegate reassembly to S3. A minimal architecture: accept each chunk PATCH, write to a temporary blob keyed by upload ID and chunk index, record the offset in a metadata store like Redis or PostgreSQL, and on CompletePart assemble or mark complete. Use object storage (S3, Cloudflare R2, Backblaze B2) for the chunk blobs rather than local disk, since load-balanced backends can't share local state easily. Garbage-collect abandoned uploads after 24 to 72 hours to reclaim storage.
Integrity Verification Per Chunk and End-to-End
Verify each chunk with a hash on upload. S3 Multipart's ETag is an MD5 hash per part (or a composite hash for the full object). For stronger integrity, compute SHA-256 per chunk client-side and send it in a header; the server stores it alongside the chunk and can verify on re-read. After all chunks upload, compute a Merkle tree root or a streaming hash of the reassembled file and return it to the client. The client compares with its own hash of the original file. Any mismatch triggers a re-upload of the offending chunks.
Encryption Interplay With Chunking
End-to-end encryption complicates chunking slightly. Each chunk needs its own nonce to avoid IV reuse in AES-GCM, and the chunk boundaries must be part of the authentication scheme. A typical approach: derive a per-chunk key via HKDF-SHA256 from a root file encryption key, using the chunk index as context info, then encrypt each chunk with AES-256-GCM and a zero or incrementing nonce. Include the chunk index and total chunk count in the AAD so attackers can't splice or reorder chunks. On decryption, verify all chunks are present and in order before releasing the plaintext.
Observability for Chunk-Based Systems
Instrument every chunk. Metrics to track: chunks uploaded per second, p50/p95/p99 chunk upload latency, retry rate per chunk, and abandoned upload rate. Dashboards in Grafana or Datadog reveal regressions quickly. Distributed traces via OpenTelemetry tie a client session to its server-side chunk handling. Log structured events in JSON so they're searchable in Loki, Elasticsearch, or Splunk. When a customer reports a slow upload, the trace shows exactly which chunks stalled and why.
Putting It Together
A production-ready chunked upload system combines 5 MB to 16 MB chunks, tus.io or S3 Multipart for the protocol, 4 to 8 parallel streams, IndexedDB-backed resume state, per-chunk SHA-256 integrity checks, and optional per-chunk E2EE with a derived key. HexaTransfer uses client-side chunked encryption with AES-256-GCM and resumable uploads to handle 10 GB files reliably over flaky connections.
Try it at hexatransfer.com — free, no account, 10 GB max.
Send large files securely with end-to-end encryption
Transfer files up to 10 GB for free with end-to-end encryption. No account required. Your files are encrypted in your browser before upload — no one else can read them.
Send a file