Distributed File Storage Explained: How It Works
Understand distributed file storage systems like IPFS and Ceph. Replication, consistency, and fault tolerance in modern storage.
Distributed file storage spreads data across many nodes so that no single machine is a bottleneck or a single point of failure. The design choices, how to place data, how to replicate it, how to handle node failures, how to stay consistent, distinguish systems like Ceph (object/block/file), GlusterFS (POSIX), HDFS (big-data batch), MinIO (S3-compatible), and content-addressed systems like IPFS and Filecoin. Each targets different workloads, and the tradeoffs are real. Here's a concrete walk through how these systems actually work.
Replication vs. Erasure Coding
Two strategies protect against node failure. Replication stores multiple full copies: Ceph's default is 3x replication, meaning a 1 GB object uses 3 GB of raw storage. Simple to reason about, fast to read, expensive on storage. Erasure coding splits data into k data chunks plus m parity chunks using Reed-Solomon codes, so a (10,4) scheme stores 14 chunks and tolerates 4 failures with only 40 percent overhead instead of 200 percent. MinIO defaults to erasure coding; Backblaze Vaults use 17+3 Reed-Solomon. Erasure coding reads are slower because reconstruction may need multiple chunks, so hot data often uses replication and cold data uses erasure coding.
Consistent Hashing and Data Placement
How does a system decide which node stores which object? Consistent hashing, introduced in academic work by Karger et al. (1997) and popularized by DynamoDB and Cassandra, hashes keys to a ring and maps each range to a node. Adding or removing a node only shuffles a fraction of keys, not the whole dataset. Ceph uses CRUSH (Controlled Replication Under Scalable Hashing), a deterministic algorithm that places objects based on a topology map (rack, row, datacenter) so replicas end up on diverse failure domains. Virtual nodes (vnodes) per physical node smooth out load imbalance.
Consistency Models: Strong, Eventual, and Causal
CAP theorem says you can't have consistency, availability, and partition tolerance simultaneously, you pick two. Strong consistency (linearizability) means reads see the latest write; systems like Spanner and etcd provide this via consensus protocols like Paxos or Raft. Eventual consistency (DynamoDB, Cassandra, S3 for some operations historically) means replicas converge given time, with stale reads possible during propagation. Causal consistency preserves cause-effect order without full linearizability. File storage often accepts eventual consistency for metadata (listing, size) with strong consistency for reads-after-writes of the same object.
Ceph Architecture: OSDs, Monitors, Managers, and MDSes
Ceph runs four daemon types. OSDs (Object Storage Daemons) store objects and replicate them; a cluster typically has 10 to 1,000 OSDs on spinning or NVMe disks. Monitors (MONs) maintain cluster state via Paxos; 3 or 5 MONs provide quorum. Managers (MGRs) expose metrics and host dashboards. MDSes (Metadata Servers) serve the CephFS POSIX filesystem layer. Object storage via RADOS Gateway (RGW) presents S3 and Swift APIs. Block storage via RBD backs OpenStack volumes and VM disks. One codebase, three personalities, tuned with /etc/ceph/ceph.conf and CRUSH map edits.
HDFS and Its Hadoop Legacy
HDFS (Hadoop Distributed File System) targets large sequential reads for MapReduce and Spark jobs. Files split into 128 MB or 256 MB blocks; each block replicates 3x by default across DataNodes. A NameNode holds all metadata in memory, limiting scale to roughly 500 million files per NameNode. HDFS Federation and HDFS Router add multi-namespace support. HDFS isn't great for small files (metadata dominates) or POSIX compatibility, but it's excellent for analytics on TB-scale datasets. It's also being displaced by object storage (S3, GCS) as compute separates from storage in the cloud era.
Content-Addressed Storage: IPFS and Filecoin
IPFS (InterPlanetary File System) identifies content by its hash (CID, Content IDentifier) rather than location. Anyone storing a file with the same content produces the same CID. Retrieval uses a DHT (Distributed Hash Table, Kademlia-based) to find nodes holding the content. Filecoin adds economic incentives, miners prove they're storing content via PoRep (Proof-of-Replication) and PoSt (Proof-of-Spacetime) and earn FIL tokens. IPFS fits archival and decentralized publishing (NFT metadata, web3 sites); it's slow for interactive workloads due to DHT lookup latency (hundreds of milliseconds to seconds).
Object Storage: S3, R2, B2, and MinIO
Object storage presents a flat key-value API: PUT an object with a key, GET it back. No directories, no POSIX semantics. This simplicity enables massive scale, AWS S3 stores trillions of objects with 11 nines durability. Clones like Cloudflare R2, Backblaze B2, Wasabi, and DigitalOcean Spaces implement the S3 API on different backends. MinIO runs on-premises as open source AGPL v3, often in Kubernetes as a StatefulSet, providing S3-compatible storage on commodity hardware. Object storage has largely won the cloud storage market because the API is simple, pricing is clear, and durability is trustworthy.
Erasure Coding in Practice: How Reconstruction Works
When a node dies in an erasure-coded system, remaining nodes reconstruct the lost chunks. For a (10,4) Reed-Solomon code, any 10 of the 14 chunks reconstruct the original 10 data chunks via matrix algebra over a finite field. The reconstruction load falls on surviving nodes, so losing 1 node in a cluster of 100 triggers reads from 10 other nodes per lost chunk. Bandwidth during rebuild is a major operational concern. Systems like Ceph rate-limit rebuilds to avoid impacting production I/O. Newer codes like Local Reconstruction Codes (LRC, used by Azure) and Hitchhiker codes reduce reconstruction bandwidth by allowing partial reads.
Tail Latency and Hedged Requests
Distributed systems have long tails. A request that hits a slow disk or congested network might take 10x the median. Google's paper "The Tail at Scale" (Dean & Barroso, 2013) formalized techniques: hedged requests send duplicate reads to two nodes, cancel whichever loses; tied requests coordinate so only one actually executes. Ceph, DynamoDB, and Spanner all use variations. For file transfer systems, reading an object via multiple replicas in parallel and taking the first response cuts p99 latency dramatically at the cost of slightly more bandwidth.
Failure Domains and Data Diversity
Placing three replicas doesn't help if all three are in the same rack and the rack's top-of-rack switch fails. Failure domain awareness means replicas land in different racks, rows, or datacenters. Ceph's CRUSH rules encode "at least 2 copies in different racks, 1 copy in a different datacenter." Cloud object storage handles this transparently, S3 Standard stores across 3+ Availability Zones. For cross-region replication, S3 Cross-Region Replication (CRR) asynchronously mirrors objects to another region, useful for disaster recovery and compliance with regional data residency.
Practical Implications for File Transfer Services
A file transfer service typically builds on object storage rather than a POSIX file system. S3-compatible backends (AWS S3, Cloudflare R2, MinIO) handle durability and scale without the team running Ceph or GlusterFS. The service adds auth, presigned URLs, metadata, and user-facing features. HexaTransfer uses an S3-compatible backend with client-side AES-256-GCM encryption, so the distributed storage layer handles durability while the app layer keeps file contents private from every tier.
Try it at hexatransfer.com — free, no account, 10 GB max.
Send large files securely with end-to-end encryption
Transfer files up to 10 GB for free with end-to-end encryption. No account required. Your files are encrypted in your browser before upload — no one else can read them.
Send a file