Distributed Systems Fundamentals
Learning Objectives
- Define a distributed system and identify its defining characteristics.
- Explain the trade-off described by the CAP theorem and why no system can escape it.
- Compare client-server, peer-to-peer, shared-disk, and shared-nothing architectures.
- Distinguish synchronous, asynchronous, and event-driven communication models.
- Identify the main challenges distributed systems introduce: consistency, latency, and partial failure.
- Apply these concepts to explain how a real service (e.g., a cloud storage system) stays available and consistent.
Quick Answer
A distributed system is a collection of independent computers that appears to its users as a single coherent system. Instead of one powerful machine doing all the work, many machines cooperate over a network, sharing the load and covering for each other when something fails. This matters because almost every large-scale service you use — search engines, social media, banking, streaming — cannot run on a single machine; it would be too slow, too fragile, and unable to serve millions of users at once. Distributed systems let us scale horizontally (add more machines) instead of vertically (buy a bigger machine), and they let a service survive the failure of any single part. The cost of this power is complexity: coordinating many machines over an unreliable network raises hard problems around consistency, latency, and partial failure that a single-machine program never has to face.
What Is a Distributed System?
A distributed system consists of multiple autonomous nodes — each with its own processor, memory, and often its own storage — that communicate over a network to accomplish a shared goal. Crucially, the system is designed so that users interact with it as if it were one machine, even though the work is happening across many.
Four properties define this kind of system:
- No shared clock or memory — nodes cannot simply read a shared variable; they must send messages.
- Independent failure — one node can crash while the rest of the system keeps running.
- Concurrency — many nodes do work at the same time, not in a strict sequence.
- Transparency — ideally, users don't need to know or care which physical machine handled their request.
Why it matters: Once you understand that "the system" is really many separate computers pretending to be one, you can predict where things go wrong — messages get delayed or lost, clocks drift, and any given node might be down at any moment. Distributed systems design is largely about hiding these realities from the user while still being correct.
Common misunderstanding: Students often think a "distributed system" just means "a system that uses a network," like a website talking to a database. The distinguishing feature is that multiple nodes cooperate to do the same job and must coordinate state — a single web server talking to a single database isn't yet distributed in the interesting sense until you add multiple database replicas or multiple app servers that must stay in sync.
Scalability
Definition: Scalability is a system's ability to handle more load — more users, more data, more requests — by adding resources rather than by rewriting the system from scratch.
Explanation: There are two ways to scale:
- Vertical scaling (scale up): give one machine more CPU, RAM, or disk. Simple, but every machine has a ceiling, and it creates a single point of failure.
- Horizontal scaling (scale out): add more machines and spread the work across them. This is how most large systems grow, because you can keep adding commodity servers almost indefinitely.
Example: A to-do list app used by 10 people can run happily on one small server. If it suddenly needs to serve 10 million people, no single machine can hold that many connections or that much data — the app must be redesigned to run across many servers that share the load.
Real-world example: During a flash sale, an e-commerce platform like Amazon automatically spins up extra web servers to absorb the traffic spike, then shuts them down once demand drops — horizontal scaling in action.
Why it matters: Scalability is what lets a service grow from a hundred users to a hundred million without a fundamental rewrite. It is one of the main reasons companies choose a distributed architecture in the first place.
Common misunderstanding: Adding more machines doesn't automatically make a system faster. If nodes must constantly coordinate (e.g., agree on a single value), the coordination overhead can grow faster than the benefit — this is why some algorithms scale well and others hit a wall no matter how many nodes you add.
Fault Tolerance
Definition: Fault tolerance is a system's ability to keep working correctly even when some of its components fail.
Explanation: In a system with thousands of machines, hardware failure isn't an edge case — it is a certainty that happens every day. Fault-tolerant design assumes failure will happen and builds in redundancy so the overall system survives it. The main technique is replication: keeping copies of data or services on multiple nodes so that if one fails, another can take over.
Example: A distributed database keeps three copies of every record on three different servers. If one server crashes, the other two still have the data, and the system can keep serving reads and writes.
Real-world example: Amazon S3 stores objects redundantly across multiple facilities within a region, so the failure of a single data center does not cause data loss or an outage.
Why it matters: Without fault tolerance, a distributed system would actually be less reliable than a single machine, because it has more components that can fail. Fault tolerance flips this: more machines, correctly designed, mean more resilience, not less.
Common misunderstanding: Students sometimes assume fault tolerance means "the system never fails." In reality it means failures are contained and recovered from — a node can and will go down; the goal is that users don't notice.
The Consistency–Availability Trade-off (CAP Theorem)
Definition: The CAP theorem states that a distributed system that has been partitioned by a network failure can guarantee at most two of the following three properties at once: Consistency (every read gets the most recent write), Availability (every request gets a response), and Partition tolerance (the system keeps working despite lost or delayed messages between nodes).
Explanation: Because network partitions are unavoidable at scale, partition tolerance is effectively mandatory for any real distributed system — so in practice CAP forces a choice between consistency and availability during a partition. A CP system will refuse to answer some requests to guarantee correctness; an AP system will answer every request but might return slightly stale data.
Example: Two data centers, one in New York and one in Tokyo, lose their network link for 30 seconds. A CP system (e.g., a strongly consistent banking ledger) will reject writes in one region until the link is restored, to avoid conflicting balances. An AP system (e.g., a shopping cart service) will keep accepting requests in both regions and reconcile any conflicts once the link comes back.
Real-world example: ATM networks favor availability — you can withdraw cash even if the ATM can't confirm your very latest balance from every branch — because refusing service entirely would be worse for the bank and the customer than a rare, quickly-reconciled discrepancy.
Why it matters: CAP explains why there is no single "best" database or architecture — the right choice depends on whether your application can tolerate stale data (favor availability) or must never show wrong data (favor consistency).
Common misunderstanding: CAP is often misquoted as "pick any two of the three, always." In practice, partition tolerance isn't optional for a real distributed system, so the actual everyday choice is consistency vs. availability specifically when a partition occurs — not a free choice among all three at all times.
Visualizing the CAP Trade-off
Architecture Models
How nodes are arranged shapes everything about a distributed system's behavior.
- Client-server: Clients send requests; a server (or server cluster) processes them and replies. Simple and centrally controlled, but the server can become a bottleneck or single point of failure. Example: a web browser requesting a page from a web server.
- Peer-to-peer (P2P): Every node is both a client and a server; there's no central authority. This scales well and resists single points of failure but is harder to secure and coordinate. Example: BitTorrent, where each participant both downloads and uploads file pieces.
- Shared-disk: Multiple compute nodes access a common storage layer. Good for coordinated access to one dataset, but the shared storage itself can bottleneck.
- Shared-nothing: Each node has its own private memory and disk; nodes communicate only via the network. This is the dominant model for large-scale systems (e.g., Hadoop) because it avoids a central bottleneck and scales horizontally.
Communication Models
Nodes need a way to exchange information, and the choice of communication style affects both performance and correctness.
- Synchronous communication: The sender waits (blocks) until it gets a response. Simple to reason about, similar to a normal function call, but a slow or dead receiver stalls the sender. Example: a remote procedure call (RPC).
- Asynchronous communication: The sender continues without waiting for a reply, often via a message queue. More resilient to slow receivers, but harder to reason about ordering and completion. Example: a message broker like RabbitMQ or Kafka.
- Event-driven communication: Nodes react to events published by others, without either side needing to know about the other directly. Example: a publish-subscribe system where a "new order" event triggers inventory, billing, and shipping services independently.
Common Mistakes
| Misconception | Why It's Wrong | Correct Understanding |
|---|---|---|
| "More servers always means better performance." | Coordination between nodes has overhead — network calls, locking, consensus — that can outweigh the benefit of added compute, especially for workloads that aren't easily parallelized. | Scalability depends on how independently the work can be split. Horizontal scaling helps most when tasks are largely independent; it helps less when nodes must constantly agree on shared state. |
| "CAP means you must pick exactly two properties permanently." | This treats CAP as a static, all-the-time choice, when it's really about behavior specifically during a network partition. | Outside of a partition, a system can often provide both consistency and availability. CAP only forces a trade-off when nodes cannot communicate. |
| "A distributed system removes single points of failure automatically." | Simply having multiple machines doesn't guarantee resilience — if a design still relies on one coordinator, one database, or one network link, that piece remains a single point of failure. | Fault tolerance must be deliberately engineered through replication, redundancy, and failover — it does not come for free just because a system has many nodes. |
Comparison and Connections
| Concept | Client-Server | Peer-to-Peer | Shared-Nothing |
|---|---|---|---|
| Control | Centralized at the server | Decentralized, every node is equal | Decentralized, each node owns its data |
| Single point of failure | Yes (the server) | No (no central node) | No, if replicated |
| Scalability | Limited by server capacity | High | High |
| Example | Web browsing | BitTorrent | Hadoop, Cassandra |
| Property | Prioritizes | Example System |
|---|---|---|
| CP (Consistency + Partition tolerance) | Correctness over uptime | Banking ledgers, distributed locks |
| AP (Availability + Partition tolerance) | Uptime over freshness | Shopping carts, DNS, ATM networks |
Key Terms
| Term | Definition |
|---|---|
| Node | An individual computer or process participating in the distributed system. |
| Horizontal scaling | Adding more machines to share the load. |
| Vertical scaling | Increasing the resources (CPU, RAM) of a single machine. |
| CAP theorem | The principle that a partitioned distributed system can guarantee at most two of consistency, availability, and partition tolerance. |
| Replication | Keeping multiple copies of data on different nodes for reliability. |
| Transparency | The property of a distributed system appearing as a single unified system to its users. |
| Partition | A break in network communication that separates nodes from each other. |
| Shared-nothing architecture | A design where each node has independent memory and storage, communicating only over the network. |
Practice Questions
Recall
- What are the four defining characteristics of a distributed system? Answer guidance: No shared memory/clock, independent node failure, concurrency, and transparency to the user.
- What does the "P" in CAP theorem stand for, and why is it usually treated as non-negotiable? Answer guidance: Partition tolerance — the ability to keep operating despite lost network messages; it's non-negotiable because network partitions are unavoidable at scale.
Understanding
- Explain why horizontal scaling is generally preferred over vertical scaling for large distributed systems. Answer guidance: Vertical scaling hits a hardware ceiling and creates a single point of failure; horizontal scaling can grow almost indefinitely by adding commodity machines and improves fault tolerance through redundancy.
- Why does a network partition force a choice between consistency and availability, rather than allowing both? Answer guidance: If nodes can't communicate, a node can either refuse to respond until it's sure it has the latest data (consistency) or respond anyway with possibly stale data (availability) — it cannot guarantee freshness and answer at the same time.
Application
- A ride-sharing app needs to show a driver's location updating every second to nearby riders. Would you design this for strong consistency or high availability, and why? Answer guidance: High availability — riders benefit from seeing frequent, slightly-stale location updates far more than they'd benefit from waiting for a perfectly synchronized value; a brief lag is harmless.
- Your team is choosing between a client-server design and a peer-to-peer design for a company file-sharing tool used by 20 employees. Which would you pick and why? Answer guidance: Client-server is simpler to secure, monitor, and manage for a small, trusted user base; P2P's decentralization benefits matter more at large scale or when avoiding a central authority is a priority.
Analysis
- Compare shared-disk and shared-nothing architectures in terms of how they would behave under heavy write load from many nodes. Answer guidance: Shared-disk can bottleneck because all nodes contend for the same storage layer; shared-nothing distributes writes across independent nodes and scales better, at the cost of needing coordination protocols to keep replicated data consistent.
- A system claims to be "fully fault tolerant" but relies on a single master node to approve every write. Evaluate this claim. Answer guidance: The claim is weak — the master is a single point of failure; true fault tolerance requires a way to promote a replacement master (failover) or avoid a single-master design altogether.
FAQ
Is the internet itself a distributed system? Yes, in the broadest sense — it's a network of independent nodes exchanging messages with no central controller. Most "distributed systems" discussed in courses are more specific: applications built on top of networks like the internet, designed to cooperate on a shared task.
Do I need multiple physical machines to have a distributed system, or can it run on one machine? Conventionally distributed systems run across multiple physical machines, since the whole point is to survive machine-level failure and share real hardware resources. Running multiple processes on one machine can simulate the concepts for learning, but it doesn't give you true fault isolation.
Why can't we just make networks perfectly reliable and avoid the CAP trade-off? Because physical networks fail — cables get cut, routers crash, data centers lose power. No amount of engineering can make partitions impossible, only rarer. CAP is about worst-case behavior, so the trade-off remains even with excellent infrastructure.
Is NoSQL always AP and SQL always CP? No — this is an oversimplification. Some NoSQL databases (like MongoDB in certain configurations) can be tuned toward consistency, and some traditional relational setups sacrifice strict consistency for availability. The database category doesn't determine the CAP choice; its configuration does.
How is fault tolerance different from scalability? Fault tolerance is about surviving failures correctly; scalability is about handling more load efficiently. They're related — replication used for fault tolerance can also help scalability by spreading read traffic — but a system can be highly scalable and still fragile, or highly fault-tolerant and still slow under load.
Quick Revision
- A distributed system is multiple independent nodes that cooperate and appear as one system to users.
- Key properties: no shared memory, independent failure, concurrency, transparency.
- Horizontal scaling adds more machines; vertical scaling upgrades one machine.
- Fault tolerance means the system survives node failures, usually through replication.
- CAP theorem: during a network partition, you can have consistency or availability, not both.
- CP systems (e.g., banking) refuse requests to stay correct; AP systems (e.g., shopping carts) stay available with possibly stale data.
- Client-server is centralized and simple; peer-to-peer is decentralized and scalable but harder to secure.
- Shared-nothing architecture avoids central bottlenecks and is the standard for large-scale systems.
- Synchronous communication waits for a response; asynchronous doesn't; event-driven reacts to published events.
- Real distributed systems: cloud platforms, social media backends, financial trading systems, scientific computing grids.
- Network latency and partial failure are unavoidable challenges that single-machine programs never face.
- More machines does not automatically mean more performance — coordination overhead matters.
Related Topics
Prerequisites: Basic computer networking (client-server communication, IP/TCP concepts), operating systems fundamentals (processes, concurrency).
Related Topics: Distributed Computing Paradigms, Cloud Computing Models, Database Replication and Consistency Models.
Next Topics: Distributed Computing Paradigms, Fault Tolerance and Scalability.