Fault tolerance is a system’s ability to keep meeting its SLOs despite expected failures—machines dying, networks flaking, processes crashing, disks filling—without human intervention. It’s the practical outcome of designing for failure: the system either continues normally or degrades gracefully when parts break.
How it differs from related terms
- Redundancy: the means (extra components/paths).
- High availability: the result (little downtime).
- Resilience: broader ability to absorb, adapt, and recover (includes human/ops).
What “good” fault tolerance looks like
- No single points of failure across failure domains (process → node → AZ → region).
- Automatic detection and recovery (health checks, failover, restarts).
- Predictable degradation (shed load, read-only mode) instead of hard outages.
- Sufficient spare capacity (N+1/N+M) to absorb losses.
Core techniques
- Redundancy & isolation: multi-replica services, spread across AZs; bulkheads to stop blast radius.
- Automated failover: leader election, health-checked load balancing, fast DNS/LB re-routing.
- Idempotency + retries with backoff/jitter; timeouts & circuit breakers to avoid cascading failure.
- Quorum/replication: e.g., Raft/Paxos, Kafka RF≥3 with
min.insync.replicas=2. - Data durability: snapshots, multi-AZ/region replicas, erasure coding.
- Graceful degradation: feature flags to disable non-critical work, serve cached results, partial results.
- Self-healing: auto-restart/replace (Kubernetes controllers, ASGs).
What to measure
- SLIs: availability, latency, error rate (per AZ/region—not just global).
- MTTR & failover time (how fast a healthy replica takes over).
- Redundancy health: quorum size, ISR status, replication lag.
- Headroom after failure: can you meet SLO with one node/AZ down?
How to verify (continuously)
- Game days / chaos tests: kill nodes, cut an AZ, block a dependency; confirm service stays within SLO and alerts are actionable.
- Runbooks & drills: rehearse promotions, restores, and traffic shifts.
- Alert on loss of tolerance: e.g., quorum at risk, only 1 AZ serving.
Common pitfalls
- Hidden SPOFs: shared DB, cache, NAT, or CI/CD path behind “redundant” apps.
- Correlated failures: all replicas in one AZ/version; dependency coupling.
- Insufficient capacity: N replicas but no spare to handle failover load.
- Unbounded retries: amplify an incident; always pair with timeouts/circuit breakers.
Concrete patterns (AWS/EKS flavored)
- Stateless services: 3+ replicas, PDBs, Pod Topology Spread across 3 AZs; ALB/NLB across subnets in all AZs; HPA with spare headroom.
- Stateful stores: RDS/Aurora Multi-AZ + tested failover; Kafka/MSK RF≥3 with rack-aware placement; Redis/ElastiCache with multi-AZ and auto-failover.
- Global stance: active-active or active-passive across regions for tier-1 APIs; Route 53 health-check failover; S3 versioning + cross-region replication where needed.
Rule of thumb: design for one unit down (node/AZ) without breaching SLOs, test it regularly, and alert when you lose that safety margin. If you share your current topology/SLOs, I can map each tier to specific configs (k8s YAML + AWS settings) to reach concrete fault-tolerance targets.
Fault tolerance Vs High Availability
