{"id":1015,"date":"2025-09-02T08:48:46","date_gmt":"2025-09-02T08:48:46","guid":{"rendered":"https:\/\/sreschool.com\/blog\/?p=1015"},"modified":"2025-09-02T08:52:41","modified_gmt":"2025-09-02T08:52:41","slug":"what-is-fault-tolerance","status":"publish","type":"post","link":"https:\/\/sreschool.com\/blog\/what-is-fault-tolerance\/","title":{"rendered":"What is Fault tolerance?"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

<\/h1>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n

Fault tolerance<\/strong> is a system\u2019s ability to keep meeting its SLOs<\/strong> despite expected<\/em> failures\u2014machines dying, networks flaking, processes crashing, disks filling\u2014without human intervention. It\u2019s the practical outcome of designing for failure: the system either continues normally<\/strong> or degrades gracefully<\/strong> when parts break.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

How it differs from related terms<\/h2>\n\n\n\n
    \n
  • Redundancy<\/strong>: the means<\/em> (extra components\/paths).<\/li>\n\n\n\n
  • High availability<\/strong>: the result<\/em> (little downtime).<\/li>\n\n\n\n
  • Resilience<\/strong>: broader ability to absorb, adapt, and recover (includes human\/ops).<\/li>\n<\/ul>\n\n\n\n

    What \u201cgood\u201d fault tolerance looks like<\/h2>\n\n\n\n
      \n
    • No single points of failure<\/strong> across failure domains (process \u2192 node \u2192 AZ \u2192 region).<\/li>\n\n\n\n
    • Automatic detection and recovery<\/strong> (health checks, failover, restarts).<\/li>\n\n\n\n
    • Predictable degradation<\/strong> (shed load, read-only mode) instead of hard outages.<\/li>\n\n\n\n
    • Sufficient spare capacity<\/strong> (N+1\/N+M) to absorb losses.<\/li>\n<\/ul>\n\n\n\n
      \"\"<\/figure>\n\n\n\n

      Core techniques<\/h2>\n\n\n\n
        \n
      • Redundancy & isolation:<\/strong> multi-replica services, spread across AZs; bulkheads to stop blast radius.<\/li>\n\n\n\n
      • Automated failover:<\/strong> leader election, health-checked load balancing, fast DNS\/LB re-routing.<\/li>\n\n\n\n
      • Idempotency + retries with backoff\/jitter; timeouts & circuit breakers<\/strong> to avoid cascading failure.<\/li>\n\n\n\n
      • Quorum\/replication:<\/strong> e.g., Raft\/Paxos, Kafka RF\u22653 with min.insync.replicas=2<\/code>.<\/li>\n\n\n\n
      • Data durability:<\/strong> snapshots, multi-AZ\/region replicas, erasure coding.<\/li>\n\n\n\n
      • Graceful degradation:<\/strong> feature flags to disable non-critical work, serve cached results, partial results.<\/li>\n\n\n\n
      • Self-healing:<\/strong> auto-restart\/replace (Kubernetes controllers, ASGs).<\/li>\n<\/ul>\n\n\n\n

        What to measure<\/h2>\n\n\n\n
          \n
        • SLIs:<\/strong> availability, latency, error rate (per AZ\/region\u2014not just global).<\/li>\n\n\n\n
        • MTTR & failover time<\/strong> (how fast a healthy replica takes over).<\/li>\n\n\n\n
        • Redundancy health:<\/strong> quorum size, ISR status, replication lag.<\/li>\n\n\n\n
        • Headroom after failure:<\/strong> can you meet SLO with one node\/AZ down?<\/li>\n<\/ul>\n\n\n\n

          How to verify (continuously)<\/h2>\n\n\n\n
            \n
          • Game days \/ chaos tests:<\/strong> kill nodes, cut an AZ, block a dependency; confirm service stays within SLO and alerts are actionable.<\/li>\n\n\n\n
          • Runbooks & drills:<\/strong> rehearse promotions, restores, and traffic shifts.<\/li>\n\n\n\n
          • Alert on loss of tolerance:<\/strong> e.g., quorum at risk, only 1 AZ serving.<\/li>\n<\/ul>\n\n\n\n

            Common pitfalls<\/h2>\n\n\n\n
              \n
            • Hidden SPOFs:<\/strong> shared DB, cache, NAT, or CI\/CD path behind \u201credundant\u201d apps.<\/li>\n\n\n\n
            • Correlated failures:<\/strong> all replicas in one AZ\/version; dependency coupling.<\/li>\n\n\n\n
            • Insufficient capacity:<\/strong> N replicas but no spare to handle failover load.<\/li>\n\n\n\n
            • Unbounded retries:<\/strong> amplify an incident; always pair with timeouts\/circuit breakers.<\/li>\n<\/ul>\n\n\n\n
              \"\"<\/figure>\n\n\n\n

              Concrete patterns (AWS\/EKS flavored)<\/h2>\n\n\n\n
                \n
              • Stateless services:<\/strong> 3+ replicas, PDBs, Pod Topology Spread across 3 AZs; ALB\/NLB across subnets in all AZs; HPA with spare<\/em> headroom.<\/li>\n\n\n\n
              • Stateful stores:<\/strong> RDS\/Aurora Multi-AZ + tested failover; Kafka\/MSK RF\u22653 with rack-aware placement; Redis\/ElastiCache with multi-AZ and auto-failover.<\/li>\n\n\n\n
              • Global stance:<\/strong> active-active or active-passive across regions for tier-1 APIs; Route 53 health-check failover; S3 versioning + cross-region replication where needed.<\/li>\n<\/ul>\n\n\n\n

                Rule of thumb:<\/strong> design for one unit down<\/em> (node\/AZ) without breaching SLOs, test it regularly, and alert when you lose that safety margin. If you share your current topology\/SLOs, I can map each tier to specific configs (k8s YAML + AWS settings) to reach concrete fault-tolerance targets.<\/p>\n\n\n\n

                Fault tolerance Vs High Availability <\/h2>\n\n\n\n
                \"\"<\/figure>\n","protected":false},"excerpt":{"rendered":"

                Fault tolerance is a system\u2019s ability to keep meeting its SLOs despite expected failures\u2014machines dying, networks flaking, processes crashing, disks […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1015","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"\nWhat is Fault tolerance? - SRE School<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/sreschool.com\/blog\/what-is-fault-tolerance\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What is Fault tolerance? - SRE School\" \/>\n<meta property=\"og:description\" content=\"Fault tolerance is a system\u2019s ability to keep meeting its SLOs despite expected failures\u2014machines dying, networks flaking, processes crashing, disks […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/sreschool.com\/blog\/what-is-fault-tolerance\/\" \/>\n<meta property=\"og:site_name\" content=\"SRE School\" \/>\n<meta property=\"article:published_time\" content=\"2025-09-02T08:48:46+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-09-02T08:52:41+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/sreschool.com\/blog\/wp-content\/uploads\/2025\/09\/image-4.png\" \/>\n\t<meta property=\"og:image:width\" content=\"801\" \/>\n\t<meta property=\"og:image:height\" content=\"401\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Rajesh Kumar\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Rajesh Kumar\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/sreschool.com\/blog\/what-is-fault-tolerance\/\",\"url\":\"https:\/\/sreschool.com\/blog\/what-is-fault-tolerance\/\",\"name\":\"What is Fault tolerance? - SRE School\",\"isPartOf\":{\"@id\":\"https:\/\/sreschool.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/sreschool.com\/blog\/what-is-fault-tolerance\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/sreschool.com\/blog\/what-is-fault-tolerance\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/sreschool.com\/blog\/wp-content\/uploads\/2025\/09\/image-4.png\",\"datePublished\":\"2025-09-02T08:48:46+00:00\",\"dateModified\":\"2025-09-02T08:52:41+00:00\",\"author\":{\"@id\":\"https:\/\/sreschool.com\/blog\/#\/schema\/person\/0ffe446f77bb2589992dbe3a7f417201\"},\"breadcrumb\":{\"@id\":\"https:\/\/sreschool.com\/blog\/what-is-fault-tolerance\/#breadcrumb\"},\"inLanguage\":\"en\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/sreschool.com\/blog\/what-is-fault-tolerance\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en\",\"@id\":\"https:\/\/sreschool.com\/blog\/what-is-fault-tolerance\/#primaryimage\",\"url\":\"https:\/\/sreschool.com\/blog\/wp-content\/uploads\/2025\/09\/image-4.png\",\"contentUrl\":\"https:\/\/sreschool.com\/blog\/wp-content\/uploads\/2025\/09\/image-4.png\",\"width\":801,\"height\":401},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/sreschool.com\/blog\/what-is-fault-tolerance\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/sreschool.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"What is Fault tolerance?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/sreschool.com\/blog\/#website\",\"url\":\"https:\/\/sreschool.com\/blog\/\",\"name\":\"SRESchool\",\"description\":\"Master SRE. Build Resilient Systems. Lead the Future of Reliability\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/sreschool.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/sreschool.com\/blog\/#\/schema\/person\/0ffe446f77bb2589992dbe3a7f417201\",\"name\":\"Rajesh Kumar\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en\",\"@id\":\"https:\/\/sreschool.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/f901a4f2929fa034a291a8363d589791d5a3c1f6a051c22e744acb8bfc8e022a?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/f901a4f2929fa034a291a8363d589791d5a3c1f6a051c22e744acb8bfc8e022a?s=96&d=mm&r=g\",\"caption\":\"Rajesh Kumar\"},\"sameAs\":[\"http:\/\/sreschool.com\/blog\"],\"url\":\"https:\/\/sreschool.com\/blog\/author\/admin\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"What is Fault tolerance? - SRE School","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/sreschool.com\/blog\/what-is-fault-tolerance\/","og_locale":"en_US","og_type":"article","og_title":"What is Fault tolerance? - SRE School","og_description":"Fault tolerance is a system\u2019s ability to keep meeting its SLOs despite expected failures\u2014machines dying, networks flaking, processes crashing, disks […]","og_url":"https:\/\/sreschool.com\/blog\/what-is-fault-tolerance\/","og_site_name":"SRE School","article_published_time":"2025-09-02T08:48:46+00:00","article_modified_time":"2025-09-02T08:52:41+00:00","og_image":[{"width":801,"height":401,"url":"https:\/\/sreschool.com\/blog\/wp-content\/uploads\/2025\/09\/image-4.png","type":"image\/png"}],"author":"Rajesh Kumar","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Rajesh Kumar","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/sreschool.com\/blog\/what-is-fault-tolerance\/","url":"https:\/\/sreschool.com\/blog\/what-is-fault-tolerance\/","name":"What is Fault tolerance? - SRE School","isPartOf":{"@id":"https:\/\/sreschool.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/sreschool.com\/blog\/what-is-fault-tolerance\/#primaryimage"},"image":{"@id":"https:\/\/sreschool.com\/blog\/what-is-fault-tolerance\/#primaryimage"},"thumbnailUrl":"https:\/\/sreschool.com\/blog\/wp-content\/uploads\/2025\/09\/image-4.png","datePublished":"2025-09-02T08:48:46+00:00","dateModified":"2025-09-02T08:52:41+00:00","author":{"@id":"https:\/\/sreschool.com\/blog\/#\/schema\/person\/0ffe446f77bb2589992dbe3a7f417201"},"breadcrumb":{"@id":"https:\/\/sreschool.com\/blog\/what-is-fault-tolerance\/#breadcrumb"},"inLanguage":"en","potentialAction":[{"@type":"ReadAction","target":["https:\/\/sreschool.com\/blog\/what-is-fault-tolerance\/"]}]},{"@type":"ImageObject","inLanguage":"en","@id":"https:\/\/sreschool.com\/blog\/what-is-fault-tolerance\/#primaryimage","url":"https:\/\/sreschool.com\/blog\/wp-content\/uploads\/2025\/09\/image-4.png","contentUrl":"https:\/\/sreschool.com\/blog\/wp-content\/uploads\/2025\/09\/image-4.png","width":801,"height":401},{"@type":"BreadcrumbList","@id":"https:\/\/sreschool.com\/blog\/what-is-fault-tolerance\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/sreschool.com\/blog\/"},{"@type":"ListItem","position":2,"name":"What is Fault tolerance?"}]},{"@type":"WebSite","@id":"https:\/\/sreschool.com\/blog\/#website","url":"https:\/\/sreschool.com\/blog\/","name":"SRESchool","description":"Master SRE. Build Resilient Systems. Lead the Future of Reliability","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/sreschool.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en"},{"@type":"Person","@id":"https:\/\/sreschool.com\/blog\/#\/schema\/person\/0ffe446f77bb2589992dbe3a7f417201","name":"Rajesh Kumar","image":{"@type":"ImageObject","inLanguage":"en","@id":"https:\/\/sreschool.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/f901a4f2929fa034a291a8363d589791d5a3c1f6a051c22e744acb8bfc8e022a?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/f901a4f2929fa034a291a8363d589791d5a3c1f6a051c22e744acb8bfc8e022a?s=96&d=mm&r=g","caption":"Rajesh Kumar"},"sameAs":["http:\/\/sreschool.com\/blog"],"url":"https:\/\/sreschool.com\/blog\/author\/admin\/"}]}},"_links":{"self":[{"href":"https:\/\/sreschool.com\/blog\/wp-json\/wp\/v2\/posts\/1015","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sreschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sreschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sreschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sreschool.com\/blog\/wp-json\/wp\/v2\/comments?post=1015"}],"version-history":[{"count":2,"href":"https:\/\/sreschool.com\/blog\/wp-json\/wp\/v2\/posts\/1015\/revisions"}],"predecessor-version":[{"id":1024,"href":"https:\/\/sreschool.com\/blog\/wp-json\/wp\/v2\/posts\/1015\/revisions\/1024"}],"wp:attachment":[{"href":"https:\/\/sreschool.com\/blog\/wp-json\/wp\/v2\/media?parent=1015"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sreschool.com\/blog\/wp-json\/wp\/v2\/categories?post=1015"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sreschool.com\/blog\/wp-json\/wp\/v2\/tags?post=1015"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}