{"id":60,"date":"2025-05-18T15:21:30","date_gmt":"2025-05-18T15:21:30","guid":{"rendered":"https:\/\/sreschool.com\/blog\/?p=60"},"modified":"2025-05-18T15:21:30","modified_gmt":"2025-05-18T15:21:30","slug":"what-is-toil","status":"publish","type":"post","link":"https:\/\/sreschool.com\/blog\/what-is-toil\/","title":{"rendered":"What is Toil?"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n
\n\n\n\n

\ud83d\udea8 What is Toil<\/em> in SRE?<\/h2>\n\n\n\n

Toil<\/strong> is a term coined by Google SREs to describe a specific class of operational work that is manual, repetitive, automatable, and scalable with service growth<\/strong> \u2014 but not with team growth.<\/p>\n\n\n\n

\n

\ud83d\udca1 Toil is the work you do that doesn’t scale and adds no enduring value.<\/strong><\/p>\n<\/blockquote>\n\n\n\n

\n\n\n\n

\ud83d\udcd8 Official Definition (from Google SRE Book)<\/h2>\n\n\n\n

\u201cToil<\/strong> is the kind of work tied to running a production service that tends to be manual<\/strong>, repetitive<\/strong>, automatable<\/strong>, tactical<\/strong>, reactive<\/strong>, and lacking enduring value<\/strong>.\u201d<\/p>\n\n\n\n

Reference: Google SRE Book \u2013 Eliminating Toil<\/a><\/p>\n\n\n\n

\n\n\n\n

\ud83e\udde9 Key Characteristics of Toil<\/h2>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Characteristic<\/th>Description<\/th><\/tr><\/thead>
Manual<\/strong><\/td>Performed by humans instead of automation<\/td><\/tr>
Repetitive<\/strong><\/td>Occurs often and follows a predictable pattern<\/td><\/tr>
Automatable<\/strong><\/td>A script or tool could handle it<\/td><\/tr>
Tactical<\/strong><\/td>Immediate fix, not a long-term solution<\/td><\/tr>
Reactive<\/strong><\/td>Triggered by alerts\/incidents<\/td><\/tr>
Non-durable<\/strong><\/td>Leaves no lasting improvement to the system<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n

\u2705 Examples of Toil in SRE<\/h2>\n\n\n\n
Example<\/th>Why it\u2019s Toil<\/th><\/tr><\/thead>
Manually restarting failed pods<\/td>Repetitive, automatable<\/td><\/tr>
Fixing disk full alerts every week<\/td>Predictable, non-durable<\/td><\/tr>
Reconfiguring firewall rules by hand<\/td>Can be automated via Terraform or scripts<\/td><\/tr>
Responding to noisy alerts<\/td>Reactive, not value-adding<\/td><\/tr>
Generating weekly reports manually<\/td>Easily scriptable<\/td><\/tr>
Approving routine access requests<\/td>Better handled with IAM automation or workflows<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n

\ud83d\udeab Not All Operational Work is Toil<\/h2>\n\n\n\n
Work Type<\/th>Toil?<\/th>Why<\/th><\/tr><\/thead>
Designing alerting strategy<\/td>\u274c<\/td>Strategic, not repetitive<\/td><\/tr>
Writing Terraform modules<\/td>\u274c<\/td>Automates infrastructure<\/td><\/tr>
Debugging complex outages<\/td>\u274c<\/td>Requires deep thinking, high-value<\/td><\/tr>
Reviewing architectural changes<\/td>\u274c<\/td>Adds long-term value<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n

\ud83e\udde0 Why Toil is Bad<\/h2>\n\n\n\n
    \n
  • Burns out engineers<\/strong><\/li>\n\n\n\n
  • Distracts from innovation<\/strong><\/li>\n\n\n\n
  • Delays projects<\/strong><\/li>\n\n\n\n
  • Doesn\u2019t scale<\/strong><\/li>\n\n\n\n
  • Leads to boredom and attrition<\/strong><\/li>\n<\/ul>\n\n\n\n

    Google recommends that SREs should spend <50% of their time on Toil<\/strong>, and ideally much less.<\/p>\n\n\n\n

    \n\n\n\n

    \ud83d\udd27 How to Identify Toil in Your Environment<\/h2>\n\n\n\n
    Signal<\/th>Interpretation<\/th><\/tr><\/thead>
    You\u2019re doing it more than twice a month<\/td>Candidate for automation<\/td><\/tr>
    Your incident postmortems are repetitive<\/td>Root cause not fixed<\/td><\/tr>
    You can document it easily<\/td>You can probably automate it<\/td><\/tr>
    Onboarding involves lots of manual steps<\/td>It’s Toil<\/td><\/tr>
    SOPs require step-by-step human execution<\/td>Replace with scripts\/pipelines<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    \n\n\n\n

    \ud83d\udee0\ufe0f How to Reduce or Eliminate Toil: A Practical Tutorial<\/h2>\n\n\n\n

    \ud83d\udd39 1. Catalog Your Repetitive Tasks<\/strong><\/h3>\n\n\n\n

    Create a list of all recurring activities:<\/p>\n\n\n\n

    - Weekly report generation\n- Alert acknowledgements\n- Manual deployment steps\n- Restarting services\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    \ud83d\udd39 2. Score Each Task on \u201cToil Scale\u201d<\/strong><\/h3>\n\n\n\n
    Create a scoring matrix:<\/p>\n\n\n\n
    Task<\/th>Manual<\/th>Repetitive<\/th>Automatable<\/th>Urgency<\/th>Toil Score<\/th><\/tr><\/thead>
    Restarting pods<\/td>\u2705<\/td>\u2705<\/td>\u2705<\/td>High<\/td>3<\/td><\/tr>
    Debugging memory leak<\/td>\u274c<\/td>\u274c<\/td>\u274c<\/td>High<\/td>0<\/td><\/tr>
    Weekly CPU report<\/td>\u2705<\/td>\u2705<\/td>\u2705<\/td>Low<\/td>3<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    Focus on tasks with highest Toil Score<\/strong>.<\/p>\n\n\n\n
    \n\n\n\n
    \ud83d\udd39 3. Automate Using Tools<\/strong><\/h3>\n\n\n\n
    Tool<\/th>Use Case<\/th><\/tr><\/thead>
    Bash\/Python<\/strong><\/td>Small scripts to start<\/td><\/tr>
    Ansible<\/strong><\/td>Automate infra\/config<\/td><\/tr>
    Terraform<\/strong><\/td>Infra provisioning<\/td><\/tr>
    Jenkins\/GitHub Actions<\/strong><\/td>CI\/CD workflows<\/td><\/tr>
    Prometheus + Alertmanager<\/strong><\/td>Alerting automation<\/td><\/tr>
    PagerDuty\/Opsgenie<\/strong><\/td>Auto-remediation hooks<\/td><\/tr>
    Runbooks + Automation<\/strong><\/td>Click-to-run fixes<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    \n\n\n\n
    \ud83d\udd39 4. Introduce Auto-Remediation<\/strong><\/h3>\n\n\n\n
    Example: Auto-restart failed pods in Kubernetes<\/p>\n\n\n\n
    apiVersion: v1\nkind: Pod\nmetadata:\n  name: resilient-app\nspec:\n  containers:\n  - name: app\n    image: my-app\n    livenessProbe:\n      httpGet:\n        path: \/healthz\n        port: 8080\n      initialDelaySeconds: 3\n      periodSeconds: 5\n<\/code><\/pre>\n\n\n\n
    With liveness probes<\/strong>, Kubernetes eliminates the need for manual restarts.<\/p>\n\n\n\n
    \n\n\n\n
    \ud83d\udd39 5. Alert Intelligently<\/strong><\/h3>\n\n\n\n
    Bad:<\/p>\n\n\n\n
      \n
    • \u201cDisk usage > 80%\u201d<\/li>\n<\/ul>\n\n\n\n
      Good:<\/p>\n\n\n\n
        \n
      • \u201cDisk usage projected to reach 100% in 3 hours\u201d<\/li>\n<\/ul>\n\n\n\n
        Even better:<\/p>\n\n\n\n
          \n
        • Automatically clean up temp files or notify owner before alerting.<\/li>\n<\/ul>\n\n\n\n
          Use suppressions<\/strong>, alert deduplication<\/strong>, and runbook links<\/strong> to avoid Toil from noisy alerts.<\/p>\n\n\n\n
          \n\n\n\n
          \ud83d\udd39 6. Measure Toil Reduction<\/strong><\/h3>\n\n\n\n
          Track metrics like:<\/p>\n\n\n\n
          Metric<\/th>Example Tool<\/th><\/tr><\/thead>
          Time spent per incident<\/td>Jira, PagerDuty logs<\/td><\/tr>
          # of repeated manual tasks<\/td>Custom spreadsheet<\/td><\/tr>
          Alert volume per on-call<\/td>Prometheus<\/td><\/tr>
          Automation coverage<\/td>CI\/CD stats<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
          Create a Toil Dashboard<\/strong> to monitor improvements.<\/p>\n\n\n\n
          \n\n\n\n
          \ud83d\udcbc Real-World Example: Toil Reduction Project<\/h2>\n\n\n\n
          Problem:<\/h3>\n\n\n\n
          Every on-call SRE had to:<\/p>\n\n\n\n
            \n
          • Manually restart a failing job<\/li>\n\n\n\n
          • Archive logs to S3<\/li>\n\n\n\n
          • Email stakeholders<\/li>\n<\/ul>\n\n\n\n
            Solution:<\/h3>\n\n\n\n
              \n
            1. Created a Jenkins pipeline triggered via API<\/li>\n\n\n\n
            2. Used kubectl<\/code> to auto-restart job<\/li>\n\n\n\n
            3. Uploaded logs via aws s3 cp<\/code><\/li>\n\n\n\n
            4. Sent Slack messages using Webhooks<\/li>\n<\/ol>\n\n\n\n
              Result:<\/strong> 80% reduction in manual toil during on-call.<\/p>\n\n\n\n
              \n\n\n\n
              \ud83d\udcca Summary: SRE Toil Elimination Mindset<\/h2>\n\n\n\n
              Step<\/th>Description<\/th><\/tr><\/thead>
              Identify<\/strong><\/td>Use logs and retros<\/td><\/tr>
              Quantify<\/strong><\/td>Track frequency and time<\/td><\/tr>
              Prioritize<\/strong><\/td>Automate high-toil tasks first<\/td><\/tr>
              Automate<\/strong><\/td>Use the right tool for the task<\/td><\/tr>
              Monitor<\/strong><\/td>Show visible reduction<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
              \n\n\n\n
              \ud83d\udd1a Final Thoughts<\/h2>\n\n\n\n
              Toil is the silent killer of productivity and innovation in SRE. Reducing toil:<\/p>\n\n\n\n
                \n
              • Boosts engineer happiness<\/li>\n\n\n\n
              • Improves system reliability<\/li>\n\n\n\n
              • Frees time for strategic projects<\/li>\n<\/ul>\n\n\n\n
                \n
                “If you do something more than twice, automate it.”<\/strong><\/p>\n<\/blockquote>\n\n\n\n
                \n\n\n\n
                <\/p>\n","protected":false},"excerpt":{"rendered":"
                \ud83d\udea8 What is Toil in SRE? Toil is a term coined by Google SREs to describe a specific class of […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-60","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"\nWhat is Toil? - SRE School<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/sreschool.com\/blog\/what-is-toil\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What is Toil? - SRE School\" \/>\n<meta property=\"og:description\" content=\"\ud83d\udea8 What is Toil in SRE? Toil is a term coined by Google SREs to describe a specific class of […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/sreschool.com\/blog\/what-is-toil\/\" \/>\n<meta property=\"og:site_name\" content=\"SRE School\" \/>\n<meta property=\"article:published_time\" content=\"2025-05-18T15:21:30+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/sreschool.com\/blog\/wp-content\/uploads\/2025\/05\/image-1024x616.png\" \/>\n<meta name=\"author\" content=\"Rajesh Kumar\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Rajesh Kumar\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/sreschool.com\/blog\/what-is-toil\/\",\"url\":\"https:\/\/sreschool.com\/blog\/what-is-toil\/\",\"name\":\"What is Toil? - SRE School\",\"isPartOf\":{\"@id\":\"https:\/\/sreschool.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/sreschool.com\/blog\/what-is-toil\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/sreschool.com\/blog\/what-is-toil\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/sreschool.com\/blog\/wp-content\/uploads\/2025\/05\/image-1024x616.png\",\"datePublished\":\"2025-05-18T15:21:30+00:00\",\"author\":{\"@id\":\"https:\/\/sreschool.com\/blog\/#\/schema\/person\/0ffe446f77bb2589992dbe3a7f417201\"},\"breadcrumb\":{\"@id\":\"https:\/\/sreschool.com\/blog\/what-is-toil\/#breadcrumb\"},\"inLanguage\":\"en\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/sreschool.com\/blog\/what-is-toil\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en\",\"@id\":\"https:\/\/sreschool.com\/blog\/what-is-toil\/#primaryimage\",\"url\":\"https:\/\/sreschool.com\/blog\/wp-content\/uploads\/2025\/05\/image.png\",\"contentUrl\":\"https:\/\/sreschool.com\/blog\/wp-content\/uploads\/2025\/05\/image.png\",\"width\":1225,\"height\":737},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/sreschool.com\/blog\/what-is-toil\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/sreschool.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"What is Toil?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/sreschool.com\/blog\/#website\",\"url\":\"https:\/\/sreschool.com\/blog\/\",\"name\":\"SRESchool\",\"description\":\"Master SRE. Build Resilient Systems. Lead the Future of Reliability\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/sreschool.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/sreschool.com\/blog\/#\/schema\/person\/0ffe446f77bb2589992dbe3a7f417201\",\"name\":\"Rajesh Kumar\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en\",\"@id\":\"https:\/\/sreschool.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/f901a4f2929fa034a291a8363d589791d5a3c1f6a051c22e744acb8bfc8e022a?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/f901a4f2929fa034a291a8363d589791d5a3c1f6a051c22e744acb8bfc8e022a?s=96&d=mm&r=g\",\"caption\":\"Rajesh Kumar\"},\"sameAs\":[\"http:\/\/sreschool.com\/blog\"],\"url\":\"https:\/\/sreschool.com\/blog\/author\/admin\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"What is Toil? - SRE School","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/sreschool.com\/blog\/what-is-toil\/","og_locale":"en_US","og_type":"article","og_title":"What is Toil? - SRE School","og_description":"\ud83d\udea8 What is Toil in SRE? Toil is a term coined by Google SREs to describe a specific class of […]","og_url":"https:\/\/sreschool.com\/blog\/what-is-toil\/","og_site_name":"SRE School","article_published_time":"2025-05-18T15:21:30+00:00","og_image":[{"url":"https:\/\/sreschool.com\/blog\/wp-content\/uploads\/2025\/05\/image-1024x616.png","type":"","width":"","height":""}],"author":"Rajesh Kumar","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Rajesh Kumar","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/sreschool.com\/blog\/what-is-toil\/","url":"https:\/\/sreschool.com\/blog\/what-is-toil\/","name":"What is Toil? - SRE School","isPartOf":{"@id":"https:\/\/sreschool.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/sreschool.com\/blog\/what-is-toil\/#primaryimage"},"image":{"@id":"https:\/\/sreschool.com\/blog\/what-is-toil\/#primaryimage"},"thumbnailUrl":"https:\/\/sreschool.com\/blog\/wp-content\/uploads\/2025\/05\/image-1024x616.png","datePublished":"2025-05-18T15:21:30+00:00","author":{"@id":"https:\/\/sreschool.com\/blog\/#\/schema\/person\/0ffe446f77bb2589992dbe3a7f417201"},"breadcrumb":{"@id":"https:\/\/sreschool.com\/blog\/what-is-toil\/#breadcrumb"},"inLanguage":"en","potentialAction":[{"@type":"ReadAction","target":["https:\/\/sreschool.com\/blog\/what-is-toil\/"]}]},{"@type":"ImageObject","inLanguage":"en","@id":"https:\/\/sreschool.com\/blog\/what-is-toil\/#primaryimage","url":"https:\/\/sreschool.com\/blog\/wp-content\/uploads\/2025\/05\/image.png","contentUrl":"https:\/\/sreschool.com\/blog\/wp-content\/uploads\/2025\/05\/image.png","width":1225,"height":737},{"@type":"BreadcrumbList","@id":"https:\/\/sreschool.com\/blog\/what-is-toil\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/sreschool.com\/blog\/"},{"@type":"ListItem","position":2,"name":"What is Toil?"}]},{"@type":"WebSite","@id":"https:\/\/sreschool.com\/blog\/#website","url":"https:\/\/sreschool.com\/blog\/","name":"SRESchool","description":"Master SRE. Build Resilient Systems. Lead the Future of Reliability","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/sreschool.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en"},{"@type":"Person","@id":"https:\/\/sreschool.com\/blog\/#\/schema\/person\/0ffe446f77bb2589992dbe3a7f417201","name":"Rajesh Kumar","image":{"@type":"ImageObject","inLanguage":"en","@id":"https:\/\/sreschool.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/f901a4f2929fa034a291a8363d589791d5a3c1f6a051c22e744acb8bfc8e022a?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/f901a4f2929fa034a291a8363d589791d5a3c1f6a051c22e744acb8bfc8e022a?s=96&d=mm&r=g","caption":"Rajesh Kumar"},"sameAs":["http:\/\/sreschool.com\/blog"],"url":"https:\/\/sreschool.com\/blog\/author\/admin\/"}]}},"_links":{"self":[{"href":"https:\/\/sreschool.com\/blog\/wp-json\/wp\/v2\/posts\/60","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sreschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sreschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sreschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sreschool.com\/blog\/wp-json\/wp\/v2\/comments?post=60"}],"version-history":[{"count":1,"href":"https:\/\/sreschool.com\/blog\/wp-json\/wp\/v2\/posts\/60\/revisions"}],"predecessor-version":[{"id":63,"href":"https:\/\/sreschool.com\/blog\/wp-json\/wp\/v2\/posts\/60\/revisions\/63"}],"wp:attachment":[{"href":"https:\/\/sreschool.com\/blog\/wp-json\/wp\/v2\/media?parent=60"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sreschool.com\/blog\/wp-json\/wp\/v2\/categories?post=60"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sreschool.com\/blog\/wp-json\/wp\/v2\/tags?post=60"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}