What is Major incident? Meaning, Architecture, Examples, Use Cases, and How to Measure It (2026 Guide)

---

Quick Definition (30–60 words)

A major incident is a service outage or degradation causing substantial business impact that requires coordinated cross-team response. Analogy: a multi-car pileup on a highway that blocks traffic and needs traffic control, tow trucks, and medical teams. Formal: an incident declared when impact thresholds and escalation policies meet pre-defined major incident criteria.

---

What is Major incident?

A major incident is an escalation tier for incidents that exceed routine on-call handling. It is NOT a routine bug, minor outage, or scheduled maintenance. It requires cross-discipline coordination, executive visibility, and often temporary mitigation work instead of immediate root cause fixes.

Key properties and constraints:

• High-impact: affects large user segments, revenue, or critical business processes.
• Fast escalation: declaration triggers specific communication and resource allocation.
• Time-bounded goal: focus on restoring service, minimizing harm, and preserving evidence.
• Governance: follows playbooks, runbooks, and accountability assignments.
• Post-incident: triggers detailed postmortem and remediations with timelines.

Where it fits in modern cloud/SRE workflows:

• Detection via SLIs and alerting rules.
• Triage by on-call or triage team.
• Major incident declared when impact thresholds met.
• War-room style coordination with incident commander, communications lead, and engineering leads.
• Temporary mitigations, rollback, or failover applied.
• Transition to remediation and postmortem with corrective actions and SLO impact accounting.

Diagram description (text-only):

• Monitoring layer detects anomaly -> Alert router evaluates severity -> If severity >= threshold, trigger major incident -> Notify incident manager, paging, and incident workspace -> Triage and implement mitigation (rollback/failover/scaling) -> Monitor restoration -> Postmortem and remediation tasks assigned.

Major incident in one sentence

A major incident is a high-severity, cross-functional outage requiring immediate, coordinated response to restore service and mitigate business impact.

Major incident vs related terms (TABLE REQUIRED)

ID	Term	How it differs from Major incident	Common confusion
T1	Outage	Less scope and impact than major incident	People call any outage major
T2	Incident	Generic term that may be low or high severity	Not all incidents are major
T3	P0	Priority label often maps to major incident but varies	P0 versus major sometimes conflated
T4	Incident report	Post-event documentation not the live response	Confused with live incident command
T5	Outage window	Scheduled downtime not an incident	People equate downtime with incident
T6	Degradation	Partial functionality loss, may or may not be major	Degradation vs outage confusion
T7	Major outage	Synonym in some organizations	Terminology varies by org
T8	Disaster recovery	Broader strategy for catastrophic events	DR is not the same as day-to-day incidents
T9	Security incident	Involves breach and special handling	Security incidents may be declared separately
T10	Crisis	Business-level emergency beyond tech scope	Crisis includes PR and legal considerations

Row Details (only if any cell says “See details below”)

• None.

---

Why does Major incident matter?

Business impact:

• Revenue loss: degraded checkout or API throughput reduces transactions.
• Brand trust: repeated major incidents reduce user retention and partner confidence.
• Regulatory risk: outages can trigger compliance reporting and fines.
• Opportunity cost: executives divert time to crisis management.

Engineering impact:

• Velocity slows as engineers shift to firefighting.
• Technical debt grows if quick fixes are not remediated.
• On-call fatigue increases, hurting retention.
• Improved practices emerge when incidents are analyzed and corrected.

SRE framing:

• SLIs and SLOs detect trends and set thresholds for major incident declaration.
• Error budgets guide trade-offs between feature delivery and reliability.
• Toil reduction is a primary SRE goal to avoid frequent major incidents.
• On-call rotations must reflect realistic major incident workload.

3–5 realistic “what breaks in production” examples:

• Global auth service failing under a schema change, blocking login globally.
• Managed database provider experiencing failover loop causing request errors.
• Kubernetes control plane API throttle due to misconfigured autoscaler.
• Edge CDN misconfiguration causing cache poisoning and serving stale content.
• Payment gateway regional outage resulting in failed transactions.

---

Where is Major incident used? (TABLE REQUIRED)

ID	Layer/Area	How Major incident appears	Typical telemetry	Common tools
L1	Edge/Network	Global connectivity loss or DDoS	High error rate, RTT spikes	WAF/CDN logs, network monitors
L2	Service/API	API 5xx surge or latency spike	5xx rate, p95 latency	APM, service metrics
L3	Application	Authentication or core flows broken	Error traces, user complaints	Tracing, logs
L4	Data/DB	DB failover or corruption	Replica lag, transaction errors	DB monitoring, backups
L5	Platform/K8s	Control plane issues or node drain	Pod failures, API errors	K8s metrics, control plane logs
L6	Serverless/PaaS	Throttling or cold-start storms	Invocation errors, throttles	Platform metrics, invocation logs
L7	CI/CD	Bad deploy causing mass failures	Deploy rollbacks, new errors	CI logs, deployment traces
L8	Security	Compromise detected with impact	Alert count, unusual activity	SIEM, IDS
L9	Observability	Telemetry gaps during outage	Missing metrics, delayed logs	Monitoring tools
L10	Billing/Cost	Unexpected cost spike causing limits	Budget alert, quota reached	Cloud billing alerts

Row Details (only if needed)

• None.

---

When should you use Major incident?

When it’s necessary:

• Service affecting large user base or revenue.
• Critical functionality broken for high-value flows.
• Multi-system outages or cross-region failures.
• Regulatory or security impact requiring expedited action.

When it’s optional:

• Localized region outage affecting subset of users.
• Single microservice degraded but can be mitigated by retries.
• Non-critical feature failures with low user impact.

When NOT to use / overuse it:

• Using major incident for every high-severity pager creates fatigue.
• Avoid declaring for routine maintenance or expected degradations.
• Do not declare when automated failover already resolved issue without human coordination.

Decision checklist:

• If 5xx rate > X and affected users > Y -> declare major incident.
• If SLO burn-rate > Z over 15m and no automated mitigation -> declare.
• If security breach with data exfiltration -> declare security incident (use specialized workflow).
• If localized and mitigated by a single owner within T minutes -> no major.

Maturity ladder:

• Beginner: Manual declaration, email/slack pages, basic runbooks.
• Intermediate: Automated detection, incident commander rotation, war-room templates.
• Advanced: Automated mitigation playbooks, multi-cloud failovers, AI-assisted triage, integrated postmortem pipelines.

---

How does Major incident work?

Components and workflow:

1. Detection: SLIs trigger alerts; anomaly detection flags unusual patterns.
2. Triage: On-call triages and determines severity.
3. Declaration: Incident commander declared; communication channels opened.
4. Coordination: Roles assigned (IC, communications, tech leads, scribe).
5. Mitigation: Execute runbooks, mitigations, rollbacks, or failovers.
6. Restoration: Monitor restoration and confirm impact reduced.
7. Recovery: Stabilize systems and transition to remediation.
8. Postmortem: Document RCA, actions, and timelines.

Data flow and lifecycle:

• Metrics and logs -> alerting engine -> incident system -> human action -> mitigation -> telemetry shows improvement -> incident closed -> postmortem logged.

Edge cases and failure modes:

• Alerting system down: fallback escalation via phone/SMS.
• Communication channel overloaded: pre-configured backup channels.
• Multiple concurrent majors: designate escalation tier and prioritize by business impact.

Typical architecture patterns for Major incident

• Centralized incident command: Single IC with global view; use when multiple teams involved.
• Federated incident hubs: Team-level ICs coordinated through a central coordinator; use in large orgs.
• Automated rollback/failover: Automated playbooks for well-defined failures.
• Circuit breaker and feature flag fallback: Use when recent deploys introduce risk.
• Multi-region failover: For cloud-native apps with active-passive regions.
• Canary isolation: Isolate failing service via routing rules and progressive traffic shifting.

Failure modes & mitigation (TABLE REQUIRED)

ID	Failure mode	Symptom	Likely cause	Mitigation	Observability signal
F1	Alert storm	Many alerts in short time	Upstream service spike or flapping	Throttle alerts, aggregate	High alert rate
F2	Missing telemetry	Dashboards blank	Logging pipeline failed	Switch to backup pipeline	Missing metrics
F3	Incorrect escalation	Wrong on-call paged	Misconfigured routing	Update escalation policy	Pager logs show misroute
F4	Runbook not found	Teams confused	Poor documentation	Create and publish runbook	Search failures
F5	Communication overload	Channel clogged	No structured updates	Use status updates cadence	Message rate spike
F6	Automated rollback fails	New errors after rollback	Incomplete rollback steps	Manual rollback path	Deploy trace shows failure
F7	Cross-region sync failure	Data inconsistency	Replication lag or network	Promote backups, re-sync	Replication lag metric

Row Details (only if needed)

• None.

---

Key Concepts, Keywords & Terminology for Major incident

(40+ terms; each line: Term — 1–2 line definition — why it matters — common pitfall)

Service Level Indicator (SLI) — A quantitative measure of service performance, e.g., success rate — Measures user-facing behavior — Pitfall: choosing irrelevant SLI Service Level Objective (SLO) — Target for an SLI over time — Sets reliability goals — Pitfall: unrealistic targets Error budget — Allowable SLO breach over time — Enables trade-offs for changes — Pitfall: not tracking consumption Incident commander (IC) — Single person coordinating response — Provides authority and decisions — Pitfall: ambiguous IC handoff War room — Communication channel for incident coordination — Centralizes info flow — Pitfall: unstructured chat noise Runbook — Step-by-step remediation guide — Speeds mitigation — Pitfall: stale runbooks Playbook — Higher-level response plan for classes of incidents — Aligns teams — Pitfall: too generic Pagerduty rotation — On-call schedule system — Ensures 24×7 coverage — Pitfall: over-alerting operators Pager fatigue — Burnout from repetitive pages — Causes retention issues — Pitfall: not addressing noisy alerts Postmortem — Detailed incident analysis document — Drives learning — Pitfall: blamelessness missing Root cause analysis (RCA) — Investigation into underlying cause — Prevents recurrence — Pitfall: premature RCA Mitigation — Temporary actions to reduce impact — Restores user service fast — Pitfall: leaving mitigation permanent Remediation — Permanent fix addressing root cause — Eliminates recurrence — Pitfall: delayed remediation SLA (Service Level Agreement) — Contractual reliability promise — Affects penalties and trust — Pitfall: misaligned SLA and SLO Observation window — Time period for evaluating SLOs — Defines measurement span — Pitfall: wrong window masking trends Alert burn rate — Rate of SLO consumption — Helps pace responses — Pitfall: miscalculation leads to wrong escalation Anomaly detection — Automated detection of abnormal behavior — Faster detection than static thresholds — Pitfall: false positives Synthetic monitoring — Simulated user checks — Detects endpoint regressions — Pitfall: false negatives vs real user flows Real-user monitoring (RUM) — Collects client-side metrics — Measures actual user impact — Pitfall: sampling bias Tracing — Distributed tracing across services — Pinpoints latency sources — Pitfall: incomplete traces Logs — Event records from systems — Essential for forensic analysis — Pitfall: not centralized Metrics — Quantitative counters and gauges — Primary input for alarms — Pitfall: cardinality issues Dashboards — Visual representations of telemetry — Rapid situational awareness — Pitfall: cluttered dashboards Escalation policy — Rules mapping alerts to responders — Ensures appropriate response — Pitfall: outdated contacts Incident lifecycle — Stages from detection to postmortem — Framework for workflows — Pitfall: skipping steps Service map — Dependency graph of services — Shows blast radius — Pitfall: not maintained Blast radius — Scope of impact from an event — Prioritizes response — Pitfall: underestimated dependencies Failover — Switching to backup system — Reduces downtime — Pitfall: failover not tested Rollback — Reverting to previous state or version — Rapid mitigation for bad deploys — Pitfall: data schema incompatibility Feature flag — Toggle to control features at runtime — Enables surgical mitigation — Pitfall: flag entanglement Canary deploy — Gradual rollout to subset of users — Limits blast radius — Pitfall: canary not representative Chaos engineering — Controlled failure injection — Validates resilience — Pitfall: inadequate safety guards Automation playbook — Scripted remediation tasks — Reduces toil and error — Pitfall: over-reliance without human oversight Incident budget — Time allocated for major incidents in SRE plan — Resource planning — Pitfall: misalignment with actual load On-call runbook — Quick actions for on-call responders — Increases speed — Pitfall: too verbose Scribe — Incident note taker — Keeps timeline record — Pitfall: missing timestamps Communications lead — Manages external and internal messaging — Maintains trust — Pitfall: inconsistent messaging Backfill — Restoring data after outage — Ensures correctness — Pitfall: silent data loss Observability debt — Missing telemetry or poor instrumentation — Hinders diagnosis — Pitfall: deferred instrumentation Post-incident action (PIA) — Tasks from postmortem — Drives remediation — Pitfall: action items not tracked to completion Blameless culture — Focus on system fixes not people — Encourages openness — Pitfall: lack of accountability Major incident playbook — Organization-specific document — Standardizes response — Pitfall: not practiced

---

How to Measure Major incident (Metrics, SLIs, SLOs) (TABLE REQUIRED)

ID	Metric/SLI	What it tells you	How to measure	Starting target	Gotchas
M1	Success rate	Fraction of successful requests	Successful responses / total	99.9% for critical APIs	Depends on traffic patterns
M2	Request latency p95	Tail latency user experiences	Measure p95 from trace or metric	<300ms for web UI	Outliers can hide distribution
M3	Error rate by code	Root cause by error class	5xx count / total requests	<0.1% for core flows	Aggregation may hide service-level issues
M4	Availability	Uptime per SLO window	Time service responds correctly / window	99.95% regionally	Maintenance windows affect calc
M5	SLO burn rate	Speed of error budget consumption	Error rate vs SLO over time	Alert at 2x burn rate	Short windows noisy
M6	MTTR	Mean time to restore service	Time from detection to fix	<60m for P0s (varies)	Depends on mitigation types
M7	MTTA	Mean time to acknowledge alerts	Time from alert to human ack	<5m for majors	High false positives inflate
M8	Incident frequency	How often majors occur	Count per quarter	<1 per quarter per service	Colocation of services skews
M9	User impact count	Number of affected users/events	Unique users with failures	Minimal acceptable per product	Privacy and sampling issues
M10	Cost impact	Cloud cost increase from incident	Delta in billing for incident window	Varies by business	Hard to compute in real time

Row Details (only if needed)

• M1: Compute on a per-endpoint basis and aggregate to service level. Use weighted averages for traffic splits.
• M5: Burn rate is best measured over multiple windows (15m, 1h, 24h).
• M6: MTTR should separate detection-to-mitigation and mitigation-to-remediation.

Best tools to measure Major incident

Tool — Observability Platform (APM/metrics tracing)

• What it measures for Major incident: Traces, latencies, error rates, distributed context
• Best-fit environment: Microservices in cloud or hybrid apps
• Setup outline:
• Instrument services for traces and metrics
• Configure sampling and trace headers
• Correlate traces with logs and metrics
• Strengths:
• Fast root cause paths
• Correlated distributed traces
• Limitations:
• High cardinality costs
• Sampling gaps may miss low-frequency errors

Tool — Logging Pipeline

• What it measures for Major incident: Application and infrastructure events
• Best-fit environment: All environments needing forensic detail
• Setup outline:
• Centralize logs
• Enrich logs with trace IDs
• Ensure retention and indexing strategy
• Strengths:
• Forensic troubleshooting
• Flexible queries
• Limitations:
• Cost and volume management
• Not real-time if ingestion lags

Tool — Synthetic Monitoring

• What it measures for Major incident: End-to-end user flows and availability
• Best-fit environment: Public-facing APIs and web UIs
• Setup outline:
• Create synthetic journeys
• Schedule checks globally
• Alert on failures and latency thresholds
• Strengths:
• Early detection of regressions
• SLA proof points
• Limitations:
• May not reflect real-user conditions
• Maintenance overhead for scripts

Tool — Incident Management Platform

• What it measures for Major incident: Pages, actions, timelines, ownership
• Best-fit environment: Organizations with distributed teams
• Setup outline:
• Integrate alerting and on-call schedules
• Define escalation policies
• Create incident templates
• Strengths:
• Coordination and audit trail
• Role-based routing
• Limitations:
• Process overhead if misused
• May centralize decision-making too much

Tool — Chaos Engineering Tools

• What it measures for Major incident: System resilience and failover behavior
• Best-fit environment: Mature teams with staging and safeguards
• Setup outline:
• Define hypotheses
• Run safe experiments
• Measure outcomes against SLIs
• Strengths:
• Proactive discovery of weak points
• Improves runbooks
• Limitations:
• Risk if experiments not scoped
• Requires investment in automation

Recommended dashboards & alerts for Major incident

Executive dashboard:

• Uptime by critical service: shows current availability vs SLO.
• Business metrics: checkout rate, payments processed, revenue delta.
• Active major incidents count and status.
• SLO burn rate summary. Why: Executives need impact, scope, and trending.

On-call dashboard:

• Real-time errors by service and region.
• Top alerts with severity and owner.
• Recent deploys and change history.
• Active mitigation steps and runbook links. Why: On-call needs immediate actionable info and context.

Debug dashboard:

• Traces for failure paths and top slow traces.
• Error logs grouped by root cause.
• Infrastructure metrics (CPU, memory, network).
• Deployment timelines and traffic routing. Why: Engineers debug and verify fixes.

Alerting guidance:

• Page for urgent (service down, data loss) incidents.
• Ticket for degradations that need attention but not immediate coordination.
• Use burn-rate alerts for SLO consumption: page when burn rate > 5x sustained over 15 minutes.
• Noise reduction: dedupe repeated alerts, group by root cause, silence known noisy windows, use alert suppression for automated mitigation.
• Use correlation rules to avoid paging for transient upstream blips.

---

Implementation Guide (Step-by-step)

1) Prerequisites – Clear SLOs for critical services. – On-call rotation and escalation policies. – Instrumentation for metrics, logs, and traces. – Incident management platform and communication channels. – Runbooks and playbooks for known failure modes.

2) Instrumentation plan – Identify critical user journeys and services. – Define SLIs and required telemetry for each SLI. – Add trace IDs to logs and propagate context across services. – Implement synthetic checks for key flows.

3) Data collection – Centralize metrics, logs, and traces. – Ensure redundancy in telemetry pipelines. – Set retention policies for postmortem evidence. – Configure alerting rules and anomaly detectors.

4) SLO design – Choose meaningful SLIs aligned to user experience. – Set SLO windows (30d, 90d) and targets based on risk tolerance. – Define error budgets and escalation thresholds.

5) Dashboards – Build three tiers: executive, on-call, debug. – Use service maps and dependency views. – Link dashboards to runbooks and incident pages.

6) Alerts & routing – Define alert priorities and what pages vs tickets. – Configure escalation policies and phone/SMS fallbacks. – Implement dedupe and grouping rules.

7) Runbooks & automation – Create runbooks for top failure scenarios. – Automate safe mitigations (traffic shift, rollback). – Keep runbooks versioned and test them in staging.

8) Validation (load/chaos/game days) – Regularly run game days simulating major incidents. – Test failover, rollback, and communication procedures. – Measure MTTR and refine processes.

9) Continuous improvement – Conduct blameless postmortems with actionable PIAs. – Track completion of action items. – Review SLOs and adjust alerts to reduce noise.

Checklists:

Pre-production checklist

• SLOs defined for critical flows.
• Synthetic checks in place.
• Rollback and deployment automation validated.
• Observability for services enabled.
• Runbooks written and accessible.

Production readiness checklist

• On-call rotation assigned.
• Escalation policies tested.
• Incident tooling integrated with chat and paging.
• Communication templates prepared.
• Backups and failover tested.

Incident checklist specific to Major incident

• Declare incident and open incident page.
• Assign IC, communications lead, scribe.
• Post initial status update within 10 minutes.
• Execute mitigation runbook and record actions.
• Monitor telemetry and update stakeholders regularly.
• Capture timeline and evidence for postmortem.

---

Use Cases of Major incident

(8–12 concise use cases)

1) Global login failure – Context: Auth service returns 500s. – Problem: Users cannot access accounts. – Why Major incident helps: Coordinates multiple teams (auth, DB, infra). – What to measure: Success rate for /login, latency, DB errors. – Typical tools: Tracing, DB monitors, incident platform.

2) Payment processing outage – Context: Payment gateway region degraded. – Problem: Transactions failing, revenue loss. – Why Major incident helps: Rapid failover and business comms. – What to measure: Transaction success, queue lengths. – Typical tools: Payment monitor, dashboard, billing alerts.

3) K8s control plane API high latency – Context: API throttle causing pod scheduling failures. – Problem: New pods failing and deployments stuck. – Why Major incident helps: Orchestrates platform and app teams. – What to measure: API latency, pod restart rate. – Typical tools: K8s metrics, control plane logs.

4) Database corruption discovered – Context: Erroneous write pattern corrupted a table. – Problem: Data integrity and customer trust at risk. – Why Major incident helps: Coordinates recovery, legal, and comms. – What to measure: Corrupt rows, replication status. – Typical tools: DB backup systems, logs.

5) CDN misconfiguration serving stale content – Context: Cache invalidation failed globally. – Problem: Users see outdated data and errors. – Why Major incident helps: Coordinates CDN provider, cache purges. – What to measure: Cache hit/miss, response headers. – Typical tools: CDN logs, synthetic checks.

6) Security breach detection – Context: Suspicious data exfiltration patterns. – Problem: Potential data leak requiring legal response. – Why Major incident helps: Engage security, legal, comms. – What to measure: Anomalous access counts, IP patterns. – Typical tools: SIEM, auth logs.

7) Billing spikes causing quotas – Context: Unexpected autoscaling increased costs and triggered quotas. – Problem: Services throttled by provider limits. – Why Major incident helps: Mitigate cost and maintain service. – What to measure: Cost per hour, quota use. – Typical tools: Cloud billing, autoscaler metrics.

8) Third-party API outage – Context: External provider returns errors. – Problem: Dependent features fail. – Why Major incident helps: Decide degrade vs wait and route traffic. – What to measure: External API success rate, fallback efficacy. – Typical tools: Synthetic monitors, circuit breaker metrics.

---

Scenario Examples (Realistic, End-to-End)

Scenario #1 — Kubernetes API throttling and app failures

Context: K8s control plane experiencing high request latencies causing scheduling failures. Goal: Restore scheduling and prevent cascading pod restarts. Why Major incident matters here: Affects many services and can create a cluster-wide blackout. Architecture / workflow: K8s API -> controllers -> kubelets -> workloads. Step-by-step implementation:

• Detect via control plane API latency alert.
• Declare major incident; assign IC and platform lead.
• Reduce create/delete activity by pausing CI/CD pipelines.
• Scale control plane or move workloads to standby cluster.
• Rollback recent control-plane-affecting changes.
• Monitor pod scheduling and API latency. What to measure: API p95, pod pending count, controller errors. Tools to use and why: K8s metrics, control plane logs, incident platform. Common pitfalls: Not pausing CI leads to continued pressure. Validation: Run create/delete load tests in staging after fix. Outcome: Cluster stabilized and scheduling restored; postmortem action to rate-limit controllers.

Scenario #2 — Serverless cold-start storm causing function timeouts

Context: Sudden traffic spike to serverless function causing high cold starts and timeouts. Goal: Reduce errors and stabilize latency. Why Major incident matters here: Many upstream services rely on low-latency functions. Architecture / workflow: API Gateway -> Serverless functions -> Downstream DB. Step-by-step implementation:

• Detect increased 5xx and latency via SLI alerts.
• Declare incident and engage platform and dev teams.
• Enable provisioned concurrency or scale warmers.
• Apply rate limiting at edge and degrade noncritical features.
• Monitor invocation success and latency. What to measure: Invocation error rate, cold start duration, provisioned concurrency fill. Tools to use and why: Platform metrics, APM, synthetic checks. Common pitfalls: Provisioned concurrency adds cost and provisioning delay. Validation: Load test with warmers and synthetic checks. Outcome: Errors reduced; plan to add adaptive concurrency and circuit breakers.

Scenario #3 — Postmortem-driven remediation after major incident

Context: Service outage due to bad schema migration. Goal: Document root cause and implement remediation. Why Major incident matters here: Data loss risk and repeated rollbacks. Architecture / workflow: Application -> DB migration -> downstream services. Step-by-step implementation:

• During incident, roll back application and restore DB from backups.
• After stabilization, declare postmortem and collect timeline.
• Analyze migration process, permissions, and testing gaps.
• Implement gatekeeping for migrations and add pre-deploy checks. What to measure: Number of failed migrations, time to rollback. Tools to use and why: DB logs, CI/CD logs, version control. Common pitfalls: Skipping postmortem or not tracking action completion. Validation: Run game day for migration path. Outcome: Migration gating implemented, reduced risk.

Scenario #4 — Cost/performance trade-off under heavy load

Context: Scaling out to handle traffic increases costs beyond budget and triggers provider quota. Goal: Balance cost with acceptable performance while restoring service. Why Major incident matters here: Financial overruns and service degradation risk. Architecture / workflow: Auto-scaling group -> instances -> load balancer. Step-by-step implementation:

• Detect cost and quota alerts; declare incident.
• Apply throttles to non-critical traffic and use degraded modes.
• Shift traffic to cheaper compute paths or reserved capacity.
• Iterate on scaling policies and implement load shedding strategies. What to measure: Cost per request, response time, queue lengths. Tools to use and why: Cloud billing, autoscaler metrics, traffic management tools. Common pitfalls: Sudden throttles harm user experience. Validation: Simulate traffic spikes and cost impact in staging. Outcome: Temporary cost controls, longer-term autoscaling policy changes.

Scenario #5 — Third-party API outage with internal fallback

Context: Payment provider API returns 503s. Goal: Keep payment flow working using fallback options. Why Major incident matters here: Direct revenue impact and refunds risk. Architecture / workflow: Checkout -> Payment provider -> Confirmation. Step-by-step implementation:

• Alert triggers major incident.
• Route traffic to backup provider or offline payment queuing.
• Notify stakeholders and initiate customer messaging.
• Monitor success rate and process queued payments. What to measure: Payment success, queue size, retry success. Tools to use and why: Payment gateway metrics, queue monitors. Common pitfalls: Data duplication or double charges. Validation: Test fallback flow with synthetic transactions. Outcome: Payments processed via fallback; contract review with primary provider.

---

Common Mistakes, Anti-patterns, and Troubleshooting

(15–25 mistakes with Symptom -> Root cause -> Fix; include 5 observability pitfalls)

1) Symptom: Pages keep firing for same error -> Root cause: alerts not deduped -> Fix: group by root cause and add suppression 2) Symptom: On-call overwhelmed -> Root cause: too many major incident declarations -> Fix: stricter declaration criteria and training 3) Symptom: Postmortems missing -> Root cause: No ownership after incident -> Fix: require postmortem within SLA and track PIAs 4) Symptom: Runbooks outdated -> Root cause: No versioning or reviews -> Fix: schedule runbook reviews and test runbooks 5) Symptom: Dashboards blank during outage -> Root cause: telemetry pipeline outage -> Fix: redundant telemetry paths and synthetic monitors 6) Symptom: Wrong person paged -> Root cause: stale escalation policy -> Fix: update on-call schedules and verify contacts 7) Symptom: Rollback fails -> Root cause: DB schema incompatible -> Fix: add backward-compatible migrations and preflight checks 8) Symptom: High MTTR -> Root cause: missing instrumentation -> Fix: add traces and logs for critical paths 9) Symptom: Executive surprises -> Root cause: no exec notification procedure -> Fix: predefine communication templates and cadence 10) Symptom: Noise from transient errors -> Root cause: low-threshold alerts -> Fix: add aggregation and anomaly detection 11) Symptom: Security incident handled like regular outage -> Root cause: lack of security-specific playbook -> Fix: create security incident process 12) Symptom: Data loss discovered late -> Root cause: insufficient backups and verification -> Fix: backup policies and restore drills 13) Symptom: Cost surge during mitigation -> Root cause: uncontrolled autoscale or fallback -> Fix: cost-aware mitigation strategies 14) Symptom: Multiple concurrent majors -> Root cause: lack of prioritization -> Fix: central coordination and business-impact scoring 15) Symptom: Blame culture after incident -> Root cause: poor postmortem facilitation -> Fix: enforce blameless language and focus on systems 16) Observability pitfall: Missing correlation IDs -> Root cause: not propagating trace IDs -> Fix: enforce propagation and enrich logs 17) Observability pitfall: High-cardinality metrics exploding cost -> Root cause: unbounded labels -> Fix: sanitize labels and sample 18) Observability pitfall: Logs not centralized -> Root cause: local logging only -> Fix: centralized logging with retention policy 19) Observability pitfall: Metrics delayed due to batching -> Root cause: long ingestion windows -> Fix: reduce batch windows for critical metrics 20) Observability pitfall: Tracing disabled in production -> Root cause: performance fears -> Fix: sampling and low-overhead tracing 21) Symptom: Communication chaos -> Root cause: no communications lead -> Fix: assign communications role on declaration 22) Symptom: Incident page lacks timeline -> Root cause: no scribe -> Fix: designate scribe and require timestamped entries 23) Symptom: Remediations never completed -> Root cause: no tracking or accountability -> Fix: assign owners and track to completion

---

Best Practices & Operating Model

Ownership and on-call:

• Define clear ownership for services and SLOs.
• Rotate IC and on-call to avoid burnout.
• Provide financial and career recognition for on-call work.

Runbooks vs playbooks:

• Runbooks: step-by-step fixes for specific failures.
• Playbooks: decision trees and coordination templates for classes of incidents.
• Keep runbooks tight, testable, and accessible.

Safe deployments:

• Use canary and progressive rollouts.
• Feature flags to disable problematic features quickly.
• Automated rollback on key error thresholds.

Toil reduction and automation:

• Automate detection-to-mitigation paths where safe.
• Reduce manual repetitive tasks such as log collection and paging.
• Audit automated actions and provide clear human override.

Security basics:

• Separate incident flows for security incidents.
• Lock down access during incidents and rotate credentials if compromised.
• Maintain audit trails and preserve forensic data.

Weekly/monthly routines:

• Weekly: review high-severity alerts and action items.
• Monthly: runbook drills and observability audits.
• Quarterly: SLO and incident frequency review with execs.

What to review in postmortems:

• Timeline and detection-to-mitigation times.
• Root cause and contributing factors.
• Action items with owners and deadlines.
• SLO impact and whether SLAs were breached.
• Lessons and process updates for future prevention.

---

Tooling & Integration Map for Major incident (TABLE REQUIRED)

ID	Category	What it does	Key integrations	Notes
I1	Metrics/Monitoring	Collects and visualizes metrics	Alerting, dashboards, tracing	Core for detection
I2	Tracing/APM	Distributed traces and span context	Logs, metrics, issue trackers	Critical for root cause
I3	Logging	Centralized logs and search	Tracing, alerts, incident pages	Forensics and audits
I4	Incident Mgmt	Pages, tracks incidents, roles	Chat, monitoring, ticketing	Orchestrates response
I5	ChatOps	Real-time coordination and automation	Incident Mgmt, CI/CD, tools	Executes runbook commands
I6	CI/CD	Deploys and can rollback releases	Feature flags, monitoring	Can trigger incidents
I7	Feature Flags	Control feature exposure at runtime	CI/CD, monitoring	Used for mitigation
I8	Chaos Tools	Inject faults to validate resilience	Scheduling, monitoring	For proactive testing
I9	SIEM	Security analysis and alerts	Logs, identity systems	For security incidents
I10	Cost/Billing	Tracks cloud spend and quotas	Monitoring, alerts	For cost-related incidents

Row Details (only if needed)

• None.

---

Frequently Asked Questions (FAQs)

What exactly qualifies as a major incident?

A major incident is declared when impact thresholds in your incident policy are met, such as significant user impact, revenue loss, or regulatory exposure.

Who should declare a major incident?

Typically the on-call engineer or triage lead can declare, but organizations may require a platform or product lead to confirm based on policy.

How does a major incident differ from a P0?

P0 is a priority label; many organizations map P0 to major incidents but definitions vary by team.

How long should a major incident stay declared?

Until the service is restored to acceptable impact levels and temporary mitigations stabilize the system; often until the first post-incidence handoff.

Are major incidents always public?

Not always. Public communication depends on impact, legal, and PR considerations; security incidents often have separate disclosure rules.

How do SLOs factor into declaring majors?

SLO breaches or rapid SLO burn rates are common triggers for escalation to major incident status.

How do you avoid alert fatigue while still detecting majors?

Use aggregation, smarter thresholds, anomaly detection, and dedupe/grouping to reduce noise without losing signal.

Should runbooks be automated?

Automate safe, repeatable steps. Manual oversight is necessary for high-risk actions like DB restores.

How do you measure success after a major incident?

Measure MTTR, recurrence, time to complete PIAs, and impact on SLO/error budget.

Who owns postmortems?

The owning team for the affected service should lead the postmortem with cross-team contributions and an executive reviewer.

How often should you do game days?

Quarterly for critical systems; more frequent for high-change environments.

What role does chaos engineering play?

It proactively reveals brittle behavior and validates mitigations before production incidents occur.

How to handle multiple concurrent major incidents?

Prioritize by business impact, assign separate ICs, and maintain a central coordinator for cross-incident dependencies.

What are reasonable starting targets for SLOs?

Start with conservative targets for critical flows (e.g., 99.9% success) and adjust based on capability and business needs.

When should executives be notified?

Notify executives for high-revenue impact, long or public-facing outages, or regulatory/security incidents.

How to track remediation completion?

Use tracked PIAs in your task system with owners, due dates, and executive visibility.

Can AI help with major incident response?

Yes; AI can assist triage, correlate signals, and summarize timelines but requires careful guardrails and human validation.

How to ensure legal/compliance needs are met during incidents?

Involve legal and compliance early for incidents impacting data or regulated services and preserve forensic evidence.

---

Conclusion

Major incidents are high-impact events that require disciplined detection, rapid coordination, and rigorous follow-through. Modern cloud-native environments demand observable systems, automated mitigations, and well-practiced playbooks. Investing in instrumentation, SLOs, role clarity, and postmortem culture reduces frequency and impact over time.

Next 7 days plan (5 bullets):

• Day 1: Audit critical SLIs and ensure telemetry exists for top 3 user journeys.
• Day 2: Verify escalation policies and on-call contacts; run a paging drill.
• Day 3: Review and update top 5 runbooks; add missing runbooks for critical flows.
• Day 4: Build or refine on-call and executive dashboards for SLO burn rates.
• Day 5–7: Run a focused game day simulating one major incident and complete a mini postmortem.

---

Appendix — Major incident Keyword Cluster (SEO)

Primary keywords

• major incident
• major incident management
• major incident response
• major incident playbook
• major incident definition

Secondary keywords

• incident command system
• SRE major incident
• major outage handling
• incident commander role
• major incident runbook

Long-tail questions

• what is a major incident in it operations
• how to declare a major incident
• major incident vs outage vs incident
• how to measure major incident impact
• best practices for major incident response
• how to write a major incident runbook
• how to recover from a major outage
• major incident communication templates
• when to notify executives during major incident
• how to measure SLO during major incident
• how to use feature flags to mitigate incidents
• automating major incident mitigation with playbooks
• running game days for major incidents
• handling security incidents during major outage
• how to prioritize multiple major incidents
• roles in major incident response team
• telemetry required for major incident detection
• major incident postmortem template
• how to track remediation after a major incident
• major incident escalation checklist

Related terminology

• SLO monitoring
• SLI definitions
• error budget burn
• postmortem analysis
• incident timeline
• war room coordination
• incident management platform
• logging and tracing
• synthetic monitoring
• chaos engineering
• canary deploy rollback
• feature flag mitigation
• automated rollback
• burn-rate alerts
• observability debt
• incident severity levels
• incident frequency metrics
• MTTR measurement
• MTTA metrics
• incident commander checklist
• communications lead duties
• runbook testing
• incident game day
• blameless postmortem
• security incident response
• legal and compliance in incidents
• cloud failover strategies
• multi-region failover
• backup and restore drills
• cost-aware incident mitigation
• CI/CD rollback procedures
• Kubernetes incident response
• serverless incident mitigation
• provider outage handling
• third-party dependency fallback
• billing and quota incident handling
• incident lifecycle stages
• incident action item tracking
• on-call fatigue mitigation
• tooling for incident response
• incident response automation