[DevOps Architecture] Incident Response

About this article

As the twelfth installment of the “DevOps Architecture” category in the series “Architecture Crash Course for the Generative-AI Era,” this article explains incident response.

Incidents always happen. Designs that pray they don’t crumble after they do. This article handles the sequence of detection, alerting, notification, response, recovery, and review, on-call regime, Severity definitions, and postmortem culture (the GitLab 2017 textbook case) - the systematization of noticing fast and recovering quickly.

What is incident response, anyway?

Imagine a fire department’s deployment system. You never know when a fire will break out, but the station has 24/7 duty rotations, deployment procedures, area maps, and past fire records ready to go. It’s not individual bravery that makes a fire department effective — it’s the system that reliably extinguishes fires.

Incident response is the fire department of system operations. When failures or abnormal events occur, it’s the activity of systematizing the entire process from detection, notification, initial response, recovery, to review. It maintains processes, tools, and training that let whoever is on duty operate at consistent quality, running a learning cycle that prevents the same incident from happening twice.

Without an incident response system, every failure becomes a matter of a veteran pulling an all-nighter to heroically fix things. If that person is on vacation, recovery is delayed by hours, and the same failures repeat over and over.

Why incident response is needed

First, because delay enlarges the damage. Whether an outage is noticed in five minutes or in an hour changes the number of affected users by an order of magnitude. Second, because letting it lock onto one person destroys reproducibility. If only one engineer can handle an incident, the organisation has no capability, only a dependency. Third, because preventing recurrence is the largest value. Handling an incident is the cost; what you take from it is the return.

The phases of incident response, and severity levels

Design incident response in 6 phases. With clear responsibles and procedures per phase, person-locking decreases.

Severity levels are the other half of that.

Stage incident severity and vary response regime. Responding to all incidents at the same intensity exhausts personnel, so per-severity escalation is required.

Without pre-documenting severity-judgment criteria, judgment varies per site and chaos ensues.

The on-call rota and the command centre

On-call uses the same thinking as a fire-station rotation. Decide an on-call schedule covering 24/7, distribute load via rotation, and avoid concentration on a single person. Standard operational form for SRE organizations, treated as a central chapter in Google’s SRE book.

On-call is heavy load, so SRE’s continuous effort is lowering on-call frequency via alert reduction and auto-recovery. Late-night calls 2+ times monthly is a sign of overload.

Promotion from an alert to an incident is the entry point.

Not all alerts are incidents. Many auto-recover or finish with response within business hours. Design discerning what should truly be an incident is needed.

Incident declaration is an explicit act, flipping the switch of “stop normal operations and concentrate when declared.”

The command centre is how a serious incident is run.

For major incidents (SEV 1-2), launch an Incident Command Center (ICC) where all gather. Split roles for parallel work, clarifying “what to do” for everyone.

It’s important that IC doesn’t make technical decisions, dedicating to overall situation grasp and decision-making. The division of labor leaving technical investigation to Operations Lead is efficient.

Notification and communication run alongside it.

Information transmission during incidents is as important as response itself. Without pre-deciding who, what, and when to communicate, info gets disparate among customers, management, and the field.

Statuspage.io and Atlassian Statuspage are SaaS for customer-facing situation announcement, important tools maintaining trust during incidents.

Postmortems and runbooks

Once an incident is resolved, always conduct review - the iron rule of Google SRE. Document “what happened,” “why it happened,” “how to improve” - turning into organizational asset.

The major principle is don’t blame people (Blameless Post-mortem). Treat as system / process flaws, not individual fault.

Root-cause analysis is the core of the postmortem.

The technique of finding the true cause beyond surface causes is RCA (Root Cause Analysis). The “5 Whys” of repeating “why” 5 times is famous, but the reality in complex systems is combinations of multiple factors rather than single causes.

Symptom: DB went down
|- why? Connections overflowed
|- why? Connection leak in new feature
|- why? Not noticed in code review
|- why? Review viewpoints have no "connection management"
|- why? Review guide stayed old

The true cause becomes the conclusion of flaw in review culture, not “code bug” - that becomes the improvement target.

Runbooks are what turn the lesson into something repeatable.

Documented typical incident-response procedures are Runbooks. Write “if this alert fires, respond this way,” letting anyone respond at the same quality.

Standard operation is placing in Notion / Confluence / Git repos, linked directly from alerts.

Three scenarios

If you are building solo or running small internal tools

UptimeRobot with email or Slack notifications is enough. No on-call rota is needed and handling things in working hours is fine. Keep a lightweight runbook in Notion so that personal knowledge is written down and does not stay in one head.

Personal / Startup: Ship in One Month Is Correcten.senkohome.com/arch-intro-case-startup/

If you are a small or mid-size SaaS

This is the stage for PagerDuty, Statuspage.io and a Slack channel. Put two or three people on a weekly on-call rotation and run blameless postmortems in Google Docs. Keeping the severity scale to three levels — SEV 1, 2 and 3 — is the trick to keeping it simple. As the team grows, add runbook-as-code, managed in Git with pull-request review, and quarterly game days.

Small-Mid SaaS - Lean on Managed and Run with Few Peopleen.senkohome.com/arch-intro-case-saas/

If you are a large or global enterprise

ServiceNow or Jira Service Management, a follow-the-sun on-call rota relaying daylight hours across regions, and AIOps. SEV 1 notifies the executive level automatically, regional on-call teams hand off between Tokyo, Europe and North America, and the reporting process to regulators has to be built in as well.

Large-Enterprise Core: Design That Holds Up for Yearsen.senkohome.com/arch-intro-case-enterprise/

Incident-response numerical gates / SLA

Note: Industry baseline values as of April 2026. Will become outdated as technology and the talent market shift, so requires periodic updates.

Incident response doesn’t function without numerically defining “what to do in how many minutes.” Below are industry-standard SLAs.

For on-call health metrics, late-night calls 2+ times monthly is a sign of overload, alert-firing rate 50%+ false positives means alert review, recurrence rate over 10% means revisiting postmortem quality. With AWS’s 4 golden signals (Latency / Traffic / Errors / Saturation) as basis, pre-define which SEV fires.

First response within 5 minutes is reliability’s lifeline. Systematize via Runbooks and on-call regime.

AI decision axes — Delegate first response and postmortems to AI

AI automation of incident initial response

The initial response during incidents (identifying impact scope, collecting related logs, listing recent deploy changes) is an area AI can automate. Configurations where PagerDuty or OpsGenie alerts trigger AI to auto-execute the following are becoming widespread:

• Aggregate error logs from the last hour and generate a summary
• Retrieve recent deploy history and summarize changes
• Identify affected SLIs and notify via Slack
• Suggest Runbook execution if a matching one exists

AI covering the minutes it takes for the human on-call engineer to wake up and grasp the situation shortens MTTR.

AI generates postmortem drafts

Creating postmortems after incident response is time-consuming work involving timeline construction, impact-scope organization, and root-cause documentation. Passing Slack logs during incident response, alert history, and deploy logs to AI and auto-generating a postmortem draft (timeline, impact scope, direct cause, root cause, action-item proposals) significantly reduces documentation burden.

Humans review the AI-generated draft and focus on accuracy verification and action-item prioritization.

Pitfalls and forbidden moves

Here are the six most dangerous ways incident response goes wrong. Every one raises the recurrence rate or exhausts the organisation.

Being greedy and “doing every preventive measure” is a failure pattern too. Resources spread thin and everything ends up half done, so narrow to the one or two with the largest effect.

Author’s note — GitLab, which did not hide it, and Uber, which did

The GitLab database-deletion incident of 31 January 2017 — an engineer mistook production for development and ran rm -rf, and four of the five backup mechanisms turned out not to work — is remembered as a case where live-streaming the recovery and publishing the full postmortem earned high regard across the industry. What became the textbook for blameless culture was not the accident but the transparency that followed it.

By contrast, the Uber data breach of 2016 was concealed at first, with the attackers paid 00,000 to keep quiet, which developed in later litigation into a settlement of 48 million. The price of concealment is far higher than the incident itself. Incidents are to be treated as something that will happen, and everything rests on being prepared to absorb them with machinery and culture (details in the appendix on major incidents).

What to decide - what is your project’s answer?

For each of the following, try to articulate your project’s answer in 1-2 sentences. Starting work with these vague always invites later questions like “why did we decide this again?”

• Severity criteria (SEV 1-4 definitions)
• On-call regime (rotation, SLA)
• Notification tools (PagerDuty / Opsgenie)
• Command-center rules (declaration criteria, roles)
• Status page (customer info dissemination)
• Postmortem rules (Blameless, scope of disclosure)
• Runbook management (location, update rules)

Related Articles

Summary

This article covered incident response, including phases, SEV levels, on-call, command center, Blameless postmortems, Runbooks, and AIOps.

Switch regime by severity, lower on-call load via alert reduction, learn via Blameless postmortems, code-ize Runbooks and entrust to AI. That is the practical answer for incident response in 2026.

Next time we’ll cover SRE practice (toil reduction, chaos engineering).

Back to series TOC -> ‘Architecture Crash Course for the Generative-AI Era’: How to Read This Book

I hope you’ll read the next article as well.