About this article
This article is the ninth deep dive in the “System Architecture” category of the Architecture Crash Course for the Generative-AI Era series, covering the system-architecture-level overall map of monitoring and operations.
Production without monitoring is flying without instruments; recovery time becomes a matter of luck. This article focuses on system-architecture-stage monitoring requirements (3 pillars of observability, 4 golden signals, platform selection, phased rollout); operational implementation (OpenTelemetry, log design, SLO operation, on-call) lives in the “DevOps Architecture” category.
Before you read this
This article uses a good deal of infrastructure vocabulary — servers, networks and so on. If that is unfamiliar, reading the primers "Servers and the Cloud" and "How a Web Service Works" first makes it far easier to follow. You can also look anything up in the glossary as you read.
What is monitoring design in the first place
Monitoring design is, roughly speaking, “setting up a system to continuously check your system’s health.”
Imagine a car’s dashboard. The speedometer (response time), fuel gauge (resource usage), and engine warning light (error alert) let you catch anomalies while driving. Without any instruments, you might not notice until the engine seizes. System monitoring works the same way — metrics, logs, and traces serve as three “instruments” for continuous checks and early anomaly detection.
Why monitoring design matters
What happens if you run production without monitoring? It’s flying without instruments — recovery time becomes a matter of luck. Systems with poor monitoring foundations collapse team morale after one major incident. Without knowing the cause, the team stumbles through response, and the anxiety of “it could happen again anytime” drags on development.
Conversely, teams with solid monitoring digest incidents as learning opportunities, building robustness over time. Monitoring is table-stakes quality — set it up before launch.
The three pillars of observability
Observability is the degree to which a system’s internal state can be inferred externally. Broader than mere “monitoring” — combining metrics, logs, traces to reach “why is it slow, where did it fail” at the root.
| Pillar | Purpose | Examples |
|---|---|---|
| Metrics | Time-series numbers | CPU, latency, error rate |
| Logs | Individual event detail | Access logs, error logs, audit logs |
| Traces | Path of a single request | Distributed tracing |
Only with all three: notice “the API is slow” in metrics, locate “which request was slow” in logs, find “which service caused it” in traces. Missing any one leaves you in the “don’t know where to start fixing” state.
The four golden signals
The four basic indicators Google’s SRE (Site Reliability Engineering) team proposed as “what to look at first.” Trying to monitor everything breaks down — anchor on these four.
| Signal | Meaning | Examples |
|---|---|---|
| Latency | Request processing time | API response time, p99 |
| Traffic | Request count, bandwidth | RPS (Requests Per Second), concurrent connections |
| Errors | Failed-request rate | 5xx error rate, exception count |
| Saturation | Resource pressure | CPU/memory/disk usage |
Latency is read on P95 / P99, not P50. Averages hide “fast for the majority but unbearable for 1%.” Outages usually start in the slow tail’s 1%.
Start with the four signals. “Monitor everything” is impossible.
Choosing a monitoring platform
Pick from three families: cloud-native vs SaaS vs OSS. Driven by ops headcount, cost tolerance, and existing stack.
| Product | Strong at | Fits |
|---|---|---|
| CloudWatch / Azure Monitor / GCP Operations | Cloud-native, free tiers | Single cloud, startup |
| Datadog | Integrated, excellent UI | Budget exists, want one pane |
| New Relic | APM-focused | App-performance focus |
| Grafana + Prometheus + Loki | OSS integrated | Cost focus, can self-operate |
| Splunk | Veteran log analytics | Enterprise, large scale |
Datadog has overwhelming feature richness but tends to hit thousands of dollars monthly. Grafana Stack (OSS) is cheap but needs ops staff. CloudWatch is the best cost-effectiveness in single-cloud environments.
Reliability targets to set at design time
Monthly SLO operation belongs to the other category, but at design time set the rough framing of “how much downtime is acceptable for this service?” This determines redundancy level, monitoring investment, and on-call need.
| Service type | Target availability | Monthly downtime budget |
|---|---|---|
| Personal blog, experimental | 99% (hard not to clear) | ~7 hours |
| General B2C web | 99.9% | ~43 minutes |
| B2B SaaS (business hours required) | 99.95% | ~22 minutes |
| Finance / payment / healthcare | 99.99%+ | ~4 minutes |
Setting SLA (contract) first locks the design too tight, so the practical order is “set rough SLO (target) first, finalize SLA from sales requirements later.” Detailed SLI selection and burn-rate design go to the other category.
At design time, set only the framing of “how many minutes of downtime is OK.” Refinement happens in another article.
A phased roadmap for rolling monitoring out
“Everything at once” is impossible; phase the buildout. Below is in order of regret-when-an-incident-happens.
| Phase | Minimum | Target / signal | Annual monitoring cost |
|---|---|---|---|
| 1 MVP | Health check + error notification (Sentry, etc.) | Daily Slack notification is fine | $0 |
| 2 Early ops | + 4 golden signals + structured logs | P95 < 500ms, error rate < 1% | $300+ |
| 3 Scale ops | + Distributed tracing (OTel), SLO / error budget | SLO 99.9%, burn-rate monitoring | $3k+ |
| 4 Multi-service | + AIOps (AI for IT Operations) tools, anomaly detection | False-positive rate <= 5%, MTTR <= 30 min | $30k+ |
| 5 On-call ops | PagerDuty/Opsgenie, runbooks, postmortems | MTTA (acknowledgement time) <= 5 min | + Headcount |
Default alert thresholds: P99 latency > 2x normal -> WARN, > 3x -> PAGE; error rate > 0.5% -> WARN, > 2% -> PAGE. Static thresholds always go stale, so switching to SLO-based within 6 months of operation is the rule.
Stage monitoring rollout with the phase. Datadog at $3k/month for an MVP is obvious over-investment.
Three scenarios
If you are building solo or at a startup
The cloud-native monitoring that comes with the platform, plus an uptime check, is enough. Configure alerts on error rate and availability only, and send them to one channel. What matters at this size is that something tells you when it is down, not the sophistication of the dashboard.
If you are a small or mid-size SaaS
The four golden signals with alerting on SLO violation, and OpenTelemetry for instrumentation so the backend stays replaceable. Separate the notification destination by severity, and review the alerts monthly so the dead ones do not accumulate.
If you are a large enterprise
Distributed tracing across services, tail latency at P95 and P99, and runbooks in Markdown so that first response can be automated. At this size the monitoring platform itself needs an owner and a budget.
AI decision axes — Can the AI read it over standard protocols?
Monitoring-platform selection lands on two axes: “can it be instrumented with standard protocols like OpenTelemetry?” and “can AI analyze the logs and metrics?”
- Instrument the 4 golden signals first (full monitoring is impossible).
- Define SLO / error budget, alert on SLO violations.
- Instrument with OpenTelemetry (avoid lock-in, AI-era ready).
- Systematize on-call and runbooks, no tribal knowledge.
Structured logs are the prerequisite for AI analysis
When delegating log analysis to AI, JSON-format structured logs are mandatory. When timestamp, level, service, trace_id, and message fields are output in a consistent format, AI can extract error patterns, identify blast radius, and infer root causes with high accuracy.
Conversely, unstructured text logs (free-format remnants of printf debugging) are expensive for AI to analyze, with frequent false positives and missed detections. Log-format unification is the first investment in an AI-era monitoring foundation.
Markdown runbooks and AI-agent auto-recovery
Managing incident-response procedures (runbooks) as Markdown or code becomes the foundation for AI agents to read procedures and execute auto-recovery. For example, if “disk usage over 90% -> delete old logs -> restart service -> verify” is structured, an AI agent auto-executing via PagerDuty or OpsGenie is becoming realistic.
Word/PDF procedure docs are human-only-readable, falling outside AI utilization scope. Being aware of “AI-readable formats” at monitoring design time connects to future operational cost reduction.
Pitfalls and forbidden moves
Here are the six most dangerous ways monitoring and alert design go wrong.
| Forbidden move | Why it is bad → what to do instead |
|---|---|
| Leaving DEBUG logs on in production | there are many cases of a five-figure monthly logging bill → restrict to INFO and above |
| Continuing to run static thresholds such as CPU at 80 percent | false positives accumulate until nobody looks → switch to alerting on SLO violation |
| Sending every alert to one channel | a real incident is buried in everyday noise → separate the destination by severity |
| Judging latency on P50, the average | the slowest one percent of users is invisible → track tail latency at P95 and P99 |
| Running microservices with no tracing | you cannot tell which service is slow and MTTR runs into days → instrument with OpenTelemetry |
| Never reviewing the alerts | an alert nobody looks at is the same as an outage that does not fire → prune the dead ones in a monthly review |
Author’s note — the alert channel you had to mute to get any work done (industry case)
A new engineer’s first day, a senior tells them with a straight face: “Mute this Slack channel or you can’t get any work done.” That channel was a noise pile of dozens of CPU-threshold alerts daily; the entire team had stopped looking.
Months later, a real incident alert sat in the same channel, detection delayed by hours.
Similar scenes recur in many teams. “Nobody looks at this channel” becomes implicit shared knowledge; only newcomers notice it every time. Lesson: “alerts’ value is decided by how many people could move, not how many fires fired.”
A channel nobody watches is the same as no channel. Cull noise, alert only on SLO violations — the courage to commit to that design saves the team in the long run.
“Alerts you’re scared to delete” are already stale.
What to decide — what is your project’s answer?
- Monitoring-platform direction (CloudWatch / Datadog / Grafana family)
- Log aggregation and retention (30 days / 90 days / annual)
- Distributed tracing (OpenTelemetry adoption)
- Reliability target (rough SLO, allowable downtime)
- Monitoring cost ceiling
Alert design, SLO operation, on-call, runbooks, and postmortems are decided in the “DevOps Architecture” category articles.
Related Articles
Summary
This article covered the monitoring and operations overall map at the system-architecture level.
The default order: 4 golden signals -> 3 observability pillars -> SLO-based alerts. “Monitor everything” is impossible — phase the rollout and have the courage to delete noise alerts.
The next article covers BCP (business continuity planning, RPO/RTO, DR strategy).
Back to series TOC -> ‘Architecture Crash Course for the Generative-AI Era’: How to Read This Book
I hope you’ll read the next article as well.
Also popular with readers
📚 Series: Architecture Crash Course for the Generative-AI Era (20/95)