[DevOps Architecture] Monitoring and Observability

About this article

As the ninth installment of the “DevOps Architecture” category in the series “Architecture Crash Course for the Generative-AI Era,” this article explains monitoring and observability.

What answers “is it running?” is monitoring, what answers “why did it break?” is observability. This article covers the 3 pillars of monitoring (Metrics/Logs/Traces), the 4 golden signals (Latency/Traffic/Errors/Saturation), OpenTelemetry, AIOps, and operational implementation (system-architecture-stage monitoring requirements live in the separate “System Architecture” article).

What are monitoring and observability, anyway?

Imagine driving a car. Speedometer, fuel gauge, engine warning light — without dashboard instruments, you can’t tell your speed, remaining fuel, or engine status. “Noticing only after the engine seizes” is too late.

Monitoring is your system’s dashboard. It’s the mechanism of continuously measuring values like CPU usage, error rate, and response time, and firing alerts when something is off. Observability goes one step further, referring to the ability to investigate “why it broke” after the fact.

Without monitoring, you first learn about incidents from user complaints. Without observability, even when you notice an incident, you can’t identify the cause, and recovery takes hours.

Why it’s needed

Damage expands when you don’t notice incidents

There are still many organizations that first notice incidents from user complaints. Without monitoring, when you notice it’s already a major accident.

Cause-tracking is hard in microservices

In systems where 100 services interact, identifying where the slowness originates requires an observability foundation like distributed tracing.

Numerical SLO/SLA

To promise “99.9% uptime,” you need mechanisms measuring actuals. The principle is what can’t be measured can’t be defended.

Three Pillars

The 3 fundamental data types of observability. Any one alone is insufficient, and the whole picture appears only when combining all 3. “3 pillars” is now classic, with the modern mainstream being multi-faceted approaches adding Events and Profiles.

Metrics show “what’s happening,” logs show “what was recorded,” traces show “how it moved.”

Metrics

Numerically-quantified time-series data, recording CPU usage, request count, error rate, latency, etc. Storage-efficient and good for aggregation/alerting - the basis of monitoring.

USE/RED are famous metric-design frameworks used as guides for “what to measure.”

Logs

Time-stamped text events recording detailed info output by apps. Structured logs (JSON format) are the modern standard, linkable with metrics and traces. Logs are covered in detail in the next article.

Logs occur in massive amounts, so storage cost easily becomes a problem - retention period, compression, and sampling strategies are operational topics.

Traces

Data tracking the process of one request transiting multiple services. In microservices, with chains like “the order API calls the inventory API and the payment API,” you can visualize where it’s slow and where it failed.

Traces have Spans (individual processing) connected by Trace ID, expressing the whole flow as a DAG. Representative tech is OpenTelemetry (industry standard), sent to Jaeger, Tempo, Datadog APM, etc.

Trace: order-abc123
 |- Span: POST /order (100ms)
 |   |- Span: check stock (30ms)
 |   |- Span: charge card (50ms)
 |       |- Span: Stripe API (45ms)

Main tools (cloud-native OSS)

OSS-based observability has Prometheus + Grafana + Loki + Tempo + OpenTelemetry as the de facto standard combination. Developing under CNCF, with high future prospects.

Integrated as the LGTM stack (Loki, Grafana, Tempo, Mimir), rapidly spreading recently.

Main tools (SaaS)

If you don’t want to build the foundation in-house, using observability SaaS is a quick choice. Pricing is high, but you get high-feature monitoring foundation with near-zero operational burden.

Datadog has the strongest features, but often hits hundreds of thousands to millions of yen monthly - small scales realistically use Grafana Cloud or New Relic free tier.

OpenTelemetry (measurement standard)

The industry standard unifying collection of metrics, logs, and traces. Born in 2019 from the merger of OpenTracing and OpenCensus, advancing from CNCF Incubating to Graduated - the modern common standard.

Using OpenTelemetry, instrumentation code can be written vendor-neutrally. Even when switching from Datadog to Grafana, no need to rewrite app-side code. Just swap send destination on the Collector side, so vendor-lock-in avoidance kicks in not as desk argument but at implementation level.

Furthermore, with unified instrumentation API, using multiple tools together becomes easy. For example: production metrics to Datadog, long-term log archive to S3 + Loki, dev-env traces to Jaeger - distribute by use case from the same instrumentation code. Custom SDKs require redoing instrumentation per tool, so this difference becomes non-negligible at scale.

For new builds, OpenTelemetry is the top candidate. Avoids vendor lock-in.

Dashboards and alerts

Just collecting monitoring data is meaningless - it becomes operationally useful only with visualization and alerting. The standard operation is overviewing state with Grafana or each SaaS dashboard, and notifying via Slack or PagerDuty when thresholds are exceeded.

Alerts have the dichotomy of too few = miss it, too many = paralyzed, so per-importance routing (Slack for warnings, PagerDuty for critical) matters.

Alert design

Conditions for good alerts: (1) response definitely needed, (2) response possible, (3) should move now - 3 satisfied. Alerts off these produce alert fatigue and cause missing real critical events.

Alerts should fire on user impact, not “symptoms.” High CPU itself doesn’t impact users.

“Whether the machine is suffering” and “whether humans are suffering” are different things - the standard lesson of monitoring design. Cases where teams set CPU usage 80% alerts to feel safe, with no one’s dashboard red but Twitter flowing with “can’t log in” reports - are commonly heard. The cause is DB connection pool exhaustion, with CPU actually idle and only the app having everyone wait - the typical pattern.

SLO-based alert implementation example

The modern way for “SLO-violation-based” alerts is firing by error-budget consumption speed (burn rate). Not mere threshold exceeding, but a mechanism detecting “at this pace, the budget will be exhausted” - recommended as standard by the Google SRE Workbook.

# Prometheus example (availability SLO 99.9%, monthly budget 43.2 min)
alert: ErrorBudgetBurnRateCritical
expr:  (1 - availability_slo:ratio_rate5m) > (14.4 * 0.001)
  and  (1 - availability_slo:ratio_rate1h) > (14.4 * 0.001)
for: 2m

“Fire when CPU exceeds 80%” is outdated. Fire by burn rate - the modern standard.

Decision criterion 1: system scale and complexity

Observability heaviness is decided by system complexity. A monolithic single server is fine with lightweight monitoring; microservices need a full set.

Decision criterion 2: operations team skills

OSS stacks have heavy operations and need a dedicated team. With SaaS, operational burden decreases but costs add up.

How to choose by case

Personal dev / small SaaS

Cloud standards (CloudWatch / Cloud Monitoring) + UptimeRobot. Runs from thousands of yen monthly. Add Grafana Cloud free tier if you want custom dashboards.

Startup / 0-1 SREs

Free tier of Datadog or New Relic + OpenTelemetry SDK. To minimize operational load, SaaS is the only choice. Instrument with OTel and you can switch backends in the future. At scale, billing spikes, so review past 100 hosts.

Mid-size SaaS / microservices-ization

Grafana Cloud (LGTM) + OpenTelemetry. There’s LGTM-stack learning cost, but considerably cheaper than Datadog. Operable with 2-3 SREs.

Large enterprise / can self-operate

Self-built Prometheus + Grafana + Loki + Tempo + OpenTelemetry. Composition avoiding vendor lock-in and not putting confidential data outside. Operations need 5+ dedicated SREs.

Monitoring-cost / alert-operation numerical gates

Note: Industry baseline values as of April 2026. Will become outdated as technology and the talent market shift, so requires periodic updates.

For monitoring, not “install and feel safe” - the key of operations is tracking cost and signal quality numerically.

Alerts firing 10+ /week is a “noise-ization sign.” Datadog over $3k/month is a guideline for over-investment at startup scale. Continuing to output DEBUG logs in production easily hits CloudWatch Logs $10k+/month - the typical accident.

Monitoring cost has 10% of infrastructure cost as upper bound. Cut via sampling and retention if exceeded.

Monitoring-design pitfalls and forbidden moves

Typical accident patterns in monitoring. All have the structure of configured but not operating.

The October 2021 Facebook/Instagram 6-hour outage (BGP config error made servers invisible from outside, and recovery delayed because internal monitoring tools and entry/exit systems all depended on the same network, estimated $60M+ loss) showed the structural problem of no monitoring of monitoring.

Monitoring fires on “human pain.” Fire on SLO violations, not CPU 80%.

AI decision axes

1. Unify measurement with OpenTelemetry - vendor-neutral, prepare for future backend switching
2. Decide SaaS vs OSS by ops regime - no dedicated SRE: Datadog, few: Grafana Cloud, rich: in-house LGTM
3. Alerts on user-impact basis - fire on SLO violations, don’t fire on CPU spike alone
4. Structured data AI can read - JSON / Traces / high cardinality, prepare for AI diagnosis

AI-driven root-cause analysis (RCA) has become practical

Datadog’s “Watchdog RCA” and New Relic’s “AI Insights” provide features that cross-analyze multiple signals (logs, metrics, traces) to estimate incident root causes. The prerequisite for these to work is that the 3 pillars (logs, metrics, traces) are correlated via trace_id.

With unified instrumentation via OpenTelemetry, the log → trace → metrics correlation is automatically built on the tool side, maximizing AI RCA accuracy.

Observability backend selection and AI feature gaps

As of 2026, AI feature maturity varies by observability tool. Datadog covers anomaly detection, RCA, and Runbook recommendation with AI; Honeycomb broadens AI intervention space with BubbleUp (high-cardinality analysis); Grafana Cloud has implemented LLM-based log-query generation. “How far AI-powered diagnostic support goes” is now an evaluation axis in tool selection.

Author’s note - cases of “all-green dashboards” with fires beneath

Cases of “monitored but couldn’t notice” have become standard talking points in operational fields.

At a certain SaaS, dashboards of CPU, memory, and network all stayed green while Twitter had hundreds of “can’t log in” reports flowing - a near-miss. The cause was DB connection pool exhaustion, with CPU actually idle and only the app having everyone wait. The trap was “feeling safe looking at infrastructure metrics,” with not measuring SLO (user impact) the root cause.

Another, the October 2021 Facebook/Instagram 6-hour outage, where BGP-config error made servers invisible from outside, but also internal monitoring tools and entry/exit systems all depended on the same network, so engineers couldn’t enter the data center and recovery was delayed - told as a case of “no monitoring of monitoring.” Estimated $60M+ in ad-revenue loss alone.

Both have design gaps in “what to measure” and “monitoring system independence” as lethal blows, slapping home that firing on user impact and separating monitoring from target are both required.

What to decide - what is your project’s answer?

For each of the following, try to articulate your project’s answer in 1-2 sentences. Starting work with these vague always invites later questions like “why did we decide this again?”

• Measurement SDK (OpenTelemetry recommended / vendor-specific)
• Backend (OSS LGTM / Datadog / New Relic)
• Metric design (USE, RED, SLI)
• Log strategy (collection scope, retention)
• Distributed tracing (all requests / sampling)
• Alert design (SLO-based, channel separation)
• Dashboards (service health, business)

Related Articles

https://en.senkohome.com/arch-intro-devops-overview/ https://en.senkohome.com/arch-intro-devops-logging/ https://en.senkohome.com/arch-intro-devops-slo/

Summary

This article covered monitoring and observability, including the 3 pillars, OpenTelemetry, main tools, SLO burn-rate alerts, and structured data for AI diagnosis.

Unify measurement with OpenTelemetry, decide SaaS vs OSS by ops regime, alerts on user-impact basis, organize structured data AI can read. That is the practical answer for monitoring and observability in 2026.

Next time we’ll cover log design (structured logs, PII protection, retention).

Back to series TOC -> ‘Architecture Crash Course for the Generative-AI Era’: How to Read This Book

I hope you’ll read the next article as well.