[DevOps Architecture] Test Design

About this article

As the sixth installment of the “DevOps Architecture” category in the series “Architecture Crash Course for the Generative-AI Era,” this article explains test design.

The goal of testing is not the coverage number, but “grasping the cause within 5 minutes when broken.” This article handles practical decisions like the test pyramid, coverage targets, TDD, and flake countermeasures - “what % coverage to aim for,” “how far to write E2E,” “what to run in CI.”

What is test design in the first place

Picture a car inspection. Brakes, lights, emissions - there are inspection criteria for each item, and you can’t drive on public roads without passing. Driving without inspection means an accident could happen at any time.

Test design is about deciding how to build mechanisms that automatically verify whether software is working correctly. You design what to inspect, at what granularity, and at what timing, creating a state where automated checks run every time code changes.

Without test design, every time you change a single line of code, you’d have to manually verify “whether other features are broken”, and fear accompanies every change. The result: nobody wants to touch the code anymore.

Why test design matters

First, to take the fear out of changing things. With automated tests, feedback comes back within minutes of a change and you can refactor with confidence. Second, to find the cause within five minutes of a failure. Looking at which test broke tells you immediately which change caused it; without tests, isolating the cause takes hours or days. Third, for assuring the quality of AI-generated code. There is a limit to how much code an AI writes that a human can review every time, and verifying it mechanically with tests is the realistic safety net.

Think of tests as three responsibilities

Organizing tests into the layers “unit,” “integration,” “E2E” (End-to-End: tests simulating user operations to verify the entire system) is the industry rule, but rather than memorizing layer names, grasping what each is responsible for guaranteeing affects judgment more.

Contract tests (machine-verify the API spec agreed between caller and callee) gain importance in microservices but are unneeded in monoliths. Choosing the layers needed per article is also the architect’s job.

The pyramid ratios, and the ice-cream cone

The test pyramid (proposed by Mike Cohn in 2009) is a model organizing tests as “fast, many, stacked low” / “slow, few, stacked high.” Today it remains the standard starting point for test strategy.

The Unit 70 / Integration 20 / E2E 10 ratio is just a guideline that shifts by domain. Logic-centric domains like payments/inventory lean toward Unit 80, while admin-panel-centric CRUD business apps lean Integration toward 30-40. What you most want to trust decides the ratio.

The inverted pyramid — the ice-cream cone — is the anti-pattern.

What’s happening in many sites is the case where the test pyramid becomes inverted. Many E2Es, thin Unit - this is called the Icecream Cone Antipattern in the industry.

  ____________
  \          /   E2E Test     60%   ← brittle, slow, maintenance abandoned
   \--------/
    \      /     Integration  30%
     \----/
      \  /       Unit Test    10%
       \/

The 2 reasons it tends to happen: “E2E feels safer” and “Unit Tests are tedious to write.” But E2Es take dozens of seconds to minutes, are easy to flake (unstable results despite same code) due to async, and skip tags accumulate as no one fixes them. You end up in the hell of 500 E2Es but half skipped, the other half 3 reds daily.

Stacking E2E to feel safe is a trap. Thin Unit can’t be compensated by E2E.

What to run in tests - phased practice

Running “all tests every time” in CI is unrealistic. Like CI/CD, split phases by timing of code touch and vary the kinds and amounts run per phase - the practical answer.

Running all E2E on PR is a bad choice. Dev speed plummets and no one adds more tests. Split E2E into post-merge smoke (minimum flow) and nightly, with Unit + Integration sufficient at PR stage - this composition functions in the field.

What coverage percentage to aim at

Coverage (the rate of code executed by tests) is used as a lower-bound line, not a target value. Aiming for 80% is good, but the field reality is meaningless tests start being mass-produced the moment 90% is chased.

Coverage has 3 types - line, branch, function - but anchoring on branch coverage (whether both arms of an if-condition were taken) is the modern rule. Looking only at line coverage, only one arm of if-statements may be taken at 90% - false positives are easy. jest --coverage and pytest-cov can output branch coverage.

Coverage realistically lines up at 80%+ for domain logic, 60% for the rest.

Coverage carries its own traps.

Because coverage is easy to measure, it tends to be reported to management as KPI, but test quality collapses the moment numbers alone are chased. Below are forbidden moves frequently happening in the field.

Coverage is an indicator viewed internally as a team’s quality sensor, and degrades when used as evaluation axis or approval blocker. For PR approval, “coverage of new code” (change-line basis) over absolute coverage value is safer - the differentiation of codecov’s project and patch is the example.

TDD and dealing with flaky tests

TDD is a development style spread by Kent Beck in the 2000s, running the 3 steps of write a failing test → minimal implementation to pass → refactor.

TDD’s essence isn’t “writing tests first” but the thinking order of verbalizing spec first then entering implementation. Thinking spec while implementing makes humanly fall into “works so OK”, and TDD functions as the device avoiding that trap.

But running TDD on all features is unrealistic. The field balance is TDD on domain-logic core (payments, inventory calculation, pricing-plan judgment) and post-write tests for UI wiring/CRUD.

Flaky tests are the other half of this.

Flake (tests that pass and fail on different runs despite same code) is the biggest enemy collapsing test reliability. With 10 tests failing 1 in 3 times, CI green only emerges probabilistically and no one trusts red. When the culture of ignoring red takes root, the org changes into one missing real bugs.

The rule for flaky tests is choosing among immediate isolation, fix, or deletion. The operation “fell again, passed on retry so OK” makes the retry feature itself a hotbed of incidents. Jest, Vitest, and Playwright have retry features, but treating “tests passing on retry as not passing” is the operation of a sound team.

Tests passing on retry are not passing. Two choices: isolation or fix.

The boundary between a test database and mocks

The most accident-prone thing in integration tests is DB connection. Replacing DB with mock (a fake in place of the real thing) frequently produces “SQL should be right but fails in production,” and substituting with locally-installed SQLite misses dialect-difference bugs from PostgreSQL/MySQL.

Today’s standard is Testcontainers. It’s a library that starts real DB/Redis/Kafka in Docker containers during testing and discards them after, realizing “running integration tests on the same middleware as production.”

“Verifying down to DB dialect” is investment 100x more valuable than raising coverage numbers.

Mocks divide into the ones not to break and the ones to use.

Mocks are convenient but a tool that erases test meaning when overused. Misjudging boundaries leads to the worst state of “tests all green but production is broken.”

• Mock targets: external SaaS (Stripe, SendGrid, Slack), time/random, costly computations
• Don’t mock: own project’s DB, code we wrote, modules in same process

A typical failure is the pattern of “mocking DB access at the repository layer.” This routes SQL bugs to production undetected. Similarly, “mocking neighbor use cases in the use-case layer test” is also a bad choice, losing the chance to make coupling visible.

Don’t mock your own code, mock only the external world - the empirical iron rule.

Contract tests sit next to them.

When sharing APIs across microservices and multiple teams, the divergence between caller and callee API specs becomes a major cause of production incidents. Contract Test is a mechanism to machine-verify whether the contract caller expects matches what callee actually returns.

Unneeded in monoliths / single team, but the line for considering introduction stands the moment microservices or BFF (Backend-for-Frontend, frontend-dedicated API aggregation layer) is operated by 2+ teams. Investment of replacing coordination cost that melts human time with machine verification.

Test-data design closes the topic.

Another underestimated thing in testing is how to create test data. Filling with bland fixed data like user_01 / product_01, when tests increase, you can’t tell “what case was this testing?”

The Factory pattern (FactoryBot, fishery, @mikro-orm/seeder etc.) is the modern standard. The style of “write only meaningful diffs, default the rest” turns tests into a form readable as spec. Writing all in fixture makes inter-test dependencies invisible, breaking down mid-term.

Three scenarios

If you are building solo or at a startup

Unit tests on the domain logic and a single smoke end-to-end test on the most important path is enough to start. Set no coverage target and write tests only for “the calculations it would hurt to get wrong.” Putting tests that touch the database on Testcontainers from the outset makes later expansion considerably easier.

Personal / Startup: Ship in One Month Is Correcten.senkohome.com/arch-intro-case-startup/

If you are a small or mid-size SaaS

This is the stage for maintaining the pyramid ratios, measuring coverage on changed lines, and quarantining flaky tests immediately. Keep unit tests and integration tests over the changed area under ten minutes on a pull request, and separate end-to-end tests into a post-merge smoke run and a nightly run. On coverage, 80 percent on the domain core and 60 percent elsewhere is a reasonable line.

Small-Mid SaaS - Lean on Managed and Run with Few Peopleen.senkohome.com/arch-intro-case-saas/

If you are a large enterprise

Where several teams run microservices, contract tests such as Pact are worth it: they verify API compatibility mechanically and cut the coordination cost between teams. In regulated industries the record of test execution is itself audit evidence, so include the retention period for CI history in the design.

Large-Enterprise Core: Design That Holds Up for Yearsen.senkohome.com/arch-intro-case-enterprise/

AI decision axes — Tests become the quality gate for AI use

Caveats when having AI write tests

When having AI generate test code, it writes happy-path tests with high accuracy. However, the following problems frequently occur:

• Missing boundary-value tests (cases for 0 items, 1 item, upper limit are omitted)
• Insufficient error-case coverage (doesn’t write cases for network errors, timeouts, insufficient permissions)
• Tests dependent on implementation internals (brittle tests that break on refactoring)
• Over-mocking (mocking even where real DB should be used, giving false confidence)

Use AI-generated tests as a “first draft” - humans review and add boundary-value and error cases. Writing test specs first as humans then delegating implementation to AI in a TDD style yields the most stable quality.

Tests function as quality gates for AI usage

When delegating code generation to AI, the existing test suite becomes the last line of defense for quality. If the automated flow of AI-written code pushed → full test run in CI → revert on failure is in place, AI can be used safely even with concerns about generation quality.

For this premise to hold, the test suite itself must have high reliability. With many flaky tests, low coverage, or mock-heavy thin integration, the probability of incorrectly AI-written code slipping through tests increases.

Pitfalls and forbidden moves

Here are the six most dangerous ways test operation goes wrong. Every one has the structure of chasing a number or a feeling of safety and losing the substance.

Author’s note - team broken by KPI-izing coverage

A widely-known industry case: a mid-size SaaS company set “coverage 80% as team KPI,” and in 3 months, getter/setter tests grew by thousands of lines while essential domain-logic bugs didn’t decrease. Numbers go up easily, but quality doesn’t change. Because “raising coverage” and “reducing bugs” are different things.

This team then switched KPI to the 2 axes of “new bug count to production” and “PR-diff coverage (change-line basis),” abolishing absolute coverage targets. As a result, test quality returned, and a suggestive outcome where getter/setter junk tests were mass-renamed and deleted.

Test value can only be measured by track record of reducing bugs.

What to decide - what is your project’s answer?

For each of the following, try to articulate your project’s answer in 1-2 sentences. Starting work with these vague always invites later questions like “why did we decide this again?”

• Test-pyramid ratio target (Unit / Integration / E2E)
• Coverage target (overall %, domain core %, branch coverage adoption)
• CI scope on PR (change scope only or all Unit)
• E2E execution timing (post-merge smoke + nightly)
• Test-DB strategy (Testcontainers / mock / shared DB)
• Flaky-test handling (isolate → fix → delete flow)
• Contract Test introduction (judge by microservices / BFF presence)
• Test-data approach (Factory / Fixture / Builder)

Related Articles

Summary

This article covered test design, including test pyramid, coverage, TDD, flake countermeasures, Testcontainers, mock boundaries, and AI-era test-first.

Be aware of the test pyramid, verify production-equivalent DB with Testcontainers, branch + change-line coverage as operational metrics, immediately isolate flake. That is the practical answer for test design in 2026.

Next time we’ll cover CI/CD (pipeline design, deploy automation).

Back to series TOC -> ‘Architecture Crash Course for the Generative-AI Era’: How to Read This Book

I hope you’ll read the next article as well.