![[Data Architecture] Streaming - Question Whether It's Truly](/uploads/2026/04/arch-intro-cat-data.png)
About this article
As the sixth installment of the “Data Architecture” category in the series “Architecture Crash Course for the Generative-AI Era,” this article explains streaming.
Question the “real-time” requirement and 90% of the time it lands at “5-min-delay batch is enough.” This article covers streaming-platform selection (Kafka/Kinesis/Pub-Sub/Flink/ksqlDB), Exactly-Once, window processing, and decision criteria - presenting the practical iron rule of questioning whether real-time is truly needed first.
Before you read this
This article uses a good deal of vocabulary from around databases. If that is unfamiliar, reading the primer "Database Basics" first makes it far easier to follow. You can also look anything up in the glossary as you read.
What is streaming
Streaming is “a mechanism that processes data in real time the instant it’s generated, continuously.”
Think of a conveyor-belt sushi restaurant. Batch processing is the “collect orders and send them to the kitchen all at once” approach. Streaming is the “the moment an order comes in, put it on the belt and it passes in front of the customer” approach. It’s used in scenarios where “even a few seconds’ delay is unacceptable” — fraud detection, stock-price updates, IoT sensors — but it costs 10x the operational overhead of batch, so the rule is to choose it only when truly necessary.
Why streaming is needed
There are situations — fraud detection, live inventory, ad bidding, IoT control — where sub-second latency between an event and a decision translates directly into business value. In global operations running around the clock there is also no window for a nightly batch, and in a microservice architecture the streaming platform that joins services by events acts as the central nervous system for inter-service communication.
| Batch processing | Streaming | |
|---|---|---|
| Processing unit | Bundled data | 1 event to a few |
| Delay | Hours to days | Milliseconds to seconds |
| Implementation | Relatively easy | Hard, heavy operation |
| Cost | Cheap | Expensive |
| Retry | Easy redo | Hard to design |
| Representative tech | Spark, dbt | Kafka, Flink |
Most business requirements are fine with batch, and scenes where streaming is truly necessary are limited. If “looks real-time-ish” is enough, substituting with 15-min microbatch often works.
Most business requirements, though, are satisfied by batch, and where the need is only to “look real-time,” a fifteen-minute micro-batch often stands in for it.
The main components
A streaming platform splits into “the layer that carries events” and “the layer that processes events.” The former is the message queue (Kafka, etc.), the latter is the stream-processing engine (Flink, etc.) - roles differ, so select them separately.
| Layer | Role | Representatives |
|---|---|---|
| Message queue | Persist and deliver events | Kafka, Kinesis, Pub/Sub |
| Stream-processing engine | Aggregate, transform, join | Flink, ksqlDB, Spark Streaming |
| Schema management | Define message types | Schema Registry, Protobuf |
The message queue — start with a managed one
Apache Kafka is OSS originating at LinkedIn and is the de facto standard for streaming platforms. The features are high throughput handling millions of events per second, designs that “persist events as a log,” and a mechanism where multiple consumers can independently read - adopted by mega-companies worldwide like Netflix, Uber, and LINE. Confluent Platform (commercial) and Confluent Cloud (managed) are also options.
The strength is “high performance and scalability,” but at the cost of extremely heavy operational load. Managing Zookeeper (KRaft today), broker partition design, consumer-group coordination - serious use is hard without a dedicated ops team.
| Pros | Cons |
|---|---|
| Overwhelming performance, track record | Heavy operational load |
| OSS, thin vendor lock-in | High learning cost |
| Rich ecosystem (Connect, Streams, etc.) | Excessive at small scale |
| Low latency (millisecond order) | High cluster-design difficulty |
If you can self-operate Kafka, it’s the strongest; if not, consider managed (Kinesis/Pub-Sub/Confluent Cloud).
The managed options are Kinesis, Pub/Sub and Event Hubs.
Cloud-vendor-provided Kafka alternatives. The cloud handles operations, eliminating worries about scaling, availability, and backup - the biggest charm is that even small teams can have a streaming platform. AWS uses Kinesis, GCP uses Pub/Sub, Azure uses Event Hubs as standard choices.
| Pros | Cons |
|---|---|
| Near-zero operations | Cloud lock-in |
| Easy to start small | Can be more expensive at large scale |
| Easy integration with other managed services | Fine-grained tuning is hard |
| Cloud handles failures | Kafka-specific features unavailable |
Representatives: Amazon Kinesis Data Streams, Google Pub/Sub, Azure Event Hubs, Confluent Cloud
The modern rule: managed first, migrate to Kafka if you hit throughput limits.
The processing engine — start from SQL and scale up
Apache Flink is OSS specialized in stateful stream processing, executing complex aggregation, join, and event-time processing at millisecond latency. Used by Uber, Alibaba, Stripe at the scale of tens of billions of events per day - the serious option that implements Exactly-Once with the highest reliability.
On the other hand, operational difficulty exceeds Kafka - checkpoint design, state-backend selection, job-restart management - learning costs are very high. Managed versions exist like AWS Kinesis Data Analytics and Aliyun Realtime Compute, and adopting via these to lower operational load is realistic.
| Pros | Cons |
|---|---|
| Low latency, high throughput | High learning cost |
| Flexible to write complex processing | High operational difficulty |
| Robust Exactly-Once | Excessive at small scale |
| Strong event-time processing | Java/Scala primary (Python also) |
ksqlDB and Kafka Streams cover the SQL and library ends.
Lightweight processing engines specific to Kafka. ksqlDB is a product that handles Kafka via SQL, expressing aggregation and filtering in SQL without writing serious Flink-class processing. Kafka Streams is a library, with the appeal of being embeddable in applications to write stream processing.
Both presuppose Kafka and can’t be used with non-Kafka queues (Kinesis, etc.). Effective for SQL-completable use cases or wanting to embed processing in existing Java apps. Can’t do as complex processing as Flink, but the appeal is “an order of magnitude lower learning cost.”
For “scales completable with Kafka + SQL,” ksqlDB is the shortest route. Migrate to Flink when complexity grows.
A typical composition looks like this.
A typical streaming-platform composition is below. From event source to BI/DB, the decisive difference from batch is flowing in real time.
The general split is left-side real-time processing and right-side analytics-bound ingestion - a two-line split.
Exactly-once and windowing — the difficulties peculiar to streaming
The most troublesome thing in streaming is realizing the “guarantee of processing a message exactly once” (Exactly-Once). Network failures, restarts, and timeouts easily cause double processing or loss. In businesses like bank transfers, payments, or inventory updates, duplication is critical.
Kafka and Flink support Exactly-Once, but “end-to-end guarantees require design,” and unless the consumer side is also designed idempotent (same input gives same result), it’s meaningless.
| Guarantee level | Meaning | Difficulty |
|---|---|---|
| At-Most-Once | Give up on failure (loss possible) | Easy |
| At-Least-Once | Reliably delivered, with possible duplicates | Mid |
| Exactly-Once | Strictly once | Hard |
To avoid double processing, the royal road is to design the consumer side idempotent. Exactly-Once is the shield, idempotency is the spear.
Windowing is the other one.
Streaming sees frequent time-bucketed aggregation (window processing) like “sales in the last 5 minutes” or “errors per hour.” What’s an easy aggregation in batch becomes a design issue in an unending stream of “where to cut.”
| Window type | Content | Example |
|---|---|---|
| Tumbling | Fixed-length, no overlap | 0-5 min, 5-10 min |
| Sliding | Fixed-length, slid forward | Last 5 min (updated every 1 min) |
| Session | Until activity breaks | One user’s visit session |
| Global | All time | Cumulative count |
Additionally, distinguishing event time (occurrence time) from processing time (arrival time) matters - network delays disorder things, and “how to handle late-arriving events” becomes a design point.
How to choose — break the freshness requirement into numbers
When somebody says “real-time,” the practical first move is to break it into numbers.
Industry baseline values as of April 2026.
| Freshness requirement | Technology to adopt | SRE headcount needed |
|---|---|---|
| Daily, or a few hours behind, is fine | batch (dbt) | none (part of another role) |
| 5 to 15 minutes behind is fine | micro-batch (dbt every 15 minutes) | none |
| One minute to a few seconds | lightweight streaming (Pub/Sub plus Lambda) | 1 |
| Under 100 ms required | full streaming (Kafka plus Flink) | 2 to 3 dedicated |
The work that genuinely needs under 100 ms is limited: payments, fraud detection, ad bidding, IoT control, exchanges. And “the practical floor for adopting full streaming is two or more dedicated SREs.” Below that, 24/7 incident response, window design and exactly-once operation melt the team. The rule of thumb is that nine in ten business requirements are met by a daily batch or a micro-batch.
Three scenarios
If you are building solo or at a startup
At this size the right answer is not to have streaming at all. A daily batch with dbt, or at most a micro-batch on a fifteen-minute cycle, is enough — and that runs with zero dedicated SREs. If all you want is “an experience that feels real-time,” shortening the polling interval achieves most of it.
If you are a small or mid-size SaaS
Once a requirement for one minute to a few seconds genuinely appears, start with lightweight streaming: Pub/Sub or Kinesis with Lambda. Staying on managed services keeps it maintainable by a single SRE, and there is no need at this stage to take on running Kafka yourself.
If you are a large enterprise
Full streaming on Kafka and Flink is justified only once the work genuinely requires under 100 ms — payments, fraud detection, ad bidding. Even then, two or three dedicated SREs, a 24/7 rota and an exactly-once design are the assumed equipment. If you cannot field that, the requirement is what should be revisited.
AI decision axes — Managed plus schema-driven is kind to AI
Managed streaming has abundant AI training data
Kinesis Data Streams and Cloud Pub/Sub have rich official documentation and sample code, so AI can accurately generate configuration code (Terraform) and Producer/Consumer code. Self-operated Kafka clusters have many project-specific settings, and there are cases where AI’s general knowledge alone can’t handle them accurately.
Schema-driven event design raises AI generation accuracy
When event schemas are registered in a Schema Registry with Avro/Protobuf, AI can accurately grasp “which fields are available in this event” and generate Consumer code. With schema-less free-form JSON, the event structure must be taught to AI each time.
Pitfalls and forbidden moves
Here are the six most dangerous patterns, all of which lead straight to lost data, double processing or a total outage.
| Forbidden move | Why it is bad → what to do instead |
|---|---|
| Adopting streaming because “the customer wants real-time” | in practice thirty minutes behind is usually fine → get the requirement in numbers |
| Running at-least-once with no idempotency | you get double payments and double stock decrements → design the consumer to be idempotent |
| Running Kafka yourself with no dedicated SRE | partition design and incident response melt the team → use a managed service |
| Running Kafka with no schema definition, JSON free-for-all | consumers break continuously → require Protobuf or Avro with a schema registry |
| Running with no dead-letter queue | failed messages retry for ever and block the pipeline → provide a DLQ from the start |
| Not distinguishing event time from processing time | late-arriving events corrupt the aggregates → design the windows precisely |
The large-scale AWS Kinesis outage of November 2020 — a long stoppage in us-east-1 that took CloudWatch and Cognito down with it — is the lesson that the moment you depend on a streaming platform, its failure stops the whole business. A real-time platform is a useful tool and a new single point of failure at the same time.
Author’s note - the real meaning of “we want real-time”
There’s a story often told about a project where the customer said “we want a real-time dashboard,” and the team “spent 3 months building a Kafka + Flink composition,” only to re-hear later that the actual business requirement was “30-min delay is fine.” For a project that cron with a 15-min schedule would have covered, the team then spent the next year being chased by midnight incident response - a typical case told paired with that punchline.
Another famous one is the November 2020 large-scale AWS Kinesis outage. In US-East region, Kinesis Data Streams went down for hours, dragging in even AWS’s own management console and CloudWatch - an event widely talked about as the lesson that the moment you depend on a streaming platform, its outage stops all your business. Real-time platforms can become “not just a convenient tool but a new single point of failure.”
I myself developed the habit, when a customer asks “we want to see numbers in real time,” to first ask back “is a few seconds’ delay troublesome?” or “what about 5 minutes?” - because I’ve taken bumps from this kind of project in the past. Both show that going off the basic “streaming only when truly needed” causes operational load and availability risk to rebound simultaneously. If microbatch suffices, it’s safest and cheapest - the practical conclusion.
When told “real-time,” break it down by numbers first. 5-min and 100ms delays are different worlds.
What to decide - what is your project’s answer?
For each of the following, try to articulate your project’s answer in 1-2 sentences. Starting work with these vague always invites later questions like “why did we decide this again?”
- Is real-time truly needed (re-confirm requirements)
- Message queue (Kafka / Kinesis / Pub-Sub)
- Processing engine (Flink / ksqlDB / Spark Streaming / not needed)
- Guarantee level (At-Least-Once / Exactly-Once)
- Schema management (Avro/Protobuf + Schema Registry)
- Window design (time types, delay tolerance)
- Monitoring/alerting (SLO, metrics, failure notifications)
Related Articles
Summary
This article covered streaming, including selection of Kafka/Kinesis/Pub-Sub/Flink/ksqlDB, Exactly-Once and window processing, the freshness x operational-cost matrix, and judgment axes for avoiding over-investment in real-time.
Question whether real-time is truly needed, prioritize managed services, make schemas explicit, and design consumers idempotent. That is the practical answer for streaming in 2026.
Next time we’ll cover data governance (master management, catalog, regulatory compliance).
Back to series TOC -> ‘Architecture Crash Course for the Generative-AI Era’: How to Read This Book
I hope you’ll read the next article as well.
Also popular with readers
📚 Series: Architecture Crash Course for the Generative-AI Era (50/95)