[Data Architecture] Data Platform

About this article

As the fourth installment of the “Data Architecture” category in the series “Architecture Crash Course for the Generative-AI Era,” this article explains data platforms.

A platform that only stores is debt; a platform you can pull data from is an asset. This article covers the characteristics of the 3 options DWH/data lake/lakehouse, BI tool integration, recommended compositions by scale, and the structure where the platform turns into a “data swamp” the moment “storing” becomes the goal.

What is a data platform

A data platform is “the foundation that collects company-wide data and makes it accessible for anyone to analyze and use.”

Picture a library’s catalog system. If each department manages its own books (data) independently, you can’t even find out whether another department’s books exist. Build a centralized catalog (DWH) and anyone can locate what they need. In the AI era, this data platform becomes the “source of truth” that LLMs and BI tools read from, so platform quality directly sets the ceiling on AI utilization.

Why a data platform is needed

There are three reasons. First, running an analytical query such as “aggregate everything” on the operational database slows the operational side down, which makes separating operations from analytics the de facto standard. Second, without a platform that joins sales, marketing and finance data across the company, the same figure comes out different in each department. Third, both machine learning and LLM retrieval only work on “data that has been put in order”, so a weak platform puts a ceiling on how far AI can be used.

The three options

There are cases where one is enough, and there are large enterprises that operate “DWH + data lake together.” Decide by scale and use case.

The warehouse — the main battlefield for aggregation

A columnar DB optimized for analysis, where structured data is prepared and loaded. A historical concept dating from the 1980s, it’s the main battlefield for “aggregation analysis” - monthly reports, executive dashboards, KPI monitoring.

All modern DWHs are cloud-managed, with BigQuery, Snowflake, and Redshift as the big three. They aggregate at TB-PB scale in seconds, are accessed via SQL, and have moderate learning costs.

Representatives: BigQuery, Snowflake, Amazon Redshift, Azure Synapse

The default is BigQuery or Snowflake. The reasons to choose Redshift are diminishing unless you’re locked to AWS.

The data lake — it takes anything, and it can turn into a swamp

“Massive storage” that holds raw data regardless of format. CSV, JSON, images, videos, PDFs, logs - the flexibility to hold anything is the feature, operated with the stance of “first store everything, think about how to use it later.” A cloud’s object storage (S3, GCS, ADLS) becomes the platform as is.

While DWH is “prepare before loading,” the data lake’s idea is load and prepare later. Machine learning, unstructured-data analysis, audit-log retention - it covers “areas DWH can’t handle.”

Representatives: Amazon S3, Google Cloud Storage, Azure Data Lake Storage

Just storing without operational rules turns into a data swamp. Always set up catalog and naming conventions.

The lakehouse — the new first candidate for a fresh build

A modern approach that places DWH features on top of a data lake. Apply SQL directly to Parquet files on S3, support ACID transactions - a “best of both” composition. Proposed by Databricks, with Delta Lake, Apache Iceberg, and Apache Hudi establishing themselves as standard data formats.

The concept was born from the issue “operating both a DWH and a data lake is too heavy,” and lakehouse is becoming the top candidate for new builds. However, operational know-how is still developing, so the difficulty rises if the team isn’t familiar.

Representatives: Databricks, Snowflake (Iceberg-supporting), BigLake

Since 2024, lakehouse has become the mainstream for new builds. Migration from existing DWH proceeds in stages.

Here is how the three compare.

The lakehouse looks balanced and new, but the reality is that organizations already running on a DWH don’t need to forcibly switch.

Working with BI tools

Even building a data platform, business departments can’t use it without visualization tools (BI). BI tools issue SQL to the DWH/lakehouse and display dashboards and reports.

Whether business departments can use it themselves decides BI penetration. Choosing one with a UI usable by non-engineers is the rule.

How to choose — a table by scale

Industry baseline values as of April 2026.

Choosing a data platform “by what is fashionable” breaks down in operation. Pick it in stages, on scale and monthly cost.

Consistency with your cloud vendor matters too: Redshift and S3 on AWS, BigQuery on GCP, Synapse on Azure. Matching the cloud you already have is easier both operationally and on the invoice. Snowflake, as the one multi-cloud warehouse, is favoured by companies that want to avoid lock-in.

On cost, watch for the query-billing blow-up. Unlimited SELECT * against BigQuery reaching several thousand dollars overnight is a standard story; control it with the three-part set of mandatory partitioning, mandatory column selection and use of the query cache. With Snowflake, fixing development to XS and auto-scaling only in production is the reliable pattern.

Three scenarios

If you are building solo or at a startup

At this stage the right answer is not to build a data platform at all. Up to about 10 GB, throwing analytical queries straight at PostgreSQL and visualising with the free tier of Metabase covers it. As the data approaches a terabyte, add BigQuery and build free dashboards in Looker Studio.

Personal / Startup: Ship in One Month Is Correcten.senkohome.com/arch-intro-case-startup/

If you are a small or mid-size SaaS

BigQuery or Snowflake at the centre, synchronised by Fivetran or Airbyte and transformed with dbt, is the reliable combination today. Run ELT from the production database daily or hourly and keep OLTP and OLAP apart. Accidents where query billing detonates the invoice are genuinely common, so make mandatory partitioning, mandatory column selection and query caching rules from the start.

Small-Mid SaaS - Lean on Managed and Run with Few Peopleen.senkohome.com/arch-intro-case-saas/

If you are a large enterprise

With many departments and 100 TB in view, Snowflake — the one multi-cloud warehouse — is the choice; for very large ML-centred work, Databricks and a lakehouse. The thing to watch here is that unless a data catalogue (DataHub, Collibra) and a governance structure are put in place alongside, you sink into the swamp of “we stored it and nobody can use it.” The organisation often matters more than the platform.

Large-Enterprise Core: Design That Holds Up for Yearsen.senkohome.com/arch-intro-case-enterprise/

AI decision axes — A maintained catalogue is the precondition for AI use

Mainstream DWHs have high AI-generated SQL accuracy

BigQuery, Snowflake, and Redshift have massive SQL patterns in training data, so AI-generated SQL accuracy is stable. With minor DWHs, dialect SQL accuracy drops and manual corrections increase. Including “can AI write for it” as an evaluation axis in platform selection is the current era.

Data platform Text-to-SQL readiness

For natural-language-to-SQL (Text-to-SQL) to work accurately, the platform needs: well-named tables/columns, rich metadata (COMMENTs, descriptions), and catalog API access. BigQuery and Snowflake both have native AI SQL-generation features (Gemini in BigQuery, Cortex in Snowflake), but their accuracy depends entirely on metadata quality. Building the data platform with “AI will query this” as a design premise from day one is the 2026 standard.

Pitfalls and forbidden moves

The moment “storing it” becomes the goal, the platform turns into a rubbish tip. Here are the six most dangerous.

Author’s note - cases of “stored but no one can use” swamps

A business unit “stored 3 years of all-department logs in S3” thinking “we’ll analyze it someday,” but with no schema or naming conventions, hundreds of millions of JSON files piled up - and the result was no one could use them, turning into a swamp. Date format differs per file, field names fluctuate with each service revision, and the same value has multiple representations. The processing cost for analysis ended up exceeding the cost of newly designing log collection - reaching the “putting the cart before the horse” punchline.

In another field, conversely, “putting everything in BigQuery is safe” led to ramming image and video binaries into the DWH, with query bills jumping to hundreds of thousands of dollars monthly - the kind of joke-like case told often. Ignoring the basic separation - DWH for structured, data lake for unstructured - rebounds on cost.

I myself once thought lightly about log design and judged “JSON for now” on a past project, only to be told six months later by the analytics person that “we can’t read this.” Both are cases that left the common lesson that when storing itself becomes the goal, the platform becomes a junkyard, not a foundation. Catalogs, naming, and use-case-based platform splits are the basic gear that prevents the platform from becoming a swamp.

A data platform’s purpose is not “storing” but excavating. Catalog and retention policy prevent debt accumulation.

What to decide - what is your project’s answer?

For each of the following, try to articulate your project’s answer in 1-2 sentences. Starting work with these vague always invites later questions like “why did we decide this again?”

• Type of platform (DWH / data lake / lakehouse)
• Cloud vendor (AWS / GCP / Azure / multi)
• BI tool (Tableau / Power BI / Metabase etc.)
• Data ingestion method (ETL / ELT / streaming)
• Retention period and cost tier (hot / cold / archive)
• Permission management method (Row-Level Security / IAM)
• Governance regime (catalog, lineage)

Related Articles

Summary

This article covered data platforms, including the 3 options DWH/data lake/lakehouse, BI tool integration, phased recommendations by scale, and catalog operations to avoid data swamps.

Separate business and analysis, phase by scale, lean on existing cloud, and build a platform AI agents touch via catalog. That is the practical answer for a data platform in 2026.

Next time we’ll cover ETL / ELT (the mechanism for extracting, transforming, and loading data).

Back to series TOC -> ‘Architecture Crash Course for the Generative-AI Era’: How to Read This Book

I hope you’ll read the next article as well.