[Data Architecture] Data Platform

About this article

As the fourth installment of the “Data Architecture” category in the series “Architecture Crash Course for the Generative-AI Era,” this article explains data platforms.

A platform that only stores is debt; a platform you can pull data from is an asset. This article covers the characteristics of the 3 options DWH/data lake/lakehouse, BI tool integration, recommended compositions by scale, and the structure where the platform turns into a “data swamp” the moment “storing” becomes the goal.

What is a data platform

In a nutshell, a data platform is “the foundation that collects company-wide data and makes it accessible for anyone to analyze and use.”

Picture a library’s catalog system. If each department manages its own books (data) independently, you can’t even find out whether another department’s books exist. Build a centralized catalog (DWH) and anyone can locate what they need. In the AI era, this data platform becomes the “source of truth” that LLMs and BI tools read from, so platform quality directly sets the ceiling on AI utilization.

What a data platform handles

BI dashboards, executive KPIs, machine learning, AI agents - the foundation of all of these is the data platform. Without a platform, you end up gathering data per individual system, and cross-department company-wide analysis becomes impossible.

Whether or not you have a data platform is the dividing line of DX success. Cutting corners here means you can’t reach company-wide AI utilization.

Why it’s needed

1. Using business DBs for analysis drops their performance

Business DBs are optimized for “fast and safe per transaction,” and running analytical queries like “aggregate all data” slows down the business side. Separating business and analysis is the de facto standard.

2. Data crosses organizations

To connect sales/marketing/accounting/CS data company-wide, “a platform that consolidates in one place” is needed. Without it, the problem of “the same number is different per department” arises.

3. AI accuracy is decided by data quality

ML models and the RAG of LLMs only stand up with “tidy data.” A poor data platform sets the ceiling on AI utilization.

3 options

There are cases where one is enough, and there are large enterprises that operate “DWH + data lake together.” Decide by scale and use case.

Data warehouse (DWH)

A columnar DB optimized for analysis, where structured data is prepared and loaded. A historical concept dating from the 1980s, it’s the main battlefield for “aggregation analysis” - monthly reports, executive dashboards, KPI monitoring.

All modern DWHs are cloud-managed, with BigQuery, Snowflake, and Redshift as the big three. They aggregate at TB-PB scale in seconds, are accessed via SQL, and have moderate learning costs.

Representatives: BigQuery, Snowflake, Amazon Redshift, Azure Synapse

The default is BigQuery or Snowflake. The reasons to choose Redshift are diminishing unless you’re locked to AWS.

Data lake

“Massive storage” that holds raw data regardless of format. CSV, JSON, images, videos, PDFs, logs - the flexibility to hold anything is the feature, operated with the stance of “first store everything, think about how to use it later.” A cloud’s object storage (S3, GCS, ADLS) becomes the platform as is.

While DWH is “prepare before loading,” the data lake’s idea is load and prepare later. Machine learning, unstructured-data analysis, audit-log retention - it covers “areas DWH can’t handle.”

Representatives: Amazon S3, Google Cloud Storage, Azure Data Lake Storage

Just storing without operational rules turns into a data swamp. Always set up catalog and naming conventions.

Lakehouse

A modern approach that places DWH features on top of a data lake. Apply SQL directly to Parquet files on S3, support ACID transactions - a “best of both” composition. Proposed by Databricks, with Delta Lake, Apache Iceberg, and Apache Hudi establishing themselves as standard data formats.

The concept was born from the issue “operating both a DWH and a data lake is too heavy,” and lakehouse is becoming the top candidate for new builds. However, operational know-how is still developing, so the difficulty rises if the team isn’t familiar.

Representatives: Databricks, Snowflake (Iceberg-supporting), BigLake

Since 2024, lakehouse has become the mainstream for new builds. Migration from existing DWH proceeds in stages.

Comparison of the 3

The lakehouse looks balanced and new, but the reality is that organizations already running on a DWH don’t need to forcibly switch.

BI tool integration

Even building a data platform, business departments can’t use it without visualization tools (BI). BI tools issue SQL to the DWH/lakehouse and display dashboards and reports.

Whether business departments can use it themselves decides BI penetration. Choosing one with a UI usable by non-engineers is the rule.

Decision criteria

1. Type of data

The optimal platform varies with data type. Structured data only is DWH only, but with images/videos/unstructured data involved, a data lake or lakehouse becomes mandatory.

2. Scale and budget

DWHs primarily charge by query billing, and useless queries spike the bill. Data lakes have extremely cheap storage and suit “store first.” Budget management philosophy also affects selection.

3. Cloud vendor

Data platforms are strongly tied to cloud vendors. If you’re already on AWS, Redshift + S3; on GCP, BigQuery; on Azure, Synapse + ADLS - aligning with existing cloud is advantageous in operations and billing.

Snowflake is the only multi-cloud DWH, supported by enterprises wishing to avoid vendor lock-in.

How to choose by case

Mid-size company executive KPI visualization

BigQuery + Metabase/Redash. Cheap (from tens of thousands of yen monthly), simple, low learning cost. Manage ETL from business DB with dbt (data build tool, the tool defining data-transformation pipelines in SQL).

Large enterprise / multi-department / multi-cloud

Snowflake. Multi-cloud support, strong permission management, contract-scale discounts. BI is Tableau or Power BI.

ML / AI utilization-focused

Lakehouse (Databricks). End-to-end including notebooks and MLOps (DevOps for ML). With existing DWH, start from parallel operation.

Startups / MVP phase

Don’t build a data platform. PostgreSQL read replicas + Metabase are enough. Once revenue grows, consider proper introduction.

Phased platform-selection table by org scale and data volume

Note: Industry baseline values as of April 2026. Will become outdated as technology and the talent market shift, so requires periodic updates.

Choosing a data platform “by trends” breaks down in operations. The practical rule is to phase by scale and monthly cost.

Query-billing blow-up pattern: Using SELECT * without limit on BigQuery is the classic story of monthly bills hitting hundreds of thousands overnight. Control costs with the triad partitioning required, column specification required, leverage query cache. Snowflake is also billed by warehouse size x runtime, so the standard is to fix dev environments at XS and Auto-scale on production only.

Snowflake at startup scale is over-investment. BigQuery free tier handles several GB to several hundred GB easily.

Author’s note - cases of “stored but no one can use” swamps

A business unit “stored 3 years of all-department logs in S3” thinking “we’ll analyze it someday,” but with no schema or naming conventions, hundreds of millions of JSON files piled up - and the result was no one could use them, turning into a swamp. Date format differs per file, field names fluctuate with each service revision, and the same value has multiple representations. The processing cost for analysis ended up exceeding the cost of newly designing log collection - reaching the “putting the cart before the horse” punchline.

In another field, conversely, “putting everything in BigQuery is safe” led to ramming image and video binaries into the DWH, with query bills jumping to hundreds of thousands of dollars monthly - the kind of joke-like case told often. Ignoring the basic separation - DWH for structured, data lake for unstructured - rebounds on cost.

I myself once thought lightly about log design and judged “JSON for now” on a past project, only to be told six months later by the analytics person that “we can’t read this.” Both are cases that left the common lesson that when storing itself becomes the goal, the platform becomes a junkyard, not a foundation. Catalogs, naming, and use-case-based platform splits are the basic gear that prevents the platform from becoming a swamp.

A data platform’s purpose is not “storing” but excavating. Catalog and retention policy prevent debt accumulation.

Data platform pitfalls and forbidden moves

Here are the typical accidents in data platforms. The moment “storing” becomes the goal, the platform becomes a junkyard.

The May 2023 Meta GDPR fine of EUR 1.2B (about JPY 200 billion) is a case showing that data-storage-location design mistakes lead directly to fines at the company-survival level. “Inputting industry-specific regulatory requirements (GDPR / HIPAA / PCI DSS) first” is the starting point of platform design.

AI decision axes

1. Separate business and analysis — don’t run analysis on the business DB; this is the starting point of everything.
2. Phase by scale — small starts on BigQuery / Snowflake alone, large migrates phased to lakehouse.
3. Lean on existing cloud — prioritize integration benefits in billing/IAM/operations; multi-cloud means Snowflake.
4. Build out catalog and lineage — aim for a platform where AI agents can introduce themselves.

Mainstream DWHs have high AI-generated SQL accuracy

BigQuery, Snowflake, and Redshift have massive SQL patterns in training data, so AI-generated SQL accuracy is stable. With minor DWHs, dialect SQL accuracy drops and manual corrections increase. Including “can AI write for it” as an evaluation axis in platform selection is the current era.

Data platform Text-to-SQL readiness

For natural-language-to-SQL (Text-to-SQL) to work accurately, the platform needs: well-named tables/columns, rich metadata (COMMENTs, descriptions), and catalog API access. BigQuery and Snowflake both have native AI SQL-generation features (Gemini in BigQuery, Cortex in Snowflake), but their accuracy depends entirely on metadata quality. Building the data platform with “AI will query this” as a design premise from day one is the 2026 standard.

What to decide - what is your project’s answer?

For each of the following, try to articulate your project’s answer in 1-2 sentences. Starting work with these vague always invites later questions like “why did we decide this again?”

• Type of platform (DWH / data lake / lakehouse)
• Cloud vendor (AWS / GCP / Azure / multi)
• BI tool (Tableau / Power BI / Metabase etc.)
• Data ingestion method (ETL / ELT / streaming)
• Retention period and cost tier (hot / cold / archive)
• Permission management method (Row-Level Security / IAM)
• Governance regime (catalog, lineage)

Related Articles

https://en.senkohome.com/arch-intro-data-datastore/ https://en.senkohome.com/arch-intro-data-etl/ https://en.senkohome.com/arch-intro-data-modeling/

Summary

This article covered data platforms, including the 3 options DWH/data lake/lakehouse, BI tool integration, phased recommendations by scale, and catalog operations to avoid data swamps.

Separate business and analysis, phase by scale, lean on existing cloud, and build a platform AI agents touch via catalog. That is the practical answer for a data platform in 2026.

Next time we’ll cover ETL / ELT (the mechanism for extracting, transforming, and loading data).

Back to series TOC -> ‘Architecture Crash Course for the Generative-AI Era’: How to Read This Book

I hope you’ll read the next article as well.