Forma Engineering Blog Series: From EAV to Zero-Dirty-Read Lakehouse
Three posts that explain a flexible data storage engine designed for the AI era
The 3 AM page
The pipeline crashed at 3 AM because the model deployed the week before had started emitting a confidence_score field that didn't exist the day before. The database rejected the write. Monitoring stayed quiet. The customers noticed first.
That is what AI-powered applications do to storage: the data structures change faster than the schema can follow.
The receipt analogy
A traditional SQL table is like a grocery receipt that lists every item the store carries, bananas and steaks and shampoo and sushi, printing a "0" next to everything you didn't buy. Selling a new item means reprinting the receipt format.
An EAV-based system lists only what you actually bought. Chips, soda, done. A new item is one more line.
That is the idea behind Forma: store only what exists, and let the schema evolve with your AI's outputs.
A note to the skeptics
If you have spent any time in data engineering, you are probably thinking: "EAV? That anti-pattern that destroys performance and makes queries a nightmare?"
The reputation is earned, and the skepticism is fair. This series is about how we tamed it. The performance problems are in Part 2, the consistency fears in Part 3, and the result is a production architecture already handling billions of records.
What is Forma?
Forma is a flexible data storage engine designed for the AI era. It rests on three technology choices:
| Technology | Purpose | Problem Solved |
|---|---|---|
| EAV pattern | Attributes stored as rows, no DDL for new fields | Schema flexibility |
| JSON Schema | AI-native data contracts, validation on write | Type safety and AI integration |
| PostgreSQL + DuckDB | OLTP and OLAP working together, hot/cold separation | Performance and cost balance |
Three problems we're solving
Problem one: AI data structures iterate fast
Your AI Agent outputs 12 fields today, 30 tomorrow, and 5 more next week. The traditional DDL workflow (file ticket, wait for approval, take downtime, run ALTER TABLE) cannot keep that pace.
Post One explains why JSON Schema plus EAV plus a hot table suits AI applications: no DDL, changes take effect immediately, and writes stay type-safe.
Problem two: the N+1 query nightmare
EAV is flexible, since a new field is just more rows and no schema change. The query performance is the problem. Fetching 100 records can take 101 database round-trips, which pushes latency past a second without much effort.
Post Two shows how PostgreSQL's CTE + JSON_AGG cuts that to a single query, and latency from 1000ms to 25ms.
Problem three: consistency across massive historical data
Once data reaches billions of records, hot/cold separation stops being optional. "Lakehouse" sounds good, but every engineer ends up asking the same thing: how do I know the data I'm querying isn't dirty?
Post Three explains how Forma uses Anti-Join and a Dirty Set to keep federated queries from reading uncommitted or inconsistent data.
Reading guide
Have common questions? Check the FAQ.
| Your scenario | Start here |
|---|---|
| Building AI applications, need flexible data storage | Post 1: AI Architecture |
| Struggling with N+1 queries, want quick performance gains | Post 2: Killing N+1 |
| Data growing, considering hot/cold separation | Post 3: Serverless Lakehouse |
| Want the whole architecture | Read all three in order |
The series
[Post 1] Why EAV is the Most Underrated Data Model for AI
TL;DR: the schema you write to constrain the model can also be the schema your storage enforces. Add a hot table and you get AI output, instant validation, and a write with no DDL behind it.
→ Read in English | 阅读中文版
[Post 2] Killing N+1: How One SQL Trick Cut Our Latency by 40x
TL;DR: PostgreSQL CTE + JSON_AGG took database round-trips from 101 to 1 and cut latency by 97%.
→ Read in English | 阅读中文版
[Post 3] Zero Dirty Reads: Building a Trustworthy Lakehouse with DuckDB
TL;DR: PostgreSQL handles the present, DuckDB and Parquet handle the past, and Anti-Join plus a Dirty Set keeps federated queries free of dirty reads.
→ Read in English | 阅读中文版
About Forma
Forma is an open-source data storage engine for AI-era workloads, built to stay flexible without giving up query performance or running up storage costs.
Issues, stars, and discussion all land in the same place.