This guide is for heads of data, data engineering leads and CTOs at mid-market companies who run a lake that people have stopped trusting, or who are about to build one. It sits under Netbase's data engineering and platforms service. It does not compare Iceberg, Delta Lake and Hudi against each other; it explains which failures any of them removes and which stay with your team.
In this guide
- What turns a data lake into a swamp?
- Which swamp failures does an open table format fix?
- What does an open table format actually add?
- Who owns the lake after the format is in place?
- When do you not need a data lake at all?
- How to pull a lake back out of the swamp
- Where AI fits
- What delivery record exists, and what does not
- Alternatives: warehouse only, managed lakehouse or plain files with conventions
- Limits of this guide
- Frequently asked questions
- Next step
What turns a data lake into a swamp?
A data lake becomes a swamp through seven repeatable failures, not through one bad decision. Each one is small when it starts, and together they produce a store where two queries over the same folder return different answers and nobody can say which is right.
Martin Fowler describes the criticism directly: a lake can become a dumping ground for data of widely varying quality, better named a data swamp. The patterns below are how that happens in practice.
- Half-written data is visible. A job fails midway and readers see part of a load, because plain files on object storage have no transaction that commits or rolls back as a whole.
- Schemas drift silently. A source adds, renames or retypes a column, new files carry the new shape, and old and new files no longer read the same way.
- Folders stand in for tables. Partitioning lives in directory names that each writer interprets its own way, so a query that filters on the wrong path scans everything or misses data.
- Small files pile up. Frequent loads write many tiny files; queries slow down and storage listings grow until jobs time out.
- Nothing is ever deleted. Old copies, failed outputs and abandoned experiments stay, so storage grows and people query the wrong copy.
- No owner, no meaning. Datasets have no named owner, description or freshness promise, so readers guess what a column means.
- Zones are only folder names. "Raw", "clean" and "curated" exist as paths, but nothing checks what may move from one to the next.
Which swamp failures does an open table format fix?
The swamp diagnostic maps each failure to what an open table format does about it. Read it as a split of the work: the left side is solved by adopting a format, the right side is solved only by people and process.
| Failure pattern | Table format fixes it | Table format helps | Still needs an owner |
|---|---|---|---|
| Half-written data visible | Yes: commits are atomic, readers see a whole snapshot or nothing | Readers never see a partial load | Retry policy for failed jobs |
| Silent schema drift | Partly: writes with a mismatched schema are rejected or evolved on purpose | Column changes are tracked by ID | Who approves a schema change |
| Folders as tables | Yes: partitions live in table metadata, not in paths | Engines prune by metadata | Choosing a partition scheme |
| Small files pile up | No | Compaction and rewrite tools exist | Scheduling and paying for compaction |
| Nothing deleted | No | Snapshot expiry and orphan file clean-up exist | Retention rules and who runs them |
| No owner, no meaning | No | Only via a catalog that holds owners and descriptions | Naming owners and writing definitions |
| Zones only in name | No | Constraints and schema checks at each boundary | Contracts between zones and producers |
Three of the seven failures disappear when tables move to a format. Two get tools that still have to be scheduled. Two are untouched: a format records what a table looks like, not who answers for it.
What does an open table format actually add?
An open table format adds a metadata layer over ordinary data files that records which files form the table, its schema and its history. Engines read that metadata instead of listing folders, so every reader sees the same committed version of the table.
Each of the three common formats keeps this record in its own way. The Apache Iceberg specification tracks tables as snapshots over manifest files. Delta Lake keeps a transaction log and, by default, validates that the schema of written data matches the table and raises an error when it does not. Apache Hudi keeps a timeline that logs every action on the table.
Two features matter most against a swamp. Schema evolution in Iceberg tracks columns by ID, so adding, renaming or reordering a column does not corrupt older files. Time travel lets a team query an earlier version after a bad load, which turns a silent data incident into a recoverable one. The data files stay in an open columnar format such as Parquet, so engines other than the one that wrote them can still read the table.
Who owns the lake after the format is in place?
A table format creates new work that someone must own: expiring old snapshots, compacting small files and deleting orphan files. The Iceberg maintenance guide lists these as regular operations; without them, history and small files grow until the lake slows down again.
Give every table three named roles. The producer owns the schema and the load. The platform team owns maintenance jobs and retention. The consumer team owns the definitions it publishes onward. A catalog with lineage sized for a small data team is where those names and definitions live, so readers can find them without asking.
When do you not need a data lake at all?
You do not need a data lake when almost all your data is structured, arrives from a handful of business systems and fits a cloud warehouse comfortably. A warehouse already gives transactions, schemas and access control in one product, and adding a lake beside it doubles the places where a number can be wrong.
A lake earns its cost when at least one of these holds: large volumes of files, logs or events that a warehouse would store expensively; semi-structured or unstructured data for machine learning; several engines that must read the same tables; or a firm requirement to keep data in an open format you control. If none holds, start with a warehouse and revisit the question when one does.
How to pull a lake back out of the swamp
-
Inventory before you migrate
List every dataset, its last write, its readers and its size; most swamps contain a large share of data nobody reads.
-
Name an owner per dataset
A dataset with no owner after the inventory is a candidate for archive, not for migration.
-
Convert the hot tables first
Move the tables that feed reports and models into an open table format, and leave cold archives as plain files until someone needs them.
-
Schedule maintenance on day one
Set snapshot expiry, compaction and orphan file clean-up as owned jobs with alerts, not as a later clean-up project.
-
Put checks on zone boundaries
Data moves from raw to clean only after schema and basic quality checks pass; ingestion from operational databases follows the method chosen in the guide on change data capture for operational databases.
-
Retire what is left
Archive or delete datasets with no owner and no reader after an agreed notice period, and record the decision.
Where AI fits
AI helps with the inventory and the documentation work that makes a lake usable, not with the storage itself. An assistant can draft column descriptions from sample data, group near-duplicate datasets for review and summarise which tables are read and by whom. Netbase works with the major commercial and open-source AI tools and models, chosen per project.
Engineers and dataset owners still approve every description, every schema change and every deletion; no model removes data or changes a table on its own.
- In delivered work
MLOps pipeline
An MLOps pipeline among the AI projects Netbase has delivered for clients that are not named; an AI delivery record, not a lake rebuild.
- Available capability
AI and data services
AI-assisted dataset inventory and documentation are a way Netbase approaches the work, not yet tied to a published case.
What delivery record exists, and what does not
Netbase has delivered AI projects for clients that are not named, including MLOps pipelines, and its named AI delivery is the 4over4 recommendation engine, which suggests products from browsing history and previous purchases. Its ERP and e-commerce work includes API sync of product catalogues, inventory, orders and customer data with ERP systems, which is the kind of source data a lake or warehouse later receives.
Delivery runs in a six-step lifecycle from discovery and architecture planning to rollout and ongoing support, and most Netbase projects are delivered on fixed-price contracts. No published Netbase record describes a data lake recovery or an open table format migration for a client, and no size, speed or cost figure is claimed here.
Alternatives: warehouse only, managed lakehouse or plain files with conventions
- Warehouse only. Load structured data straight into a cloud warehouse and skip the lake. Choose it when data is mostly tabular, sources are few and one engine is enough.
- Managed lakehouse platform. Use a vendor platform that runs an open table format, catalog and maintenance for you. Choose it when the team is small and paying for operations beats staffing them; check how easily tables can be read from outside the platform.
- Plain files with strict conventions. Keep Parquet files with documented paths, write-once folders and a single writer per dataset. Choose it for append-only archives that one team writes and few people read.
- Open table format on your own storage. Run Iceberg, Delta Lake or Hudi with your own engines and catalog. Choose it when several engines share tables, data must stay in an open format you control, and the team can own maintenance.
Limits of this guide
- It explains failures and countermeasures; it does not rank Iceberg, Delta Lake or Hudi, whose fit depends on your engines and catalog.
- No price, cost or performance figure is given; storage and compute cost depend on volume, query patterns and the cloud contract.
Plan the next step with a Netbase consultant
Frequently asked questions
Only part of it. A table format removes half-written loads, folder-based partitions and silent schema breaks, but it does not name owners, write definitions or delete data nobody needs. Those still need people and scheduled jobs.
Often not. If your data is mostly structured, comes from a few business systems and one engine reads it, a warehouse alone is simpler and safer. Add a lake when file volumes, machine learning or several engines make the warehouse the wrong store.
Regular snapshot expiry, compaction of small files and removal of orphan files. Without these jobs the table history and file count keep growing, queries slow down and storage cost rises, so each job needs an owner and an alert.
No. Convert the tables that feed reports and models first, archive datasets with no owner or reader, and leave cold files as they are until someone needs them.
Next step
If your lake has stopped being trusted, Netbase can run the swamp diagnostic with your team and propose which tables to convert, own or retire first: book a solution review.
Related services and solutions
Data engineering platforms for reporting and AI Netbase builds data pipelines and analytics foundations for companies whose data is spread across a store, an ERP, a CRM and spreadsheets. We collect it, clean it, model it and check its quality, so reports agree with each other and AI projects start from data people trust. Data engineering is a capability we are growing alongside the platforms we deliver.
Learn More
AI and data services, from readiness check to production feature Netbase provides AI and data services for companies that want to move from an AI idea to a feature that works inside a real product or process: readiness assessment, data preparation, model and automation development, and production support. Delivered AI includes 4over4's recommendation engine, plus RAG, document AI and MLOps projects for unnamed clients; the rest is growing.
Learn More
Enterprise integration hub ready for AI agents: one monitored layer for commerce, ERP and CRM data An enterprise integration hub is a central layer that moves orders, customers, stock and invoices between your commerce, ERP and CRM systems, with monitoring, retries and replay in one place, and scoped APIs that AI agents can call safely. Netbase builds it as a custom layer on the stack you already run, starting from the flows that break most often.
Learn More
Work with Netbase remotely
Every Netbase engagement is delivered remotely, so clients in any country and any industry can work with us; Netbase has worked with clients in 80 countries. Contracts and invoices are issued by Netbase JSC, and payment is made by bank transfer to Netbase JSC's company account or PayPal Business. Talk to the CEO, Huy Nguyen (David), on WhatsApp or by email.
Discuss a project
Netbase JSC helps organizations design, build, modernize, and operate digital products and AI-enabled business systems.project@netbasejsc.com
+84 937 869 689
91 Nguyen Chi Thanh, Dong Da, Hanoi, Vietnam
Get in touch
Tell us what you want to build, modernize, or operate.