Skip to main content

What are you looking for?

Explore our services and discover how we can help you achieve your goals

Client Workspace

Why data lakes turn into swamps and how open table formats prevent it

A data lake turns into a swamp when files land without table rules, owners or clean-up, so nobody trusts what a query returns. Open table formats fix the table mechanics: atomic writes, schema checks and history. Netbase pairs them with named owners and maintenance jobs, because the format cannot fix ownership.

Book a solution review See the related solution

By Netbase Commerce and Platform Engineering Desk · Reviewed by David (CEO) · Updated · 11 min read

star

This guide is for heads of data, data engineering leads and CTOs at mid-market companies who run a lake that people have stopped trusting, or who are about to build one. It sits under Netbase's data engineering and platforms service. It does not compare Iceberg, Delta Lake and Hudi against each other; it explains which failures any of them removes and which stay with your team.

In this guide

What turns a data lake into a swamp?

A data lake becomes a swamp through seven repeatable failures, not through one bad decision. Each one is small when it starts, and together they produce a store where two queries over the same folder return different answers and nobody can say which is right.

Martin Fowler describes the criticism directly: a lake can become a dumping ground for data of widely varying quality, better named a data swamp. The patterns below are how that happens in practice.

  1. Half-written data is visible. A job fails midway and readers see part of a load, because plain files on object storage have no transaction that commits or rolls back as a whole.
  2. Schemas drift silently. A source adds, renames or retypes a column, new files carry the new shape, and old and new files no longer read the same way.
  3. Folders stand in for tables. Partitioning lives in directory names that each writer interprets its own way, so a query that filters on the wrong path scans everything or misses data.
  4. Small files pile up. Frequent loads write many tiny files; queries slow down and storage listings grow until jobs time out.
  5. Nothing is ever deleted. Old copies, failed outputs and abandoned experiments stay, so storage grows and people query the wrong copy.
  6. No owner, no meaning. Datasets have no named owner, description or freshness promise, so readers guess what a column means.
  7. Zones are only folder names. "Raw", "clean" and "curated" exist as paths, but nothing checks what may move from one to the next.

Which swamp failures does an open table format fix?

The swamp diagnostic maps each failure to what an open table format does about it. Read it as a split of the work: the left side is solved by adopting a format, the right side is solved only by people and process.

Failure pattern Table format fixes it Table format helps Still needs an owner
Half-written data visible Yes: commits are atomic, readers see a whole snapshot or nothing Readers never see a partial load Retry policy for failed jobs
Silent schema drift Partly: writes with a mismatched schema are rejected or evolved on purpose Column changes are tracked by ID Who approves a schema change
Folders as tables Yes: partitions live in table metadata, not in paths Engines prune by metadata Choosing a partition scheme
Small files pile up No Compaction and rewrite tools exist Scheduling and paying for compaction
Nothing deleted No Snapshot expiry and orphan file clean-up exist Retention rules and who runs them
No owner, no meaning No Only via a catalog that holds owners and descriptions Naming owners and writing definitions
Zones only in name No Constraints and schema checks at each boundary Contracts between zones and producers

Three of the seven failures disappear when tables move to a format. Two get tools that still have to be scheduled. Two are untouched: a format records what a table looks like, not who answers for it.

What does an open table format actually add?

An open table format adds a metadata layer over ordinary data files that records which files form the table, its schema and its history. Engines read that metadata instead of listing folders, so every reader sees the same committed version of the table.

Each of the three common formats keeps this record in its own way. The Apache Iceberg specification tracks tables as snapshots over manifest files. Delta Lake keeps a transaction log and, by default, validates that the schema of written data matches the table and raises an error when it does not. Apache Hudi keeps a timeline that logs every action on the table.

Two features matter most against a swamp. Schema evolution in Iceberg tracks columns by ID, so adding, renaming or reordering a column does not corrupt older files. Time travel lets a team query an earlier version after a bad load, which turns a silent data incident into a recoverable one. The data files stay in an open columnar format such as Parquet, so engines other than the one that wrote them can still read the table.

Who owns the lake after the format is in place?

A table format creates new work that someone must own: expiring old snapshots, compacting small files and deleting orphan files. The Iceberg maintenance guide lists these as regular operations; without them, history and small files grow until the lake slows down again.

Give every table three named roles. The producer owns the schema and the load. The platform team owns maintenance jobs and retention. The consumer team owns the definitions it publishes onward. A catalog with lineage sized for a small data team is where those names and definitions live, so readers can find them without asking.

When do you not need a data lake at all?

You do not need a data lake when almost all your data is structured, arrives from a handful of business systems and fits a cloud warehouse comfortably. A warehouse already gives transactions, schemas and access control in one product, and adding a lake beside it doubles the places where a number can be wrong.

A lake earns its cost when at least one of these holds: large volumes of files, logs or events that a warehouse would store expensively; semi-structured or unstructured data for machine learning; several engines that must read the same tables; or a firm requirement to keep data in an open format you control. If none holds, start with a warehouse and revisit the question when one does.

How to pull a lake back out of the swamp

  1. Inventory before you migrate

    List every dataset, its last write, its readers and its size; most swamps contain a large share of data nobody reads.

  2. Name an owner per dataset

    A dataset with no owner after the inventory is a candidate for archive, not for migration.

  3. Convert the hot tables first

    Move the tables that feed reports and models into an open table format, and leave cold archives as plain files until someone needs them.

  4. Schedule maintenance on day one

    Set snapshot expiry, compaction and orphan file clean-up as owned jobs with alerts, not as a later clean-up project.

  5. Put checks on zone boundaries

    Data moves from raw to clean only after schema and basic quality checks pass; ingestion from operational databases follows the method chosen in the guide on change data capture for operational databases.

  6. Retire what is left

    Archive or delete datasets with no owner and no reader after an agreed notice period, and record the decision.

Where AI fits

AI helps with the inventory and the documentation work that makes a lake usable, not with the storage itself. An assistant can draft column descriptions from sample data, group near-duplicate datasets for review and summarise which tables are read and by whom. Netbase works with the major commercial and open-source AI tools and models, chosen per project.

Engineers and dataset owners still approve every description, every schema change and every deletion; no model removes data or changes a table on its own.

What delivery record exists, and what does not

Netbase has delivered AI projects for clients that are not named, including MLOps pipelines, and its named AI delivery is the 4over4 recommendation engine, which suggests products from browsing history and previous purchases. Its ERP and e-commerce work includes API sync of product catalogues, inventory, orders and customer data with ERP systems, which is the kind of source data a lake or warehouse later receives.

Delivery runs in a six-step lifecycle from discovery and architecture planning to rollout and ongoing support, and most Netbase projects are delivered on fixed-price contracts. No published Netbase record describes a data lake recovery or an open table format migration for a client, and no size, speed or cost figure is claimed here.

Alternatives: warehouse only, managed lakehouse or plain files with conventions

  • Warehouse only. Load structured data straight into a cloud warehouse and skip the lake. Choose it when data is mostly tabular, sources are few and one engine is enough.
  • Managed lakehouse platform. Use a vendor platform that runs an open table format, catalog and maintenance for you. Choose it when the team is small and paying for operations beats staffing them; check how easily tables can be read from outside the platform.
  • Plain files with strict conventions. Keep Parquet files with documented paths, write-once folders and a single writer per dataset. Choose it for append-only archives that one team writes and few people read.
  • Open table format on your own storage. Run Iceberg, Delta Lake or Hudi with your own engines and catalog. Choose it when several engines share tables, data must stay in an open format you control, and the team can own maintenance.

Limits of this guide

  • It explains failures and countermeasures; it does not rank Iceberg, Delta Lake or Hudi, whose fit depends on your engines and catalog.
  • No price, cost or performance figure is given; storage and compute cost depend on volume, query patterns and the cloud contract.

Plan the next step with a Netbase consultant

Frequently asked questions

Only part of it. A table format removes half-written loads, folder-based partitions and silent schema breaks, but it does not name owners, write definitions or delete data nobody needs. Those still need people and scheduled jobs.

Often not. If your data is mostly structured, comes from a few business systems and one engine reads it, a warehouse alone is simpler and safer. Add a lake when file volumes, machine learning or several engines make the warehouse the wrong store.

Regular snapshot expiry, compaction of small files and removal of orphan files. Without these jobs the table history and file count keep growing, queries slow down and storage cost rises, so each job needs an owner and an alert.

No. Convert the tables that feed reports and models first, archive datasets with no owner or reader, and leave cold files as they are until someone needs them.

Next step

If your lake has stopped being trusted, Netbase can run the swamp diagnostic with your team and propose which tables to convert, own or retire first: book a solution review.

Data engineering platforms for reporting and AI Data engineering platforms for reporting and AI

Netbase builds data pipelines and analytics foundations for companies whose data is spread across a store, an ERP, a CRM and spreadsheets. We collect it, clean it, model it and check its quality, so reports agree with each other and AI projects start from data people trust. Data engineering is a capability we are growing alongside the platforms we deliver.

Learn More
line
AI and data services, from readiness check to production feature AI and data services, from readiness check to production feature

Netbase provides AI and data services for companies that want to move from an AI idea to a feature that works inside a real product or process: readiness assessment, data preparation, model and automation development, and production support. Delivered AI includes 4over4's recommendation engine, plus RAG, document AI and MLOps projects for unnamed clients; the rest is growing.

Learn More
line
Enterprise integration hub ready for AI agents: one monitored layer for commerce, ERP and CRM data Enterprise integration hub ready for AI agents: one monitored layer for commerce, ERP and CRM data

An enterprise integration hub is a central layer that moves orders, customers, stock and invoices between your commerce, ERP and CRM systems, with monitoring, retries and replay in one place, and scoped APIs that AI agents can call safely. Netbase builds it as a custom layer on the stack you already run, starting from the flows that break most often.

Learn More
line

Work with Netbase remotely

Every Netbase engagement is delivered remotely, so clients in any country and any industry can work with us; Netbase has worked with clients in 80 countries. Contracts and invoices are issued by Netbase JSC, and payment is made by bank transfer to Netbase JSC's company account or PayPal Business. Talk to the CEO, Huy Nguyen (David), on WhatsApp or by email.

Contact Netbase

Discuss a project

Netbase JSC helps organizations design, build, modernize, and operate digital products and AI-enabled business systems.
Project enquiries

project@netbasejsc.com

WhatsApp

+84 937 869 689

Office address

91 Nguyen Chi Thanh, Dong Da, Hanoi, Vietnam

Get in touch

Tell us what you want to build, modernize, or operate.

Tell us what you want to build, modernize, or operate.

Contact Netbase