Skip to main content

What are you looking for?

Explore our services and discover how we can help you achieve your goals

Client Workspace

Data catalog and lineage: what a 20-person data team actually needs

A data team of about twenty people needs a small catalog of its most-used tables, each with an owner, and table-level lineage that answers "what breaks if I change this?"; a full governance platform pays off only after specific triggers. Netbase scopes catalog and lineage work from the questions the team must answer, not from a tool list.

Book a solution review See the related solution

By Netbase Commerce and Platform Engineering Desk · Reviewed by David (CEO) · Updated · 11 min read

star

This guide is for heads of data, analytics leads and CTOs at mid-market companies who are being asked to buy a catalog or governance platform and want to know what is enough. It sits under Netbase's data engineering and platforms service, and it pairs with the guides on change data capture for operational databases and on streaming or batch latency per business metric, whose outputs (capture method and freshness tier) belong in the catalog entry of each table.

In this guide

What does a 20-person data team actually need from a catalog?

A team of this size needs three things from a catalog: people can find the right table without asking in chat, every important table has a named owner, and an engineer can see what depends on a table before changing it. Search across every asset, automated classification and approval workflows are useful later, but they are not what slows a team of twenty down.

At this size the usual failure is not a missing tool. It is two tables with similar names, no one sure which is current, and a dashboard that breaks after a column rename because nobody knew it read that column. A catalog that covers the tables people actually query, with an owner and a status on each, removes most of that friction.

Public guidance for lean teams makes the same point: with one warehouse, clear naming, a short data dictionary and the schema browser of the BI tool often cover the need, and a dedicated catalog becomes worth it once nobody can reliably say which table is the right one.

What is the minimum viable catalog?

The minimum viable catalog is one entry per table people query, kept in the same repository as the pipeline code, so it is reviewed with every change. It starts with the most-queried tables and the tables behind board or finance figures, not with every table in the warehouse.

  • Name and status

    What it holds
    Table name, and whether it is current, deprecated or experimental
    Who fills it
    Data engineer who builds it
  • Description

    What it holds
    One or two plain sentences: what one row means and what the table is for
    Who fills it
    Table owner
  • Owner

    What it holds
    A named role or person who answers questions and approves changes
    Who fills it
    Head of data assigns
  • Source and method

    What it holds
    Source system and how data arrives (batch export, API, change data capture)
    Who fills it
    Data engineer
  • Freshness tier

    What it holds
    How often it is refreshed and the freshness consumers rely on
    Who fills it
    Table owner with consumers
  • Sensitivity

    What it holds
    Whether it holds personal or restricted data, and the access rule
    Who fills it
    Table owner, checked by security
  • Key consumers

    What it holds
    The dashboards, reports or services that read it
    Who fills it
    Generated from lineage where possible
  • Known limits

    What it holds
    Gaps, late data, definitions that differ from another team's
    Who fills it
    Table owner

The rule of thumb: an entry without an owner is not finished, and a table without an entry should not feed a dashboard. Transformation tools such as dbt already let teams write descriptions next to the model code and generate a browsable documentation site with a dependency graph, which is often the first catalog a team needs.

Which lineage use cases pay back?

Lineage is worth its upkeep only where it answers a question someone asks often or where a wrong answer is expensive. The matrix below ranks the common use cases by the level of lineage each needs.

Use case Lineage level needed Payback Choose it when
Impact analysis before a change Table level, from source to dashboard High: prevents broken reports after renames and drops Always; it is the first use case to support
Incident root cause Table level with job run history High: shortens the search for which job produced a bad figure Several pipelines feed shared tables
Regulated or contested figures Column level, for a small set of fields High for those fields only A figure goes to auditors, regulators or the board
Migration or tool change Table level, a full inventory Medium, once per migration A warehouse or BI move is planned
Access and deletion requests Column level for personal data fields Medium to high where personal data is held Personal data spreads across many tables

Column-level lineage across the whole warehouse rarely pays back for a team of twenty: it is costly to keep accurate, and most questions are answered at table level. Keep column-level lineage for the few fields where a wrong answer has legal or financial weight.

Catalog in code, open-source catalog or commercial platform?

Option Strength Risk Choose it when
Catalog in code and the transformation tool Lowest running cost; entries are reviewed with code changes Covers only what the tool builds; weak for sources and BI objects One warehouse, one transformation project, one data team
Open-source catalog (for example DataHub) Search and lineage across sources, warehouse and BI; no licence fee Someone must run, upgrade and connect it A second team produces data, or consumers cannot be listed
Commercial governance platform Workflows, policy, classification and support included Licence cost and a long rollout; adoption often lags Compliance asks for it, or many teams share regulated data

Open frameworks such as OpenLineage let jobs emit lineage events in a standard format, so the lineage you collect while on the first option can be loaded into a catalog later instead of being redrawn.

When should a data team not buy a catalog platform?

Do not buy a catalog platform when one team owns one warehouse and one transformation project, when nobody has time to keep owners and descriptions current, or when the real problem is unclear table names and duplicate models. A platform indexes the confusion; it does not remove it, and an empty or stale catalog teaches people to stop looking.

The triggers that justify a platform are organisational, not a table count: a second team producing data, consumers no one can list, a regulated report, an upcoming migration, or repeated incidents caused by unknown dependencies. When two or more apply, evaluate an open-source or commercial catalog against the entries you already keep in code.

How to roll out a minimum viable catalog

  1. List the tables people query

    Use warehouse query logs or BI usage to find the most-read tables and the tables behind finance and board figures; that list is the first scope.

  2. Assign an owner per table

    The head of data names an owner for each table on the list; a table nobody will own is a candidate for deprecation.

  3. Write the entries next to the code

    Add the eight fields to the model files or a catalog file in the same repository, and review them in the same pull request as the pipeline change.

  4. Turn on the lineage you already have

    Use the dependency graph of the transformation tool, and add lineage events from orchestration jobs where the tool does not see them.

  5. Add one check in continuous integration

    Fail a change that adds a table without an owner and description, or that drops a column a dashboard still reads.

  6. Mark deprecated tables

    Give every replaced table a status and a successor, and remove it after its consumers have moved.

  7. Review the triggers twice a year

    Check the upgrade triggers above; move to a catalog platform only when they apply, and load the existing entries into it.

Where AI fits

AI helps with the writing and checking around the catalog, not with deciding ownership. An assistant can draft table descriptions from model code and column names, suggest which columns may hold personal data, and flag entries whose description no longer matches the model. Netbase works with the major commercial and open-source AI tools and models, chosen per project.

Table owners still approve every description and sensitivity label, and engineers still decide which lineage is recorded; no model changes an owner, an access rule or a pipeline on its own.

What delivery record exists, and what does not

Netbase has delivered AI projects for clients that are not named, including MLOps pipelines, and its named AI delivery is the 4over4 recommendation engine. Its ERP and e-commerce work includes API sync of product catalogues, inventory, orders and customer data with ERP systems, where the question of which system owns a record comes up first.

Delivery runs in a six-step lifecycle from discovery and architecture planning to rollout and ongoing support, and most Netbase projects are delivered on fixed-price contracts. No published Netbase record describes a data catalog or governance platform rollout for a client, and no adoption, cost or time figure is claimed here.

Alternatives: other ways to make data findable

  • A shared data dictionary. A single page listing the main tables and their meaning works for a very small team; choose it when there is one analyst group and few tables, and replace it once entries drift from the code.
  • BI semantic layer descriptions. Describing metrics and fields in the BI tool helps business users; choose it when most questions are about dashboards, not tables, and keep the table entries in code anyway.
  • Warehouse-native catalogs. Cloud warehouses and lakehouse platforms ship their own catalogs with access control and some lineage; choose them when nearly all data lives on one platform.
  • A naming convention first. Clear prefixes for staging, intermediate and reporting tables remove much of the "which table?" problem; choose it before any tool.

Limits of this guide

  • The field set and the payback matrix are planning aids, not benchmarks; adapt them to your regulation and your team's structure.
  • "Twenty people" is a size band. A team with several separate business units may hit the upgrade triggers earlier.
  • No tool is recommended here; the right catalog depends on your warehouse, your BI tool and who will run it.

Plan the next step with a Netbase consultant

Frequently asked questions

Often not at first, because dbt documentation already holds descriptions next to the model code and shows a dependency graph. A separate catalog becomes useful when sources, BI objects or a second team fall outside what the dbt project can describe.

Usually only for a small set of fields, such as those behind regulated, financial or contested figures and fields that hold personal data. For most other questions, table-level lineage gives the same answer at a fraction of the upkeep.

The table owner, usually the analytics engineer or data engineer responsible for the model, owns its entry, and the head of data assigns owners. An entry without a named owner goes stale quickly and should block the table from feeding a dashboard.

Move when two or more triggers apply: a second team producing data, consumers nobody can list, a regulated report, a planned migration or repeated incidents from unknown dependencies. Load the entries you already keep in code into the new platform rather than starting over.

Next step

If you want a second opinion on how much catalog and lineage your team needs, Netbase can review your most-used tables and owners and propose a minimum catalog and an upgrade path: book a solution review.

Data engineering platforms for reporting and AI Data engineering platforms for reporting and AI

Netbase builds data pipelines and analytics foundations for companies whose data is spread across a store, an ERP, a CRM and spreadsheets. We collect it, clean it, model it and check its quality, so reports agree with each other and AI projects start from data people trust. Data engineering is a capability we are growing alongside the platforms we deliver.

Learn More
line
AI and data services, from readiness check to production feature AI and data services, from readiness check to production feature

Netbase provides AI and data services for companies that want to move from an AI idea to a feature that works inside a real product or process: readiness assessment, data preparation, model and automation development, and production support. Delivered AI includes 4over4's recommendation engine, plus RAG, document AI and MLOps projects for unnamed clients; the rest is growing.

Learn More
line
Enterprise integration hub ready for AI agents: one monitored layer for commerce, ERP and CRM data Enterprise integration hub ready for AI agents: one monitored layer for commerce, ERP and CRM data

An enterprise integration hub is a central layer that moves orders, customers, stock and invoices between your commerce, ERP and CRM systems, with monitoring, retries and replay in one place, and scoped APIs that AI agents can call safely. Netbase builds it as a custom layer on the stack you already run, starting from the flows that break most often.

Learn More
line

Work with Netbase remotely

Every Netbase engagement is delivered remotely, so clients in any country and any industry can work with us; Netbase has worked with clients in 80 countries. Contracts and invoices are issued by Netbase JSC, and payment is made by bank transfer to Netbase JSC's company account or PayPal Business. Talk to the CEO, Huy Nguyen (David), on WhatsApp or by email.

Contact Netbase

Discuss a project

Netbase JSC helps organizations design, build, modernize, and operate digital products and AI-enabled business systems.
Project enquiries

project@netbasejsc.com

WhatsApp

+84 937 869 689

Office address

91 Nguyen Chi Thanh, Dong Da, Hanoi, Vietnam

Get in touch

Tell us what you want to build, modernize, or operate.

Tell us what you want to build, modernize, or operate.

Contact Netbase