This guide is for heads of data, CTOs and analytics leads at mid-market companies who receive "we need it in real time" requests and must decide what to build. It sits under Netbase's data engineering and platforms service, and it pairs with the guide on change data capture for operational databases, which covers how changes leave a source system once you know how fresh they must be.
In this guide
- How fresh does a business metric really need to be?
- What is the latency-to-value worksheet?
- Streaming, micro-batch or batch: how do the four tiers compare?
- When should you not stream a metric?
- How to roll out the worksheet across a data platform
- Where AI fits
- What delivery record exists, and what does not
- Alternatives: other ways to get fresher numbers
- Limits of this guide
- Frequently asked questions
- Next step
How fresh does a business metric really need to be?
A metric needs to be as fresh as the fastest decision anyone takes with it, plus the time that person needs to act. A daily revenue figure reviewed in a morning meeting gains nothing from second-level updates. A stock level that triggers an automatic reorder may need minutes. A payment that must be blocked before it completes needs seconds.
The request "real time" usually describes a feeling, not a decision. The useful question to ask the requester is what they would do differently if the number were fifteen minutes old, and what that delay would cost. When the honest answer is "nothing", the metric belongs on batch.
Freshness also has a price. A streaming pipeline runs all day, has to handle late and out-of-order events, and needs someone on call when it stalls. A batch job starts, finishes and can be re-run from the same input. Public engineering guides make the same point: decide by the latency the use case requires, not by which approach looks more modern.
What is the latency-to-value worksheet?
The worksheet is one row per metric. It turns a freshness request into a decision you can defend, and it is short enough to fill in with the metric owner in one meeting.
-
Metric and owner
- Question to answer
- Which number, and who acts on it?
- Example answer
- Orders awaiting dispatch, warehouse lead
-
Decision it drives
- Question to answer
- What action changes when the number changes?
- Example answer
- Add a picking shift
-
Decision cadence
- Question to answer
- How often is that action actually taken?
- Example answer
- Twice a day
-
Cost of acting late
- Question to answer
- What is lost if the action comes an hour later?
- Example answer
- Some orders ship the next day
-
Data arrival
- Question to answer
- Does the source produce events continuously, in bursts or in files?
- Example answer
- Orders arrive continuously
-
Correction rule
- Question to answer
- Can a late or corrected record change yesterday's value?
- Example answer
- Yes, cancellations
-
Proposed tier
- Question to answer
- Daily, hourly, micro-batch or streaming?
- Example answer
- Hourly batch
The rule of thumb that falls out of the worksheet: the tier is set by the decision cadence, and it moves one tier faster only when the cost of acting late is concrete and the owner can name it. In our experience of scoring metrics this way, most dashboard, finance and marketing metrics land on hourly or daily batch.
Streaming, micro-batch or batch: how do the four tiers compare?
| Tier | Freshness and strength | Risk | Choose it when |
|---|---|---|---|
| Daily batch | Next morning; simplest to test, re-run and reconcile | Stale for anything acted on within the day | Finance close, trend reporting, model training sets |
| Hourly batch | Within the hour; fresh enough for most operational dashboards and still re-runnable | Load spikes if many jobs start on the hour | Sales, stock and campaign dashboards reviewed during the day |
| Micro-batch | A few minutes; near-live numbers with batch-style recovery | More scheduling and state than hourly jobs | Operations screens where a few minutes changes an action |
| Streaming | Seconds or less; acts on each event as it arrives | Continuous running cost, late-event handling, on-call load | Fraud blocking, live alerts, automated reactions to events |
Micro-batch engines such as Spark Structured Streaming process a stream as a series of small batches, which is why they sit between the two extremes. Change data capture is a separate choice: it can feed any tier, and log-based tools such as Debezium can deliver changes with very low delay even when the warehouse loads them hourly.
When should you not stream a metric?
Do not stream a metric when its owner reviews it on a schedule, when its value can be corrected later by refunds, returns or cancellations, or when the source system itself only updates in files or nightly exports. In each case a faster pipeline shows a number that is either ignored or wrong until it settles.
Do not stream when the team has no one to own the pipeline outside office hours. A batch job that fails at night can be re-run in the morning; a stream that falls behind keeps producing stale numbers that look live. And do not stream to "future-proof" a dashboard: moving one metric up a tier later is cheaper than running every metric at the fastest tier from day one.
How to roll out the worksheet across a data platform
-
List the metrics, not the pipelines
Start from the dashboards and automated actions people use, and write one worksheet row per metric with its owner, as recorded in the data catalog.
-
Fill the decision columns with the owner
Ask what action the metric drives, how often it is taken and what a late action costs. Leave the tier blank until those are answered.
-
Set every metric to hourly batch by default
Move a metric to daily batch when nobody acts on it within the day, and to micro-batch or streaming only when the cost of acting late is written down.
-
Check the source
Confirm how each source produces data; a source that exports files cannot feed a stream, and a database may need change data capture before a faster tier is possible.
-
Build the batch layer first
Model, test and reconcile the metric in batch, so a faster tier later reuses the same definitions and can be checked against them.
-
Promote on evidence
Move a metric up one tier when the owner shows a decision that was missed or made late because of its freshness, and record the reason in the worksheet.
-
Review twice a year
Decision cadences change; demote metrics nobody acts on quickly any more and retire the faster pipelines they used.
Where AI fits
AI helps with the reading and checking around the worksheet, not with deciding the tier. An assistant can draft worksheet rows from dashboard usage logs and meeting notes, summarise which metrics are opened and when, and flag a pipeline whose freshness no longer matches its documented tier. Netbase works with the major commercial and open-source AI tools and models, chosen per project.
Engineers and metric owners still set every tier, approve every promotion to streaming and own the reconciliation between batch and faster pipelines; no model changes a schedule or a pipeline on its own.
- In delivered work
MLOps pipeline
An MLOps pipeline among the AI projects Netbase has delivered for clients that are not named; an AI delivery record, not a streaming analytics deployment.
- Available capability
AI and data services
AI-assisted worksheet drafting and freshness checks are a way Netbase approaches the work, not yet tied to a published case.
What delivery record exists, and what does not
Netbase has delivered AI projects for clients that are not named, including MLOps pipelines, and its named AI delivery is the 4over4 recommendation engine. Its ERP and e-commerce work includes API sync of product catalogues, inventory, orders and customer data with ERP systems, which is where freshness questions about stock and orders first appear.
Delivery runs in a six-step lifecycle from discovery and architecture planning to rollout and ongoing support, and most Netbase projects are delivered on fixed-price contracts. No published Netbase record describes a streaming analytics platform built for a client, and no latency, cost or result figure is claimed here.
Alternatives: other ways to get fresher numbers
- Query the source directly. A dashboard that reads a read replica of the operational database can be current without any pipeline, as in many SaaS in-product dashboards. Choose it for a few low-traffic screens, not for heavy queries across many systems.
- Operational alerts in the application. A threshold check inside the order or payment system can act in seconds while the analytics layer stays on batch. Choose it when one action needs speed and the reporting does not.
- Shorter batch windows. Moving one job from daily to hourly is often enough and needs no new tooling. Choose it before any new streaming component.
- Managed streaming services. A cloud provider's managed stream and processing services reduce the operations work of streaming. Choose them when the worksheet shows a real need and the team prefers less to run.
The cloud migration cutover guide covers the data-sync side when a pipeline change coincides with a platform move.
Limits of this guide
- The tiers and freshness ranges are planning bands, not benchmarks; measure your own pipelines before you commit to a figure in a service agreement.
- The worksheet covers analytics and decision metrics. Systems that must react to each event, such as payment authorisation, are product features and need their own design review.
- No tool is recommended here; the right engine depends on your cloud, your team's skills and the sources involved.
Plan the next step with a Netbase consultant
Frequently asked questions
For most operations dashboards it is, because people review them a few times a day and act on trends rather than single events. Hourly batch becomes too slow when a named action, such as stopping a shipment or blocking a payment, must happen within minutes of the event.
Usually it does, because a streaming pipeline runs continuously, handles late and out-of-order events and needs someone to watch it outside office hours. A batch job uses resources only while it runs and can be re-run from the same input after a failure.
Yes, and it is the safer order. Building the metric in batch first fixes its definition and tests, and a faster pipeline added later can be reconciled against the batch result before anyone relies on it.
Ask what decision the number drives, how often that decision is taken and what is lost if the number is fifteen minutes or an hour old. The answers fill the worksheet row and usually settle the tier.
Next step
If you want a second opinion on which of your metrics need a faster pipeline, Netbase can run the worksheet with your metric owners and propose a tier per metric: book a solution review.
Related services and solutions
Data engineering platforms for reporting and AI Netbase builds data pipelines and analytics foundations for companies whose data is spread across a store, an ERP, a CRM and spreadsheets. We collect it, clean it, model it and check its quality, so reports agree with each other and AI projects start from data people trust. Data engineering is a capability we are growing alongside the platforms we deliver.
Learn More
AI and data services, from readiness check to production feature Netbase provides AI and data services for companies that want to move from an AI idea to a feature that works inside a real product or process: readiness assessment, data preparation, model and automation development, and production support. Delivered AI includes 4over4's recommendation engine, plus RAG, document AI and MLOps projects for unnamed clients; the rest is growing.
Learn More
Enterprise integration hub ready for AI agents: one monitored layer for commerce, ERP and CRM data An enterprise integration hub is a central layer that moves orders, customers, stock and invoices between your commerce, ERP and CRM systems, with monitoring, retries and replay in one place, and scoped APIs that AI agents can call safely. Netbase builds it as a custom layer on the stack you already run, starting from the flows that break most often.
Learn More
Work with Netbase remotely
Every Netbase engagement is delivered remotely, so clients in any country and any industry can work with us; Netbase has worked with clients in 80 countries. Contracts and invoices are issued by Netbase JSC, and payment is made by bank transfer to Netbase JSC's company account or PayPal Business. Talk to the CEO, Huy Nguyen (David), on WhatsApp or by email.
Discuss a project
Netbase JSC helps organizations design, build, modernize, and operate digital products and AI-enabled business systems.project@netbasejsc.com
+84 937 869 689
91 Nguyen Chi Thanh, Dong Da, Hanoi, Vietnam
Get in touch
Tell us what you want to build, modernize, or operate.