This guide is for heads of data, analytics leads and CTOs at mid-market companies who have agreed that a dashboard must refresh within minutes and now have to choose how to build it. It sits under Netbase's data engineering and platforms service. If the freshness target itself is still under debate, start with the guide on deciding the latency a business metric really needs, and come back here once the number is agreed.
In this guide
- What does "near-real-time" mean for a dashboard?
- Which hop sets your freshness floor?
- How do the three architectures compare?
- When should you not build a near-real-time dashboard?
- How to choose and roll out the architecture
- Where AI fits
- What delivery record exists, and what does not
- Alternatives: other ways to get fresher numbers on screen
- Limits of this guide
- Frequently asked questions
- Next step
What does "near-real-time" mean for a dashboard?
For a dashboard, near-real-time means the number on screen is a few seconds to about fifteen minutes behind the event it reports, measured from the moment the event happens to the moment the tile changes. It is not a property of one tool; it is the sum of every hop between the source system and the browser.
That end-to-end view matters because teams often measure only one hop. A stream processor can report a lag of two seconds while the dashboard still shows ten-minute-old data, because the BI tool caches its query results or the warehouse loads files on a schedule. Agree on one definition before choosing anything: event time to rendered tile.
The word also hides very different needs. A warehouse floor screen that changes a picking plan twice an hour is well served at ten minutes. A support queue wall board may need under a minute. Treat each as its own target.
Which hop sets your freshness floor?
The freshness budget splits the agreed target across the five hops a number travels, so a team can see which hop sets the floor and where money buys the most freshness. Fill it in for the current setup first, then for each candidate architecture.
-
Capture
- What adds delay
- Nightly exports, polling intervals, file drops
- Typical lever
- Change data capture or events from the application
-
Transport
- What adds delay
- Batch file loads, queue backlog
- Typical lever
- Continuous loading or a message stream
-
Transform
- What adds delay
- Full rebuilds of models, scheduled jobs
- Typical lever
- Incremental models or views updated on insert
-
Serve
- What adds delay
- Query time on large tables, BI result caches
- Typical lever
- Pre-aggregated tables, a store built for fast reads
-
Render
- What adds delay
- Browser polling interval, manual refresh
- Typical lever
- Shorter polling or pushed updates
The slowest hop sets the floor. Speeding up any other hop costs money without moving the number on screen, which is a common waste in "real-time" requests: a streaming transport feeding a dashboard that a BI tool refreshes on a schedule.
How do the three architectures compare?
| Architecture | Freshness and strength | Running cost and burden | Choose it when |
|---|---|---|---|
| A. Warehouse micro-batch plus BI refresh | Roughly 5 to 15 minutes; reuses the existing warehouse, models and tests | Warehouse compute on every load and refresh; low on-call load, jobs can be re-run | The warehouse already holds the data and minutes are fine |
| B. Change data capture into an incremental serving store | Roughly under a minute to a few minutes; fresh numbers from operational databases without touching the application | A capture pipeline and a second store to run, monitor and reconcile with the warehouse | Sources are databases and the dashboard is read often by many people |
| C. Event stream with processing and push to the screen | Seconds; each event updates the tile as it arrives | Brokers, stream processors and push connections run all day; late events and lag need an owner on call | A named action must follow an event within seconds |
Architecture A keeps one platform. The warehouse loads data every few minutes, models update incrementally, and the BI tool refreshes only recent partitions, as incremental refresh in Power BI does. Its cost driver is warehouse compute: every refresh is a query, and a busy dashboard with a short refresh interval can cost more than the load itself.
Architecture B moves the dashboard's reads off the warehouse. Changes leave the source databases through change data capture and land in a store that updates aggregates as rows arrive. In ClickHouse, for example, an incremental materialized view runs its query on each block of inserted data, so the read is cheap. The cost is a second system that must stay consistent with the warehouse.
Architecture C treats the dashboard as a live application. Events flow through a broker, a processor keeps the current state, and the browser receives updates instead of polling. It is the only option for second-level tiles, and it is the one that needs the most people to keep running.
When should you not build a near-real-time dashboard?
Do not build one when nobody acts on the number within the hour, when values are corrected later by refunds, returns or cancellations, or when the source only produces nightly files. In each case the faster dashboard shows a number that is ignored or wrong until it settles.
Do not move to architecture B or C when the team has no one to own a continuously running pipeline outside office hours. A late batch job can be re-run in the morning; a stream that falls behind shows stale numbers that look live. And do not make a whole dashboard fast because one tile needs it: mix tiers on one screen instead.
How to choose and roll out the architecture
-
Write down the freshness target per tile
Use event time to rendered tile, and give each tile a target and an owner; most dashboards have one or two fast tiles and many slow ones.
-
Fill the freshness budget for today's setup
Measure each hop and find the one that sets the floor before you buy or build anything.
-
Fix the slowest hop with the cheapest lever
Shorter load schedules, incremental models and BI refresh of recent partitions often reach the target inside architecture A.
-
Add change data capture only where sources are databases
Pick the capture method per table, as the guide on change data capture for operational databases explains, and land changes in a serving store only when warehouse reads become the floor or the bill.
-
Reserve streaming for tiles with a named action
Build architecture C for the few tiles where an action must follow an event within seconds, and keep every other tile on A or B.
-
Reconcile fast numbers against the warehouse
Compare each fast tile with its batch version daily, so a drift in the faster path is caught before anyone relies on it.
-
Review cost and usage every quarter
Demote tiles nobody opens quickly any more, and lengthen refresh intervals where the owner no longer acts within minutes.
On PostgreSQL, a simple serving layer for architecture A is a set of materialized views; refreshing them concurrently lets readers keep querying the old version while the new one is built, so a frequent refresh does not lock the dashboard.
Where AI fits
AI helps around the dashboard, not inside the freshness decision. An assistant can draft the freshness budget from pipeline logs and job schedules, summarise which tiles are opened and how often, and flag a tile whose measured freshness drifts from its target. Netbase works with the major commercial and open-source AI tools and models, chosen per project.
Engineers and tile owners still set every target, approve each move to a faster architecture and own the reconciliation between fast and batch numbers; no model changes a schedule or a pipeline on its own.
- In delivered work
MLOps pipeline
An MLOps pipeline among the AI projects Netbase has delivered for clients that are not named; an AI delivery record, not a near-real-time dashboard deployment.
- Available capability
AI and data services
AI-assisted freshness budgets and drift checks are a way Netbase approaches the work, not yet tied to a published case.
What delivery record exists, and what does not
Netbase has delivered AI projects for clients that are not named, including MLOps pipelines, and its named AI delivery is the 4over4 recommendation engine. Its ERP and e-commerce work includes API sync of product catalogues, inventory, orders and customer data with ERP systems, which is where requests for fresher stock and order numbers usually start.
Delivery runs in a six-step lifecycle from discovery and architecture planning to rollout and ongoing support, and most Netbase projects are delivered on fixed-price contracts. No published Netbase record describes a near-real-time dashboard platform built for a client, and no latency, cost or result figure is claimed here.
Alternatives: other ways to get fresher numbers on screen
- Query a read replica directly. A dashboard that reads a replica of the operational database is current without any pipeline. Choose it for a few low-traffic screens on one source, not for queries that join many systems.
- Live queries against the warehouse. A BI tool in DirectQuery mode sends each visual's query to the source, so freshness equals the load schedule. Choose it when users are few and query cost is watched.
- Alerts inside the application. A threshold check in the order or payment system can act in seconds while the dashboard stays on batch. Choose it when one action needs speed and the reporting does not.
- Polling plus a cache. Keeping browser polling but serving it from a cache or read model avoids a new pipeline. Choose it for bounded user counts and relaxed freshness, and plan for cache invalidation.
For customer-facing dashboards inside a SaaS product, the guide on embedded analytics: build, buy or hybrid covers the product side of the same choice.
Limits of this guide
- The freshness bands are planning ranges, not benchmarks; measure your own hops before committing to a figure in a service agreement.
- No price or cost figure is given; running cost depends on data volume, query patterns and the cloud contract.
- Tools are named as examples of a pattern, not as recommendations; the right engine depends on your cloud, sources and team skills.
Plan the next step with a Netbase consultant
Frequently asked questions
Usually not. A warehouse that loads every few minutes, incremental models and a BI refresh of recent data can reach five minutes on the existing platform. Streaming earns its running cost only when a tile must change within seconds of an event.
Because another hop sets the floor. A BI result cache, a scheduled model rebuild or a browser that refreshes only on demand can hold the number back however fast the stream is. Measure event time to rendered tile and fix the slowest hop first.
Yes, and it is usually the cheapest design. Historical and trend tiles read daily or hourly tables, while the one or two tiles that drive a fast action read a faster store, each labelled with its last update time.
Mostly continuous compute and people. Faster tiers run all day, refresh queries hit the warehouse more often, and a pipeline that never stops needs someone to watch lag and late events outside office hours.
Next step
If you want a second opinion on which architecture your dashboard needs, Netbase can fill the freshness budget with your team and propose the slowest design that meets each tile's target: book a solution review.
Related services and solutions
Data engineering platforms for reporting and AI Netbase builds data pipelines and analytics foundations for companies whose data is spread across a store, an ERP, a CRM and spreadsheets. We collect it, clean it, model it and check its quality, so reports agree with each other and AI projects start from data people trust. Data engineering is a capability we are growing alongside the platforms we deliver.
Learn More
AI and data services, from readiness check to production feature Netbase provides AI and data services for companies that want to move from an AI idea to a feature that works inside a real product or process: readiness assessment, data preparation, model and automation development, and production support. Delivered AI includes 4over4's recommendation engine, plus RAG, document AI and MLOps projects for unnamed clients; the rest is growing.
Learn More
Enterprise integration hub ready for AI agents: one monitored layer for commerce, ERP and CRM data An enterprise integration hub is a central layer that moves orders, customers, stock and invoices between your commerce, ERP and CRM systems, with monitoring, retries and replay in one place, and scoped APIs that AI agents can call safely. Netbase builds it as a custom layer on the stack you already run, starting from the flows that break most often.
Learn More
Work with Netbase remotely
Every Netbase engagement is delivered remotely, so clients in any country and any industry can work with us; Netbase has worked with clients in 80 countries. Contracts and invoices are issued by Netbase JSC, and payment is made by bank transfer to Netbase JSC's company account or PayPal Business. Talk to the CEO, Huy Nguyen (David), on WhatsApp or by email.
Discuss a project
Netbase JSC helps organizations design, build, modernize, and operate digital products and AI-enabled business systems.project@netbasejsc.com
+84 937 869 689
91 Nguyen Chi Thanh, Dong Da, Hanoi, Vietnam
Get in touch
Tell us what you want to build, modernize, or operate.