Skip to main content

What are you looking for?

Explore our services and discover how we can help you achieve your goals

Client Workspace

Near-real-time dashboards: architecture options and the freshness they cost

A near-real-time dashboard can be built three ways: warehouse micro-batch with a BI refresh, change data capture into an incremental serving store, or an event stream pushed to the screen. Each step down in latency adds running cost and on-call work, so Netbase picks the slowest architecture that still meets the agreed freshness.

Book a solution review See the related solution

By Netbase Commerce and Platform Engineering Desk · Reviewed by David (CEO) · Updated · 10 min read

star

This guide is for heads of data, analytics leads and CTOs at mid-market companies who have agreed that a dashboard must refresh within minutes and now have to choose how to build it. It sits under Netbase's data engineering and platforms service. If the freshness target itself is still under debate, start with the guide on deciding the latency a business metric really needs, and come back here once the number is agreed.

In this guide

What does "near-real-time" mean for a dashboard?

For a dashboard, near-real-time means the number on screen is a few seconds to about fifteen minutes behind the event it reports, measured from the moment the event happens to the moment the tile changes. It is not a property of one tool; it is the sum of every hop between the source system and the browser.

That end-to-end view matters because teams often measure only one hop. A stream processor can report a lag of two seconds while the dashboard still shows ten-minute-old data, because the BI tool caches its query results or the warehouse loads files on a schedule. Agree on one definition before choosing anything: event time to rendered tile.

The word also hides very different needs. A warehouse floor screen that changes a picking plan twice an hour is well served at ten minutes. A support queue wall board may need under a minute. Treat each as its own target.

Which hop sets your freshness floor?

The freshness budget splits the agreed target across the five hops a number travels, so a team can see which hop sets the floor and where money buys the most freshness. Fill it in for the current setup first, then for each candidate architecture.

  • Capture

    What adds delay
    Nightly exports, polling intervals, file drops
    Typical lever
    Change data capture or events from the application
  • Transport

    What adds delay
    Batch file loads, queue backlog
    Typical lever
    Continuous loading or a message stream
  • Transform

    What adds delay
    Full rebuilds of models, scheduled jobs
    Typical lever
    Incremental models or views updated on insert
  • Serve

    What adds delay
    Query time on large tables, BI result caches
    Typical lever
    Pre-aggregated tables, a store built for fast reads
  • Render

    What adds delay
    Browser polling interval, manual refresh
    Typical lever
    Shorter polling or pushed updates

The slowest hop sets the floor. Speeding up any other hop costs money without moving the number on screen, which is a common waste in "real-time" requests: a streaming transport feeding a dashboard that a BI tool refreshes on a schedule.

How do the three architectures compare?

Architecture Freshness and strength Running cost and burden Choose it when
A. Warehouse micro-batch plus BI refresh Roughly 5 to 15 minutes; reuses the existing warehouse, models and tests Warehouse compute on every load and refresh; low on-call load, jobs can be re-run The warehouse already holds the data and minutes are fine
B. Change data capture into an incremental serving store Roughly under a minute to a few minutes; fresh numbers from operational databases without touching the application A capture pipeline and a second store to run, monitor and reconcile with the warehouse Sources are databases and the dashboard is read often by many people
C. Event stream with processing and push to the screen Seconds; each event updates the tile as it arrives Brokers, stream processors and push connections run all day; late events and lag need an owner on call A named action must follow an event within seconds

Architecture A keeps one platform. The warehouse loads data every few minutes, models update incrementally, and the BI tool refreshes only recent partitions, as incremental refresh in Power BI does. Its cost driver is warehouse compute: every refresh is a query, and a busy dashboard with a short refresh interval can cost more than the load itself.

Architecture B moves the dashboard's reads off the warehouse. Changes leave the source databases through change data capture and land in a store that updates aggregates as rows arrive. In ClickHouse, for example, an incremental materialized view runs its query on each block of inserted data, so the read is cheap. The cost is a second system that must stay consistent with the warehouse.

Architecture C treats the dashboard as a live application. Events flow through a broker, a processor keeps the current state, and the browser receives updates instead of polling. It is the only option for second-level tiles, and it is the one that needs the most people to keep running.

When should you not build a near-real-time dashboard?

Do not build one when nobody acts on the number within the hour, when values are corrected later by refunds, returns or cancellations, or when the source only produces nightly files. In each case the faster dashboard shows a number that is ignored or wrong until it settles.

Do not move to architecture B or C when the team has no one to own a continuously running pipeline outside office hours. A late batch job can be re-run in the morning; a stream that falls behind shows stale numbers that look live. And do not make a whole dashboard fast because one tile needs it: mix tiers on one screen instead.

How to choose and roll out the architecture

  1. Write down the freshness target per tile

    Use event time to rendered tile, and give each tile a target and an owner; most dashboards have one or two fast tiles and many slow ones.

  2. Fill the freshness budget for today's setup

    Measure each hop and find the one that sets the floor before you buy or build anything.

  3. Fix the slowest hop with the cheapest lever

    Shorter load schedules, incremental models and BI refresh of recent partitions often reach the target inside architecture A.

  4. Add change data capture only where sources are databases

    Pick the capture method per table, as the guide on change data capture for operational databases explains, and land changes in a serving store only when warehouse reads become the floor or the bill.

  5. Reserve streaming for tiles with a named action

    Build architecture C for the few tiles where an action must follow an event within seconds, and keep every other tile on A or B.

  6. Reconcile fast numbers against the warehouse

    Compare each fast tile with its batch version daily, so a drift in the faster path is caught before anyone relies on it.

  7. Review cost and usage every quarter

    Demote tiles nobody opens quickly any more, and lengthen refresh intervals where the owner no longer acts within minutes.

On PostgreSQL, a simple serving layer for architecture A is a set of materialized views; refreshing them concurrently lets readers keep querying the old version while the new one is built, so a frequent refresh does not lock the dashboard.

Where AI fits

AI helps around the dashboard, not inside the freshness decision. An assistant can draft the freshness budget from pipeline logs and job schedules, summarise which tiles are opened and how often, and flag a tile whose measured freshness drifts from its target. Netbase works with the major commercial and open-source AI tools and models, chosen per project.

Engineers and tile owners still set every target, approve each move to a faster architecture and own the reconciliation between fast and batch numbers; no model changes a schedule or a pipeline on its own.

What delivery record exists, and what does not

Netbase has delivered AI projects for clients that are not named, including MLOps pipelines, and its named AI delivery is the 4over4 recommendation engine. Its ERP and e-commerce work includes API sync of product catalogues, inventory, orders and customer data with ERP systems, which is where requests for fresher stock and order numbers usually start.

Delivery runs in a six-step lifecycle from discovery and architecture planning to rollout and ongoing support, and most Netbase projects are delivered on fixed-price contracts. No published Netbase record describes a near-real-time dashboard platform built for a client, and no latency, cost or result figure is claimed here.

Alternatives: other ways to get fresher numbers on screen

  • Query a read replica directly. A dashboard that reads a replica of the operational database is current without any pipeline. Choose it for a few low-traffic screens on one source, not for queries that join many systems.
  • Live queries against the warehouse. A BI tool in DirectQuery mode sends each visual's query to the source, so freshness equals the load schedule. Choose it when users are few and query cost is watched.
  • Alerts inside the application. A threshold check in the order or payment system can act in seconds while the dashboard stays on batch. Choose it when one action needs speed and the reporting does not.
  • Polling plus a cache. Keeping browser polling but serving it from a cache or read model avoids a new pipeline. Choose it for bounded user counts and relaxed freshness, and plan for cache invalidation.

For customer-facing dashboards inside a SaaS product, the guide on embedded analytics: build, buy or hybrid covers the product side of the same choice.

Limits of this guide

  • The freshness bands are planning ranges, not benchmarks; measure your own hops before committing to a figure in a service agreement.
  • No price or cost figure is given; running cost depends on data volume, query patterns and the cloud contract.
  • Tools are named as examples of a pattern, not as recommendations; the right engine depends on your cloud, sources and team skills.

Plan the next step with a Netbase consultant

Frequently asked questions

Usually not. A warehouse that loads every few minutes, incremental models and a BI refresh of recent data can reach five minutes on the existing platform. Streaming earns its running cost only when a tile must change within seconds of an event.

Because another hop sets the floor. A BI result cache, a scheduled model rebuild or a browser that refreshes only on demand can hold the number back however fast the stream is. Measure event time to rendered tile and fix the slowest hop first.

Yes, and it is usually the cheapest design. Historical and trend tiles read daily or hourly tables, while the one or two tiles that drive a fast action read a faster store, each labelled with its last update time.

Mostly continuous compute and people. Faster tiers run all day, refresh queries hit the warehouse more often, and a pipeline that never stops needs someone to watch lag and late events outside office hours.

Next step

If you want a second opinion on which architecture your dashboard needs, Netbase can fill the freshness budget with your team and propose the slowest design that meets each tile's target: book a solution review.

Data engineering platforms for reporting and AI Data engineering platforms for reporting and AI

Netbase builds data pipelines and analytics foundations for companies whose data is spread across a store, an ERP, a CRM and spreadsheets. We collect it, clean it, model it and check its quality, so reports agree with each other and AI projects start from data people trust. Data engineering is a capability we are growing alongside the platforms we deliver.

Learn More
line
AI and data services, from readiness check to production feature AI and data services, from readiness check to production feature

Netbase provides AI and data services for companies that want to move from an AI idea to a feature that works inside a real product or process: readiness assessment, data preparation, model and automation development, and production support. Delivered AI includes 4over4's recommendation engine, plus RAG, document AI and MLOps projects for unnamed clients; the rest is growing.

Learn More
line
Enterprise integration hub ready for AI agents: one monitored layer for commerce, ERP and CRM data Enterprise integration hub ready for AI agents: one monitored layer for commerce, ERP and CRM data

An enterprise integration hub is a central layer that moves orders, customers, stock and invoices between your commerce, ERP and CRM systems, with monitoring, retries and replay in one place, and scoped APIs that AI agents can call safely. Netbase builds it as a custom layer on the stack you already run, starting from the flows that break most often.

Learn More
line

Work with Netbase remotely

Every Netbase engagement is delivered remotely, so clients in any country and any industry can work with us; Netbase has worked with clients in 80 countries. Contracts and invoices are issued by Netbase JSC, and payment is made by bank transfer to Netbase JSC's company account or PayPal Business. Talk to the CEO, Huy Nguyen (David), on WhatsApp or by email.

Contact Netbase

Discuss a project

Netbase JSC helps organizations design, build, modernize, and operate digital products and AI-enabled business systems.
Project enquiries

project@netbasejsc.com

WhatsApp

+84 937 869 689

Office address

91 Nguyen Chi Thanh, Dong Da, Hanoi, Vietnam

Get in touch

Tell us what you want to build, modernize, or operate.

Tell us what you want to build, modernize, or operate.

Contact Netbase