This guide is for CTOs, heads of engineering and platform leads who run a web, SaaS or commerce product in production and want to see why it is slow or failing without paying to store every byte it emits. It sits beside Netbase's cloud platform engineering service, where observability is one of the platform layers Netbase sets up.
In this guide
- Why do observability stacks get expensive before they get useful?
- Which signal should you instrument first?
- How does OpenTelemetry fit into the stack?
- The telemetry budget worksheet
- Choosing the backend shape
- Rolling out the stack
- When should you not build a full observability stack?
- Where AI fits in observability
- What delivery record exists, and what does not
- Alternatives: other ways to get visibility
- What are the limits of this guide?
- Frequently asked questions
- Next step
Why do observability stacks get expensive before they get useful?
Observability stacks usually get expensive first because teams switch everything on at once. Automatic instrumentation emits spans for every call, debug logs ship from production, and a metric label such as a user ID turns one time series into millions. The bill grows with volume, while the questions the team can answer stay the same.
The three signals do different jobs. OpenTelemetry describes traces as the path of a request through an application, metrics as measurements captured at runtime, and logs as timestamped text or structured records. Metrics tell you that something is wrong, traces show where, and logs explain why. A stack that collects all three without a plan stores the same failure three times and still cannot connect them.
The fix is an order and a budget. Each step of the order answers one operational question, and each signal gets a written rule for how much of it is kept and for how long.
Which signal should you instrument first?
Start with request metrics at the edge of every service: rate, errors and duration per endpoint. Google's SRE book names latency, traffic, errors and saturation as the four golden signals, and they answer the first question on call: is the user affected right now? They are cheap because a metric aggregates many requests into one time series.
The order Netbase recommends, with the question each step must answer before the next one starts:
-
Request metrics and a service map
Are users affected, and which service is it?
- Move on when
- Every service exposes rate, errors and duration, and alerts fire on user impact
-
Structured logs with trace and span IDs
Why did this request fail?
- Move on when
- Logs are JSON, carry the trace ID and contain no secrets or personal data
-
Traces on the revenue path
Where in the chain is the time or the error?
- Move on when
- Checkout, sign-up or order flows are traced end to end across services
-
Business events
Did the failure cost orders, sign-ups or jobs?
- Move on when
- A small set of named events links technical faults to business impact
-
Profiling and deep traces
Which code path burns the CPU or memory?
- Move on when
- Only for services with a proven performance problem
Profiling comes last on purpose. OpenTelemetry lists profiles among the signals still under development, and most teams never need code-level resource recordings on every service.
How does OpenTelemetry fit into the stack?
OpenTelemetry is the instrumentation and transport layer, not the storage. Applications use its SDKs and semantic conventions to name attributes the same way everywhere, and the W3C Trace Context header carries the trace ID from service to service. The backend that stores and queries the data is a separate decision.
The Collector is the piece that makes cost control possible. The OpenTelemetry project calls it a vendor-agnostic way to receive, process and export telemetry data. Placing it between applications and the backend gives one place to drop health-check spans, redact fields, apply sampling and route each signal to the store that suits it, without a code change in every service. It also keeps the backend replaceable: switching vendors changes an exporter, not the instrumentation.
The telemetry budget worksheet
Write one row per signal before you connect a backend. The worksheet names what drives volume, the control that caps it and the person who approves an exception.
-
Metrics
- Volume driver
- Number of label combinations
- Default control
- An allow-list of labels; no user IDs, emails, order IDs or raw URLs as labels
- Exception needs approval from
- Platform lead
-
Logs
- Volume driver
- Log level and request rate
- Default control
- Info and above in production; debug only per service and with an expiry
- Exception needs approval from
- Service owner
-
Traces
- Volume driver
- Request rate and spans per request
- Default control
- Head sampling for normal traffic; keep every error and slow trace
- Exception needs approval from
- Platform lead
-
Business events
- Volume driver
- Event count
- Default control
- A named list; no free-form events
- Exception needs approval from
- Product owner
-
Retention
- Volume driver
- Days kept per tier
- Default control
- Short hot retention, longer cheap archive for audit logs only
- Exception needs approval from
- Security and finance
The Prometheus documentation warns against labels that store high-cardinality dimensions such as user IDs or email addresses. That one rule prevents the most common runaway metric bill. Request-specific detail belongs in a trace or a log, where it is sampled or expires.
Sampling has two forms. Head sampling decides at the start of a request whether to keep its trace; tail sampling decides after the request completes, so it can keep every error and slow trace. The OpenTelemetry documentation notes that tail sampling is harder to implement and operate, so start with head sampling and add tail rules in the Collector when the error rate makes it worth it.
Choosing the backend shape
| Option | Strength | Risk | Choose it when |
|---|---|---|---|
| Vendor SaaS behind a Collector | No storage to operate; fast start | Cost grows with volume; pricing models differ by signal | A small team, moderate volume, no one to run storage |
| Self-hosted open source | Lower marginal storage cost; full control | You operate its storage, upgrades and availability | A platform team exists and volume is high or data must stay in your cloud |
| Hybrid | Hot data in a vendor, bulk logs or archive in your own storage | Two systems to query and secure | Volume is uneven: few critical services, many noisy ones |
| Cloud-provider native tools | Already in the account; simple for one cloud | Harder to correlate across clouds or vendors | The product runs on one cloud and will stay there |
Whichever shape you pick, instrument with OpenTelemetry and route through the Collector. The backend then stays a decision you can revisit rather than a commitment.
Rolling out the stack
-
Pick the journey
List the user-facing journeys and pick the one that loses money when it breaks.
-
Measure it
Add request metrics and alerts on user impact to every service on that journey.
-
Correlate the logs
Switch logs to structured JSON with trace and span IDs; strip secrets and personal data.
-
Deploy the Collector
Add health-check filtering, redaction and head sampling in one place.
-
Trace end to end
Trace the chosen journey, then add the next one only when the first is useful on call.
-
Budget the volume
Fill in the telemetry budget worksheet and review volume per signal at each release review.
-
Extend on evidence
Add tail sampling, business events or profiling only when a real incident shows the gap.
When should you not build a full observability stack?
A full metrics, logs and traces stack is the wrong first move for a single monolith with a handful of endpoints, a product without real traffic yet, or a team with no on-call owner. Distributed tracing pays off when a request crosses several services. With one process, request metrics plus structured logs answer almost every question.
Do not adopt a new backend to fix a noisy one either. If the current tool is expensive because of debug logs and unbounded labels, the same data will be expensive in the next tool. Fix the worksheet first.
Where AI fits in observability
AI helps in triage. An AI assistant can group alerts that share a trace, summarise the logs of a failing request and draft a first hypothesis for the engineer on call, so the investigation starts from a prepared timeline. Netbase works with the major commercial and open-source models and chooses them per project.
Engineers still decide what counts as an incident, confirm every root cause in the traces themselves and own the alert rules. Telemetry sent to a model is redacted in the Collector first. For AI features that are themselves the workload, the AI agent observability and cost control guide covers token budgets and kill switches.
- In delivered work
MLOps pipeline
An MLOps pipeline Netbase delivered for a client that is not named; an AI delivery record, not an observability rollout.
- Available capability
AI agents in operations
AI-assisted alert triage on an OpenTelemetry stack is a way Netbase approaches the work, not yet tied to a published case.
What delivery record exists, and what does not
Netbase works as offshore development partner on a multi-tenant cloud ERP delivered as SaaS to small and mid-sized businesses for a US client, not named, and builds and runs its own products, such as Cloodo Workspace. Netbase names AWS, Google Cloud, DigitalOcean and Cloudflare as the cloud providers it uses, and lists disaster recovery and vulnerability scanning among its security practices.
Delivery runs in a six-step lifecycle with weekly reviews and KPI dashboards, and most Netbase projects are delivered on fixed-price contracts. No published Netbase record describes an OpenTelemetry rollout for a client, and no uptime, recovery time or telemetry cost saving is claimed.
Alternatives: other ways to get visibility
| Alternative | Fits when | Selection criterion |
|---|---|---|
| Uptime checks and logs only | One service, low traffic, no on-call rota | You only need to know that the site is down |
| A single vendor agent per host | Speed of setup matters more than portability | You accept vendor-specific instrumentation |
| Cloud-native monitoring | One cloud, managed services, small team | Cross-cloud correlation is not needed |
| Full OpenTelemetry stack | Several services, a revenue path, an owner | You need traces across services and a replaceable backend |
Teams that are also reviewing their release and reliability practices can score observability alongside the rest with the DevOps maturity assessment scorecard, and teams fighting the wider cloud bill can start with cloud cost controls for SaaS. When the work includes CI/CD and environments as well, Netbase's DevOps and platform engineering service covers the full setup, and cloud migration and modernization is the place to add observability during a move.
What are the limits of this guide?
- It is a design order, not a product comparison; no backend is ranked or priced.
- The worksheet controls volume; it does not replace a vendor's own pricing model, which changes over time.
- OpenTelemetry signal maturity differs by language; check the status of each SDK before you rely on it.
Plan the next step with a Netbase consultant
Frequently asked questions
No. Start with request metrics and alerts on user impact, then structured logs with trace IDs. Add traces when a request crosses several services and the team needs to see where time or errors appear in the chain.
No. OpenTelemetry standardises how telemetry is produced and transported; a backend still stores and queries it. Instrumenting with OpenTelemetry keeps that backend replaceable without changing application code.
Write a budget per signal: a label allow-list for metrics, a log-level policy for production, head sampling for traces that keeps errors, and short hot retention. Review volume per signal at every release review and approve exceptions by name.
Start with head sampling because it is simple and cheap. Add tail sampling in the Collector when you need to keep every error and slow trace, accepting that it is harder to run.
Next step
If you want a second opinion on your instrumentation order or telemetry budget, book a solution review with Netbase.
Related services and solutions
AI-ready cloud platform engineering for SaaS and commerce workloads Netbase provides cloud platform engineering for SaaS and commerce teams that need a secure, repeatable home for their software and its AI features, on AWS, Google Cloud, DigitalOcean or Cloudflare. We design the architecture, write the environments as code, connect managed and AI services and hand over a platform your team can run, built the way we run ours.
Learn More
AI-assisted DevOps and platform engineering for routine releases Netbase provides DevOps and platform engineering for SaaS and commerce teams to make releases routine instead of risky. We set up CI/CD pipelines, repeatable environments and observability on AWS, Google Cloud, DigitalOcean or Cloudflare, with AI assistance for tests, reviews and incident triage, so every change ships the same way and problems surface before customers notice.
Learn More
SaaS product accelerator: launch an AI-ready SaaS on proven Netbase modules A SaaS product accelerator is a set of reusable Netbase modules for accounts, billing, roles and integrations that helps founders and product teams launch subscription software faster, with room for in-product AI from the first release. Reusing these modules can shorten development time, and the approach is proven on Printcart, Netbase's own web-to-print SaaS.
Learn More
Work with Netbase remotely
Every Netbase engagement is delivered remotely, so clients in any country and any industry can work with us; Netbase has worked with clients in 80 countries. Contracts and invoices are issued by Netbase JSC, and payment is made by bank transfer to Netbase JSC's company account or PayPal Business. Talk to the CEO, Huy Nguyen (David), on WhatsApp or by email.
Discuss a project
Netbase JSC helps organizations design, build, modernize, and operate digital products and AI-enabled business systems.project@netbasejsc.com
+84 937 869 689
91 Nguyen Chi Thanh, Dong Da, Hanoi, Vietnam
Get in touch
Tell us what you want to build, modernize, or operate.