Skip to main content

What are you looking for?

Explore our services and discover how we can help you achieve your goals

Client Workspace

On-call and incident response for small engineering teams: runbooks and escalation

A small engineering team can run sustainable on-call with a rota that matches its size, a severity table that decides who is paged, one-page runbooks for every paging alert and a blameless review after each serious incident. Netbase treats these four pieces as one operating design, not as a paging tool setup.

Book a solution review See the related solution

By Netbase Commerce and Platform Engineering Desk · Reviewed by David (CEO) · Updated · 11 min read

star

This guide is for CTOs, heads of engineering and engineering leads of teams of roughly three to fifteen engineers who run their own production system without a dedicated SRE function. It sits beside Netbase's cloud platform engineering service and the DevOps and platform engineering work that sets up the alerts on-call answers.

In this guide

What does a small team actually need before it starts on-call?

A small team needs four things before the first night on call: a written answer to "what hours do we promise", alerts that point at a user-visible symptom, a named person with the authority to make a business call, and a place where the live incident is written down.

If no customer contract or business process depends on the system outside office hours, a business-hours rota with a best-effort phone tree is honest and cheap. If orders, payments or customer logins run at night, someone must be reachable, and the rota has to be built for that.

Google's SRE book sets a useful ceiling: no more than 25% of an engineer's time on call, and at most two incidents in a 12-hour shift, because each incident needs time for analysis, fixes and a review. A small team can copy the ceiling, if not the staffing: when incidents regularly exceed one shift's capacity, the fix is fewer or better alerts.

Which rota pattern fits a team of your size?

Rota pattern How it works Main cost Choose it when
Business hours plus best effort One engineer owns alerts in office hours; nights use a phone tree with no response promise Night incidents wait No contract or process needs a night response
Weekly primary with secondary One primary takes pages for a week; a secondary is called if the primary does not answer Each engineer is on call often in a small team Five or more engineers share the rota and nights matter
Follow-the-sun Two groups in different time zones each cover their own daytime Two groups must share runbooks and handovers The team already works across two regions with overlap hours
Shared rota with a partner An external team covers part of the hours under written runbooks Context lives in two organisations The system is stable, documented and the partner has run it

Google's guidance names eight engineers as the minimum for a single-site rota covering every hour, or six per site for two sites. Most small teams are below that, which is why the business-hours pattern is often the correct one rather than a compromise. Move to a weekly primary with a secondary when the business promise requires it, and lighten that person's project work for the week.

Follow-the-sun removes night shifts, which the SRE book links to health effects, but it only works when both groups can fix the same faults. Each shift ends with a handover note: open incidents, silenced alerts and production changes.

How should severity decide who is paged?

A severity table turns a stressful judgement into a lookup; PagerDuty's incident response guide publishes one example. Keep three levels, define each by user impact, and write who is paged, who decides and when the business owner hears about it.

  • SEV-1

    Definition
    Customers cannot complete a core journey, or data is lost or exposed
    Who is paged
    Primary, secondary and the engineering lead
    Who is told
    Business owner within the first half hour
  • SEV-2

    Definition
    A core journey is degraded or a workaround exists
    Who is paged
    Primary
    Who is told
    Business owner in the next daily update
  • SEV-3

    Definition
    Internal impact only, or a single customer
    Who is paged
    No page; a ticket for office hours
    Who is told
    Nobody outside the team

The escalation path follows the table. If the primary does not acknowledge a page, the secondary is paged automatically; if neither answers, the engineering lead is. A primary who has worked a problem without progress should escalate rather than continue alone. Google's incident guidance gives three triggers for declaring an incident: a second team is needed, the outage is visible to customers, or the issue is unsolved after an hour of concentrated analysis.

Name the incident roles even when one person holds two. Google separates the incident commander, who holds the overall state, from the operations lead who applies the fix and the person who communicates with stakeholders. In a team of five, the lead often commands while the primary fixes, so the person fixing is not interrupted.

What belongs in a runbook an engineer can follow at night?

Every alert that can page someone needs a runbook, and each runbook should fit on one page. Use the same five headings every time:

  1. Symptom. What the alert means in user terms, and which journey is affected.
  2. Confirm. The dashboard, query or log search that proves the alert is real.
  3. Mitigate. The safe first actions in order: roll back the last release, fail over, scale, restart, disable a feature flag.
  4. Escalate. Who to call next and the point at which to call them.
  5. Afterwards. What to record in the incident document before handing over.

Write runbooks from real incidents and link each from its alert. Have someone who did not write a runbook reread it regularly; if they cannot follow it, it is wrong. The alert signals themselves come from the telemetry choices described in observability stack design with OpenTelemetry.

Setting up on-call in seven steps

  1. Write the promise

    Agree with the business owner which hours and which journeys need a response, and write it down.

  2. Cut the alerts

    Keep only alerts tied to a user-visible symptom; move the rest to dashboards or tickets.

  3. Pick the rota pattern

    Choose from the table above by team size and promise, and publish the schedule where everyone can see it.

  4. Write the severity table

    Three levels, defined by impact, with who is paged and who is told.

  5. Write the first runbooks

    One per paging alert, in the five-heading format, linked from the alert.

  6. Rehearse once

    Run a planned failure in office hours and let the person on call use only the runbook.

  7. Review after every SEV-1 and SEV-2

    Use the template below and track every action to closure.

The post-incident review template

A blameless review looks for contributing causes in the system, the process and the information people had, not for someone to blame. Google's SRE book describes a postmortem as a written record of the incident, its impact, the actions taken, the root causes and the follow-up actions, and lists triggers such as user-visible downtime, any data loss, a rollback by the on-call engineer and a monitoring failure.

  • Summary

    What to write
    Two sentences: what users saw and for how long
    Who writes it
    Incident commander
  • Timeline

    What to write
    Times of detection, escalation, mitigation and resolution, from the incident document
    Who writes it
    Primary on call
  • Impact

    What to write
    Journeys, customers and data affected
    Who writes it
    Engineering lead with the business owner
  • Contributing factors

    What to write
    Technical, process and information gaps, without names
    Who writes it
    Everyone involved
  • What went well

    What to write
    Alerts, runbooks and decisions that helped
    Who writes it
    Everyone involved
  • Actions

    What to write
    Each action with one owner, a due date and a priority
    Who writes it
    Engineering lead

An action without an owner and a due date is not an action. Review open actions in planning until they close; the share of reviews with every action closed says more than the incident count. The incident row of the DevOps maturity assessment scorecard scores exactly this evidence.

When should a small team not run 24-hour on-call?

Do not run night on-call when no contract, customer journey or business process needs a night response; a published support window is better than an exhausted team. Do not start a rota while alerts fire for causes nobody can act on at night; people learn to ignore pages.

Do not put one engineer on a permanent rota because they know the system best. Write the runbooks and spread the rota instead. Give time off after a night with pages, so the next day starts rested.

Where AI fits in incident response

AI helps around the incident, not inside the decision. An assistant can summarise a long incident channel into a first timeline, draft a post-incident review from the incident document and suggest which existing runbook matches an alert. Netbase works with the major commercial and open-source AI tools and models, chosen per project.

Engineers still confirm every timeline entry, choose and run each mitigation, decide severity and escalation by name and own the review's actions; no model acknowledges a page, rolls back a release or closes an incident.

What delivery record exists, and what does not

Netbase built Cloodo, an AI-powered digital workplace that is live at cloodo.com, so its engineers work on a system that has real users. Netbase names AWS, Google Cloud, DigitalOcean and Cloudflare as the cloud providers it uses.

Delivery runs in a six-step lifecycle that ends in ongoing support, with weekly reviews and KPI dashboards. Most Netbase projects are delivered on fixed-price contracts. No published Netbase record describes an on-call rota or an incident history, and no response time, uptime or result figure is claimed here.

Alternatives: other ways to cover incidents

Alternative Fits when Selection criterion
Managed services from the cloud provider Most incidents come from infrastructure the team runs itself Moving a database or queue to a managed service removes a class of pages
A support partner for first response The system is stable and documented The partner can follow your runbooks and escalate to your engineers
A published support window No night promise exists Customers accept a response the next working day
Investing in release safety first Most incidents follow a release Rollback in one step and smaller releases cut incidents at the source

When incidents follow releases, start with the pipeline; a cloud migration cutover plan uses the same rollback-point thinking for one large change.

What are the limits of this guide?

  • It gives patterns and decision rules, not staffing numbers for a particular team or contract.
  • Labour law and employment contracts set rules for night work and on-call compensation; check them locally.
  • Paging tools differ in how they escalate; check your tool's documentation before relying on automatic escalation.

Plan the next step with a Netbase consultant

Frequently asked questions

Google's SRE guidance names eight engineers for a single-site rota covering every hour, or six per site for two sites. A smaller team can still cover nights, but each person is on call more often, so cut alerts first.

Yes, when the runbooks and alerts are in place first. Engineers paged for their own services fix causes faster, but a rota without runbooks burdens whoever knows the system best.

It is a written review that looks for the system, process and information gaps that let an incident happen, without blaming a person. It ends with a short list of actions, each with one owner and a due date, tracked until it closes.

Escalate when the primary and secondary do not acknowledge a page, when customers cannot complete a core journey, when data is lost or exposed, or when the primary has worked a problem for a while without progress. The severity table should say this in advance.

Next step

If your team is about to start on-call, Netbase can review alerts, runbooks and the escalation path with you as part of a cloud platform engineering engagement. Book a solution review.

AI-ready cloud platform engineering for SaaS and commerce workloads AI-ready cloud platform engineering for SaaS and commerce workloads

Netbase provides cloud platform engineering for SaaS and commerce teams that need a secure, repeatable home for their software and its AI features, on AWS, Google Cloud, DigitalOcean or Cloudflare. We design the architecture, write the environments as code, connect managed and AI services and hand over a platform your team can run, built the way we run ours.

Learn More
line
AI-assisted cloud migration with a cut-over plan and a way back AI-assisted cloud migration with a cut-over plan and a way back

Netbase provides cloud migration services for companies running business applications on ageing servers or hosting to move them to AWS, Google Cloud, DigitalOcean or Cloudflare without losing data or control. AI speeds up the code and dependency assessment; every migration still has an inventory, a rehearsed cut-over, cost controls from day one and a tested rollback point.

Learn More
line
AI-assisted DevOps and platform engineering for routine releases AI-assisted DevOps and platform engineering for routine releases

Netbase provides DevOps and platform engineering for SaaS and commerce teams to make releases routine instead of risky. We set up CI/CD pipelines, repeatable environments and observability on AWS, Google Cloud, DigitalOcean or Cloudflare, with AI assistance for tests, reviews and incident triage, so every change ships the same way and problems surface before customers notice.

Learn More
line
SaaS product accelerator: launch an AI-ready SaaS on proven Netbase modules SaaS product accelerator: launch an AI-ready SaaS on proven Netbase modules

A SaaS product accelerator is a set of reusable Netbase modules for accounts, billing, roles and integrations that helps founders and product teams launch subscription software faster, with room for in-product AI from the first release. Reusing these modules can shorten development time, and the approach is proven on Printcart, Netbase's own web-to-print SaaS.

Learn More
line

Work with Netbase remotely

Every Netbase engagement is delivered remotely, so clients in any country and any industry can work with us; Netbase has worked with clients in 80 countries. Contracts and invoices are issued by Netbase JSC, and payment is made by bank transfer to Netbase JSC's company account or PayPal Business. Talk to the CEO, Huy Nguyen (David), on WhatsApp or by email.

Contact Netbase

Discuss a project

Netbase JSC helps organizations design, build, modernize, and operate digital products and AI-enabled business systems.
Project enquiries

project@netbasejsc.com

WhatsApp

+84 937 869 689

Office address

91 Nguyen Chi Thanh, Dong Da, Hanoi, Vietnam

Get in touch

Tell us what you want to build, modernize, or operate.

Tell us what you want to build, modernize, or operate.

Contact Netbase