This guide is for a CTO, head of engineering or founder whose product already runs in production and who wants to know what to fix first. It sits under cloud platform engineering, and the work it usually leads to is DevOps and platform engineering.
In this guide
- Why do most DevOps maturity scores mislead?
- What do the four levels mean?
- The scorecard: delivery practices
- The scorecard: reliability practices
- The scorecard: security practices
- How do you turn fifteen scores into one level?
- Running the assessment
- Who should run the assessment?
- When should you not run a maturity assessment?
- Where AI fits in a maturity assessment
- What delivery record exists, and what does not?
- Alternatives: other ways to judge a delivery team
- What are the limits of this scorecard?
- Frequently asked questions
- Next step
Why do most DevOps maturity scores mislead?
Most maturity questionnaires ask a team to rate itself, and a team rates its intentions. "We do code review" scores the same whether every change is reviewed or only the risky ones. The fix is simple: a level counts only when the reviewer has seen the artefact for it, such as a pipeline run, a restore-test record or a scan report.
The second problem is averaging. A team with an excellent deployment pipeline and no tested backups averages out to "good", yet one bad incident shows that it is not. This scorecard therefore caps the overall level at the weakest area.
The third problem is scope. Many models score delivery speed and treat reliability and security as extras. The DORA metrics themselves pair throughput with instability measures, so speed alone is never the score. Here, delivery, reliability and security carry equal weight.
What do the four levels mean?
The levels describe how a practice is run, not how advanced the tools are.
- Level 1, ad hoc: the practice depends on a person; nothing is written down or repeatable.
- Level 2, repeatable: the practice is written down and followed, but run by hand.
- Level 3, automated: the practice runs in tooling on every change or on a schedule, and leaves a record.
- Level 4, measured: the record is reviewed on a fixed cadence, and the results change what the team does next.
A level with no artefact scores the level below. Level 4 is not a goal for every practice; a small product can stop at level 3 where the risk is low.
The scorecard: delivery practices
-
Version control and review
- Level 2 evidence
- Branch rules document
- Level 3 evidence
- Merge blocked without an approving review
- Level 4 evidence
- Review-time trend discussed in a retrospective
-
Build and test pipeline
- Level 2 evidence
- Written build steps
- Level 3 evidence
- Pipeline run log for each change
- Level 4 evidence
- Failed-build trend with actions taken
-
Deployment
- Level 2 evidence
- Release checklist
- Level 3 evidence
- Scripted deployment from the pipeline
- Level 4 evidence
- Deployment frequency and lead time reviewed monthly
-
Environments
- Level 2 evidence
- List of environments and owners
- Level 3 evidence
- Environments created from code
- Level 4 evidence
- Drift report checked on a schedule
-
Rollback
- Level 2 evidence
- Written rollback steps
- Level 3 evidence
- Rollback run in the pipeline in a test
- Level 4 evidence
- Rollback time recorded after real releases
The scorecard: reliability practices
-
Monitoring and alerting
- Level 2 evidence
- List of alerts and who receives them
- Level 3 evidence
- Alerts defined in code, linked to runbooks
- Level 4 evidence
- Alert noise review with alerts removed
-
Service level objectives
- Level 2 evidence
- Written target for the main user journey
- Level 3 evidence
- Indicator measured on a dashboard
- Level 4 evidence
- Error budget used in release planning
-
Backup and restore
- Level 2 evidence
- Backup schedule document
- Level 3 evidence
- Restore test log with date and result
- Level 4 evidence
- Restore tests repeated and timed on a cadence
-
Incident response
- Level 2 evidence
- Contact list and escalation steps
- Level 3 evidence
- Incident tickets with timeline
- Level 4 evidence
- Post-incident reviews with closed actions
-
Capacity and cost
- Level 2 evidence
- Known limits written down
- Level 3 evidence
- Usage and cost dashboards per service
- Level 4 evidence
- Capacity review feeding the roadmap
The service level objective row follows the Google SRE book definition: an indicator is a measure of the service, and an objective is a target for that measure. The incident row follows its postmortem guidance: written reviews with tracked actions. For the monitoring row, see what to instrument first.
The scorecard: security practices
-
Secrets
- Level 2 evidence
- Secrets kept out of the repository by rule
- Level 3 evidence
- Secrets in a vault, injected at runtime
- Level 4 evidence
- Rotation log and access review
-
Dependencies
- Level 2 evidence
- Dependency list per service
- Level 3 evidence
- Dependency scan report on each build
- Level 4 evidence
- Open findings tracked to closure dates
-
Code security
- Level 2 evidence
- Secure review checklist
- Level 3 evidence
- Static analysis results in the pipeline
- Level 4 evidence
- Finding trend reviewed with the team
-
Access control
- Level 2 evidence
- Role list for production
- Level 3 evidence
- Role-based access with MFA enforced
- Level 4 evidence
- Periodic access review record
-
Build integrity
- Level 2 evidence
- Build steps documented
- Level 3 evidence
- Build runs on a hosted platform with provenance
- Level 4 evidence
- Provenance verified before deployment
The build-integrity row borrows from the SLSA build levels, where a hosted platform signs the provenance and harder levels add isolation. The security rows map to the four practice groups of the NIST Secure Software Development Framework.
How do you turn fifteen scores into one level?
- Score each practice at the highest level whose evidence the reviewer has opened.
- Take the lowest practice score inside each area as that area's floor.
- Average the five practice scores per area, then cap the area at its floor plus one.
- Take the lowest of the three area scores as the overall level.
- List the three practices whose next level removes the cap, with an owner for each.
The cap is the point of the method. A level 4 pipeline feeding level 2 security mostly ships vulnerabilities faster, so the overall score says level 2 and the next actions are security work, not more pipeline work.
Running the assessment
-
Agree the scope
One product or one platform, its production environment and the team that runs it.
-
Collect artefacts before the session
Ask for links to pipeline runs, dashboards, restore logs and scan reports.
-
Score together in one session
The reviewer opens each artefact; anything not opened scores the level below.
-
Record the score as data
Keep the fifteen scores and their artefact links in a file that can be compared later.
-
Pick three upgrades
Choose the practices that lift the capped area, each with an owner and a review date.
-
Re-score after the work ships
Re-run the same scorecard, not a new one, so the trend means something.
Who should run the assessment?
| Option | Strength | Risk | Choose it when |
|---|---|---|---|
| Self-assessment by the team | Fast, cheap, builds ownership | Scores intentions unless a peer checks the evidence | The team is stable and wants a baseline |
| Outside assessment | Independent reading of the evidence | Assessor may not know the product's constraints | A board, buyer or investor needs a neutral view |
| Score from a vendor tool | Quick and repeatable from connected systems | Measures what the tool can see, misses manual practices | Most practices already run in connected tooling |
A mixed route works well: the team self-scores with artefact links, then an outside reviewer checks the links for the capped area only.
When should you not run a maturity assessment?
Skip it when the product has not found its market yet; the right practices then are a pipeline, backups and secrets kept out of code, and a score adds little. Skip it when a migration or rebuild is already decided; assess the target design instead, through cloud migration and modernization. And skip it when one person runs everything, because every practice will score level 1 for the same reason.
Where AI fits in a maturity assessment
AI helps in the evidence step. An AI assistant can read pipeline configurations, alert rules and incident tickets and draft a first score with the artefact links it found, so the session starts from a prepared sheet. Netbase works with the major commercial and open-source models and chooses them per project.
Engineers still open every artefact, decide each score and own the upgrade plan. An AI draft that cannot point to an artefact scores nothing.
- In delivered work
Product recommendation engine
Built for 4over4's online printing store from browsing and purchase history; a delivered AI feature named with a client, not a DevOps assessment.
- Available capability
AI agents in delivery
AI-assisted review of pipeline and infrastructure configuration is a capability Netbase offers, not yet tied to a published assessment case.
What delivery record exists, and what does not?
Netbase works as offshore development partner on a multi-tenant cloud ERP delivered as SaaS on AWS for a US client, not named, and builds and runs its own SaaS products, such as Cloodo Workspace. Netbase names AWS, Google Cloud, DigitalOcean and Cloudflare as the providers it uses, and lists security practices such as secure code review, role-based access control, MFA for admin dashboards, vulnerability scanning and disaster recovery.
Delivery runs in a six-step lifecycle with weekly reviews and KPI dashboards. Most Netbase projects are delivered on fixed-price contracts. No published Netbase record describes a maturity assessment for a client, and no uptime, recovery time or cost result is claimed.
Alternatives: other ways to judge a delivery team
A metrics-first view tracks a few delivery numbers over time; the guide on measuring engineering delivery health explains which to pick. It shows trends but not which practice causes them. A release-gate view checks each release against acceptance criteria; see software quality governance. It protects releases but does not score reliability. When the assessment shows a gap the team cannot close alone, the comparison of cloud consulting and DevOps companies lists criteria for choosing help.
What are the limits of this scorecard?
- The fifteen practices fit a web or SaaS product on a public cloud; regulated or embedded systems need more rows.
- The levels are ordinal; an average of 2.6 is a reading aid, not a measurement.
- Evidence proves a practice exists, not that it works well; pair the score with incident history.
Plan the next step with a Netbase consultant
Frequently asked questions
It depends on how ready the evidence is. When the team collects artefact links before the session, scoring fifteen practices fits one working session; most of the effort is in finding the evidence, not in the scoring.
No. The DORA metrics measure delivery outcomes such as lead time and change failure rate. A maturity score rates the practices that produce those outcomes, and it adds reliability and security practices the metrics do not show.
Most product teams are well served at level 3 in every area, with level 4 only where an incident or a customer requirement justifies the extra review effort.
Yes. Ask the vendor for the same artefact links and score them the same way; a vendor that cannot show a pipeline run or a restore log scores the level below.
Next step
If you want a second reader for your scorecard, book a solution review and bring the artefact links.
Related services and solutions
AI-ready cloud platform engineering for SaaS and commerce workloads Netbase provides cloud platform engineering for SaaS and commerce teams that need a secure, repeatable home for their software and its AI features, on AWS, Google Cloud, DigitalOcean or Cloudflare. We design the architecture, write the environments as code, connect managed and AI services and hand over a platform your team can run, built the way we run ours.
Learn More
AI-assisted cloud migration with a cut-over plan and a way back Netbase provides cloud migration services for companies running business applications on ageing servers or hosting to move them to AWS, Google Cloud, DigitalOcean or Cloudflare without losing data or control. AI speeds up the code and dependency assessment; every migration still has an inventory, a rehearsed cut-over, cost controls from day one and a tested rollback point.
Learn More
AI-assisted DevOps and platform engineering for routine releases Netbase provides DevOps and platform engineering for SaaS and commerce teams to make releases routine instead of risky. We set up CI/CD pipelines, repeatable environments and observability on AWS, Google Cloud, DigitalOcean or Cloudflare, with AI assistance for tests, reviews and incident triage, so every change ships the same way and problems surface before customers notice.
Learn More
SaaS product accelerator: launch an AI-ready SaaS on proven Netbase modules A SaaS product accelerator is a set of reusable Netbase modules for accounts, billing, roles and integrations that helps founders and product teams launch subscription software faster, with room for in-product AI from the first release. Reusing these modules can shorten development time, and the approach is proven on Printcart, Netbase's own web-to-print SaaS.
Learn More
Work with Netbase remotely
Every Netbase engagement is delivered remotely, so clients in any country and any industry can work with us; Netbase has worked with clients in 80 countries. Contracts and invoices are issued by Netbase JSC, and payment is made by bank transfer to Netbase JSC's company account or PayPal Business. Talk to the CEO, Huy Nguyen (David), on WhatsApp or by email.
Discuss a project
Netbase JSC helps organizations design, build, modernize, and operate digital products and AI-enabled business systems.project@netbasejsc.com
+84 937 869 689
91 Nguyen Chi Thanh, Dong Da, Hanoi, Vietnam
Get in touch
Tell us what you want to build, modernize, or operate.