This guide is for the AI lead or engineering manager who has decided to fine-tune and must turn tickets, documents or past decisions into training examples. It sits under our generative AI development service; the data side of the work, from source audits to labelling, is part of our AI and data services.
In this guide
- Why does fine-tuning data need its own quality gate?
- Which five gates must the data pass before training?
- How do you check for train-test leakage?
- How many examples do you need?
- Who should label: experts, a labelling team or a model?
- How to prepare a fine-tuning set, step by step
- Where AI fits
- What delivery record exists, and what does not?
- Alternatives: when fine-tuning data work is the wrong investment
- What are the limits of this guide?
- Frequently asked questions
- Next step
Why does fine-tuning data need its own quality gate?
A fine-tuned model learns whatever pattern the examples contain, including the mistakes. If two reviewers label the same ticket type differently, the model learns the disagreement. If test examples also sit in the training set, the score looks better than the model is.
These are not rare problems. A study of ten widely used benchmark test sets by Northcutt, Athalye and Mueller found label errors in at least 3.3% of examples on average, and showed that correcting them can change which model ranks best. Lee et al. found a single 61-word sentence repeated more than 60,000 times in a large public training corpus, and train-test overlap in more than 4% of standard validation sets. Company data is usually messier, so the gate comes before the training budget.
Which five gates must the data pass before training?
Our rule: no training run starts until every gate below passes, and each gate has a named owner.
-
Source and permission
Where each record comes from, whether contracts and privacy rules allow training on it, and which personal fields must go
- Passes when
- Every source has a written basis for use and a removal rule for personal data
-
Curation
Exact and near-duplicates, templates, empty or broken records, off-task examples, outdated policy
- Passes when
- Duplicates are collapsed, off-task records are removed and the remaining mix matches real traffic
-
Labelling
A written labelling guide with definitions, edge cases and an "unsure" option; a second reviewer on a sample
- Passes when
- Two reviewers agree on the sampled examples at a level the team set in advance, and disputes have an owner
-
Split and leakage
Train, validation and test sets split by customer, document and date before any transformation
- Passes when
- The leakage checks in the next section find nothing, and the test set is frozen
-
Sizing
A pilot set trained and scored on the frozen test set
- Passes when
- The pilot shows a gain over the prompted base model, and the error analysis says which labels to grow
If ownership of the data, the test set or the trained weights is unclear between you and a vendor, settle it at gate 1; our guide to contract clauses for outsourced AI projects lists the three clauses to add.
How do you check for train-test leakage?
Leakage means information that will not be available when the model runs in production has reached training. The scikit-learn guide to common pitfalls puts the rule plainly: split the data into train and test sets first, before any preprocessing, and never fit anything on the test data. A review of leakage across 17 research fields traced errors in 329 papers to eight leakage types. For company records, five checks catch most of it:
-
Exact duplicates
- How to detect it
- Hash each normalised record and compare across splits
- What to do
- Keep one copy, in one split
-
Near-duplicates
- How to detect it
- Text similarity or embedding similarity above a threshold, then a human look at the clusters
- What to do
- Collapse each cluster and assign it to one split
-
Entity leakage
- How to detect it
- Group by customer, account or document ID before splitting
- What to do
- Split by group, not by row
-
Time leakage
- How to detect it
- Compare date ranges across splits
- What to do
- Train on the past, test on the most recent period
-
Selection leakage
- How to detect it
- Count how often the test set was scored during development
- What to do
- Tune on the validation set; score the frozen test set once per release
Also check inside each example: a status field or reviewer note can reveal the answer.
How many examples do you need?
There is no universal number, but two public reference points give a sense of scale. OpenAI's supervised fine-tuning guide accepts as few as 10 examples, recommends starting with 50 well-crafted demonstrations, and advises rethinking the task or prompt if 50 good examples make no difference. The LIMA paper tuned a large model on 1,000 carefully curated prompts and responses and argued that most knowledge comes from pretraining.
Our sizing guidance is staged, not fixed:
- Pilot. Label a small reviewed set that covers every label or output type, including the rare ones, and train once.
- Measure. Score the pilot against the prompted base model on the frozen test set, and sort the errors by label and input type.
- Grow where it errs. Add examples only to the labels and input types that fail, and stop when another round no longer moves the test score.
Who should label: experts, a labelling team or a model?
| Criterion | In-house experts | Managed labelling team | Model-drafted labels with human review |
|---|---|---|---|
| Best fit | Judgement tasks where policy knowledge matters | High-volume tasks with clear, stable definitions | Large backlogs where a strong model gets most cases right |
| Label consistency | High if the guide is written down | Depends on training and spot checks | High on easy cases, risky on edge cases |
| Data exposure | Stays inside the company | Needs access controls and an agreement | Depends on where the drafting model runs |
| Main risk | Bottleneck and reviewer fatigue | Shallow labels on ambiguous cases | Reviewers rubber-stamp the model's errors |
Most projects mix the routes: experts write the guide and label hard cases, while a team or a model handles volume under sampled review.
How to prepare a fine-tuning set, step by step
-
Write the task definition and labelling guide
Define every label, add edge cases and an "unsure" option, and name who settles disputes.
-
Inventory and clear the sources
List each source, the basis for training on it and the personal fields to mask.
-
Split before you touch the data
Group records by customer and document, order them by date, and assign train, validation and test sets.
-
Curate each split
Remove duplicates across splits, drop broken and off-task records, and rebalance to match real traffic.
-
Label and double-check
Label the test set with your best reviewers first; a second reviewer labels a sample, and disagreements are settled against the guide.
-
Run the leakage checks
Repeat the checks on the final files and freeze the test set with a version tag.
-
Train a pilot and analyse errors
Compare against the prompted base model, then grow only the weak labels.
-
Record the dataset version
Store sources, guide version, split rule and passed checks with the trained model.
Steps 6 to 8 feed straight into the release pipeline in our guide to MLOps and LLMOps for production AI systems, where each retraining reuses the same frozen test set.
Where AI fits
AI helps three ways here: drafting labels for reviewers to correct, grouping near-duplicates by meaning, and flagging likely label errors. Engineers and domain reviewers still write the labelling guide, approve every training label, set the split rule and sign off the leakage report. Netbase works with commercial and open-weight models from OpenAI, Anthropic, Google and Meta, among others, chosen per project. Each item below states how mature it is at Netbase.
- In delivered work
Document AI platform
Document AI delivered for a client that is not named, the kind of system where field labels and reviewed examples from confidence-based review routing drive quality.
- In delivered work
MLOps pipeline
An MLOps pipeline delivered for a client that is not named, where dataset versions and frozen test sets belong.
- Available capability
Fine-tuning data preparation
The five-gate checklist, leakage checks and staged sizing described on this page; not yet tied to a published case.
What delivery record exists, and what does not?
- What exists. Netbase has delivered retrieval-augmented knowledge assistants, document AI and MLOps pipelines for clients that are not named, and built a WhatsApp chatbot that combined a large language model with a separate intent engine, also for a client that is not named. The named delivered AI feature is 4over4's recommendation engine, which suggests products from browsing history and previous purchases. Most Netbase projects are delivered on fixed-price contracts.
- What does not. No published Netbase case reports a fine-tuning dataset size, a label agreement rate or an accuracy gain from fine-tuning, and none is offered here. The gates, checks and sizing stages are method guidance; the figures on this page come from the cited sources.
Alternatives: when fine-tuning data work is the wrong investment
- Better prompts and examples in the prompt. Choose them when the task still changes every few weeks or when a handful of examples in the prompt already meets the bar; a labelled set built for a moving task goes stale.
- Retrieval over your documents. Choose it when wrong answers come from missing or changing knowledge, not from the model's style or format. Our retrieval quality tuning guide covers how to measure and improve what the model is given.
- A smaller model trained for one narrow task. Choose it when the task has a fixed output and high volume; the same data gates apply, and our guide to small language models for narrow business tasks covers the size decision.
- Rules or classic machine learning. Choose them when the pattern is fixed, such as known codes or structured fields, and a simple classifier already passes the test set.
What are the limits of this guide?
- It covers supervised fine-tuning on text examples; preference data for reinforcement-style tuning needs extra checks on rater agreement.
- The external figures were measured on public datasets; your own records decide the real error and duplicate rates.
Plan the next step with a Netbase consultant
Frequently asked questions
Start with a small reviewed pilot that covers every label, train once and score it on a frozen test set. OpenAI's guide suggests 50 well-crafted examples as a starting point; grow the set only where the pilot makes errors.
Yes, as drafts. A model can propose labels quickly, but people must review every label that enters training, and the test set should be labelled by your best human reviewers so it can catch the model's errors.
Leakage is when test examples, or information the model will not have in production, reach the training data. It makes the test score look better than real performance, so split by customer and date first and check for duplicates across splits.
Next step
Send us a sample of the records you plan to train on and a description of the task, and we will book a solution review to run the five gates on it and plan a pilot set. You can also see the intelligent document processing solution, where reviewed field labels decide extraction quality.
Related services and solutions
Generative AI development with evaluation and human review Netbase builds generative AI features for product and operations teams: drafting, summarising, rewriting and content generation inside the software your people and customers already use. Every feature ships with an evaluation set, a human review step where output matters and limits on cost. Generative AI is a capability we are growing, and we label its maturity honestly.
Learn More
AI and data services, from readiness check to production feature Netbase provides AI and data services for companies that want to move from an AI idea to a feature that works inside a real product or process: readiness assessment, data preparation, model and automation development, and production support. Delivered AI includes 4over4's recommendation engine, plus RAG, document AI and MLOps projects for unnamed clients; the rest is growing.
Learn More
Responsible AI and MLOps for AI in production MLOps and responsible AI keep an AI feature trustworthy after launch. Netbase's practice for monitored, governed AI in production covers evaluation before every release, versioning of models, prompts and data, monitoring of quality and cost, and incident controls with a named owner. It is a growth capability, backed by an MLOps pipeline delivered for an unnamed client.
Learn More
AI knowledge assistant: answers from company knowledge, with sources and access control An AI knowledge assistant is an internal tool that answers employees' questions from your company's own documents and systems, cites the source behind every answer and shows each person only what they may see. Netbase offers it as a growth capability, designed with retrieval, evaluation and human review from the first pilot, and has delivered one for an unnamed client.
Learn More
Work with Netbase remotely
Every Netbase engagement is delivered remotely, so clients in any country and any industry can work with us; Netbase has worked with clients in 80 countries. Contracts and invoices are issued by Netbase JSC, and payment is made by bank transfer to Netbase JSC's company account or PayPal Business. Talk to the CEO, Huy Nguyen (David), on WhatsApp or by email.
Discuss a project
Netbase JSC helps organizations design, build, modernize, and operate digital products and AI-enabled business systems.project@netbasejsc.com
+84 937 869 689
91 Nguyen Chi Thanh, Dong Da, Hanoi, Vietnam
Get in touch
Tell us what you want to build, modernize, or operate.