Skip to main content

What are you looking for?

Explore our services and discover how we can help you achieve your goals

Client Workspace

How to prepare fine-tuning data: curation, labelling and leakage checks

Prepare fine-tuning data by passing it through five gates before any training run: confirm the right to use each source, curate out duplicates and noise, label against a written guide, split by customer and date so nothing leaks, and size the set from a small pilot. Netbase works through this checklist with a client before training is scheduled.

Book a solution review See the intelligent document processing solution

By Netbase AI and Automation Engineering Desk · Reviewed by David (CEO) · Updated · 10 min read

star

This guide is for the AI lead or engineering manager who has decided to fine-tune and must turn tickets, documents or past decisions into training examples. It sits under our generative AI development service; the data side of the work, from source audits to labelling, is part of our AI and data services.

In this guide

Why does fine-tuning data need its own quality gate?

A fine-tuned model learns whatever pattern the examples contain, including the mistakes. If two reviewers label the same ticket type differently, the model learns the disagreement. If test examples also sit in the training set, the score looks better than the model is.

These are not rare problems. A study of ten widely used benchmark test sets by Northcutt, Athalye and Mueller found label errors in at least 3.3% of examples on average, and showed that correcting them can change which model ranks best. Lee et al. found a single 61-word sentence repeated more than 60,000 times in a large public training corpus, and train-test overlap in more than 4% of standard validation sets. Company data is usually messier, so the gate comes before the training budget.

Which five gates must the data pass before training?

Our rule: no training run starts until every gate below passes, and each gate has a named owner.

  1. Source and permission

    Where each record comes from, whether contracts and privacy rules allow training on it, and which personal fields must go

    Passes when
    Every source has a written basis for use and a removal rule for personal data
  2. Curation

    Exact and near-duplicates, templates, empty or broken records, off-task examples, outdated policy

    Passes when
    Duplicates are collapsed, off-task records are removed and the remaining mix matches real traffic
  3. Labelling

    A written labelling guide with definitions, edge cases and an "unsure" option; a second reviewer on a sample

    Passes when
    Two reviewers agree on the sampled examples at a level the team set in advance, and disputes have an owner
  4. Split and leakage

    Train, validation and test sets split by customer, document and date before any transformation

    Passes when
    The leakage checks in the next section find nothing, and the test set is frozen
  5. Sizing

    A pilot set trained and scored on the frozen test set

    Passes when
    The pilot shows a gain over the prompted base model, and the error analysis says which labels to grow

If ownership of the data, the test set or the trained weights is unclear between you and a vendor, settle it at gate 1; our guide to contract clauses for outsourced AI projects lists the three clauses to add.

How do you check for train-test leakage?

Leakage means information that will not be available when the model runs in production has reached training. The scikit-learn guide to common pitfalls puts the rule plainly: split the data into train and test sets first, before any preprocessing, and never fit anything on the test data. A review of leakage across 17 research fields traced errors in 329 papers to eight leakage types. For company records, five checks catch most of it:

  • Exact duplicates

    How to detect it
    Hash each normalised record and compare across splits
    What to do
    Keep one copy, in one split
  • Near-duplicates

    How to detect it
    Text similarity or embedding similarity above a threshold, then a human look at the clusters
    What to do
    Collapse each cluster and assign it to one split
  • Entity leakage

    How to detect it
    Group by customer, account or document ID before splitting
    What to do
    Split by group, not by row
  • Time leakage

    How to detect it
    Compare date ranges across splits
    What to do
    Train on the past, test on the most recent period
  • Selection leakage

    How to detect it
    Count how often the test set was scored during development
    What to do
    Tune on the validation set; score the frozen test set once per release

Also check inside each example: a status field or reviewer note can reveal the answer.

How many examples do you need?

There is no universal number, but two public reference points give a sense of scale. OpenAI's supervised fine-tuning guide accepts as few as 10 examples, recommends starting with 50 well-crafted demonstrations, and advises rethinking the task or prompt if 50 good examples make no difference. The LIMA paper tuned a large model on 1,000 carefully curated prompts and responses and argued that most knowledge comes from pretraining.

Our sizing guidance is staged, not fixed:

  • Pilot. Label a small reviewed set that covers every label or output type, including the rare ones, and train once.
  • Measure. Score the pilot against the prompted base model on the frozen test set, and sort the errors by label and input type.
  • Grow where it errs. Add examples only to the labels and input types that fail, and stop when another round no longer moves the test score.

Who should label: experts, a labelling team or a model?

Criterion In-house experts Managed labelling team Model-drafted labels with human review
Best fit Judgement tasks where policy knowledge matters High-volume tasks with clear, stable definitions Large backlogs where a strong model gets most cases right
Label consistency High if the guide is written down Depends on training and spot checks High on easy cases, risky on edge cases
Data exposure Stays inside the company Needs access controls and an agreement Depends on where the drafting model runs
Main risk Bottleneck and reviewer fatigue Shallow labels on ambiguous cases Reviewers rubber-stamp the model's errors

Most projects mix the routes: experts write the guide and label hard cases, while a team or a model handles volume under sampled review.

How to prepare a fine-tuning set, step by step

  1. Write the task definition and labelling guide

    Define every label, add edge cases and an "unsure" option, and name who settles disputes.

  2. Inventory and clear the sources

    List each source, the basis for training on it and the personal fields to mask.

  3. Split before you touch the data

    Group records by customer and document, order them by date, and assign train, validation and test sets.

  4. Curate each split

    Remove duplicates across splits, drop broken and off-task records, and rebalance to match real traffic.

  5. Label and double-check

    Label the test set with your best reviewers first; a second reviewer labels a sample, and disagreements are settled against the guide.

  6. Run the leakage checks

    Repeat the checks on the final files and freeze the test set with a version tag.

  7. Train a pilot and analyse errors

    Compare against the prompted base model, then grow only the weak labels.

  8. Record the dataset version

    Store sources, guide version, split rule and passed checks with the trained model.

Steps 6 to 8 feed straight into the release pipeline in our guide to MLOps and LLMOps for production AI systems, where each retraining reuses the same frozen test set.

Where AI fits

AI helps three ways here: drafting labels for reviewers to correct, grouping near-duplicates by meaning, and flagging likely label errors. Engineers and domain reviewers still write the labelling guide, approve every training label, set the split rule and sign off the leakage report. Netbase works with commercial and open-weight models from OpenAI, Anthropic, Google and Meta, among others, chosen per project. Each item below states how mature it is at Netbase.

What delivery record exists, and what does not?

  • What exists. Netbase has delivered retrieval-augmented knowledge assistants, document AI and MLOps pipelines for clients that are not named, and built a WhatsApp chatbot that combined a large language model with a separate intent engine, also for a client that is not named. The named delivered AI feature is 4over4's recommendation engine, which suggests products from browsing history and previous purchases. Most Netbase projects are delivered on fixed-price contracts.
  • What does not. No published Netbase case reports a fine-tuning dataset size, a label agreement rate or an accuracy gain from fine-tuning, and none is offered here. The gates, checks and sizing stages are method guidance; the figures on this page come from the cited sources.

Alternatives: when fine-tuning data work is the wrong investment

  • Better prompts and examples in the prompt. Choose them when the task still changes every few weeks or when a handful of examples in the prompt already meets the bar; a labelled set built for a moving task goes stale.
  • Retrieval over your documents. Choose it when wrong answers come from missing or changing knowledge, not from the model's style or format. Our retrieval quality tuning guide covers how to measure and improve what the model is given.
  • A smaller model trained for one narrow task. Choose it when the task has a fixed output and high volume; the same data gates apply, and our guide to small language models for narrow business tasks covers the size decision.
  • Rules or classic machine learning. Choose them when the pattern is fixed, such as known codes or structured fields, and a simple classifier already passes the test set.

What are the limits of this guide?

  • It covers supervised fine-tuning on text examples; preference data for reinforcement-style tuning needs extra checks on rater agreement.
  • The external figures were measured on public datasets; your own records decide the real error and duplicate rates.

Frequently asked questions

Start with a small reviewed pilot that covers every label, train once and score it on a frozen test set. OpenAI's guide suggests 50 well-crafted examples as a starting point; grow the set only where the pilot makes errors.

Yes, as drafts. A model can propose labels quickly, but people must review every label that enters training, and the test set should be labelled by your best human reviewers so it can catch the model's errors.

Leakage is when test examples, or information the model will not have in production, reach the training data. It makes the test score look better than real performance, so split by customer and date first and check for duplicates across splits.

Next step

Send us a sample of the records you plan to train on and a description of the task, and we will book a solution review to run the five gates on it and plan a pilot set. You can also see the intelligent document processing solution, where reviewed field labels decide extraction quality.

Generative AI development with evaluation and human review Generative AI development with evaluation and human review

Netbase builds generative AI features for product and operations teams: drafting, summarising, rewriting and content generation inside the software your people and customers already use. Every feature ships with an evaluation set, a human review step where output matters and limits on cost. Generative AI is a capability we are growing, and we label its maturity honestly.

Learn More
line
AI and data services, from readiness check to production feature AI and data services, from readiness check to production feature

Netbase provides AI and data services for companies that want to move from an AI idea to a feature that works inside a real product or process: readiness assessment, data preparation, model and automation development, and production support. Delivered AI includes 4over4's recommendation engine, plus RAG, document AI and MLOps projects for unnamed clients; the rest is growing.

Learn More
line
Responsible AI and MLOps for AI in production Responsible AI and MLOps for AI in production

MLOps and responsible AI keep an AI feature trustworthy after launch. Netbase's practice for monitored, governed AI in production covers evaluation before every release, versioning of models, prompts and data, monitoring of quality and cost, and incident controls with a named owner. It is a growth capability, backed by an MLOps pipeline delivered for an unnamed client.

Learn More
line
AI knowledge assistant: answers from company knowledge, with sources and access control AI knowledge assistant: answers from company knowledge, with sources and access control

An AI knowledge assistant is an internal tool that answers employees' questions from your company's own documents and systems, cites the source behind every answer and shows each person only what they may see. Netbase offers it as a growth capability, designed with retrieval, evaluation and human review from the first pilot, and has delivered one for an unnamed client.

Learn More
line

Work with Netbase remotely

Every Netbase engagement is delivered remotely, so clients in any country and any industry can work with us; Netbase has worked with clients in 80 countries. Contracts and invoices are issued by Netbase JSC, and payment is made by bank transfer to Netbase JSC's company account or PayPal Business. Talk to the CEO, Huy Nguyen (David), on WhatsApp or by email.

Contact Netbase

Discuss a project

Netbase JSC helps organizations design, build, modernize, and operate digital products and AI-enabled business systems.
Project enquiries

project@netbasejsc.com

WhatsApp

+84 937 869 689

Office address

91 Nguyen Chi Thanh, Dong Da, Hanoi, Vietnam

Get in touch

Tell us what you want to build, modernize, or operate.

Tell us what you want to build, modernize, or operate.

Contact Netbase