This guide is for the CTO, AI lead or product owner whose large-model pilot works but is too slow, too costly at volume or too hard to keep inside the data rules. It sits under our generative AI development service; for text classification, extraction and search built into business systems, see NLP development.
In this guide
- What counts as a small language model?
- Which task profile favours a small model?
- Small model, large model or a hybrid router?
- Which business tasks fit, and where does a small model fail?
- How do you prove a small model is good enough?
- Where AI fits
- What delivery record exists, and what does not?
- Alternatives: when a small model is the wrong choice
- What are the limits of this guide?
- Frequently asked questions
- Next step
What counts as a small language model?
There is no official size line. In practice, a small language model is one that a team can fine-tune, host on a single modest server or run on a device, often an open-weight model, a distilled version of a larger one, or an encoder model trained only to classify or extract. A large model is a general model reached through a provider's API.
Two research results show why the size question is worth asking. The DistilBERT paper reports a model 40% smaller than BERT that keeps 97% of its language understanding and runs 60% faster. The Phi-3 technical report describes a 3.8-billion-parameter model small enough to run on a phone, and states plainly that it cannot store much factual knowledge, which shows on trivia questions.
That is the trade-off: a small model gives up breadth and open-ended reasoning for speed, control and lower running cost on the task it was shaped for.
Which task profile favours a small model?
Score the task, not the model. Each signal below that is true for your task is one point toward a small model.
-
Narrow task
- What to check
- Can you write the task definition on one page?
- Points to a small model when
- The definition is stable and the same for every input
-
Fixed output
- What to check
- Is the answer a label, a set of fields or a short summary?
- Points to a small model when
- Outputs can be checked against a schema or a label list
-
High volume
- What to check
- How many calls per day, now and in a year?
- Points to a small model when
- Calls are steady and numerous enough that cost per call matters
-
Tight latency
- What to check
- What response time does the user or the next system need?
- Points to a small model when
- The task sits inside a user flow or a real-time pipeline
-
Data location
- What to check
- Must the text stay in your cloud, on premises or on a device?
- Points to a small model when
- Rules or contracts keep the data out of third-party APIs
-
Labelled examples
- What to check
- Do you have, or can you produce, reviewed examples?
- Points to a small model when
- A reviewed example set exists or can be labelled from past work
A seventh question works as a veto: does the task need broad world knowledge or multi-step reasoning across varied inputs? If yes, a small model alone will fail, whatever the score.
Our scoring rule: four or more points and no veto, run a bake-off between a small and a large model. Two or three points, design a hybrid router where a small model takes the routine share and hands the rest to a large one. Zero or one point, stay with a large model and tune prompts and retrieval instead.
Small model, large model or a hybrid router?
| Criterion | Small model, fine-tuned | Large general model | Hybrid router |
|---|---|---|---|
| Best task shape | One narrow task with a fixed output | Varied, open-ended or knowledge-heavy requests | Mixed traffic where most requests are routine |
| Start-up effort | Labelling, training and evaluation before launch | Prompt and evaluation work only | Both, plus a routing rule and a threshold |
| Running cost at volume | Lowest per call once hosted | Grows with every token | Small-model cost for the routine share |
| Latency | Fast, can run near the data or on a device | Depends on provider load and output length | Fast for routed requests, slower for escalations |
| Data control | Can stay entirely in your environment | Depends on the provider's terms and region | Escalated requests leave your environment |
| Keeping it current | Retrain when the task or the data drifts | Provider updates the model, which can shift behaviour | Monitor both and the routing share |
| Main risk | Confident errors outside the training range | Cost and latency at scale | A wrong threshold sends hard cases to the small model |
Research supports the small-model column for narrow tasks. The LoRA Land report fine-tuned 310 models across task types and found the adapted small models beat their base models by 34 points and GPT-4 by 10 points on average across its tasks. An NVIDIA research position paper argues that agent systems, which repeat a few specialised calls many times, should use small models by default and call a large model only when general conversation is needed.
Which business tasks fit, and where does a small model fail?
| Task type | Small-model fit | Where it fails |
|---|---|---|
| Ticket, email or document routing to a fixed set of queues | Strong | New queue names need relabelling and retraining |
| Field extraction from invoices, orders or forms | Strong when formats repeat | Rare layouts and handwritten fields |
| Content moderation and policy flags | Strong with a reviewed label set | New abuse patterns until relabelled |
| Short summaries of one record or one call | Good | Summaries that must weigh many documents |
| On-device or offline assistance | Often the only option | Anything that needs fresh or broad knowledge |
| Open-ended drafting and multi-step reasoning | Weak | Use a large model |
| Questions over broad or changing knowledge | Weak alone | Pair any model with retrieval |
For knowledge questions, model size is rarely the fix: grounding answers in your documents matters more. Our RAG architecture guide covers grounding, and retrieval quality tuning covers how to measure and improve what the model is given.
How do you prove a small model is good enough?
-
Freeze the task definition
Write the label list or field schema, the edge cases and who decides a disputed example.
-
Build a labelled test set from real traffic
Draw it from recent production or pilot inputs, include the rare cases on purpose, and keep it apart from any training data.
-
Run the large model first
Its score on the test set is the bar the small model has to meet, and its outputs can help draft labels for training, with people reviewing every one.
-
Train or adapt the small model
Start from an open-weight or distilled model, fine-tune on reviewed examples, and keep the training run reproducible. Parameter-efficient fine-tuning trains only a small number of extra parameters, which cuts compute and storage.
-
Compare on more than accuracy
Record the worst error class, latency at the slow end of the range, and cost per thousand tasks from your own volumes and hosting quote.
-
Set an escalation threshold
If the small model's confidence is below the line, send the request to the large model or to a person; tune the line on the test set, not on intuition.
-
Ship with monitoring and a rollback
Track the escalation share and sampled errors, and keep the large-model path ready.
Steps 6 and 7 belong in the release pipeline described in our guide to MLOps and LLMOps for production AI systems.
Where AI fits
AI shows up twice in this job: as the small and large models being compared, and as a helper that drafts labels and grades outputs during the bake-off. Engineers still own the task definition, review every label that goes into training, choose the escalation threshold and sign off the comparison. Netbase works with commercial and open-weight models from OpenAI, Anthropic, Google and Meta, among others, chosen per project. Each item below states how mature it is at Netbase.
- In delivered work
WhatsApp AI chatbot with CRM integration
A chatbot that pairs a large language model with a dedicated intent engine for conversation flows, synchronised with the client's CRM; the client is not named.
- In delivered work
AI content moderation on a classifieds platform
AI-powered filtering that flags offensive content into an admin moderation dashboard, a narrow fixed-output task; the client is not named.
- Available capability
Small-model fine-tuning and bake-offs
Task scoring, labelled test sets and side-by-side model comparison as described on this page; not yet tied to a published case.
What delivery record exists, and what does not?
- What exists. Netbase has delivered retrieval-augmented knowledge assistants, document AI and MLOps pipelines for clients that are not named. It built a WhatsApp chatbot that combined a large language model with a separate intent engine, and AI content filtering for a classifieds platform. The named delivered AI feature is 4over4's recommendation engine, which suggests products from browsing history and previous purchases. Most Netbase projects are delivered on fixed-price contracts.
- What does not. No published Netbase case compares a small and a large model on the same task, and no latency, cost or accuracy figure from client work is offered here. The scoring rule and bake-off are method guidance; the figures on this page come from the cited research.
Alternatives: when a small model is the wrong choice
- Better prompting of a large model. Choose it when volume is low or the task definition still changes every few weeks; a bake-off is wasted on a moving target.
- Retrieval over your documents. Choose it when wrong answers come from missing knowledge, not from the model's skill; a smaller model does not fix that.
- Rules or classic machine learning. Choose them when the pattern is fixed, such as order numbers or known codes, or when a simple classifier on structured data already meets the bar.
- Stay with the API. Choose it when no one on your side can own a hosted model: patching, monitoring and retraining are ongoing work, and the responsible AI and MLOps side of the build must be staffed.
What are the limits of this guide?
- It covers text tasks; vision and speech models follow the same logic, but their test sets and hosting differ.
- The research figures were measured on public benchmarks; your inputs, labels and edge cases decide the real result.
- Model releases move fast, so re-run the bake-off when a new small or large model is a serious candidate.
Plan the next step with a Netbase consultant
Frequently asked questions
Not always. A small model is usually cheaper per call once hosted, but labelling, training, hosting and retraining are fixed costs. At low volume an API call to a large model can cost less overall, so compute both from your own volumes before deciding.
It depends on the number of labels and how varied the inputs are. Start with a modest reviewed set per label and grow it where the model errs; the test set tells you when the model is good enough, and our fine-tuning data checklist covers labelling and leakage.
Yes. Open-weight and distilled models can be hosted in your cloud account, on premises or on a device, which keeps the text away from third-party APIs. You then own patching, monitoring and capacity.
Rarely at first. Most teams get the best result from a hybrid router: a small model handles the routine share of requests and passes low-confidence cases to a large model or a person.
Next step
Send us one task your large model handles today, with a sample of inputs and the volume you expect, and we will book a solution review to score it against the six signals and plan a first bake-off. You can also see the customer service AI solution, where routing and triage are typical narrow tasks.
Related services and solutions
Generative AI development with evaluation and human review Netbase builds generative AI features for product and operations teams: drafting, summarising, rewriting and content generation inside the software your people and customers already use. Every feature ships with an evaluation set, a human review step where output matters and limits on cost. Generative AI is a capability we are growing, and we label its maturity honestly.
Learn More
Responsible AI and MLOps for AI in production MLOps and responsible AI keep an AI feature trustworthy after launch. Netbase's practice for monitored, governed AI in production covers evaluation before every release, versioning of models, prompts and data, monitoring of quality and cost, and incident controls with a named owner. It is a growth capability, backed by an MLOps pipeline delivered for an unnamed client.
Learn More
NLP development: language AI that reads, sorts and answers from your text Netbase builds natural language processing (NLP) into the systems you already run: classifying emails and tickets, extracting names, amounts and dates from documents, scoring sentiment in reviews, searching by meaning and summarising long text. We choose between a large language model, a smaller fine-tuned model or plain rules per task, and measure each on your own text.
Learn More
Intelligent document processing with AI: from orders, invoices and forms to validated data Intelligent document processing is AI-driven intake that reads orders, invoices and forms, extracts the fields your systems need, validates them against business rules and routes each record to the right queue, with people reviewing anything uncertain. Netbase offers it as a growth capability built on its computer vision and automation work, including a document AI platform for an unnamed client.
Learn More
Work with Netbase remotely
Every Netbase engagement is delivered remotely, so clients in any country and any industry can work with us; Netbase has worked with clients in 80 countries. Contracts and invoices are issued by Netbase JSC, and payment is made by bank transfer to Netbase JSC's company account or PayPal Business. Talk to the CEO, Huy Nguyen (David), on WhatsApp or by email.
Discuss a project
Netbase JSC helps organizations design, build, modernize, and operate digital products and AI-enabled business systems.project@netbasejsc.com
+84 937 869 689
91 Nguyen Chi Thanh, Dong Da, Hanoi, Vietnam
Get in touch
Tell us what you want to build, modernize, or operate.