Why the right model for the job is rarely the biggest one
Small and Domain-Specific Language Models
The article at a glance
This article challenges the “one powerful foundation model everywhere” default.
It argues that production-grade agentic systems have heterogeneous task requirements—some need broad open-ended reasoning, but many need fast, consistent, narrow decisions (classification, routing, filtering, rule checks). In those cases, smaller or domain-specific models can be more accurate, cheaper, and easier to evaluate.
This blog is part of a series.
Read previous part: Designing the Agent Data Plane
Read next part: Designing Multi-Modal Agentic Systems
One of the most common assumptions in early agentic AI deployments is that more capability is always better. If a foundation model is powerful, it is tempting to place it at the centre of every decision an agent makes. Why introduce complexity by using multiple models when one capable model could handle everything? It is a reasonable instinct. It is also one that tends to break down quickly in production.
Bigger models increase capability. They do not guarantee better systems. The reality of building reliable agentic systems is that different tasks within a pipeline have genuinely different requirements. Some require broad reasoning across open-ended problems. Others require precise, consistent classification within a narrow domain. Others require fast, low-cost filtering at high volume.
Using the same model for every task is like using the same tool for every job. It works, but not as well as the right tool would, and at considerably greater cost.
Small and domain-specific language models are not a compromise. They are a deliberate architectural choice that makes agentic systems more accurate, more efficient, and easier to validate.
What we mean by small and domain-specific
These two categories are related but distinct, and it is worth being precise about each.
- Small language models
These are models with significantly fewer parameters than foundation models. They are faster, cheaper to run, and can often be deployed closer to where the data lives. Their capability is narrower, but for well-defined tasks that narrowness is an advantage. A small model asked to classify a support ticket into a fixed set of categories will often match or exceed the accuracy of a foundation model on that task, while being significantly faster and cheaper.
- Domain-specific models
These are models trained or fine-tuned on data from a particular field: clinical records, legal documents, financial filings, engineering specifications, regulatory text. They have internalised the vocabulary, conventions, and reasoning patterns of that domain. When accuracy within a domain matters, they typically outperform general-purpose models while being more predictable and easier to evaluate.
In practice, these categories often overlap. A small model fine-tuned on domain data combines both advantages: efficiency and consistency.
Where they fit in an agentic pipeline
The most effective agentic architectures treat models as components within a pipeline, each used where its strengths are most relevant. A foundation model may orchestrate the overall task, handle open-ended reasoning, and generate outputs. Smaller and domain-specific models handle specific functions within that pipeline. Models are not the system. They are components within it.
- Validation and checking
Rather than asking a model to verify its own output, a separate validation model reviews results against domain rules, flags inconsistencies, and identifies uncertainty. Self-review by a single model is unreliable. Independent validation is significantly more robust. - Classification and routing
Determining categories, selecting tools, and routing tasks through workflows are well-suited to small, specialised classifiers. These tasks benefit from consistency more than broad reasoning. - Filtering
High-volume pipelines benefit from lightweight models that filter or prioritise inputs before they reach more expensive reasoning layers. This reduces cost, latency, and noise. - Domain judgment
When an agent operates in a specialised field, domain-specific models provide more reliable assessments. Whether evaluating a clinical document, reviewing a contract, or analysing a financial transaction, models trained on domain data apply judgment more consistently than general-purpose models.
The trade-offs worth understanding
Small and domain-specific models bring clear advantages, but they also introduce considerations that need to be managed deliberately.
- Maintenance: A fine-tuned model reflects the data it was trained on. As domains evolve and rules change, models need to be updated. Without this, outputs drift from current requirements.
- Evaluation: Specialised models are easier to evaluate because their tasks are narrower and criteria clearer. But this only matters if evaluation is built into the process. A model that is deployed without systematic evaluation is not more reliable.
- Composition: Multi-model pipelines introduce coordination complexity. Interfaces must be defined clearly, information flow must be governed, and the system must be tested as a whole. A pipeline where each model performs well individually can still fail at the points where they interact.
These are not reasons to avoid specialised models. They are reasons to use them deliberately.
What this means for organisations
Model selection in agentic systems should be made at the level of individual tasks, not just at the system level.
For each function within a pipeline, the relevant questions are:
- What does this task require?
- What level of accuracy is needed?
- What are the cost and latency constraints?
- Is domain-specific knowledge required?
The answer will not always be a small or specialised model. Some tasks require the broad reasoning capability of a foundation model. But many do not. Designing as if they do leads to systems that are slower, more expensive, and harder to validate than necessary.
The most reliable agentic systems are not built with the most powerful models. They are built with the right models, used in the right places, with clear responsibilities at each stage of the pipeline.
Meet the Author:
Alison Hodivala
Director of AI & Data Portfolio Strategy at Bell Integration
Alison Hodivala is the Director of AI & Data Portfolio Strategy at Bell Integration, where she focuses on ensuring the company’s AI and data services reflect market needs and are supported by delivery models that scale reliably. She works across product, delivery and commercial teams to develop offerings that are practical, relevant and capable of delivering business impact.
Her career spans more than 25 years in start-ups and global organisations, covering consultative sales and delivery, enablement and change management. Drawing on a background in organisational psychology, Alison applies an evidence-based approach to AI adoption, aligning people, processes and technology to achieve effective and responsible outcomes.
