LLM ENGINEERING
LLM fine-tuning services with the proof attached. We build the evaluation suite first, then tune or distill a smaller, cheaper, faster model until it clears your quality bar - and keep the suite running afterward, because models drift and so does your data.
What fine-tuning does, and what evaluation decides
Fine-tuning is further training of an existing model on your own examples so it behaves the way you need - in a consistent format, in your terminology, on your narrow task. Distillation is the related move of teaching a small model to reproduce a larger one’s behavior on that task, at a fraction of the cost and latency.
Evaluation is what makes either defensible. Without a scored suite, “the fine-tuned model is better” is an opinion formed from a handful of examples someone happened to look at. With one, it is a number you can put in front of a CFO, and a gate that stops a regression reaching production.
We lead with evaluation deliberately, and often the suite alone is the deliverable. It is common for a scored comparison to show that a better prompt and better retrieval close the gap, and that fine-tuning is not needed at all. That finding costs a fortnight and saves a project.
You probably need this if you recognize these
- The system works, and it is too slow or too expensive at your volume.
- Output format is nearly consistent, which is worse than either extreme.
- You are paying for a frontier model to do one narrow, repetitive task.
- The model does not use your internal terminology however carefully you prompt it.
- Nobody can prove whether last month’s changes improved anything.
- A provider update changed behavior and you found out from a customer.
What you get
An evaluation suite
Real inputs, expected behaviors, scoring rubrics and pass thresholds, versioned alongside your code.
A scored baseline
What the current setup achieves, per case category, before anything is changed.
A dataset
Curated, deduplicated and split so training examples cannot leak into the test set. Usually the most laborious part, and the part that decides the outcome.
The tuned or distilled model
Trained, compared against the baseline and against a well-prompted larger model, on quality, latency and cost together.
Regression gates
Wired into CI, so a prompt, model or data change is scored before it merges.
A drift watch
Live sampling and scheduled reruns, with alarms when quality moves.
A rollback path
Model versions pinned and reversible, because the fastest fix for a bad model is the previous one.
How we decide whether to tune at all
LLM fine-tuning and evaluation
Define the bar
What "good" means, per case category, agreed in writing before any modeling.
Self-healed - retried with fallback tool. Human not required.
Build the suite
Real inputs, expected behavior, and a scoring method that a person would agree with.
Baseline
The current system is scored. So is a well-prompted large model and a well-prompted small one.
Decide
If prompting and retrieval close the gap, we say so and stop. This happens often.
Tune or distill
If they do not, we curate the dataset and train, comparing against every baseline.
Gate and watch
The winner ships behind regression gates, with drift alarms and a pinned rollback.
Tasks where a smaller tuned model usually wins
Ticket Triage Agent
Classification into your categories: narrow, repetitive and high volume, which is the classic distillation case.
Code Review Agent
Comment style and severity ranking tuned to your conventions rather than generic advice.
Invoice Matching Agent
Extraction from your suppliers’ actual layouts, where a tuned small model beats a general large one on both cost and accuracy.
What this work touches
We keep training data and evaluation sets inside your boundary by default. They are usually the most sensitive artifacts in the project.
- Model providers and fine-tuning APIs
- Self-hosted inference
- Training and experiment tracking
- Data warehouse and labeling tools
- CI/CD, for regression gates
- Observability
- Model registries
- Cost platforms
Platform names are shown as examples of the categories agents connect to. They are not partnerships or endorsements.
How model changes are controlled
Autonomy boundary
No model reaches production without clearing the suite. A model that passes on average but fails a protected category does not ship.
Approval gates
Adopting a new base model, changing a threshold or retiring a rule is your decision, presented with the scored comparison.
What stays human
Setting the quality bar, labeling the reference set, and judging the cases the rubric cannot express.
Logging
Dataset version, base model, training configuration, evaluation scores and deployment date recorded together, so any production behavior can be traced to the artifacts that produced it.
How the work runs
Typical ranges from our engagement model (doc 04 §5), not a quote.
| Stage | Typical | What happens |
|---|---|---|
| Pilot | 3-10 days audit, then 2-4 weeks | The suite, the baseline and a recommendation on whether tuning is warranted at all. |
| Build | 3-8 weeks | Dataset curation, training runs, comparison, regression gates and the drift watch. |
| Release | 1-2 weeks | Shadow inference alongside the incumbent model, compared case by case before the switch. |
| Managed | ongoing, optional | Suites rerun against new base models, periodic retraining, drift alarms watched. |
What we measure
- Suite first
- The evaluation set exists before any model is trained (Target)
- 3-way comparison
- Tuned model, prompted large model, prompted small model - always scored together (Target)
- Cost and latency
- Reported next to quality, because a better model you cannot afford is not better (Target)
- Your bar
- The pass threshold you set, per case category, before the work begins (Yours)
“ Until then these are design targets and measurement commitments, not results.
What this looks like in practice
Reconciliation hive
Reference scenario · FinTech. Extraction and matching where a tuned smaller model carries the repetitive volume.
Reference scenario - a composite build illustrating our method. Figures are modeled and the model is shown.
Frequently asked questions
How do we know whether we need fine-tuning at all?
You measure. We build the suite and score three options: your current setup, a well-prompted large model and a well-prompted small one. Frequently the gap closes without any training, and we tell you that even though it is the smaller engagement. Fine-tuning earns its cost on narrow, repetitive, high-volume tasks with consistent output requirements.
How much training data do we need?
Fewer examples than most people expect for behavior and format, and more than they expect for genuinely new capability. Quality and coverage of the awkward cases matter far more than volume. We assess what you have during the pilot, and where the honest answer is "not yet", we say so rather than training on a thin set and presenting a flattering average.
Will fine-tuning stop it making things up?
Not reliably, and treating it as the fix for hallucination is a common and expensive mistake. Tuning shapes behavior and format; missing knowledge is a retrieval problem, handled by RAG and knowledge systems. A tuned model with no grounding will confabulate more fluently, which is worse.
What happens when the base model is deprecated?
The suite makes this survivable. We rerun it against candidate replacements, compare on quality, cost and latency, and migrate on evidence with a pinned rollback. Without a suite, a deprecation is a rewrite done under time pressure, which is how most teams first discover they needed one.
Can we run the tuned model ourselves?
Depending on the base model's license and your infrastructure, yes, and self-hosting a small tuned model is often the cheapest option at volume. We are candid about the trade-off: you take on inference operations and GPU capacity planning in exchange for lower unit cost and full data control.
Is our training data used anywhere else?
No. Training data and evaluation sets stay within your boundary, we contract for zero training rights with every provider we use, and artifacts are versioned under your control. These are typically the most sensitive things in the project, and they are treated that way.
Related services
Conversational AI & Voice Agents
Where tuned models and latency budgets meet directly.
LLM Engineering
The layer this work sits inside.
Generative AI Development
The products these models end up serving.