Choosing a model for a customer-facing agent: OpenAI, Anthropic, Gemini or open weights
The choice is less about benchmarks than about latency, instruction-following, safety behaviour, data handling and price at your volume. A practical way to decide, and why the answer is often two models.
12 Feb 2026 · 2 min read · by HeiSyn
Model comparisons online are benchmark tables; agent behaviour in production is about whether the model follows your rules on the thousandth conversation, answers fast enough for a phone call, and handles your customers' data the way your contracts require. The right choice comes from testing your own conversations, not from a leaderboard.
What to test with your own data
Run your evaluation set through each candidate. Look at instruction-following — does it stay in scope, does it escalate when told to. Look at tone in your language and register. Look at latency to first token for voice. Look at how it fails: a model that says 'I'm not sure' is safer than one that answers confidently and wrong.
Data handling and residency
Where the data is processed, whether it is retained, whether it is used for training, what the contract says. Regulated industries and some regions need specific answers, and the vendors differ. Open-weight models hosted on your own infrastructure answer these questions completely, at the cost of running them.
Price at your volume
Estimate monthly conversations and turns, price each candidate at that volume, and remember that a cheaper model with a longer prompt can cost more than a pricier model with a shorter one. Prompt caching changes the numbers substantially for agents with large stable prompts.
Use two models
A small, fast model for classification, routing and the guardrail check; a capable model for the conversation itself. This is cheaper and safer than one model for everything, and it means you can swap either without rebuilding the agent. Keep the prompts and the evaluation set vendor-neutral so switching is a configuration change, not a project.
Questions people ask next
- Do we need the most capable model for a support agent?
- Usually not. A capable mid-tier model with a good prompt, a clean knowledge base and a guardrail check handles most support and qualification conversations well. Reserve the largest models for the hardest reasoning or for grading the smaller one.
- How hard is it to switch models later?
- Easy if the prompts, the tools and the evaluation set are vendor-neutral from the start: switching becomes a configuration change plus an evaluation run. Hard if the agent is built inside one vendor's proprietary agent framework.
- Are open-weight models good enough for customer conversations?
- For classification, extraction and well-scoped support with a strong knowledge base, current open-weight models are capable and give full control over data. For open-ended sales conversations, the leading hosted models still hold an edge. Test both on your own evaluation set.
Want this built for your team?
A free scoping call: we map your motion, name the number that should move, and quote a fixed-scope build.
