1. The Demo-to-Production Gap
An agent demo takes an afternoon. An agent in production takes weeks, and the difference has almost nothing to do with the model.
The demo runs on a curated question against a clean document, observed by someone hoping it works. Production runs on the question a frustrated customer actually typed, against a document set with three contradictory versions of the refund policy, and nobody is watching.
The engineering between those two states is the entire job. Model choice is perhaps ten percent of it, and it is the ten percent that gets discussed most.
2. Agents Are Tools, Not Prompts
The single largest determinant of whether an agent is useful is what it can do, not what it can say.
A chat box that answers questions from a knowledge base is a search interface with better manners. It is genuinely useful for deflecting repeat questions, and it is not an agent. An agent has tools: functions it can call against your live systems, with real permissions and real consequences.
In practice, a useful tool set looks like:
lookup_order(order_id)— reads from your actual order system, not a stale exportcheck_availability(service, date_range)— hits the live calendarcreate_ticket(summary, priority, customer_id)— writes into your helpdeskescalate_to_human(reason, context)— always present, always available
That last one matters more than the others. An agent without a clean escalation path will improvise when it hits the edge of its competence, and improvisation is precisely the behaviour you do not want at the edge.
Tool design rules that hold up
Make tools narrow. update_customer_record(field, value) is a footgun. update_shipping_address(address) is a tool. Narrow tools are easier to permission, easier to log, and dramatically easier to evaluate.
Return structured errors. When a tool fails, it should return a machine-readable reason the agent can act on — "order not found" versus "database timeout" lead to entirely different correct behaviours.
Make reads free and writes expensive. Reads can be liberal. Every write should be deliberate, logged, and reversible or gated.
3. Retrieval Is Where Most Agents Fail
When an agent gives a confidently wrong answer, the cause is usually retrieval, not generation. The model faithfully summarised the wrong document.
Things that reliably break retrieval in real deployments:
- Contradictory sources. The 2023 policy, the 2025 policy, and a Slack message where someone made an exception. All three are in the index. All three are retrievable. The model has no principled way to prefer one.
- Chunking that severs context. A chunk containing "customers are eligible for a refund" without the preceding sentence "before 2024," is worse than no chunk at all.
- Semantic similarity that is not relevance. "How do I cancel?" retrieves the cancellation policy, the cancellation fee schedule, and a blog post about cancelling meetings. Two of those actively degrade the answer.
The fixes are unglamorous and effective: curate the index rather than dumping a drive into it, attach metadata (effective dates, document status, owner) and filter on it before ranking, and re-rank retrieved chunks with a dedicated model rather than trusting cosine similarity alone.
Most retrieval quality problems are content-governance problems wearing a technical costume. Cleaning the source is nearly always higher leverage than tuning the retriever.
4. Evaluation Before Deployment
You cannot improve what you cannot measure, and "it seemed good when I tried it" is not measurement.
A workable evaluation harness needs three parts:
A golden set. Fifty to two hundred real inputs with known-correct outputs. Draw them from actual support tickets, real queries, genuine edge cases. Include the ugly ones — the ambiguous, the hostile, the ones with typos and missing context. A golden set of polite, well-formed questions measures nothing useful.
Automated scoring. For structured outputs, assert directly. For free text, use a model-based judge with a rubric — but validate the judge against human ratings on a sample first, or you are measuring the judge.
Regression runs in CI. Every prompt change, model swap, or tool modification runs the full set. Prompt engineering without regression testing is a random walk that feels like progress.
The number that matters is not the average score. It is the tail: what fraction of cases fail, and what is the worst thing that happens when they do.
5. Guardrails and the Irreversibility Test
The design question is not "will the agent make mistakes?" It will. The question is which mistakes are recoverable.
Sort every action the agent can take by reversibility:
- Freely reversible — drafting text, reading records, suggesting options. Let the agent act autonomously.
- Reversible with effort — creating a ticket, scheduling an appointment. Let it act, log everything, make undo easy.
- Irreversible — sending customer communications, moving money, deleting data, making commitments. Human approval, every time, no exceptions for convenience.
That third category is where "we will add approvals later" turns into an incident. Later never arrives before the first mistake does.
Alongside that, three guardrails earn their cost in every deployment: an allowlist of callable tools per agent context, hard limits on iterations and spend per session, and full logging of every tool call with inputs and outputs so any interaction can be replayed and understood after the fact.
6. Cost and Latency Are Design Constraints
An agent that costs more per interaction than the human it replaced is a demo, not a system.
Instrument cost per completed workflow — not per token, not per call. A workflow that takes nine model calls at low cost each may be more expensive than one call to a stronger model, and it is usually slower. Route by task difficulty: cheap, fast models for classification and routing; stronger models reserved for the reasoning step that actually needs them.
Latency deserves the same discipline. Users tolerate a few seconds for something visibly hard and abandon at two seconds for something that looks trivial. Stream partial output where you can, and design the interaction so waiting is legible.
None of this is exotic. It is ordinary engineering discipline applied to a component that happens to be probabilistic — which is exactly the framing that makes agents boring to operate, and boring to operate is the goal.
Run the readiness assessment
Eight questions on your data, process, and tooling. Three minutes, no system access, no sales call.