Standing up an AI support agent in a week

Support is the first place most companies put an agent, and the first place a bad agent does damage. Answering a customer wrong is worse than answering slowly. So the question is not how fast you can ship an agent. It is how fast you can get one onto real tickets with a person checking every answer.

We stand up support agents in a week. Not because a week is impressive, but because a week is what it takes to reach that supervised state. The learning starts there. Everything before that is guessing about what customers ask, and the guesses are usually wrong.

This playbook is what we run for Automate partners with an existing help desk and at least a few hundred historical tickets. With less history than that, the week is mostly writing the answers by hand, which is a different job.

What a week buys you

Day five is not a finished agent. It is an agent that drafts answers for a defined set of intents, hands everything else to a person, and logs enough to tell you where it is wrong. The finished version comes from weeks two through six, driven by those logs. We would rather have a narrow agent on real traffic than a broad one in staging.

Day by day

• Day 1: export. Help-center articles, macros, the last six months of tickets, and the escalation policy. We tag the top twenty intents by volume.
• Day 2: knowledge. Articles become a retrieval index. Macros become answer templates. Anything contradictory gets flagged for you to resolve, because the agent cannot.
• Day 3: the agent. Intent router, retrieval, answer draft, confidence score, hand-off rules. Deployed to a staging queue fed with replayed tickets.
• Day 4: evals. Two hundred held-out tickets scored against the human answer. We fix the worst intents and remove any the agent cannot handle safely.
• Day 5: live, supervised. The agent drafts, a support lead approves or edits, and every edit becomes a training example and an eval case.

The scope we hold

Twenty intents is the number. On most help desks it covers the bulk of the volume, and it is small enough to eval by hand. Order status, password resets, refund policy, plan changes, shipping windows. The long tail stays with people until the logs show a cluster worth adding.

The agent never takes an action in week one. It reads, drafts, and routes. Refunds, cancellations, and account changes require a person to click. Actions come later, one at a time, each behind its own approval rule, starting with the most reversible. Sending a tracking link is usually first. Issuing a refund is usually last.

Guardrails

• Confidence threshold. Below it, the agent writes nothing and routes to a person with a one-line summary.
• Refusal list. Legal threats, safety issues, anything naming a regulator, and anything you add. Always human.
• Source citations. Every drafted answer cites the article or macro it came from. No source, no draft.
• Tone lock. A style sheet built from your best hundred replies, applied at draft time.
• Rate limit. A cap on drafts per hour in week one so a bad prompt cannot flood the queue.
• Kill switch. One flag, checked on every request, owned by the support lead, not by us.

The agent spec

The whole configuration fits in one file. It is the same shape as the support-triage template in our Agent library, with your intents and your thresholds. The mode field is the one that changes over the following weeks.

agent: support-triagemode: draft_only            # draft_only | assist | autonomousintents:  - order_status  - password_reset  - refund_policy  - plan_change  - shipping_windowretrieval:  sources: [help_center, macros]  min_score: 0.72handoff:  below_confidence: 0.65  always_human: [legal, safety, regulator, chargeback]tone:  style_sheet: partner/support-voice.mdlimits:  drafts_per_hour: 120kill_switch: flags.support_agent_enabled

What we measure

• Draft acceptance rate: how often the person sends the draft unchanged. The single best signal.
• Edit distance on accepted drafts: small edits mean tone; large edits mean the answer was wrong.
• Hand-off rate by intent: which intents the agent is not ready for.
• Time to first response: usually the first number to move, and the one customers notice.
• Satisfaction on agent-drafted tickets versus human-only, matched by intent so the comparison is fair.

We read these daily in week one and weekly after that. If acceptance on an intent is under half after two weeks, the intent comes out. Removing an intent is not a failure. It is the system working as designed, and the intent can come back when the knowledge base catches up.

Week two and after

Weeks two through four run the same loop at lower supervision. The support lead reviews a sample instead of everything, then only low-confidence drafts. Actions arrive one at a time. The eval set grows from every edit and every escalation, and it runs on every change to the prompt or the knowledge base.

By week six most partners run the agent in assist mode on the first twenty intents and draft mode on a second twenty. The support team handles fewer tickets and more judgment. That is the goal. Nobody on the team should be typing an order-status reply by hand, and nobody should be trusting an agent with a chargeback.

The eval harness that keeps the agent honest after week one has its own playbook. Read the two together. One gets the agent live; the other keeps it from quietly getting worse.