November 10, 2026 · Post-mortem
We turned our AI support bot off after two weeks
Week one looked like a triumph: a third of conversations closed without a human. Week two cost us a distributor. The interesting part is why the dashboard could not see it coming.
Writing this one is less fun than writing the wins. But we spent $1,400 and three weeks on it, and the useful part is the shutdown, not the launch.
The setup: 400 SKUs, about 60 tickets a week across email and WhatsApp, two people on support. We connected a bot to our help centre and let it answer anything it recognised. Escalation was "if the confidence score is below 0.7, hand it to a human".
Week one looked like a triumph. Thirty-four percent of conversations closed without a human. Average first response: eleven seconds. The dashboard had a green line going up and to the right, and I sent it to the founder with the word "working" in the subject line.
Week two is where it fell apart
A customer asked whether our half-inch fitting was rated for potable water. The bot said yes. The knowledge base article it pulled from was written in 2023, before we changed the O-ring supplier. It was not a hallucination in the technical sense — the model read a real document that was out of date. That is worse, because no confidence score catches it.
A second customer asked where his order was. The bot, which had no access to the order system, wrote a confident paragraph about typical transit times and implied a delivery date. He waited. Then he called.
The third one cost money. A distributor asked about a volume discount. The bot invented a tier structure that was close to our real one but not ours, and the distributor quoted it to his own customer before anyone caught it.
The root causes, in order of how much they mattered
The knowledge base was stale and nobody owned it. The bot was accurately repeating two-year-old truth. Every support tool on the market assumes you have a living knowledge base; almost nobody does.
No hard escalation rule. Confidence scores measure the model's certainty, not the world's. It was 0.9 confident and wrong.
It was allowed to answer questions it had no data for. Order status, pricing, lead times. Three things it should have refused outright, and refusing is a product decision, not a model setting.
What we kept
Draft suggestions for human agents. The bot writes a first reply, the agent edits and sends. Same model, no autonomy, and it cut handle time by about forty percent with zero invented facts, because a human reads every word before it leaves.
That is the lesson I would pass on. The value was not in letting the bot talk to customers. It was in letting it type while a human watched. We paid $1,400 to learn that, which is cheap compared with the distributor.
If you are shopping for support tooling, the ones in the library under customer care are worth reading for the escalation controls alone. Sierra is the one that takes outcome accountability seriously; it is also priced for companies much larger than ours.
Why publish the failure. I would rather write this than a case study with a logo on it. If you want more post-mortems like it — the ones vendors do not write — you can support the site here. No amount is too small, and none of it buys a better review.
We publish the pricing model, minimum commitment and export-readiness notes for every tool in the library, and we flag it when a vendor changes one without changing the headline price.