Blog
We built an AI support assistant. Tickets went up.
Purpl's AI support assistant did what we designed it to do, and support tickets still rose 25% in its first month. What we learned about measuring a chatbot.
The pitch for an AI support chatbot is simple. Members ask the bot, the bot answers, and your support inbox gets quieter. We built one for Purpl, a discounts service for disabled people with over 100,000 members, and set it a target of cutting the tickets it could affect by 85%.
Thirty days in, those tickets were up 25%.
The assistant hadn't broken. It was doing what we'd designed it to do. The trouble was what we'd chosen to count. If you're thinking about a support chatbot for your own business, this is the month one review we wish we'd read first.
What we built
The chatbot sits on Purpl's website and in its app. It answers only from Purpl's help centre, links the articles it actually read, and can check a member's account status. When someone is upset, asks for a person or has a problem specific to their account, it answers first and then offers a support ticket with the conversation attached.
Before launch we replayed twenty real, already-solved tickets through it. Twelve answers were good, seven were OK and one was bad. After changes, it missed none of the cases where it should have handed over to a person. It went live on 9 September 2026.
What the numbers said
The measure was support tickets the chatbot can affect: those raised on the website, in the app and through forms. Email and Facebook Messenger were left out, because the assistant doesn't sit on those.
| Period | Tickets | From chatbot handovers | Raised another way |
|---|---|---|---|
| 4 weeks before launch | 554 | n/a | 554 |
| First 4 weeks | 692 (+25%) | 263 | 429 (−23%) |
| Latest 4 weeks (to 26 Oct) | 700 (+26%) | 456 | 244 (−56%) |
So on the headline measure the project was off track, and we marked it that way in the 30-day review.
Two things were happening at once. Fewer people were raising tickets without going through the chatbot: down 23% in the first month and 56% in the latest four weeks. But the chatbot's own handovers more than made up the difference.
One caveat matters. Since 9 October every contact route sends people through the chatbot first, so tickets that used to arrive directly now show up as handovers. That means the two right-hand columns aren't comparable before and after that date. The total is the fair headline, and the total went up.
Making help easy to ask for creates more asks
Here's our reading. When asking for help gets easier, more people ask. A member who might have given up on a contact form, or never got round to emailing, can now type a question and be offered a ticket a few seconds later. Some of those tickets are people the chatbot couldn't help. Some are people who wouldn't have got in touch at all before.
We can't yet tell those apart, and that's the honest gap in our month one figures. The 90-day review is meant to separate "the chatbot dealt with it" from "the chatbot made it easier to raise".
We built the handover in on purpose, because a support bot that traps upset people is worse than no bot. Judging it on ticket count alone was our mistake.
Measure a support bot by what it resolves
If you're setting targets for a chatbot, these are the things we'd sort out before launch rather than after.
- Count resolutions, not tickets. A ticket count rises when the bot hands over well and falls when it stonewalls people. What you want to know is how many questions ended with the member's problem solved and no person involved.
- Take a clean baseline. Record tickets by channel for at least four weeks before launch. We'd also note any change to how people reach support, because changing the routes changes what the numbers mean.
- Tag every handover. Know which tickets the bot raised and what they were about. That's the list of questions it could learn to answer, and the list it should keep handing over.
- Decide the framing up front. Agree before launch whether handovers count against the bot. We left that open, and the 30-day review had to argue it out.
- Test on real, solved tickets. Replaying twenty old tickets told us more than any demo. It showed us where the bot was weak and whether it knew when to step aside.
Write the handover rules as carefully as the answers
Most of the effort on a support bot goes into its answers. The handover rules decide your ticket numbers just as much. For each type of question, write down:
- Can the bot answer this from what it knows, without guessing?
- Does it need to look something up about this person's account?
- Is this someone who should always reach a person, whatever the bot could say?
- When it hands over, does the ticket carry the conversation so nobody has to ask again?
Then look at the handovers each week. Purpl's next step is going through what the escalations are about, by chat category, to see which ones the chatbot could answer itself.
What's changing before 8 December
We've added a nightly routine in which the model reviews escalated tickets and learns from them.
The 90-day review is due on 8 December, with a decision to keep, change or remove the assistant. Before then we're setting up a checked sample of tickets raised outside the assistant, so the review can judge the chatbot on what it resolved rather than on the raw total.
When a support bot isn't worth it
- When the help centre is thin or out of date. The chatbot answers only from Purpl's help centre. If yours doesn't have the answers, a bot will either make things up or hand everything over.
- When the questions are mostly account problems a person has to fix. The bot can only offer a ticket.
- When the only number you'll judge it on is ticket count. You'll probably be disappointed, for the reasons above.
The short version
An AI support assistant that hands over properly can push your ticket count up, at least at first. Decide what a resolution is, count those, and treat the handovers as the to-do list for the next version.
If there's a dull job like this in your business, we build tools that do it.