Before letting AI receive support inquiries, define the conditions for handing them back: measure wrong resolutions, not just answer rate

A&A perspective

A&A's suggestion for owners of small companies is to judge an AI introduction by the movement of cases a person has to pick up later, not by the increase in the count of AI responses. Answer rate only moves upward, but the count of cases picked up later can move in either direction. Before choosing a product, decide on a way to measure which direction it moves in. This is not a fight over the numbers themselves; it is a way to separate which direction actually helps the business.

A&A perspective

The important qualification is that the case is centered on the share Fin returns and how that share was operated upward. Under what conditions the roughly half that Fin does not resolve is routed to a person, and how many of the returned answers turn out to be wrong and how they are detected, are not the subject of that document. Reading the case is useful for enumerating what your own company should measure, not for adopting the same rate as a promise. Smaller shops are more exposed to a headline rate because the denominator is small.

Anthropic ↗

A&A perspective

A&A's reading is that what a smaller company should carry away from a large case like Intercom's is the design view that "AI answered," "AI handed to a person," and "a person had to pick this back up" can and should be tracked as three separate signals rather than folded into one resolution rate. If they are folded, the effort of picking back up disappears from the report.

A&A perspective

Translating this into a small company's inquiry desk, the first task is to write down, before choosing an AI product, which kinds of inquiries must go to a person. Read one to two weeks of the actual inquiry history and mark the cases the answering employee treated as beyond the first line. Refunds, cancellation fees, communication that touches regulation, changes to an existing customer's contract terms, and requests that require identity verification are typical. For these, decide that AI performs only intake and acknowledgement, and is out of scope for the reply itself.

A&A perspective

With the handoff conditions decided in advance, comparison of AI products narrows to "within the scope you actually delegate." A "90% resolution" product that raises its number by answering across categories you did not intend to delegate is not a useful number for you. If you plan to delegate only simple FAQ, it suffices to check two things: response quality within that scope, and the accuracy of the handoff when a case leaves that scope for a person.

A&A perspective

An inquiry received by AI ends in one of three ways. First, AI answers, the customer accepts, and the reply is consistent with internal rules. Second, AI answers, the customer accepts in the moment but a discrepancy becomes visible later. Third, AI does not decide and passes the case to a person. What is good for the business is the first case growing, the second case shrinking, and the third case being one where "the cases that ought to come back reliably come back." Vendor resolution rates are often defined so the first and second are summed in the numerator and the third is removed from the denominator; without checking the definition, the invisibility of the second case remains.

A&A perspective

The third case, "cases passed to a person," cannot be judged by count alone either. Even if their count is low, if any of the customer's own words, the content AI presented, or the internal source AI consulted is missing at handoff, the person has to ask the customer the same question again. From the customer's perspective this is a duplicated contact; from the handler's perspective the time per case can be longer than it would have been without AI. Record "average handling time for handed-back cases" and "number of re-questions to the customer," alongside your pre-AI baseline.

A&A perspective

Before signing with an AI product, put on a single sheet what must accompany a handoff. As a minimum, six items should arrive as one unit: the customer identifier, the inquiry text, the full text of what AI replied, the ID of any knowledge article AI referenced, the reason AI decided to pass the case to a person, and candidate categories for the human response. With these six items present, the handler can reconstruct the situation from internal records before asking the customer any additional question.

A&A perspective

With a handoff format decided, AI-product comparison can proceed on "does it export in that format?" and "when a field is missing, which item is dropped?" Many vendors have their own standard UI, but check whether records can be written in the shape the handler actually reads, not the shape of the standard UI. This check separates into cases resolvable by AI-side configuration and cases that require a receiving surface in your internal customer-management tool. If the estimate does not include the latter where relevant, you will end up in a state where "cases return but cannot be read."

A&A perspective

The trial that supports the introduction decision does not need to be a large A/B test. Restrict AI to a narrow delegated scope for one or two weeks and count that period's inquiries under the three-way split above. In parallel, take a random sample of inquiries that arrived in the non-delegated scope and have a handler re-read them to confirm none were mishandled through AI. Ten to twenty percent of the total is enough to reveal a tendency while keeping the handler's added workload bounded.

A&A perspective

At the end of the trial, do not look only at the change in response rate. Cases detected after the fact should be at zero or no higher than the pre-AI baseline. Average handling time for cases handed back should be shorter than the baseline, or at least equal. The random sample from the non-delegated scope should show no mishandling. If any one of these fails, either narrow the delegated scope or rebuild the handoff format. The decision to stop the introduction is also made at this stage, not later.

A&A perspective

When A&A is asked to help with inquiry design, the same trial design and the same articulation of the delegate-versus-return boundary are the starting points. The boundary is not there to reject a given AI product; it is there to keep the product within the range where it is actually good. At the three-month and six-month marks after introduction, check whether the burden of handed-back cases is quietly eating into the gross margin on the business side. The related article "When identical questions stop your team's hands: making support capacity with the Intercom case" handles the same Fin case from the side of building capacity.