How to audit what your AI support agent promised: Gorgias AI Agent, Zendesk AI and Intercom Fin (2026)
To audit an AI support agent, you match every refund, credit or discount it promised in a ticket to what actually happened in Shopify and Stripe. Teams do this four ways: a manual spot check, a spreadsheet, a conversation QA tool, or Recoup, which checks every AI-closed ticket against the ledger and flags promised but not issued, issued twice, wrong amount and outside policy. Recoup is new, with no customer results yet, and we make it, so read this with that in mind.
Four ways to audit AI agent refunds
| Method | What it checks | Every AI-closed ticket? | Finds promised, not issued | Finds double refunds and wrong amounts | Cost |
|---|---|---|---|---|---|
| Recoup | Each money promise in the ticket, matched to the Shopify order and refund and the Stripe refund, and checked against your written policy | Yes | Yes | Yes, plus credits outside policy | Free in-browser audit. Founding audit $299 after the report, free under $299 found |
| Manual spot check | A person reads a sample of tickets and looks up each order | No, only the sample | In the sample | In the sample, if someone compares amounts | Staff time |
| Spreadsheet reconciliation | Ticket export and order export joined on order number | Possible | Only if someone tags which tickets promised money | Depends on your formulas | Staff time to build and keep it up |
| Conversation QA tools (Oversai, EvaluAgent, Scorebuddy) | Grade what the agent said: quality, tone, accuracy | Depends on plan | Not mentioned on their pricing pages | Not mentioned on their pricing pages | EvaluAgent from $35 per user per month; Oversai platform from $500 a month billed annually |
QA tool details from their own pricing pages, read 19 October 2026, as listed in Best tools to audit AI support agent refunds. Recoup prices from our pricing page.
What to export from each AI agent
- Gorgias AI Agent: closed tickets and the agent's replies. Gorgias AI Agent refund audit
- Zendesk AI agents: solved tickets and comments. Zendesk AI agent refund audit
- Intercom Fin: conversations Fin resolved. Intercom Fin refund audit
- Shopify: orders, refunds and discounts. Stripe: payments and refunds, if you use it.
Manual spot checks: how to do one well
Pull a random set of AI-closed tickets that mention a refund, credit, discount or replacement. For each one, copy the exact sentence the agent wrote, open the order in Shopify, and write down what the ledger shows: nothing refunded, the right amount, the wrong amount, or two refunds. Check any discount or credit against your written policy.
It is free and teaches you what your agent says. The limit is coverage: a missed refund looks like a solved ticket, so anything outside your sample only surfaces as a follow-up or a chargeback.
Where Recoup fits
Recoup does the same check on every AI-closed ticket. It finds each sentence where the agent promised money, ties it to the order and the payment, and checks it against your policy using plain written rules, not a language model, so the same files always give the same answer. Every finding shows the sentence and the ledger line behind it. See a sample report (60 tickets, 21 mismatches, $744.25 at stake).
Honest limits: Recoup checks money, not tone, so it does not replace conversation QA. It is new and onboarding its first brands by hand.
Frequently asked
How do I check if my AI support agent actually issued the refunds it promised?
Export the tickets the agent closed, find every sentence where it promised money, and match each one to the order and refund in Shopify and Stripe. You can do that by hand on a sample, in a spreadsheet, or with Recoup, which reads the exports, matches every promise to the ledger and lists what never happened, what happened twice, wrong amounts and anything outside your policy.
Is a manual spot check enough?
A spot check shows whether problems exist, but it only covers the tickets you read. A promise that was never issued looks exactly like a solved ticket in the help desk, so misses in the tickets you did not sample stay hidden until the customer follows up or disputes the charge.
Do Gorgias, Zendesk or Intercom check refunds against Shopify or Stripe?
Recoup reads closed tickets and agent replies from Gorgias AI Agent, conversations Fin resolved in Intercom, and solved tickets and comments from Zendesk AI agents, then checks them against Shopify orders, refunds and discounts and Stripe payments and refunds. The help desk marks a ticket solved when the agent replies, while the refund happens in another system, which is the gap Recoup checks.
Does Recoup change anything in my store?
No. Recoup is read-only. It never issues, edits or reverses a refund. The self-serve audit runs in your browser and your files never leave it. Anything it finds, your team fixes in your own Shopify or Stripe admin.
How much does a refund audit cost?
The in-browser audit is free. The founding audit, where we run your last 90 days for you, is $299, paid only after you see the report, and free if we find less than $299 at stake. Monitor is $249 a month for 3 months as a founding price, then $500 a month up to 2,000 orders.
Who wrote this?
Recoup, made by AGUNA WELLNESS LTD, so Recoup is listed first. QA tool details come from our sourced comparison, read on 19 October 2026. Corrections to support@recoupaudit.com.
Check every promise, not a sample
Run the audit on your own exports with files kept in your browser, or let us run your last 90 days: $299 after you see the report, free if we find under $299.