# How to audit the refunds your AI support agent handles URL: https://recoupaudit.com/guides/how-to-audit-ai-support-agent-refunds/ Updated: 2026-10-17 Recoup / Guides / How to audit AI refunds # How to audit the refunds your AI support agent handles Updated 17 Oct 2026 · 6 min read To audit an AI support agent's refunds, export the tickets it closed, pull out every money promise (amount, type and order), match each one to refunds in Shopify and Stripe, and classify it as matched, missing, duplicated, wrong amount or outside policy. Then fix the gaps and change the agent's rules so they stop recurring. You can do this by hand in a spreadsheet with the six steps below. Steps - Export the tickets - Extract the promise - Match to Shopify and Stripe - Classify each promise - Apply your policy rules - Fix and prevent ## Why reading conversations is not enough Most AI agent reviews read transcripts. That catches wrong answers and bad tone. It cannot catch a refund that was promised and never made, because the transcript looks fine. A HackerNoon essay makes the same point about agents in general: an API can return "200 OK" when the wrong thing happened, such as a refund on the wrong transaction or a second refund after a timeout retry (HackerNoon; the author presents these as hypothetical examples). Proof lives in the order system and the payment processor, not in the chat. ## Step 1: Export the tickets the AI agent closed Export 30 to 90 days of tickets where the AI agent replied, with the reply text, ticket ID, customer email, any order number and timestamps. Filter to tickets that mention money: refund, credit, discount, code, gift card, replace, reship, cancel. If the volume is large, take a random sample of 200 money tickets rather than the first 200. ## Step 2: Extract the promise For each ticket, write down what the customer was told in one row: promise type (full refund, partial refund, credit, discount code, replacement), amount, currency, order number and the time of the reply. If the agent hedged ("I've asked the team to refund you"), record it as a promise anyway, because the customer will read it as one. ## Step 3: Match to Shopify refunds and Stripe refunds Find the order in a Shopify orders export and note its refunded amount, financial status and any discount code. If Stripe takes payments directly, find refunds on the customer's charges and note amount, charge, created time and status. Look only at refunds created after the promise. Count how many refunds exist for that order. ## Step 4: Classify each promise Give every row one label: matched, promised but not issued, issued twice, wrong amount, or outside policy. Use the table below as the rule. Where you had to match on email instead of an order number, mark the row for a second look. | | What you see | Label | One refund, same amount, after the promise | Matched | No refund, credit or code after the promise | Promised, not issued | Two or more refunds for one promise, in one system or across Shopify and Stripe | Issued twice | A refund whose amount differs from the promise | Wrong amount | Correct refund that breaks a written rule | Outside policy ## Step 5: Apply your policy rules Write your refund policy as checkable rules: return window in days, maximum amount without approval, final-sale products, and how many claims one customer can make in a period. Run every refund, matched or not, through them. List customers with several claims as repeat claimants; it is a prompt to look, not proof of abuse. ## Step 6: Fix and prevent Issue the missing refunds and tell those customers. Decide whether to recover duplicates or write them off. Then change the agent: escalate any partial refund it cannot action, block refunds outside the window or above a ceiling, and make it say "a teammate will process this" rather than "refunded" when it has not run an action. ## Spreadsheet columns to use A sheet with these columns covers all six steps. Build it once and reuse it each month. | | Column | Filled in at | Example | Ticket ID and link | Step 1 | 48213 | Reply time | Step 1 | 2026-09-14 10:22 | Customer email | Step 1 | a@example.com | Promise type and amount | Step 2 | Partial refund, 18.00 USD | Order number | Step 2 | #10442 | Shopify refunded amount and count | Step 3 | 0.00, 0 refunds | Stripe refund amount, charge, status | Step 3 | none | Label | Step 4 | Promised, not issued | Policy rule broken | Step 5 | none | Money gap | Step 4 | 18.00 owed | Fix and owner | Step 6 | Refund issued, Sam Example values are an illustration, not real data. ## How long does it take? By hand, steps 2 and 3 take the time, because each promise needs reading and a lookup in two systems. As a rough guide, a few hundred tickets can take an afternoon and a few thousand can take a week. That is why sampling makes sense when you check by hand. The in-browser audit runs steps 2 to 5 on your exports automatically, and the files stay in your browser. If you would rather not do it at all, the free 90-day audit is done for you. Recoup is new and we are onboarding the first brands by hand. Help desk notes: Gorgias AI Agent, Intercom Fin, Zendesk AI. Terms are defined in the glossary, and real cases are in the incident log. ## FAQ How do I check whether refunds my AI support agent promised were actually issued? Export the tickets it closed, note each promised amount and order, then look for a matching refund on that order in Shopify, and in Stripe if it takes payments directly. No refund after the promise means promised but not issued. What guardrails should an AI support agent have for refunds? Escalate partial refunds it cannot action, block refunds outside the return window or above a set amount, exclude final-sale items, flag repeat claimants, and never say "refunded" unless an action actually ran. How many tickets should I sample? If you check by hand, a random sample of about 200 money-related tickets per month is a practical start. If you find gaps, widen it or check every ticket with a tool. Is it safe to let an AI agent issue refunds? It can be, with limits on amount and window, escalation for anything it cannot action, and a regular check that the refunds it promised match what was issued. The check is the part most teams skip. Who wrote this guide? Recoup, built by AGUNA Wellness Ltd (RC 9590421). Recoup is new. Questions go to support@recoupaudit.com. ## Skip the spreadsheet Run the audit on your own exports, with files kept in your browser. Or let us run the first 90 days for you, free. Run the audit in your browser →Get the free 90-day audit →