Running Regression Testing for your AI Agent

Overview

Quack AI empowers you to train and release AI agents with confidence. Our scheduled regression testing automatically checks your agent’s performance and notifies you if accuracy drops, so you can act quickly and maintain quality.

  • Automatic regression tests: Scheduled to run weekly or monthly

  • Agent Home Page: Instantly view your agent’s current accuracy rate

  • Customizable scheduling: Choose frequency and preferred day/time

💡 Pro Tip: Regression testing helps you catch accuracy drops caused by training data changes, platform updates, or model drift—protecting your customer experience.

Why Use Scheduled Regression Tests?

  • Continuous assurance: Stay confident your agent isn’t drifting

  • Actionable alerts: Get notified if performance degrades

  • Insightful metrics: Review accuracy rates from previous positive interactions

How to Configure Scheduled Regression Testing

  1. Go to the Training Center:
    From the Quack AI dashboard, click Training Center in the left navigation.

  2. Select Your AI Agent:
    Browse your agents list. Click the agent you want to configure (e.g., Helper Hannah).

  3. Open Regression Test Settings:
    On the agent home page, find the Accuracy Rate panel and click Set Up Regression Test.

  4. Choose a Schedule:

    • Weekly: Select your preferred day (e.g., Monday)

    • Monthly: Test will run on the 1st day of each month at 9am UTC

  5. Save Your Configuration:
    Click Save Settings. Your schedule is now active.

💡 Pro Tip: You can update your test schedule anytime in the agent’s Settings panel.

Reviewing Regression Test Results

  1. Open the Agent Home Page:
    After a scheduled run, accuracy results appear in the Accuracy Rate section.

  2. View Test Runs:
    Click any test run to see a detailed overview - metrics, stats, and performance trends.

  3. Drill Into Results:
    Select a specific question to review the agent’s response and see where improvements may be needed.

  4. Act on Insights:
    Use these findings to retrain your agent or update training data for higher accuracy.

💡 Pro Tip: Link regression test results directly to the Training Panel for quick retraining actions.