How to Create and Edit Scorecards in Quack AI

Scorecards in Quack AI let you define what “good” looks like for your AI evaluations. You can set clear quality criteria, guide AI scoring behavior, and validate results with human review.

Follow these steps to create, configure, and edit your scorecards.

Create a Scorecard

  1. Go to the Scorecards page: From your Quack AI dashboard, navigate to Quality → Scorecards.

  2. Click “Create a scorecard”: A new scorecard is automatically generated with a default name.

  3. Rename your scorecard: Click the name field and enter something descriptive - e.g., Email Scorecard if you’re evaluating email interactions.

💡 Pro Tip: Use clear names by channel or use case so it’s easier to track performance later.

Define What to Score

  1. Choose the scope:

  • All interactions: Apply your scorecard to every interaction.

  • Custom interactions: Target specific interaction types (e.g., only email).

Example: Select Custom interactions → set Channel = Email to evaluate only email-based interactions.

Add Evaluation Questions

  1. Click “Add question”: Define the criteria you want to score.

  2. Write your question: Example: “Did the agent provide a clear and complete answer?”

  3. Select a category: Choose a relevant category like Communication, Accuracy, or Problem-Solving.

    • Categories group similar questions, making reports cleaner and easier to interpret.

  4. Choose question type:

    • Rating: Use a scale to measure performance.

    • Binary: Choose between Achieved or Not achieved.

  5. Add a brief for the AI: Provide a short explanation to help the AI interpret your question correctly. For example:
    Question: “Did the agent listen attentively to the customer’s concerns?”

    Brief: “Evaluate whether the agent clearly understood the customer’s specific issue and responded directly to it. Give full credit if the response addresses the customer’s described concern without ignoring details, interrupting, or making assumptions.”

💡 Pro Tip: A clear brief ensures consistent AI evaluations across all interactions.

Set Up Validation

  1. Configure validation settings:

    • Default: 50 interactions are validated weekly.

    • Custom: Adjust the number and frequency (daily, weekly, bi-weekly, or monthly).

  2. Add advanced logic (optional): Create specific validation rules, such as:

    • Only validate interactions with very negative sentiment.

    • Validate by resolution status or channel type.

  3. Add multiple validation sets: Use more than one validation set to compare different QA samples within the same scorecard.

Publish and View the Scorecard

  1. Click “Publish”: Once your scorecard is complete, publish it to activate scoring.

  2. View your scorecard: Access it from the Scorecards board. From this point forward, Quack AI will automatically score interactions according to your setup.

Edit an Existing Scorecard

  1. Open the Scorecards board: Find the scorecard you want to update.

  2. Click the three-dot menu → select Edit, or open the scorecard and click Edit inside the detail view.

  3. Update settings as needed:

    • Change the target (e.g., apply to new channels)

    • Add or edit evaluation questions

    • Adjust validation rules or frequency

  4. Save your changes: For example, switch a question type from Rating to Achieved / Not achieved.

Next Steps

Scorecards help you maintain quality control while giving AI clear instructions on what “good” looks like. Explore our QA Scorecard Questions Suggestions and start building your ideal QA scorecard today.