If you signed up before Aug 9, 2021, please click Previous plans to view your applicable plans.

This article provides a complete guide to testing, monitoring behavior via analytics, and troubleshooting the Email AI Agent.


This article contains:

About Email AI Agent Testing

Before deploying your Email AI Agent to handle live customer queries, it’s essential to ensure it delivers accurate and contextually relevant responses. The Email AI Agent provides a preview widget that allows you to validate your AI Agent’s real-time performance. You can test common queries and review the full response flow without affecting live customer conversations.


You can use the preview to:

  • Validate the AI agent’s accuracy before deployment
  • Conduct end-to-end testing of the customer email experience
  • Identify incorrect or incomplete responses

Prerequisites:

The Email AI Agent testing environment generates responses using your Knowledge Base articles. Before testing, ensure that:

  • You have created and published Knowledge Base articles.
  • The article visibility is set to All Users.

For details, see Create and Organize Knowledge Base.


How does the Email AI Agent preview widget work?

The Email AI Agent Preview widget replicates real-world customer interactions. It lets you test how it understands, interprets, and responds to customer emails, without sending or receiving real messages.

  • Setup and learning phase
    When you open the Email AI Agent for the first time, it begins learning from your connected Knowledge Base sources. A “Training in progress” message appears until the widget is ready for testing, which may take up to 30 minutes. Once training is complete, you can begin testing.
  • Query testing
    When you submit a test query and select a preconfigured portal email address, the widget simulates a customer email. The AI Agent then generates a response using the Knowledge Base linked to the selected portal.
  • Response display
    The widget displays the reply along with relevant Knowledge Base articles. 
  • Feedback collection
    You can rate responses using the thumbs-up or thumbs-down icons to provide feedback.

Test the Email AI Agent

You can test the Email AI Agent in two ways:

  • Use the Preview Widget with the Email AI Agent turned off 
  • Enable the Email AI Agent for internal testing only

Method 1: Using the preview widget with Email AI Agent turned off

You can test the Email AI Agent even when it’s turned off. In this case, the widget generates responses using the last saved configuration (rules and events) and the selected portal email address.

  1. Log in to your account as an admin.
  2. Go to Admin > Freddy > Email AI Agent.
  3. Click Configure.
  4. In the Preview Responses tab on the right pane:
    • Choose the preferred portal email address.
      Note: The list displays product-specific portal email IDs. If multiple products exist, all linked portal addresses will appear. Responses are based on the Knowledge Base associated with that portal.
    • Enter the subject and add the query in the email body.
  5. Click Send.


What happens next?

After you send the test query, the preview widget displays the Email AI Agent’s simulated response. This includes:

  • Response Summary: A short answer based on the Knowledge Base linked to the selected portal.
  • Reference Articles: Links to articles used to generate the response, helping you trace the source of information.
  • Feedback Options: Thumbs-up or thumbs-down icons to rate the response’s accuracy and relevance of the repsonse. This includes:
    • Yes, close my ticket: Resolves and closes ticket.
    • Not, really: No further action is taken. An agent will review and respond to the ticket.

      Note: The feedback choices are displayed only for evaluating the responses of the Email AI Agent. Selecting Not really does not create any new ticket.

If the preview widget cannot interpret the query, you may see the following message. You can modify the query and retry.

            

View past queries

The Past queries option in the preview widget lets you track and analyze how the Email AI Agent preview widget handled your test runs. This helps you validate accuracy, identify gaps, and refine your configuration before going live.

  1. Click on the past queries option in the preview widget:
  2. Here’s what you can see:
    • Query Status: Check if the query was answered or unanswered.
    • Query preview: Preview of the query asked.
    • Feedback given: Shows the feedback given by agents for the response generated.


Method 2: Testing with AI Agent Turned On (Internal Testing Only)

Besides the preview widget, you can test responses with the Email AI Agent enabled in a live-like setup while limiting it to internal team emails. This prevents it from responding to real customer tickets during testing.

  1. Go to Admin > Freddy.
  2. Enable Email AI Agent and click Configure.
  3. To restrict responses to internal users, add an AND condition:
    In Contacts/Tickets if Requester email is [internal email address]
  4. Click Update.
  5. Send a test email to your support inbox from the internal requester’s address (example, emily.gracia@acme.com)
  6. Ensure the email body contains a clear question or intent that matches a topic already documented in your Knowledge Base.
    Example:
    "What’s my refund policy? "(relevant to an existing published article)

Best practices for testing

  1. Test real scenarios
    Try a mix of common and complex customer queries.
    Example:
    “How do I reset my password?”
    “Is there a refund policy for annual plans?"
  2. Use different portals
    While using the preview widget, test using all the preconfigured support email addresses to ensure the Email AI Agent responds from the correct portal Knowledge Base. 
  3. Check edge cases
    Include unclear or irrelevant queries to confirm proper if the reposnes is "We couldn't generate a response"
    Example:
    “Test 123”
    “Please ignore this email."
  4. Validate linked articles
    Verify that linked articles are correct, current, and relevant to the portal.

Email AI Agents Performance Report

Email AI Agent's effectiveness can be measured using the curated Email AI Agents Performance Report available. This curated dashboard is designed to help admins and leaders track adoption, engagement, and ROI metrics over time.


Tab 1: Absolute Metrics

Understand the number of email tickets the Email AI Agent is handling, how effective its responses are, and where improvements can be made.

This tab gives you a direct view of the Email AI Agent’s contribution to support operations, from engagement volume to helpfulness and deflection trends.

Filters Available:

  • Date Range: Focus your analysis on specific weeks or months.

Key Widgets and Their Insights:

  1. Tickets Given to AI Agent 
    Track the number of email tickets passed to the Email AI Agent after satisfying trigger conditions, which is also bounded by the configured daily response limit. Use this to measure total Email AI Agent reach over the selected time range.
  2. Tickets Answered by AI Agent
    View how many of those tickets were successfully answered by the Email AI Agent. This helps you understand delivery success and configuration gaps. Choose how you want the data to be displayed - as Numbers or in Tabular format.
  3. AI Agent Answer Rate
    Analyze what percentage of tickets were answered by the Email AI Agent.
  4. AI Agent Customer Feedback
    Review how many Email AI Agent responses were rated Helpful by customers. This metric is key to evaluating the quality and impact of responses. Feedback can originate from two sources—Email, when customers rate responses directly from their inbox, and Answers, when feedback is submitted via the support portal.
  5. Ticket Deflection Trend
    Visualize how many tickets were auto-resolved without agent involvement.
  6. AI Agent Response Rate Breakdown
    Understand the distribution of Total tickets received with source as email, Tickets given to Email AI Agent, and Tickets answered by the Email AI Agent. The graph highlights areas that may need better templates or content tuning.

Tab 2: Sessions Comparison

Measure how many sessions the Email AI Agent consumes and when. This is useful for quota tracking and usage trend analysis.

Session usage is critical for licensing and planning. Each AI Agent-triggered reply counts as one session, and this tab helps monitor consumption over time.

Filters Available:

  • Date Range: Pinpoint usage spikes and surges.

Key Widgets and Their Insights:

  1. Sessions Consumed
    See the total number of Email AI Agent sessions used in the selected time period. Use this to track remaining quota or investigate overages.
  2. Sessions Consumption Trend
    View a trendline of session consumption over weeks or months. Use grouping filters (e.g., language or trigger) to diagnose usage drivers.

Use The Analytics Insights

Utilize the Email AI Agent Performance Report to proactively fine-tune your AI deployment strategy. These insights can help you:

  • Identify underperforming templates, languages, or KB articles.
  • Detect high-volume automation opportunities through deflection trends.
  • Justify ROI and AI investment by comparing engagement vs resolution.
  • Monitor whether you're reaching session thresholds earlier than expected.
Note: The performance report is available only on select Pro and Enterprise plans with Freddy Copilot enabled. Data is refreshed every 24 hours.


Troubleshooting Common Issues

Admins may occasionally encounter configuration or runtime issues. Here’s how to identify and resolve them quickly.


  1. Email AI Agent Not Responding
    Check that:
    • Email AI Agent is enabled
    • Trigger rules are valid
    • Daily limit isn’t exhausted - Admins receive an email notification when the limit is reached, and they can choose to increase the limit or disable the Email AI Agent.
    • Ticket meets the rule conditions
  2. Templates Not Saving
    Common causes:
    • Missing feedback widget
    • Unresolved placeholders
    • Offline connection
  3. Missing Languages or Placeholder Errors
    • Ensure the language is supported and enabled
    • Validate all placeholders are correctly formatted with double curly braces
  4. Email AI Agent Auto-Disabled

            Happens when:

  • Templates are deleted
  • All supported languages are disabled