01What the AQuA AI agent is and how it works

The AQuA AI agent is a free, open source tool from Google that watches your other AI. AQuA stands for Ambient Quality Agent, and it reads every conversation your agent has with customers, finds the mistakes nobody noticed and tells you the exact line in your agent's instructions to fix.

The problem it solves is a quiet one. A confident wrong answer looks exactly like a right one. In one demo conversation, an employee asks an agent to file an expense and the agent says it is done.

But it made up the employee ID when it should have asked. Nothing crashed, so no alarm went off.

AQuA works in five steps:

  1. Read: it grabs up to a thousand recent conversations.
  2. Check: it checks each one against a nine-point checklist of common mistakes, plus goals you write in plain English.
  3. Group: it puts together the conversations that went wrong in the same way.
  4. Double check: a separate AI with a fresh memory rereads the real conversations and throws out any problem that does not hold up.
  5. Track: it keeps score of what is new, what keeps coming back and what you have fixed.

So you do not get a thousand alerts. A thousand conversations become a handful of problems.

02Main features of Google's AQuA agent

AQuA is in Google's ADK recipes on GitHub. It is built for agents made with Google's Agent Development Kit, running on Google Cloud. Julian Goldie ran it on a demo agent for this episode, and these are the features that stood out:

  • It sits off to the side. It never gets in the way of a customer, and nothing in your agent changes.
  • It never edits your agent on its own. It shows you the file, the exact line and the fix it would make. You approve it.
  • It explains each problem in plain English. You see what should have happened and what actually happened.
  • You can replay conversations. Under each problem are the real conversations, and you can step through any of them.
  • It finds root causes. It goes into your agent's own instructions to find out why the mistake happened.
  • It shows you a timeline. Each bar is one day of conversations, so you can see which day things broke.

Google calls this the outer loop. Before launch, teams test against a set of questions and get the agent to about 80% pretty fast. After launch, real customers ask things nobody tested, the model gets upgraded and a tool changes.

AQuA watches after launch, and whatever it finds can go back into your tests.

03How to get started with AQuA

Your path depends on where your agent lives. There are three ways to begin:

  • Just trying it: there is a demo mode that runs on your laptop, with no cloud account needed. That is what Julian used.
  • Agent on Google Cloud with Google's ADK: your developer adds it with a few commands. Nothing in your agent changes, and you just read the report.
  • Any other chatbot: AQuA will not plug into a website widget, a booking tool or a no-code builder, but the idea still works.

Here is the version to use if you are not on Google Cloud. It is all plain English:

  1. Export your last 100 chatbot conversations. Most tools let you download them.
  2. Give them to Claude along with your goals in plain English, such as never making up a name, number or date.
  3. Ask it to group the failures by type and count them.
  4. Open a brand new chat, so the AI has no memory of the first one. Ask it to check each problem against the real conversations and throw out any that do not hold up.

That last step is the bit that stops the AI from inventing problems. Then fix what is left and run it again next week to track what changed.

04Walking through the AQuA AI agent demo

The demo is a company travel and expense agent that books trips and files expenses. AQuA read a month of its conversations and found 16 problems on its own. Nobody had to read a single chat.

The latest check passed 69%, so roughly one in three conversations had something wrong. None of it showed up as an error. It was just wrong.

Three of the examples show what this looks like:

  • The double booking: the employee said yes, and the agent sent a second identical booking.
  • The rounding bug: an employee uploaded a receipt, and the agent rounded the amount before filing it. It was a tiny change, and it happened every single time.
  • The dropped destination: the agent lost the destination when it moved from answering a policy question to making a booking.

The rounding bug is the most useful. AQuA found out why it happened. The rule about amounts was written like a suggestion, so the AI treated it as optional.

When the same conversation was replayed with the fixed rule, the agent did the right thing at the point where it used to go wrong.

This is the kind of mistake you would normally hear about from an angry customer, or from finance three weeks later.

05Results from the morning report and Google's own test

The morning report is the screen to check each day if you own an agent. It shows what broke overnight, why it broke and which line to fix.

In the demo, failures sat around a hundred a day or less for about two weeks. Then the last two days jumped to 194 and then 293. Something had changed.

The worst run had a 50% pass rate, and AQuA named the cause, so you know which day it broke and exactly what broke.

The bigger question is whether fixing what it finds really works. Google tested it on one of their own travel agents with 32 test conversations. AQuA came back with six verified problems.

They fixed the lines it pointed to and ran the same conversations again.

  • Biggest problem: down from 15 conversations to two.
  • Fully clean conversations: up from five to 13.

That is more than double the perfect conversations, just by fixing a few lines of instructions. There was no new model and no rebuild. Six lines of instructions made that difference.

06The must test and other tips to avoid silent failures

The biggest lesson from this episode works even if you never install AQuA. Julian calls it the must test, and it takes about five minutes.

Go into your chatbot's instructions and look for soft words. These read as optional to an AI:

  • Try to
  • Should
  • Ideally
  • Where possible

If something has to happen every time, write it as a rule using always, never or must. Then tell the AI what to do instead. One soft word in your instructions can quietly break your AI for weeks.

A few more tips:

  • Write your own goals. Good examples are never confirming anything you have not checked, never making up a name, number or date, and never promising something that is not in your policy.
  • Always use a fresh reviewer. A second AI with no memory of the first run is what keeps made-up problems out of your report.
  • Fix the top three first. Then run the check again next week and watch the clean conversations go up.
  • Know the limit. AQuA itself only plugs into agents built with Google's ADK on Google Cloud.

07Who the AQuA AI agent is for and why it matters

The AQuA AI agent is for teams with an agent on Google Cloud, but the idea helps anyone running a chatbot. Silent failures hide in every kind of agent, and each one looks fine from far away:

  • Support agent: offering a refund it is not allowed to give.
  • HR agent: quoting last year's holiday policy with total confidence.
  • Sales agent: pitching an offer that ended last month.
  • Booking agent: telling two customers they are confirmed for one slot.
  • Lead agent: writing a lovely message to Sarah, whose name is Sophie.

None of these throws an error. Usually the first you hear about it is an angry customer. With a check like this, you hear about it the next morning, before it spreads.

You also do not need to be technical. On Google Cloud, your developer does the setup and you read the report. Anywhere else, the steps are plain English.

If your agent hardly gets any complaints, that does not prove it is fine. Most people do not complain. They just leave.

Complaints are the problems you heard about, and this shows you the ones you did not.