Same mistake, one week later: why AI agent observability isn't enough

Ticket #4812: when AI agents in production get it wrong

Monday, 9:14 AM: the CFO's office asks the company's analytics agent for last quarter's net revenue in the North region, and within seconds the agent writes the query, runs it on the data warehouse and returns a clean table with a confident summary. The number goes straight into a board deck, and it is wrong.

On Thursday someone in finance spots the discrepancy and opens a ticket, which ends up where tickets about AI agents in production usually end up: with IT. IT finds nothing broken, because the model, the query engine and the warehouse connection all work as expected.

The problem is that the warehouse holds hundreds of tables, with ambiguous names and relationships only the data owners know, and the agent joined two that look related but are not: one records orders at the time of purchase, the other at the time of invoicing. Only the people who own that data know they must never be joined and that net revenue is read from the second. That rule was never written down anywhere. The limit is not the technology. It is the context.

This is the kind of error that costs the most in enterprise AI: not a crash, but a fluent, plausible answer that quietly erodes trust. After two or three episodes like this, finance stops using the agent and goes back to emailing the data team, and the project that was meant to scale becomes yet another pilot. An article published in Harvard Business Review in June 2026 captures the point: most companies now have the same models and the same tools, and what separates those who scale their agents from those stuck in pilots is the ability to make expert judgment explicit.

Ticket #4812 will be fixed, reopened and fixed again more than once before anything really changes. To understand what the turning point is, and why it comes from where nobody is looking, it is worth following it to the end.

AI agent observability sees everything except the why

In the story of ticket #4812, the first mandatory step is AI agent observability, the industry's answer to errors like this one. And it is a good answer: the trace shows the question, the tables the agent read, the SQL it generated and the answer it returned. Together with response evaluation, it is the first layer of AI agent reliability.

But observability is a witness, not a detective: it can say exactly what happened at 9:14, but not why the agent chose those tables, or who knows which ones it should have used. Multiply ticket #4812 by a few hundred conversations a week, and the result is a backlog of production failures that someone has to analyse by hand.

Regression tests: when the alarm is not enough

The industry's standard response to the limits of observability is evaluation. LangChain, for example, argues that production failure traces are the best raw material for offline evaluation datasets, so that every error becomes a regression test; LangChain itself, with LangSmith Engine, goes further, grouping errors and proposing fixes automatically.

It is a solid practice, and every serious team should follow it. But it catches the error rather than eliminating it. Turning ticket #4812 into a test means saving the CFO's question together with the correct number and rerunning it with every new version of the agent: if the result is wrong again, the test flags it. The test, however, does not tell the agent how those tables should be read, so until someone gives it that rule, the error will be flagged every week, exactly the same. A test that fails every Monday is an alarm, not a solution.

Automatically generated fixes hit the same wall from another side. They can rework what is already in the system, refining a prompt or tuning a retrieval parameter, but they cannot add a rule that exists only in a data owner's head.

When the missing piece is know-how, the fix has to come from a person, and the real question becomes how to get it from that person to the agent, quickly and safely.

The why has a name: failure attribution

At this point ticket #4812 has been fixed, turned into a test and reopened, but nobody has yet asked why the agent gets it wrong. Finding that why is called failure attribution, and it is one of the hardest problems in managing AI agents in production. When a rule is missing, the agent does not say “I don't know”: it answers with the same confidence it uses when it hallucinates a fact, except that here there is no wrong fact to disprove, only a missing rule. And the same wrong number can come from a missing rule, a misleadingly named table, a skipped step or partial data, each requiring a different fix.

That is why attribution matters more than it seems: its real output is not a label, but a name. Fixing the wrong cause, for example tweaking a prompt when the agent was missing knowledge, brings the error back in a slightly different form; finding the right one tells you who can solve it: the policy owner for a missing rule, the process owner for a skipped step, the team that manages the integration for a wrong tool call.

Most tools today can tell whether an answer was right, while explaining why it was wrong, and who should fix it, at scale, is still frontier work.

This is where ticket #4812 changes course. Syllotips finds the issues that need an expert's judgement and traces them back to their cause, whether missing or wrong knowledge or the agent's behaviour, grouping related ones so that a recurring problem becomes a single task instead of fifty tickets. The expert corrects it once, and the approved knowledge helps the agent handle the next conversation.

Whether it is a customer service assistant, an agent that handles invoices and orders or one that queries company data, the path is the same: every error is traced back to its cause and fixed by whoever owns that rule or procedure.

Ticket #4812 is a knowledge failure: the agent reasoned correctly, but it was missing a rule a data owner would have applied without thinking. HBR describes the same pattern in customer-facing agents, which go off track when faced with a pricing exception or a request just outside policy, because nobody ever wrote down how that decision is really made.

Tacit knowledge: what nobody writes down

The most obvious solution to this kind of problem is to ask the data team to document every table, every rule and every exception, but it rarely works.

Tacit knowledge is made of the shortcuts, exceptions and intuitions experts use every day and almost never put into words, least of all when the question comes in the abstract.

And even a heroic documentation effort produces a static knowledge base, one that goes out of date the day a table is added or a business rule changes.

HBR points to a better trigger: real situations and concrete edge cases, in which experts have to decide and explain why, which is exactly the moment tacit knowledge becomes explicit.

So ticket #4812 is not just a problem, but the perfect edge case: real, specific and already documented in the trace. Errors also tend to cluster, so a single expert answer often solves a whole family of them; and when a business rule changes, the first error is precisely the signal that knowledge needs updating. Every agent error is an invitation to expert judgement, as long as the request reaches the right person.

Expert-in-the-loop feedback: sending it to the people who know

Today that invitation usually takes the long way round: finance reports the problem, IT investigates, a data engineer gets involved, someone tweaks a prompt and everyone hopes for the best. HBR frames it as an operating model problem, in which business expertise has to pass through technical teams before it reaches the agent.

Human-in-the-loop feedback works when the loop is short. In Syllotips the right expert, in this case the revenue data owner, receives a targeted task with the evidence attached: the question, the query, the tables used and the cause identified. The correction comes from the person who knows the rule, not from someone who has to guess it, and weeks of hand-offs become a few hours.

Can AI agents remember what experts teach them?

Fixing ticket #4812 is not enough: if the correction lives only in that conversation, next quarter the same question will get the same wrong number. Retraining the model is the slow road, and a hard one to verify, and the latest research, from Thinking Machines' reflections on “frozen” AI to Databricks' work on MemAlign, points to agent memory as the alternative: a memory in which expert feedback is kept and reused without changing the model.

Syllotips applies the same principle, with governance on top. Expert-approved contributions go into governed memory and are provided at runtime to the existing agent, whether it is built on Copilot Studio, LangChain, LangGraph or custom code, while the AI team keeps control of integration and deployment and the business decides what goes in, what changes and what comes out when the rules shift.

The following Monday the CFO's office asks the same question, and this time the number is right: ticket #4812 is closed for the last time, because the rule the data owners taught the agent is now part of what it knows.

This is continuous improvement for AI agents, and the most durable path to AI agent reliability: not an ever-longer test suite, but company knowledge that grows with every correction and makes each correction faster than the last.

Four questions before scaling AI agents

• Where does a correction go when the ticket is closed? If the answer is “nowhere”, the agent will never see it.
• Does the same error come back the following week? Then the system fixes, but does not improve.
• When an error repeats, does anyone know who owns the missing rule? If the answer is “IT”, attribution stopped halfway.
• Does expert know-how accumulate or evaporate? The first is an asset, the second a cost that keeps coming back.

Glossary

AI agent observability:

the ability to see what an AI agent does in production (which sources it reads, which tools it uses, which answers it gives) in order to find where it goes wrong.

AI agent reliability:

an agent's ability to give correct answers consistently, in real working conditions and not only in tests.

Failure attribution:

the analysis that traces an error back to its cause, distinguishing knowledge failures from behavioural failures.

Tacit knowledge:

the knowledge experts apply without ever having written it down: criteria, exceptions and intuitions born of experience.

Expert feedback:

the correction provided by someone who really knows the answer, in the context of the error.

Governed memory:

the memory in which Syllotips keeps expert-approved contributions, reused by the agent at runtime and controlled by the business.

Continuous improvement:

the improvement of an agent that happens while it is in use, with every correction, rather than with every new release.