THE INSTITUTE / RESEARCH TOPICS
Agent Observability
How to reconstruct what an AI agent did, judge the result and cost, and what to check before buying a tool for it.
Explore this topic
Agent Observability
These four types are sorted by one thing: where the product sits in your stack. Most can trace and evaluate. The difference is what else they can see and where the record is kept.
Dedicated agent tools
Built around model and agent runs: step-level traces, evaluation datasets and prompt versions. The deepest view of the agent itself, and the shallowest view of the databases and services around it.
Buying question: Does it instrument the frameworks and model providers you use, not only its own?
Monitoring suites
Application monitoring vendors adding agent traces beside service, database and network data. Useful when an agent failure may start in a system the agent depends on.
Buying question: Is agent tracing part of your current contract, and how is it metered?
Cloud agent platforms
Observability included in the cloud service that runs the agent. Setup can be lighter for agents already hosted there.
Buying question: What can you see for agents, tools and models that run outside this cloud?
AI gateways
A proxy between your applications and the models they call. It records each model request with tokens, cost and errors, and can cap request rates. It sees model traffic, not the agent’s internal steps or the business result.
Buying question: Which agent steps and tool calls never pass through the gateway?
Example: Cloudflare AI Gateway ↗
A deployment may combine several types. Underneath all four, the OpenTelemetry GenAI conventions are the common record format, so prefer tools that emit and accept them. Then decide which system holds the record you would show an auditor.
Choose a layer to understand the tools and buying questions.
The quick answer
Agent observability is the ability to reconstruct what an AI agent did and the information it acted on, then connect that record to checks of quality, cost and business outcome. It helps teams find failures and investigate decisions. It does not stop or reverse an action; that takes controls in the agent and in the systems it uses.
What it is
Application monitoring shows whether services are available, fast and free of errors. Agents add a different problem. A model chooses the next step, so the same request can take a different path each time, a mistake can surface several steps after it was made, and the agent can change records in other systems. Agent observability adds a record of each run and links it to checks of the result.
The Institute frames the subject as six questions about any agent run, set out under How it works. Observability answers the first four with records and checks. For the last two, whether an action was allowed and whether it can be stopped, it shows whether identity and control systems worked; it does not replace them.
Read it alongside What are AI agents? That guide explains what an agent is. This one explains how to know what it did.
Know the difference
Several kinds of product call themselves observability. They answer different questions, and buying one does not buy the others.
| Capability | Answers | Misses |
|---|---|---|
| Application monitoring | Is the service up, fast and free of errors? | Whether the agent chose the right action. |
| Model call logging | What went into and came out of each model call? | How calls, tools and handoffs combined into one decision. |
| Agent tracing | What path did the agent take, step by step? | Whether the result was correct or permitted. |
| Evaluation | Did an output meet a defined standard? | What happened on runs that were not evaluated. |
| Agent observability | Can we reconstruct a run and judge its result and cost? | Whether an action can be blocked or reversed; that needs controls. |
A quick test: if a trace shows every tool call succeeded, can you still tell whether the customer received the right refund? If not, you have tracing without evaluation.
How it works
The six questions are a coverage check, not a build order. To investigate a finished run, work through them as listed. Before a deployment, settle questions five and six first: decide what the agent may do, and how to stop it, before it can act.
| Question | What to record | Typical owner |
|---|---|---|
| 1. What happened? | Each step, model call, tool call and handoff, with inputs, outputs, timing, token use, and the model and prompt version. | Engineering |
| 2. What shaped the action? | The instructions, retrieved data, tool results and policy rules present when the agent acted. | Engineering and the process owner |
| 3. Was it right? | A check against the agreed acceptance standard: the resulting system state, a business rule, a reviewer’s decision or an automated evaluation score. | Business owner |
| 4. What did it cost? | Model and tool charges from the trace, plus infrastructure, retries, failed attempts and review time, totalled per business case. | Finance and operations |
| 5. Was it allowed? | The identity the agent acted under, the permissions it used, the data it touched and any policy exceptions. | Security and compliance |
| 6. Can we stop it? | Alerts, approval gates, a way to pause the agent and a record of what each action changed. Undoing a change needs a recovery step in the affected system. | Operations |
What tracing covers. Traces supply question one, most of question two and part of question four, since they carry timing and token use. Question three needs an acceptance check from outside the trace. For questions five and six, the trace is evidence that identity and control systems worked, not a substitute for them.
A trace is not an explanation. It shows what the agent was given and what it did. Any reasoning the model reports is evidence to check, not proof of what drove the action. Anthropic’s research found that reasoning models often leave out factors that influenced their answers.
Follow the business case, not the run. One order, claim or ticket can involve several runs, retries and human handoffs, and one run can touch several records. Put the business case identifier on every run, trace, action and approval, so the full history can be retrieved and cost totalled per case rather than per run.
Types of tools
These four types are sorted by one thing: where the product sits in your stack. Most can trace and evaluate. The difference is what else they can see and where the record is kept.
Dedicated agent tools
Built around model and agent runs: step-level traces, evaluation datasets and prompt versions. The deepest view of the agent itself, and the shallowest view of the databases and services around it.
Buying question: Does it instrument the frameworks and model providers you use, not only its own?
Monitoring suites
Application monitoring vendors adding agent traces beside service, database and network data. Useful when an agent failure may start in a system the agent depends on.
Buying question: Is agent tracing part of your current contract, and how is it metered?
Cloud agent platforms
Observability included in the cloud service that runs the agent. Setup can be lighter for agents already hosted there.
Buying question: What can you see for agents, tools and models that run outside this cloud?
AI gateways
A proxy between your applications and the models they call. It records each model request with tokens, cost and errors, and can cap request rates. It sees model traffic, not the agent’s internal steps or the business result.
Buying question: Which agent steps and tool calls never pass through the gateway?
Example: Cloudflare AI Gateway ↗
A deployment may combine several types. Underneath all four, the OpenTelemetry GenAI conventions are the common record format, so prefer tools that emit and accept them. Then decide which system holds the record you would show an auditor.
Platforms and tools
Examples of each type, from vendor documentation checked October 24, 2026. This is a map of roles, not a ranking, and none were independently tested. Every card gives the same three things: role, deployment and the main limitation to investigate.
LangSmith
Dedicated agent tool
Traces, evaluation datasets and monitoring from the LangChain team, with integrations for many frameworks.
Deployment: Cloud, hybrid or self-hosted.
Investigate: Which plan includes self-hosting, and trace depth for agents not built with LangChain.
Langfuse
Dedicated agent tool
Tracing, evaluation and prompt management. MIT-licensed core; owned by ClickHouse since January 2026.
Deployment: Cloud or self-hosted.
Investigate: Which self-hosted features need a commercial license key.
Arize Phoenix
Dedicated agent tool
Tracing and evaluation built on OpenTelemetry, with experiment comparison.
Deployment: Self-hosted.
Investigate: Its Elastic License 2.0 is source-available, not open source; check the terms.
Datadog Agent Observability
Monitoring suite
Agent traces, evaluations, and cost and latency dashboards beside Datadog application monitoring.
Deployment: Datadog’s hosted service.
Investigate: Billed per LLM span ingested; estimate volume at production scale.
Amazon Bedrock AgentCore Observability
Cloud agent platform
Session, latency, token and error metrics, with traces and logs in CloudWatch in an OpenTelemetry compatible format.
Deployment: AWS.
Investigate: Coverage outside AgentCore; evaluation is not part of the observability documentation.
Microsoft Foundry
Cloud agent platform
OpenTelemetry tracing with token use, latency and retries, stored in Azure Monitor Application Insights.
Deployment: Azure.
Investigate: Generally available for prompt and hosted agents; workflow and external agents are in preview.
Google Cloud Agent Platform
Cloud agent platform
Metrics, traces and logs in OpenTelemetry format, with online quality and safety monitors.
Deployment: Google Cloud.
Investigate: Documented for agents on Google’s Agent Runtime; confirm coverage elsewhere and each feature’s launch stage.
Cloudflare AI Gateway
AI gateway
Logs model requests, tokens, cost and errors, and adds caching and rate limits.
Deployment: Cloudflare’s network.
Investigate: Documented for model traffic; confirm whether tool calls and agent steps are covered.
Business uses
Pick one agent and one business case, then decide what you need to see before choosing a tool. Agree the acceptance standard, and when it will be judged, before the agent runs. These are evaluation scenarios with illustrative thresholds, not measured customer results.
Consumer goods replenishment
An agent proposes store orders when demand is unusual, such as a local event the forecast does not cover. Routine orders stay with the fixed replenishment rule.
Agent decides: Whether the forecast needs adjusting, and by how much.
Accepted when: No stockout and under 5% of the extra stock unsold seven days after delivery.
Record: Data and rules used, quantity, approval path, and every run and retry for the order.
Stop point: Orders above 100 cases need a buyer’s approval; any order can be cancelled until the truck is dispatched.
Customer service
An agent resolves refund and account requests.
Agent decides: Which policy applies and which records to check.
Accepted when: The account matches the approved resolution, and the customer does not contact you again on the same issue within 14 days.
Record: Policy applied, systems changed, amount refunded.
Stop point: Refunds above a set amount and policy exceptions go to a person.
Finance operations
An agent matches invoices to purchase orders and proposes payment.
Agent decides: How to resolve differences between the invoice and the purchase order.
Accepted when: No duplicate or incorrect payment appears at the next reconciliation.
Record: Documents compared, exceptions found, proposed action.
Stop point: Any release of payment needs a person.
IT operations
An agent investigates an alert and proposes or applies a fix.
Agent decides: Which evidence to gather and which fix to propose.
Accepted when: The incident does not recur within the agreed window and the change is not rolled back.
Record: Evidence gathered, change applied, identity used.
Stop point: Production changes outside a pre-approved runbook need approval.
Benefits and costs
What it buys. Faster diagnosis when an agent fails, evidence for audits and disputes, earlier detection of quality drift, and the records needed to calculate what an agent actually costs.
Cost per accepted outcome
All costs of the evaluated workflow, including failed attempts ÷ business cases that meet the acceptance standard.
Observability links each run to the acceptance check and to cost records, so cost can be counted per accepted case rather than per run. The acceptance check itself comes from the business. Without one you can count attempts, not accepted outcomes.
What it costs. Telemetry ingestion and storage, which grow with every step an agent takes. Evaluation compute, especially when one model grades another. Reviewer time for sampled human checks. The work of redacting sensitive data and enforcing retention. Price tools against your expected trace volume and retention period, not a demonstration workload.
Set the cost boundary before you measure: which charges count, and whether failed attempts, retries and review time are included. This is an evaluation approach, not a claim of measured savings.
Limits and risks
A complete trace is not a correct result. Every tool call can succeed while the business outcome is wrong. Correctness needs a check against a resulting state, a rule or a reviewer.
Traces hold sensitive data. Prompts, tool arguments and outputs can contain customer and employee information, as Microsoft’s tracing documentation notes. Decide what to capture, redact and retain before turning tracing on in production.
Sampling hides rare failures. Evaluating a small share of runs keeps cost down but can miss infrequent, expensive errors. Pair sampling with rules that always capture actions above a value or risk threshold.
Traces break at boundaries. When an agent calls another vendor’s agent or tool, the record can stop at your edge. Ask whether trace context passes across those calls.
Retention can become a legal duty. Under the EU AI Act, high-risk systems must allow automatic logging over their lifetime (Article 12). Providers (Article 19) and deployers (Article 26) must keep the logs under their control for a period suited to the system’s purpose, of at least six months unless other EU or national law, in particular data protection law, provides otherwise. The Digital Omnibus moved the application date for stand-alone high-risk systems to December 2, 2027, and for high-risk AI in regulated products to August 2, 2028. Confirm classification and dates with counsel.
Seeing is not stopping. An alert that arrives after an order ships is a record, not a control. Decide which actions need a gate before they run. Recording what an action changed does not create a way to undo it; recovery needs its own step in the affected system.
What the evidence shows
External research · LangChain
Most teams trace their agents. Fewer evaluate live traffic.
In LangChain’s State of Agent Engineering survey, most respondents reported observability and step-level tracing. Stricter evaluation was less common, especially of live traffic.
Among teams with agents in production, the gap narrows but remains: 94% reported observability, 71.5% step-level tracing and 44.8% online evaluation, while 22.8% reported no evaluation. LangChain does not give the size of that group.
Read the survey and its method ↗
Read the conditions. Public survey of 1,340 respondents, fielded November 18 to December 2, 2025. 63% worked in technology and 49% at organizations with fewer than 100 people. Respondents could report several practices, so the shares overlap. The survey measures whether teams use a practice, not how often or how well. LangChain sells an observability product, so its users may be overrepresented. Treat the figures as directional, not as an enterprise benchmark.
Source status. Report page dated June 12, 2026.
How we assess it
This page is a source-based introduction, not a vendor ranking or performance study. The six questions are an Institute framework drawn from the linked standards, regulation and product documentation. Product descriptions come from vendor documentation checked October 24, 2026; none were independently tested.
To evaluate tools for your own agents, test each one with the integrations you would actually deploy, not the tool alone. Use preserved past runs that include known failures: missing context, an unauthorized action, a tool call that succeeded with a wrong result, a retry that repeated an action and an interrupted run. If you re-run anything, use an isolated test environment or simulated external actions.
For each of the six questions, record whether the setup answers it directly, retrieves the answer from another system or leaves it to manual investigation. Then measure evidence completeness, whether reviewers reached the right conclusion, investigation time, integration effort, whether intended gates actually stopped an action, and operating cost at your expected volume and retention.
Follow the methodology attached to each Institute report for that report’s actual scope and findings.
Questions before you buy
For the executive
- Which agent decisions would we need to explain to a customer, auditor or regulator?
- Which agents may act without human approval today, and who agreed to that?
- Do we know the cost per accepted outcome, or only the cost per run?
For the business owner
- What result counts as accepted, and when is it judged?
- Which errors cost the most, and are those runs always captured rather than sampled?
- Which actions can be paused, cancelled or recovered, and who is allowed to do it?
For the implementing engineer
- Can one business case identifier link every run, model call, tool call and approval behind it?
- Which version of the OpenTelemetry GenAI conventions does the tool emit and accept, and can we export traces if we change vendors?
- What happens to the trace when the agent calls an external agent or tool?
For risk and compliance
- Under which identity does each agent act, and is it separate from the person it serves?
- Where are traces stored, who can read them and how long are they kept?
- How is personal data redacted, and can a record be altered after the fact?
Engineering can show what happened. The business owner defines a right result, risk decides what must be provable and the executive decides what an agent may do alone. Choose a tool against all four.
Common questions
Is agent observability the same as LLM observability?
No. LLM observability usually means logging individual model calls. Agent observability follows a whole run across many calls, tools and handoffs, and links it to outcome checks, identity records and cost. It does not include the controls themselves.
We already have an application monitoring tool. Do we need anything else?
Possibly not for infrastructure health. You still need step-level agent traces, an acceptance check for results and a record of the identity each agent acted under. Some monitoring vendors now add these; check which are included in your edition.
Should we log every prompt and response?
Not by default. Capture what you need to answer the six questions, redact sensitive fields and set retention deliberately. Full content capture raises storage cost and privacy exposure.
Who should own it?
Engineering usually runs the tooling. The business owner defines what a correct result is, and risk defines what must be kept and provable. Without all three, the tool gets bought and the questions stay unanswered.
Can observability make an agent reliable?
No. It shows where an agent fails and how often. Reliability comes from changing the agent, its tools, its permissions or the process around it, then measuring again.
Sources and reading
- LangChain: State of Agent Engineering Survey of 1,340 respondents, November to December 2025. A vendor survey; read the conditions under Evidence.
- OpenTelemetry: GenAI semantic conventions Common telemetry names for agent, model, tool and MCP operations. Still evolving; check the version in use.
- Anthropic: Reasoning models don’t always say what they think Evidence that a model’s reported reasoning can omit what influenced its answer. Published April 3, 2025.
- EU AI Act Service Desk: Article 12, Record-keeping Logging capabilities required of high-risk systems. Retention duties are in Articles 19 and 26, linked under Limits and risks.
- Regulation (EU) 2026/1744, the Digital Omnibus on AI Revised application dates for high-risk obligations.
Product documentation is linked in the product cards and was checked October 24, 2026. Documentation describes a provider’s offering; it is not independent proof of performance. The business examples are evaluation scenarios with illustrative thresholds, not case studies. Regulatory notes are general information, not legal advice.
What changed
October 24, 2026, revision: Narrowed the definition to separate observability from controls. Reframed the second question around the information that shaped an action. Corrected the survey headline. Sorted product types by one principle and added AI gateways and Google Cloud. Gave every product card the same fields. Added acceptance standards to the business cases. Tightened the EU retention note. Redrew the worked example for phone screens.
October 24, 2026: First version.