by Datadog
Traces, evaluates and monitors LLM apps and AI agents inside Datadog, billing only on LLM spans, with up to 40K of them free each month.
Pick it if your team already runs Datadog and wants agent traces, experiments and evals beside service metrics. Skip it if you need a self-hosted open-source tracer or more than 15 days of retention without an add-on. Only LLM calls are billed, so tool-heavy agents stay cheaper to trace.
Agent traces share one account with backend services, infrastructure and user sessions.
Datadog keeps LLM traces in the same account as its infrastructure and application monitoring, per its product page.
Main trade-off
Retention is 15 days on the public plans and pricing above 100K LLM spans is not published.
| Plan | Monthly price | What it includes |
|---|---|---|
| Free | Free | Up to 40K LLM spans a month, 15-day retention of traces and experiments, Full feature access, evals included |
| Pro | $160/mo | Up to 100K LLM spans a month, On-demand usage beyond 100K spans, Same feature set as Free |
Datadog bills per LLM span, and neither plan gates features. Free costs $0 and Pro costs $160 a month, both in US dollars. Month-to-month and annual commitments are discounted. The product page prints no rate for on-demand usage or for retention add-ons. Checked on 2026-11-22.
Datadog LLM Observability, now headlined by Datadog as Agent Observability, is a closed-source tracing and evaluation product for LLM apps and AI agents. It bills only on LLM spans, runs inside the Datadog platform, and its Python SDK auto-instruments OpenAI, LangChain, AWS Bedrock and Anthropic calls.
End-to-end tracing
Follows each request across prompts, retrieval steps, tool calls and agent decisions, with latency, token usage, retries and errors per step.
Datasets from production traces
Turns real traces into versioned datasets that experiments can be rerun against; Datadog keeps datasets for three years.
Experiments and playground
Compares prompts, models and configurations against the same data before a change reaches users.
Evaluators and annotation
Offers built-in and custom evaluators plus human review, and flags hallucinations, prompt injection attempts and PII exposure.
Sensitive data scanning
Sensitive Data Scanner is included and scales with LLM usage, so personal data in prompts can be redacted.
Correlation with the rest of Datadog
Links agent traces to backend services, infrastructure and user sessions in the same account.
| Tool | Price from | Free plan | Runs on | Rating |
|---|---|---|---|---|
| $160/mo | Yes | Web | 3.6 | |
| $200/mo | Yes | Web | -- | |
| $499/mo | Yes | Mac · Windows | 3.5 | |
| Quote | No | Web | 3.5 |
Proprietary
Datadog prices it per LLM span, meaning a single call to a model provider. Free is $0 a month with up to 40K LLM spans, and Pro is $160 a month with up to 100K. Usage past 100K is billed on demand, and retention add-ons are billed per 10K spans. The product page prints no rate for either, and annual commitments are discounted.
Yes, within a limit. The Free plan allows up to 40K LLM spans a month with 15-day retention and the full feature set, including evals. Teams that outgrow it move to Pro or on-demand billing.
An LLM span is one call to an LLM provider such as OpenAI or Anthropic. Spans for tools, workflows, agents, embeddings and retrieval are not billed. One agent run can therefore contain many spans but only a few billed ones.
Datadog lists OpenAI, Anthropic, Gemini, Vertex AI, Bedrock, LangChain, CrewAI, Pydantic, LiteLLM and Strands Agents. SDKs cover Python, Node.js and Java, and OpenTelemetry or the HTTP API handle other stacks.
Free and Pro keep traces, spans and experiment data for 15 days. Add-ons extend traces to 30, 60 or 90 days and experiments to 6, 9 or 12 months. Datasets are kept for three years, versioned for repeat runs.
Datadog LLM Observability is the AI tracing and evaluation product from Datadog, the monitoring vendor co-founded by CEO Olivier Pomel. Datadog's product page and docs now headline it as Agent Observability, and the old name still appears in the page title and the docs address. It sits in the AI agent observability tools category, where its main pull is that LLM traces live in the same account as a team's infrastructure and application monitoring.
Each request your app handles becomes a trace made of spans: an LLM call with its tokens, latency and errors, a fixed workflow of calls, or a dynamic run by an agent. The Python SDK auto-instruments frameworks and providers such as OpenAI, LangChain, AWS Bedrock and Anthropic, and Datadog's docs say this needs no code changes. CrewAI and LiteLLM are on the vendor's supported list, and any stack that can emit OpenTelemetry spans or call the HTTP API can send data too.
Beyond tracing it covers datasets built from production traces, experiments that compare prompts and models, built-in and custom evaluators, human annotation, and checks for prompt injection and sensitive data. It suits platform and ML engineers who already run Datadog and want agent traces next to service traces and user sessions. The three jobs of agent observability guide explains where tracing ends and testing begins.
It is a weaker pick for a solo developer who wants a self-hosted, open-source tracer, or for a team whose main job is evaluation rather than monitoring. Confident AI is built around evals and regression tests, Openlayer leans toward governance evidence, and Kashikoi focuses on simulating agents before release. Langfuse, MIT-licensed and free to self-host according to its pricing page is the open-source alternative for teams that must run the tracer themselves. Datadog's docs also leave its government regions out of scope. Datadog's product page lists HIPAA compliance among its controls and offers local tracing on your own machine with no signup, which is the cheapest way to test it, though Hokai has not run that trial.
Bottom line: choose it if your team already pays for Datadog; choose Confident AI if evals are the main job, or Langfuse if the tracer has to be self-hosted.
Product facts on this page (pricing, plans, integrations, compliance) are read from Datadog's own pages and documentation. The verdict is Hokai's reading of them, not the vendor's claim.