At a Glance:
OpenLLMetry is an open-source observability framework built as OpenTelemetry extensions, providing standardized tracing for LLM providers, vector databases, and agent frameworks while connecting to existing observability backends like Datadog, Grafana, and Honeycomb.
Overview:
OpenLLMetry is an open-source observability solution specifically designed for LLM-powered applications. It extends OpenTelemetry with custom instrumentations that automatically trace calls to LLM providers such as OpenAI, Anthropic, and Ollama, vector databases including Pinecone and Chroma, and frameworks like LangChain and LlamaIndex. Because it outputs standard OpenTelemetry data, it connects directly to over 20 supported observability destinations without vendor lock-in. The project is maintained by Traceloop under the Apache 2.0 license and can be used either through a streamlined SDK or by adding individual instrumentations to existing OpenTelemetry setups.
Key Decision Points:
Standardized observability: Outputs OpenTelemetry-compliant data, allowing integration with existing observability stacks rather than requiring a proprietary monitoring platform.
SDK vs. direct instrumentations: Users can adopt the full SDK for quick setup or selectively add instrumentations to an already-instrumented OpenTelemetry environment.
Multi-component tracing: Covers LLM providers, vector databases, and agent frameworks, giving visibility across the entire LLM application stack.
Backend flexibility: Supports over 20 tested destinations, including Datadog, Grafana, Honeycomb, New Relic, and the OpenTelemetry Collector, without forcing a specific vendor.
Telemetry collection discontinued: As of v0.49.2, the SDK and instrumentations no longer collect or log any telemetry data.
Core Features:
LLM provider instrumentations: Automatic tracing for OpenAI, Anthropic, Bedrock, Cohere, Gemini, Mistral AI, Ollama, Vertex AI, and others.
Vector database instrumentations: Tracing support for Chroma, Pinecone, Qdrant, Weaviate, Milvus, LanceDB, and Marqo.
Framework instrumentations: Built-in support for LangChain, LlamaIndex, Haystack, CrewAI, LangGraph, LiteLLM, and OpenAI Agents.
MCP protocol support: Instrumentation for the Model Context Protocol.
Standard OpenTelemetry output: All traces are emitted as standard OTel data, compatible with any OpenTelemetry-compliant backend.
SDK with local debugging option: The SDK includes a configuration to disable batch sending for immediate trace visibility during local development.
Use Cases:
Developers building LLM applications who need to trace calls across multiple providers, vector databases, and frameworks in a unified observability pipeline.
Teams already using OpenTelemetry who want to add LLM-specific instrumentation without adopting a new observability system.
Operators debugging LLM application performance who need visibility into model API calls, embedding operations, and agent workflow execution.
Open-Source Alternative Value:
OpenLLMetry provides open-source LLM observability that integrates with existing monitoring infrastructure by extending OpenTelemetry rather than introducing a proprietary collection layer. Developers can trace their full LLM stack—providers, vector databases, and frameworks—using standardized instrumentation that outputs vendor-neutral data. This allows connection to over 20 observability backends without being locked into a single monitoring platform. The ability to use individual instrumentations directly also makes it compatible with environments that already have OpenTelemetry deployed, reducing adoption friction for teams with established observability practices.
