At a Glance:
Quickwit is an open-source cloud-native search engine for observability data (logs, traces) that provides sub-second search on object storage like AWS S3, an Elasticsearch-compatible API, and native OTLP ingestion, positioned as an alternative to Datadog, Elasticsearch, Loki, and Tempo.
Overview:
Quickwit is a distributed search engine purpose-built for log management and distributed tracing on cloud storage. It decouples compute from storage using stateless indexers and searchers to deliver sub-second full-text search and aggregation queries directly against Amazon S3, Azure Blob Storage, or Google Cloud Storage. The project provides an Elasticsearch-compatible API for ingestion and querying, native Jaeger and OpenTelemetry support for trace analytics, and a Grafana data source plugin, targeting platform engineers and developers who manage observability pipelines with object storage infrastructure.
Key Decision Points:
Elasticsearch API compatibility: Supports a subset of Elasticsearch endpoints and query DSL, allowing existing log shippers like Vector or Fluent Bit to send data to Quickwit without changing ingestion pipelines.
Stateless architecture: Indexers and searchers operate as stateless components with compute and storage decoupled, enabling independent scaling on Kubernetes via the official Helm chart.
Cloud storage as primary storage: Index data structures are optimized for direct search on object storage (AWS S3, Azure Blob, GCS), not local block storage or filesystem-based indices.
High availability limitations: HA search is available, but HA indexing requires a Kafka source and is not supported with other ingestion methods.
Core Features:
Full-text search and aggregation queries: Sub-second search performance on object storage with support for Elasticsearch-compatible aggregations.
Elasticsearch-compatible ingest and search API: Accepts data from existing Elasticsearch/OpenSearch clients and log shippers like Vector, Fluent Bit, and Syslog.
Jaeger-native distributed tracing: Provides backend storage for Jaeger trace data and supports trace analytics with Grafana.
OTEL-native ingestion: Accepts logs and traces directly via OpenTelemetry Protocol (OTLP).
Kafka / Kinesis / Pulsar native ingestion: Supports reading data directly from streaming platforms for production data pipelines.
Multi-tenancy with retention policies: Indexes support partitioning, lifecycle management through retention rules, and delete tasks for GDPR compliance.
Use Cases:
Platform engineers migrating observability workloads from Elasticsearch to a stateless, cloud-storage-native backend for log management and distributed tracing.
Developers integrating trace analytics into existing Grafana instances with Quickwit as the Jaeger storage backend and data source.
Operators running observability infrastructure on Kubernetes who need an Elasticsearch-compatible ingestion API without managing stateful index nodes.
Teams ingesting data from streaming platforms like Kafka or Amazon Kinesis into a searchable log and trace store on cloud object storage.
Open-Source Alternative Value:
Quickwit is licensed under Apache 2.0 and offers a stateless search engine alternative that lands directly on object storage infrastructure rather than requiring locally attached storage or managed block volumes. Its Elasticsearch-compatible API means teams can evaluate replacing existing Elasticsearch, Loki, or Tempo backends without immediately changing log shipper configurations or query patterns. The project's architecture choices around decoupled compute and native cloud storage optimization, combined with native Jaeger and OTLP integration, make it a targeted option for observability workloads where storage cost and horizontal scaling on Kubernetes are primary evaluation criteria.
