At a Glance:
AutoMQ is a stateless Apache Kafka alternative that stores data directly on S3 or S3-compatible object storage, offering cost-effective scaling in seconds and zero cross-AZ traffic costs by replacing the traditional disk-based storage engine with a cloud-native shared storage architecture.
Overview:
AutoMQ is a fork of Apache Kafka that has been redesigned as a diskless, stateless streaming platform by replacing the local disk storage layer with a cloud-native engine that writes directly to S3. It addresses two core limitations of standard Kafka: the complex, slow scaling of stateful brokers and the high cloud costs associated with EBS volumes and cross-AZ data transfer. The project is built for users running streaming workloads on AWS, GCP, or any environment with S3-compatible storage, and it maintains full client compatibility with the Apache Kafka protocol.
Key Decision Points:
Stateless broker architecture: Brokers are stateless, so the computing layer can scale in or out in seconds to match workloads without manual data movement or partition reassignment.
Direct-to-S3 storage: The primary storage is cloud object storage instead of local disks, which eliminates management of disk capacity and enables a shared storage architecture.
Zero cross-AZ traffic: A rack-aware router provides clients with partition metadata for their own Availability Zone, and data is exchanged through object storage rather than direct broker-to-broker replication, removing the cross-AZ traffic fees typical on AWS and GCP.
Auto-balancing: A built-in auto balancer component handles partition and traffic scheduling between brokers, removing the need for external tools or manual reassignment operations.
Enterprise latency trade-off: The open-source version runs directly on S3 with hundreds of milliseconds of latency; single-digit millisecond latency is only available in the enterprise edition.
Core Features:
S3Stream storage library: A shared streaming storage library that combines write-ahead logs, LogCache, and BlockCache to optimize frequent writes and reduce S3 API costs while improving read performance.
S3 Storage Adapter: Reimplements UnifiedLog, LocalLog, and LogSegment classes to create logs directly on S3, while still supporting traditional local disk storage if required.
Auto balancer: Automatically monitors and balances partitions and network traffic among brokers, removing the need for manual partition reassignment or external cruise control.
Rack-aware router: Provides availability zone-specific partition metadata to clients, avoiding cross-AZ traffic fees by routing data exchanges through the shared object storage layer.
Built-in metrics export: Natively exports Prometheus and OpenTelemetry metrics in both push and pull modes, replacing traditional JMX-based cluster monitoring.
Table Topic: A stream-and-table unifying feature that currently supports Apache Iceberg, integrating with AWS Glue, HMS, and Rest catalog services, including native support for AWS S3 tables.
Use Cases:
Cloud-based streaming platforms that need to avoid high EBS storage costs and cross-AZ data transfer fees on AWS or GCP.
Data engineers who require Kafka-compatible streaming with the ability to scale broker capacity in seconds to handle variable workloads.
Teams using S3-compatible storage like MinIO who want to run a Kafka-compatible stream layer without managing local disk capacity.
Open-Source Alternative Value:
AutoMQ provides an open-source, Apache 2.0-licensed alternative to Apache Kafka that replaces the stateful, disk-dependent broker model with a stateless architecture built on S3. For users already running object storage, it offers a way to operate a Kafka-compatible streaming system without manual partition reassignment, with built-in auto-balancing, and with a storage layer that scales independently of compute. The project reduces operational complexity by removing cruise control dependencies and replacing JMX monitoring with native Prometheus and OpenTelemetry metrics export.