Overview:
AutoMQ is a stateless fork of Apache Kafka that replaces local disk storage with S3 or any S3-compatible object storage. It addresses two core issues with traditional Kafka: the high cost of cloud hosting and the difficulty of scaling stateful brokers. By leveraging a shared storage architecture and an auto-balancer, AutoMQ reduces cross-AZ traffic costs and enables scaling in seconds. It is positioned as a cost-effective and operationally efficient Kafka alternative for organizations running streaming workloads in the cloud or on-premise, provided S3-compatible storage is available.
Core Features:
Cost-effective object storage engine: Replaces Kafka's local disk storage with S3, designed to cut cloud billing by up to 90%.
Auto scaling in seconds: The stateless broker layer can scale in or out within seconds, moving toward a pay-as-you-go model.
Built-in Auto Balancer: Automatically schedules partitions and network traffic between brokers, eliminating the need for manual partition reassignment or external tools.
Zero cross-AZ traffic: Uses cloud object storage as the primary storage to eliminate cross-AZ traffic costs on AWS and GCP for producer, consumer, and replication traffic.
100% Kafka compatible: A fork of Apache Kafka, supporting all standard Kafka features and protocols.
Built-in Metrics Export: Natively exports Prometheus and OpenTelemetry metrics via both push and pull modes, replacing legacy JMX-based monitoring.
Use Cases:
Cloud-hosted data streaming: Teams looking to reduce Kafka costs on AWS or GCP, particularly where cross-AZ data transfer is a significant expense.
Rapidly scaling workloads: Developers operating streaming pipelines that require frequent scaling up or down, where stateful broker rebalancing is a bottleneck.
Replacing self-managed or managed Kafka clusters: Organizations using Apache Kafka or services like AWS MSK that want to simplify cluster operations and storage management via object storage.
High-durability log storage: Use cases requiring 11 nines (99.999999999%) durability and zero RPO by leveraging S3 as the durable backend.
Why It Matters:
AutoMQ modifies Kafka's core architecture by substituting its local disk storage with a shared object storage layer. This change makes brokers stateless, allowing the system to scale in seconds and reduce cross-AZ traffic costs. The built-in auto-balancer removes the need for manual partition management or separate tools like Cruise Control. For teams that need a Kafka-compatible streaming platform in cloud environments, AutoMQ offers a transparent, self-hostable alternative that directly addresses the high cost and operational complexity of stateful Kafka clusters.