At a Glance:
Dragonfly is a high-throughput, multi-threaded in-memory data store fully compatible with Redis and Memcached APIs that demonstrates up to 25X greater throughput than Redis and 30% better memory efficiency in benchmarking.
Overview:
Dragonfly is an in-memory data store designed for modern application workloads that implements both Redis and Memcached APIs without requiring code changes. It uses a shared-nothing architecture to partition the keyspace across threads, enabling vertical scaling on multi-core instances where single-threaded Redis reaches CPU bottlenecks. Dragonfly delivers higher throughput, lower tail latency, and can operate on significantly fewer resources for equivalent workloads. It targets engineers and operators running cache and data store workloads who need Redis or Memcached compatibility with improved resource efficiency on cloud instances.
Key Decision Points:
API compatibility: Fully compatible with Redis and Memcached wire protocols, allowing drop-in replacement without application code changes.
Architecture: Multi-threaded, shared-nothing design partitions data across threads, eliminating the single-threaded CPU bottleneck present in Redis.
Deployment model: Runs as a single process via CLI or Docker, with support for common Redis configuration arguments and additional Dragonfly-specific flags.
Memory efficiency: Demonstrated 30% better memory efficiency than Redis in idle state with no visible memory increase during snapshotting in benchmarks.
Cluster support: Currently supports only emulated cluster mode, with native replication under active design.
HTTP console access: Provides a native HTTP console and Prometheus-compatible metrics endpoint on the main TCP port by default, which should be disabled if exposing the port externally.
Core Features:
Redis API compatibility: Supports approximately 185 Redis commands, covering APIs through Redis 5.0.
Memcached API compatibility: Supports all Memcached commands except CAS via a configurable port.
Multi-threaded execution: Shared-nothing architecture partitions keyspace across threads for vertical scaling on larger instances.
Adaptive cache eviction: A unified caching algorithm with zero memory overhead that evicts items when near the maxmemory limit when cache mode is enabled.
Fork-less snapshotting: Snapshotting algorithm that avoids memory duplication during backup operations.
Prometheus metrics: Exposes Prometheus-compatible metrics on the main TCP port for integration with monitoring systems like Grafana.
Cron-based automatic snapshots: Supports automatic backup snapshots scheduled via standard cron expressions with minute granularity.
Use Cases:
Developers replacing Redis in cloud workloads on multi-core instances to overcome single-threaded CPU limitations.
Operators running Memcached-compatible caching layers seeking higher write throughput and memory efficiency.
Teams deploying in-memory data stores on AWS instances who need to reduce resource usage while maintaining API compatibility.
Open-Source Alternative Value:
Dragonfly provides an open-source in-memory data store that maintains full Redis and Memcached wire protocol compatibility while leveraging a modern multi-threaded architecture. The project transparently documents its design decisions, including its shared-nothing sharding model, Dash-based hashtable structure, and VLL-inspired transactional framework, all drawn from published academic research. Users can build from source, run via Docker, and access the HTTP console and Prometheus metrics directly. Benchmarking published in the README shows performance and efficiency improvements over single-threaded Redis and Memcached on comparable AWS instances.