SlateDB

At a Glance:

SlateDB is an embedded log-structured merge-tree storage engine that writes directly to object storage including S3, GCS, and MinIO, supporting bottomless capacity and configurable durability with official bindings for Rust, Go, Java, Python, and TypeScript.

Overview:

SlateDB is an embedded key-value storage engine built on a log-structured merge-tree architecture. Unlike traditional LSM-tree engines that rely on local disk, SlateDB persists string-sorted tables directly to object storage such as S3, GCS, Azure Blob Storage, and MinIO. This design provides bottomless storage capacity and simplifies replication through object storage's native durability guarantees. The engine batches writes into periodic memtable flushes to manage object storage API costs, while read performance is addressed through block caches, bloom filters, compression, and optional local disk caches. Developers embed SlateDB through its Rust crate or official language bindings, controlling durability behavior on a per-write basis.

Key Decision Points:

• Object storage as persistence layer: All data writes to remote object storage rather than local disk, trading higher latency for bottomless capacity and built-in replication through the underlying storage service.

• Configurable write durability: The standard put() call resolves when data is durably persisted to object storage, while put_with_options with await_durable set to false offers lower latency at the cost of some durability guarantees.

• Embeddable with multi-language bindings: The core engine is a Rust crate, with official bindings available for Go, Java, Node.js/TypeScript, and Python, plus community bindings for .NET and Ruby.

• LSM-tree caching for read performance: Mitigates object storage read costs through in-memory block caches, bloom filters, compression, and optional local SSD disk caches.

• Forward/backward storage compatibility: Adjacent versions guarantee compatibility for storage formats, though compile-time API compatibility is not currently guaranteed across releases.

Core Features:

• Basic CRUD operations: Provides get, put, and delete operations against a key-value data model.

• Range queries: Supports range-based key queries for scanning ordered data.

• Memtable batching with periodic SST flushes: Accumulates writes in memory and flushes sorted string tables to object storage at configurable intervals to reduce PUT API costs.

• Compactions: Implements compaction to reorganize and merge SSTs, managing read amplification and reclaiming space from overwritten or deleted keys.

• Transactions and merge operator: Supports transactional operations and user-defined merge operators for combining values under the same key.

• Change data capture: Emits change events, enabling consumers to react to data modifications as they occur.

Use Cases:

• Developers building applications that need an embeddable key-value store with effectively unlimited storage capacity backed by cloud object storage.

• Projects that already use S3, GCS, or MinIO and want to avoid managing separate storage volumes or local disk provisioning for their database layer.

• Rust applications that benefit from a native embedded storage engine with the option to expose data access through Go, Java, Python, or TypeScript bindings.

Open-Source Alternative Value:

SlateDB provides an open-source embedded storage engine that shifts persistence from local disk to object storage, allowing projects to scale storage independently without provisioning or managing disk resources. The engine is available as a Rust crate with multiple official language bindings, and its adoption by organizations including Dropbox, Gadget, and Tensorlake indicates real-world use. Developers can inspect the full source, integrate the engine directly into their process, and configure durability behavior per write operation based on their latency and consistency requirements.