At a Glance:
Elementary OSS is an open-source CLI tool for dbt-native data observability that generates data observability reports, surfaces anomalies and failed tests, and sends alerts to Slack and Microsoft Teams based on metadata collected by the Elementary dbt package.
Overview:
Elementary OSS is an open-source command-line tool designed for dbt-native data observability. It connects to a data warehouse to read metadata, artifacts, and test results gathered by the companion Elementary dbt package. Using this information, the tool generates a basic data observability report highlighting anomalies and failed tests, tracks model and test performance trends over time, and sends alerts to Slack and Microsoft Teams. It functions as a configuration-as-code solution, where observability settings are managed directly within dbt project code. The project is targeted at data teams that use dbt and need to monitor data quality, model performance, and pipeline execution without relying on a separate SaaS platform, providing a foundation that can be extended to the Elementary Cloud for additional enterprise-scale capabilities.
Key Decision Points:
CLI-Based Operation: The tool operates as a command-line interface, requiring direct interaction with the terminal and integration into workflows that can trigger CLI processes.
dbt-Native Architecture: Observability configuration is managed as code within dbt projects, making it a natural fit for teams already managing their data transformations with dbt and who prefer infrastructure-as-code practices.
Alerting Channels: Alerts can be sent to Slack and Microsoft Teams, with support for custom channels and owner tagging, which defines the scope of incident notification integration.
Reliance on dbt Artifacts: The tool depends on the Elementary dbt package to collect metadata and test results during dbt runs, meaning it must be integrated into the dbt execution pipeline to function.
Observability Dashboard: A single interface is provided for viewing data monitoring results, test results, and model performance over time, consolidating various data reliability signals.
Core Features:
Anomaly Detection Tests: Collects data quality metrics and detects anomalies using native dbt tests.
Automated Monitors: Provides out-of-the-box monitors to detect freshness, volume, and schema issues in data.
End-to-End Data Lineage: Enriches data lineage with the latest test results to enable impact and root cause analysis of data issues.
Data Quality Dashboard: Delivers a single interface for viewing all data monitoring results, test outcomes, and model performance trends.
Configuration-as-Code: Manages all Elementary observability settings directly within the dbt project code.
Alerts to Slack and Teams: Sends actionable alerts with support for custom channels and tagging of data owners.
Use Cases:
Data Engineers: Monitoring dbt model runs, test failures, and performance over time from the command line and receiving failure alerts in Slack or Teams.
dbt Users: Embedding anomaly detection tests and metadata collection into an existing dbt workflow to generate an observability report and track data quality trends.
Open-Source Alternative Value:
Elementary OSS provides a code-based, locally executable data observability layer that integrates directly with an existing dbt setup. The open-source CLI, in combination with the Elementary dbt package, allows users to generate observability reports, run anomaly detection tests, and set up alerting without depending on an external monitoring SaaS platform. The configuration resides within the user's dbt codebase, giving developers a way to manage data quality and pipeline monitoring through version-controlled code. The tool's reports and alerts are generated from metadata already available in the user's warehouse, making it a self-contained extension of the dbt workflow.