datafusion

Scalable datastore for metrics, events, and real-time analytics

🚀2.3x faster than MinIO for 4KB object payloads. RustFS is an open-source, S3-compatible high-performance object storage system supporting migration and coexistence with other S3-compatible platforms such as MinIO and Ceph.

Production-grade Rust-native trading engine with deterministic event-driven architecture

📊 Cube Core is open-source semantic layer for AI, BI and embedded analytics

Open-source developer platform to power your entire infra and turn scripts into webhooks, workflows and UIs. Fastest workflow engine (13x vs Airflow). Open-source alternative to Retool and Temporal.

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

Cloud-native search engine for observability. An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.

Visualize, query, and stream to train on multimodal robotics data.

Developer-friendly OSS embedded retrieval library for multimodal AI. Search More; Manage Less.

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

Simple, Elastic-quality search for Postgres

Apache DataFusion SQL Query Engine

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

The open-source Observability 2.0 database. One engine for metrics, logs, and traces — replacing Prometheus, Loki & ES.

Space and Time | Proof of SQL

Readyset is a MySQL and Postgres wire-compatible caching layer that sits in front of existing databases to speed up queries and horizontally scale read throughput. Under the hood, ReadySet caches the results of cached select statements and incrementally updates these results over time as the underlying data changes.

One SQL interface over APIs, files, and live sources — built for agents.

Distributed stream processing engine in Rust

Restate is the platform for building resilient applications that tolerate all infrastructure faults w/o the need for a PhD.

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

A native Rust library for Delta Lake, with bindings into Python

An extensible, state-of-the-art framework for columnar compression, and the fastest FOSS columnar file format. Formerly at @spiraldb, now an Incubation Stage project at LFAI&Data, part of the Linux Foundation.

A portable accelerated SQL query, search, and LLM-inference engine, written in Rust, for data-grounded AI apps and agents.

Fastest library to load data from DB to DataFrames in Rust and Python

Parseable is an observability datalake built from first principles.

Apache DataFusion Ballista Distributed Query Engine

The Feldera Incremental Computation Engine

Apache Kafka® compatible broker with S3, PostgreSQL, SQLite, Apache Iceberg and Delta Lake

The Auron accelerator for distributed computing framework (e.g., Spark) leverages native vectorized execution to accelerate query processing

A cloud-native open source distributed time series database with high performance, high compression ratio and high availability.

Apache Iceberg

High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.

Spec-driven framework for composable applications in Rust.

PyTorch Single Controller

Lakehouse native graph engine with git-style workflows

Flow-Like: Strongly Typed Enterprise Scale Workflows. Built for scalability, speed, seamless AI integration and rich customization.

A Vector Database Tutorial (over CMU-DB's BusTub system)

Version control system for AI agents

Scalable graph analytics database powered by a multithreaded, vectorized temporal engine, written in Rust

Elodin simulation and flight software monorepo.

Analytical database for data-driven Web applications 🪶

A single-node analytical database engine with geospatial as a first-class citizen

Protocol and libraries for sending and receiving OpenTelemetry data using Apache Arrow

Fully Managed, Streaming Ingestion (CDC) into your Lakehouse

A SQL transformation engine that type-checks your whole pipeline and catches breaking changes before they run — branches, replay, column-level lineage, compile-time contracts, per-model cost. Adapters: Databricks, Snowflake, BigQuery, DuckDB. Single static Rust binary. Apache 2.0.

The native Rust implementation for Apache Hudi, with C++ & Python API bindings.

Unofficial rust implementation of Apache Iceberg with integration for Datafusion

View parquet files online

DataFusion TableProviders for reading data from other systems

Batteries included CLI, TUI, and server implementations for DataFusion.