GitHub topics

datafusion

Repositories

32

Total stars

31,165

Trending

lakehq

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

Created Dec 2023

apache-iceberg apache-spark arrow artificial-intelligence big-data
Total3,503
30 days +46
nisshi-io

Apache Kafka® compatible broker with S3, PostgreSQL, SQLite, Apache Iceberg and Delta Lake

Created Aug 2024

apache-arrow apache-iceberg apache-kafka built-with-rust datafusion
Total1,933
30 days +41
lakesoul-io

An open lakehouse engine for agents — Postgres-native, vector-native and Python-native. Speaks PostgreSQL, runs on your object storage, written in Rust.

Created Dec 2021

Total3,261
30 days +7
arkflow-rs

High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.

Created Mar 2025

Total1,317
30 days +6
roapi

Create full-fledged APIs for slowly moving datasets without writing a single line of code.

Created Dec 2020

analytics arrow blob-storage cloud-native columnar
Total3,435
30 days +3

Top

lakehq

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

Created Dec 2023

apache-iceberg apache-spark arrow artificial-intelligence big-data
Total3,503
roapi

Create full-fledged APIs for slowly moving datasets without writing a single line of code.

Created Dec 2020

analytics arrow blob-storage cloud-native columnar
Total3,435
arkflow-rs

High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.

Created Mar 2025

Total1,317
kamu-data

Next-generation decentralized data lakehouse and a multi-party stream processing network

Created May 2019

data-as-code data-management data-science datafusion flink
Total351
clement-tourriere

Fast psql-style database CLI/workbench for PostgreSQL, MySQL, SQLite, ClickHouse, MongoDB, Elasticsearch, and SQL over Parquet/CSV/JSON — with optional AI, SSH/Vault, Python & Django.

Created Jul 2025

Total92
jorgecarleitao

A Python library to run analytics workloads with the performance of Rust, the flexibility of Python and O(1) cost in moving data between the two. Uses Apache Arrow in-memory format and respective query engine DataFusion.

Created Jul 2020

datafusion python rust udfs
Total61
madesroches

Unified observability for logs, metrics and traces — low overhead instrumentation (Rust, Unreal Engine 5), OTLP ingestion, SQL analytics on DataFusion over Parquet, presentation through notebooks, Grafana and Python. Self-hosted.

Created Jan 2024

Total56
NOFireAI

A distributed OpenTelemetry-native columnar datastore whose only backend is object storage.

Created Aug 2026

apache-2-0-license arrow columnar-storage datafusion datastore
Total50

Related

All topics →