GitHub topics

data-engineering

10% of these repos are in Data processing, 10% in MLOps.

Repositories

19

Total stars

32,782

Trending

cocoindex-io

Incremental engine for long horizon agents 🌟 Star if you like it!

Created Mar 2025

agentic-data-framework ai ai-agents change-data-capture codebase-intelligence
Total11,916
30 days +212
slothflowlabs

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 437 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

Created May 2026

cdc connectors data-engineering data-integration data-orchestration
Total1,559
30 days +68
risingwavelabs

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

Created Jan 2022

apache-iceberg data-engineering database etl-pipeline event-streaming
Total9,425
30 days +54
lakehq

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

Created Dec 2023

apache-iceberg apache-spark arrow artificial-intelligence big-data
Total3,503
30 days +46
estuary

🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊

Created Feb 2020

change-data-capture data data-collection data-engineering data-integration
Total993
30 days +10
rocky-data

A SQL transformation engine that type-checks your whole pipeline and catches breaking changes before they run — branches, replay, column-level lineage, compile-time contracts, per-model cost. Adapters: Databricks, Snowflake, BigQuery, DuckDB. Single static Rust binary. Apache 2.0.

Created Apr 2026

bigquery column-lineage dagster data-contracts data-engineering
Total311
30 days +4

Top

cocoindex-io

Incremental engine for long horizon agents 🌟 Star if you like it!

Created Mar 2025

agentic-data-framework ai ai-agents change-data-capture codebase-intelligence
Total11,916
risingwavelabs

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

Created Jan 2022

apache-iceberg data-engineering database etl-pipeline event-streaming
Total9,425
lakehq

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

Created Dec 2023

apache-iceberg apache-spark arrow artificial-intelligence big-data
Total3,503
slothflowlabs

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 437 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

Created May 2026

cdc connectors data-engineering data-integration data-orchestration
Total1,559
estuary

🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊

Created Feb 2020

change-data-capture data data-collection data-engineering data-integration
Total993
rocky-data

A SQL transformation engine that type-checks your whole pipeline and catches breaking changes before they run — branches, replay, column-level lineage, compile-time contracts, per-model cost. Adapters: Databricks, Snowflake, BigQuery, DuckDB. Single static Rust binary. Apache 2.0.

Created Apr 2026

bigquery column-lineage dagster data-contracts data-engineering
Total311
AkashPriyadarshii

High-throughput synthetic and pretraining dataset sifter for TypeSafe Jev. Rust streaming core, Parquet and JSONL I/O, typed Choice/Score/Noul judgments, speculative fan-out, 24.0 rows/sec measured.

Created Sep 2026

arrow cli data-cleaning data-engineering dataset-curation
Total115
SpaceCell

Fast, minimal, and ergonomic zero-copy Apache Arrow tabular data implementation in Rust, with PyO3, Python bindings, NdArray, XArray, shared-memory and low-latency SIMD for data-engineering and columnar workloads. Bridge to Polars and Arrow-rs over FFI effortlessly.

Created Aug 2025

apache apache-arrow arrow arrow-rs bindings
Total103
contrastdata

Stream PostgreSQL changes into standard Apache Iceberg tables, built in Rust.

Created Sep 2026

apache-iceberg cdc change-data-capture data-engineering iceberg
Total89
sqlstack-io

SQL transformation framework with custom compile-time checks, end-to-end testing and safe renames that keep your table history.

Created May 2026

analytics-engineering bigquery cli data-engineering data-testing
Total50

Related

All topics →