GitHub topics

parquet

Repositories

32

Total stars

26,402

Trending

lakehq

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

Created Dec 2023

apache-iceberg apache-spark arrow artificial-intelligence big-data
Total3,505
30 days +46
nisshi-io

Apache Kafka® compatible broker with S3, PostgreSQL, SQLite, Apache Iceberg and Delta Lake

Created Aug 2024

apache-arrow apache-iceberg apache-kafka built-with-rust datafusion
Total1,933
30 days +38
parseablehq

Parseable is an open source, unified infrastructure observability platform built in Rust on a data lake architecture. It tracks logs, metrics, traces, and events across apps, agents, and systems, reducing storage costs by up to 90% through columnar telemetry compression.

Created Aug 2022

agent-observability alerting apm cloud-native dashboards
Total2,581
30 days +26

Top

lakehq

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

Created Dec 2023

apache-iceberg apache-spark arrow artificial-intelligence big-data
Total3,505
roapi

Create full-fledged APIs for slowly moving datasets without writing a single line of code.

Created Dec 2020

analytics arrow blob-storage cloud-native columnar
Total3,435
parseablehq

Parseable is an open source, unified infrastructure observability platform built in Rust on a data lake architecture. It tracks logs, metrics, traces, and events across apps, agents, and systems, reducing storage costs by up to 90% through columnar telemetry compression.

Created Aug 2022

agent-observability alerting apm cloud-native dashboards
Total2,581
CrunchyData

Copy to/from Parquet in S3, Azure Blob Storage, Google Cloud Storage, http(s) stores, local files or standard inout stream from within PostgreSQL

Created Sep 2024

azure-storage columnar data-ingestion data-migration google-cloud-storage
Total696
jorgecarleitao

Fastest and safest Rust implementation of parquet. `unsafe` free. Integration-tested against pyarrow

Created Mar 2021

parallelism parquet rust safe
Total389
pacman82

A command line tool to query an ODBC data source and write the result into a parquet file.

Created Aug 2020

odbc parquet
Total258
abyo-software

GPU-accelerated transparent compression S3-compatible storage gateway. Drop-in replacement for AWS S3 endpoints; cuts your S3 bill 50-80% with no app changes (Rust, nvCOMP, zstd).

Created May 2026

apache-2 aws-s3 compression cost-optimization cuda
Total159
AkashPriyadarshii

High-throughput synthetic and pretraining dataset sifter for TypeSafe Jev. Rust streaming core, Parquet and JSONL I/O, typed Choice/Score/Noul judgments, speculative fan-out, 24.0 rows/sec measured.

Created Sep 2026

arrow cli data-cleaning data-engineering dataset-curation
Total115
clement-tourriere

Fast psql-style database CLI/workbench for PostgreSQL, MySQL, SQLite, ClickHouse, MongoDB, Elasticsearch, and SQL over Parquet/CSV/JSON — with optional AI, SSH/Vault, Python & Django.

Created Jul 2025

Total92
dweve-ai

Token-efficient data serialization for LLM/AI. 50% fewer tokens than JSON, 93% better value/token. Rust, schema validation, LSP.

Created Jan 2026

ai-ml cli csv data-format json-alternative
Total80
buoyant-data

Collection of AWS Lambdas for creating and managing Delta tables

Created May 2023

datalake deltalake lambda parquet rust
Total59
madesroches

Unified observability for logs, metrics and traces — low overhead instrumentation (Rust, Unreal Engine 5), OTLP ingestion, SQL analytics on DataFusion over Parquet, presentation through notebooks, Grafana and Python. Self-hosted.

Created Jan 2024

Total56

Related

All topics →