GitHub topics

crawler

16% of these repos are in Web programming.

Repositories

18

Total stars

7,982

Trending

fastcrw

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.

Created Mar 2026

ai ai-agents crawler data-extraction docker
Total1,279
30 days +118
janreges

SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).

Created Oct 2023

Total1,015
30 days +51
0x676e67

An ergonomic, privacy-aware Rust HTTP Client

Created Sep 2024

akamai censorship-resistant client crawler fingerprint
Total1,116
30 days +46

Top

fastcrw

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.

Created Mar 2026

ai ai-agents crawler data-extraction docker
Total1,279
0x676e67

An ergonomic, privacy-aware Rust HTTP Client

Created Sep 2024

akamai censorship-resistant client crawler fingerprint
Total1,116
janreges

SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).

Created Oct 2023

Total1,015
oscar-project

:spider: The pipeline for the OSCAR corpus

Created Feb 2021

common-crawl commoncrawl corpus-linguistics crawler fasttext
Total178
sshwy

🎓a Better BlackBoard for PKUers. 北京大学教学网命令行工具(🖥️Win/🐧Linux/🍏Mac)

Created Mar 2025

blackboard-learn cli command-line-tool compio crawler
Total167
spronta

Fast, free, open-source technical SEO + GEO crawler: built for humans and agents.

Created Jun 2026

aeo crawler geo marketing-tools mcp
Total130
hominee

Dyer is designed for reliable, flexible and fast web crawling, providing some high-level, comprehensive features without compromising speed.

Created Feb 2021

crawler rust rust-programming-language spider web-crawler
Total126

Related

All topics →