GitHub topics

web-scraping

13% of these repos are in Browsers and Browser Automation, 13% in Web, 10% in Web programming.

Repositories

29

Total stars

36,583

Trending

YOLOW92

Local-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, verifiable data graphs, and evidence-grounded AI context.

Created Oct 2026

ai-agents browser-automation crawl4ai-alternative data-extraction document-parsing
Total288
30 days +286
fastcrw

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.

Created Mar 2026

ai ai-agents crawler data-extraction docker
Total1,279
30 days +114
0xMassi

Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.

Created Mar 2026

ai-agents ai-scraping apify-alternative cli crawl4ai-alternative
Total2,438
30 days +71
janreges

SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).

Created Oct 2023

Total1,017
30 days +49
deedy5

HTTP client that can impersonate web browsers

Created Apr 2024

akamai akamai-fingerprint curl fingerprint http
Total634
30 days +20

Top

0xMassi

Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.

Created Mar 2026

ai-agents ai-scraping apify-alternative cli crawl4ai-alternative
Total2,438
fastcrw

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.

Created Mar 2026

ai ai-agents crawler data-extraction docker
Total1,279
janreges

SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).

Created Oct 2023

Total1,017
deedy5

HTTP client that can impersonate web browsers

Created Apr 2024

akamai akamai-fingerprint curl fingerprint http
Total634
buyukakyuz

Discover and verify professional emails using names + domains

Created Apr 2025

cold-email contact-enrichment dns email-verification lead-generation
Total429
YOLOW92

Local-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, verifiable data graphs, and evidence-grounded AI context.

Created Oct 2026

ai-agents browser-automation crawl4ai-alternative data-extraction document-parsing
Total288
xberg-io

Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.

Created Mar 2026

crawling csharp elixir ffi golang
Total197
konippi

A self-contained browser engine that fetches, renders, and extracts web content as Markdown, JSON, or screenshots — no Chromium, no API key, no setup.

Created Apr 2026

Total164
yfedoseev

Stealth headless browser engine in Rust — native BoringSSL TLS/JA4 fingerprint, real JS via deno_core, no Chromium, no CDP. Python + MCP bindings. A camoufox alternative.

Created Apr 2026

ai-agents anti-bot anti-detect-browser boringssl browser-fingerprint
Total131
hominee

Dyer is designed for reliable, flexible and fast web crawling, providing some high-level, comprehensive features without compromising speed.

Created Feb 2021

crawler rust rust-programming-language spider web-crawler
Total126
proofofbots

Headless Akamai & Kasada solver that runs without a browser, and the reversing toolkit it was built with

Created Aug 2026

akamai akamai-bypass akamai-sensor-generator akamai-solver captcha-bypass-api
Total122
sderosiaux

Web tasks that compile. Other browser tools tell your AI agent the click worked. chrome-agent reads the page back after every action and answers in JSON what actually happened. One 3 MB Rust binary, no Node, no cloud.

Created Mar 2026

agent-tools ai-agents browser-automation cdp chrome-devtools-protocol
Total99
Murrough-Foley

Fast, accurate web content extraction in Rust. ML page-type classification, per-type extraction, confidence scoring. F1=0.966 on ScrapingHub (#1), F1=0.859 across 2,008 annotated pages (1,497 development + 511 held-out test

Created Jan 2026

content-extraction machine-learning nlp rust search-engine-optimization
Total71
cortex-works

A unified web extraction and stateful automation engine for AI. Replaces heavy testing frameworks with token-optimized browser control, deep research, and HITL.

Created Aug 2025

ai-agents anti-bot-bypass automated-testing dev-tools knowledge-graph
Total71
saeedkolivand

Local-first AI desktop assistant that scrapes job boards, matches roles to your resume, and auto-generates resumes & cover letters. Offline with Ollama, BYO API key, or route it through an AI CLI agent (Claude Code, Codex, Gemini CLI).

Created May 2026

ai ats desktop-app job-search local-first
Total64
nodemaven

Rust SDK for the NodeMaven residential and mobile proxy gateway: builds the gateway username, validates it client-side, and refuses parameters the gateway would silently drop. Also checks a connection and calls the account API.

Created Sep 2026

http-proxy mobile-proxy nodemaven proxy proxy-api
Total55

Related

All topics →