web-scraping
13% of these repos are in Browsers and Browser Automation, 13% in Web, 10% in Web programming.
Repositories
29
Total stars
36,583
Trending
Local-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, verifiable data graphs, and evidence-grounded AI context.
Created Oct 2026
Trap AI web scrapers in an endless poison pit.
Created Mar 2026
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
Created Mar 2026
Foundational low latency web data collecting in Rust
Created Jan 2018
SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).
Created Oct 2023
ZTE U60 Pro (MU5250) 5G router toolkit — Rust on-device REST agent + native iOS/Android/web companions. Reverse-engineered, 143-endpoint coverage, OpenWrt 23.05.
Created Mar 2026
Top
Foundational low latency web data collecting in Rust
Created Jan 2018
Trap AI web scrapers in an endless poison pit.
Created Mar 2026
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
Created Mar 2026
SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).
Created Oct 2023
Discover and verify professional emails using names + domains
Created Apr 2025
Local-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, verifiable data graphs, and evidence-grounded AI context.
Created Oct 2026
A self-contained browser engine that fetches, renders, and extracts web content as Markdown, JSON, or screenshots — no Chromium, no API key, no setup.
Created Apr 2026
ZTE U60 Pro (MU5250) 5G router toolkit — Rust on-device REST agent + native iOS/Android/web companions. Reverse-engineered, 143-endpoint coverage, OpenWrt 23.05.
Created Mar 2026
Stealth headless browser engine in Rust — native BoringSSL TLS/JA4 fingerprint, real JS via deno_core, no Chromium, no CDP. Python + MCP bindings. A camoufox alternative.
Created Apr 2026
Headless Akamai & Kasada solver that runs without a browser, and the reversing toolkit it was built with
Created Aug 2026
A web crawling programming language
Created May 2024
Web tasks that compile. Other browser tools tell your AI agent the click worked. chrome-agent reads the page back after every action and answers in JSON what actually happened. One 3 MB Rust binary, no Node, no cloud.
Created Mar 2026
Track changes on websites via git
Created May 2021
Fast, accurate web content extraction in Rust. ML page-type classification, per-type extraction, confidence scoring. F1=0.966 on ScrapingHub (#1), F1=0.859 across 2,008 annotated pages (1,497 development + 511 held-out test
Created Jan 2026
A unified web extraction and stateful automation engine for AI. Replaces heavy testing frameworks with token-optimized browser control, deep research, and HITL.
Created Aug 2025
Local-first AI desktop assistant that scrapes job boards, matches roles to your resume, and auto-generates resumes & cover letters. Offline with Ollama, BYO API key, or route it through an AI CLI agent (Claude Code, Codex, Gemini CLI).
Created May 2026
Web search CLI for AI agents — 13 providers rank-fused into one Rust binary. Search, extract, verify, check credits.
Created Mar 2026
Spider ported to Node.js
Created Nov 2023
AI agent browser for web reverse engineering
Created Apr 2026
Rust SDK for the NodeMaven residential and mobile proxy gateway: builds the gateway username, validates it client-side, and refuses parameters the gateway would silently drop. Also checks a connection and calls the account API.
Created Sep 2026