Crawler AI Agent Skills

An adaptive web scraping framework that handles everything from a single request to a full-scale crawl. Its parser learns from website changes and automatically relocates your elements when pages update. Its fetchers bypass anti-bot systems like Cloudflare Turnstile.

Turn URLs into clean markdown or structured JSON with one engine for search, scrape, map, crawl, and extract. Run it locally as a small Rust binary or use the managed API. Get 1000 free credits → · Install · No credit card. Continue with GitHub. Runs local and free, no account needed. To use the Cloud, paste your key into the same command and it installs the binary, connects the key, and registers

MCP Server for Driflyte. The Driflyte MCP Server exposes tools that allow AI assistants to query and retrieve topic-specific knowledge from recursively crawled and indexed web pages. With this MCP server, Driflyte acts as a bridge between diverse, topic-aware content sources (web, GitHub, and more) and AI-powered reasoning, enabling richer, more accurate answers. - Deep Web Crawling: Recursively f

Fast, token-efficient web content extraction for AI agents - converts websites to clean Markdown. Existing MCP web crawlers are slow and consume large quantities of tokens. This pauses the development process and provides incomplete results as LLMs need to parse whole web pages. This MCP package fetches web pages locally, strips noise, and converts content to clean Markdown while preserving links.