A Model Context Protocol (MCP) server implementation that integrates with Olostep for web scraping, content extraction, and search capabilities. To set up Olostep MCP Server, you need to have an API key. You can get the API key by signing up on the Olostep website. - Scrape website content in HTML, Markdown, JSON or Plain Text (with optional parsers) - Parser-based web search with structured resul
Add this skill
npx mdskills install olostep/olostep-mcp-serverComprehensive web scraping MCP with 9 well-documented tools, excellent setup guides, and rich examples
1# Olostep MCP Server23[](https://hub.docker.com/r/olostep/mcp-server)4[](https://www.npmjs.com/package/olostep-mcp)5[](https://opensource.org/licenses/ISC)67A Model Context Protocol (MCP) server implementation that integrates with [Olostep](https://olostep.com) for web scraping, content extraction, and search capabilities.8To set up Olostep MCP Server, you need to have an API key. You can get the API key by signing up on the [Olostep website](https://olostep.com/auth).91011## Features1213- Scrape website content in HTML, Markdown, JSON or Plain Text (with optional parsers)14- Parser-based web search with structured results15- AI Answers with citations and optional JSON-shaped outputs16- Batch scraping of up to 10k URLs17- Autonomous site crawling from a start URL18- Website URL discovery and mapping (with include/exclude filters)19- Country-specific request routing for geo-targeted content20- Configurable wait times for JavaScript-heavy websites21- Comprehensive error handling and reporting22- Simple API key configuration2324## Installation2526### 🐳 Running with Docker (Recommended)2728The easiest way to run the Olostep MCP server:2930```bash31docker pull olostep/mcp-server3233docker run -i --rm \34 -e OLOSTEP_API_KEY="your-api-key" \35 olostep/mcp-server36```3738#### Local-only Docker build (no Docker Hub required)3940If the Docker Hub image isn’t available from your environment, you can build and run the image locally from this repository:4142```bash43cd olostep-mcp-server44npm install45npm run build46docker build -t olostep/mcp-server:local .4748docker run -i --rm -e OLOSTEP_API_KEY="your-api-key" olostep/mcp-server:local49```5051#### Local smoke test (initialize + tools/list)5253This MCP server uses **stdio** transport. You can validate it starts and lists tools without needing a working API key:5455On Windows (PowerShell):5657```powershell58cd .\olostep-mcp-server59powershell -ExecutionPolicy Bypass -File .\scripts\smoke-test.ps160```6162To actually call tools successfully, provide `OLOSTEP_API_KEY` when running the container.6364#### Using Docker with Claude Desktop6566Add this to your `claude_desktop_config.json`:6768```json69{70 "mcpServers": {71 "olostep": {72 "command": "docker",73 "args": [74 "run", "-i", "--rm",75 "-e", "OLOSTEP_API_KEY=YOUR_API_KEY_HERE",76 "olostep/mcp-server"77 ]78 }79 }80}81```8283#### Using Docker with Cursor8485Add an MCP server with:86- **Name:** `olostep`87- **Type:** `command`88- **Command:** `docker run -i --rm -e OLOSTEP_API_KEY=your-api-key olostep/mcp-server`8990### Running with npx9192```bash93env OLOSTEP_API_KEY=your-api-key npx -y olostep-mcp94```9596On Windows (PowerShell):9798```powershell99$env:OLOSTEP_API_KEY = \"your-api-key\"; npx -y olostep-mcp100```101102On Windows (CMD):103104```cmd105set OLOSTEP_API_KEY=your-api-key && npx -y olostep-mcp106```107108### Manual Installation109110```bash111npm install -g olostep-mcp112```113114### Running on Claude Desktop115116Add this to your `claude_desktop_config.json`:117118```json119{120 "mcpServers": {121 "mcp-server-olostep": {122 "command": "npx",123 "args": ["-y", "olostep-mcp"],124 "env": {125 "OLOSTEP_API_KEY": "YOUR_API_KEY_HERE"126 }127 }128 }129}130```131132Or for a more straightforward way you can install via the Smithery CLI by running the following code in your device terminal133134```135npx -y @smithery/cli install @olostep/olostep-mcp-server --client claude136```137138### Running on Windsurf139140Add this to your `./codeium/windsurf/model_config.json`:141142```json143{144 "mcpServers": {145 "mcp-server-olostep": {146 "command": "npx",147 "args": ["-y", "olostep-mcp"],148 "env": {149 "OLOSTEP_API_KEY": "YOUR_API_KEY_HERE"150 }151 }152 }153}154```155156### Running on Cursor157158To configure Olostep MCP in Cursor:1591601. Open Cursor Settings1612. Go to Features > MCP Servers1623. Click "+ Add New MCP Server"1634. Enter the following:164 - Name: "olostep-mcp" (or your preferred name)165 - Type: "command"166 - Command: `env OLOSTEP_API_KEY=your-api-key npx -y olostep-mcp`167168Replace `your-api-key` with your Olostep API key.169170### Running on Metorial171172**Option 1: One-Click Installation (Recommended)**1731741. Open [Metorial](https://metorial.com) dashboard1752. Navigate to MCP Servers directory1763. Search for "Olostep"1774. Click "Install" and enter your API key178179**Option 2: Manual Configuration**180181Add this to your Metorial MCP server configuration:182183```json184{185 "olostep": {186 "command": "npx",187 "args": ["-y", "olostep-mcp"],188 "env": {189 "OLOSTEP_API_KEY": "YOUR_API_KEY_HERE"190 }191 }192}193```194195The Olostep tools will then be available in your Metorial AI chats.196197## Configuration198199### Environment Variables200201- `OLOSTEP_API_KEY`: Your Olostep API key (required)202- `ORBIT_KEY`: An optional key for using Orbit to route requests.203204## Available Tools205206### 1. Scrape Website (`scrape_website`)207208Extract content from a single URL. Supports multiple formats and JavaScript rendering.209210```json211{212 "name": "scrape_website",213 "arguments": {214 "url_to_scrape": "https://example.com",215 "output_format": "markdown",216 "country": "US",217 "wait_before_scraping": 1000,218 "parser": "@olostep/amazon-product"219 }220}221```222223#### Parameters:224225- `url_to_scrape`: The URL of the website you want to scrape (required)226- `output_format`: Choose format (`html`, `markdown`, `json`, or `text`) - default: `markdown`227- `country`: Optional country code (e.g., US, GB, CA) for location-specific scraping228- `wait_before_scraping`: Wait time in milliseconds before scraping (0-10000)229- `parser`: Optional parser ID for specialized extraction230231#### Response (example):232233```json234{235 "content": [236 {237 "type": "text",238 "text": "{\n \"id\": \"scrp_...\",\n \"url\": \"https://example.com\",\n \"markdown_content\": \"# ...\",\n \"html_content\": null,\n \"json_content\": null,\n \"text_content\": null,\n \"status\": \"succeeded\",\n \"timestamp\": \"2025-11-14T12:34:56Z\",\n \"screenshot_hosted_url\": null,\n \"page_metadata\": { }\n}"239 }240 ]241}242```243244### 2. Search the Web (`search_web`)245246Search the Web for a given query and get structured results (non-AI, parser-based).247248```json249{250 "name": "search_web",251 "arguments": {252 "query": "your search query",253 "country": "US"254 }255}256```257258#### Parameters:259260- `query`: Search query (required)261- `country`: Optional country code for localized results (default: `US`)262263#### Response:264265- Structured JSON (as text) representing parser-based results266267### 3. Answers (AI) (`answers`)268269Search the web and return AI-powered answers in the JSON structure you want, with sources and citations.270271```json272{273 "name": "answers",274 "arguments": {275 "task": "Who are the top 5 competitors to Acme Inc. in the EU?",276 "json": "Return a list of the top 5 competitors with name and homepage URL"277 }278}279```280281#### Parameters:282283- `task`: Question or task to answer using web data (required)284- `json`: Optional JSON schema/object or a short description of the desired output shape285286#### Response includes:287288- `answer_id`, `object`, `task`, `result` (JSON if provided), `sources`, `created`289290### 4. Batch Scrape URLs (`batch_scrape_urls`)291292Scrape up to 10k URLs at the same time. Perfect for large-scale data extraction.293294```json295{296 "name": "batch_scrape_urls",297 "arguments": {298 "urls_to_scrape": [299 {"url": "https://example.com/a", "custom_id": "a"},300 {"url": "https://example.com/b", "custom_id": "b"}301 ],302 "output_format": "markdown",303 "country": "US",304 "wait_before_scraping": 500,305 "parser": "@olostep/amazon-product"306 }307}308```309310#### Response includes:311312- `batch_id`, `status`, `total_urls`, `created_at`, `formats`, `country`, `parser`, `urls`313314### 5. Create Crawl (`create_crawl`)315316Autonomously discover and scrape entire websites by following links.317318```json319{320 "name": "create_crawl",321 "arguments": {322 "start_url": "https://example.com/docs",323 "max_pages": 25,324 "follow_links": true,325 "output_format": "markdown",326 "country": "US",327 "parser": "@olostep/doc-parser"328 }329}330```331332#### Response includes:333334- `crawl_id`, `object`, `status`, `start_url`, `max_pages`, `follow_links`, `created`, `formats`, `country`, `parser`335336### 6. Create Map (`create_map`)337338Get all URLs on a website. Extract all URLs for discovery and analysis.339340```json341{342 "name": "create_map",343 "arguments": {344 "website_url": "https://example.com",345 "search_query": "blog",346 "top_n": 200,347 "include_url_patterns": ["/blog/**"],348 "exclude_url_patterns": ["/admin/**"]349 }350}351```352353#### Response includes:354355- `map_id`, `object`, `url`, `total_urls`, `urls`, `search_query`, `top_n`356357### 7. Get Webpage Content (`get_webpage_content`)358359Retrieves webpage content in clean markdown format with support for JavaScript rendering.360361```json362{363 "name": "get_webpage_content",364 "arguments": {365 "url_to_scrape": "https://example.com",366 "wait_before_scraping": 1000,367 "country": "US"368 }369}370```371372#### Parameters:373374- `url_to_scrape`: The URL of the webpage to scrape (required)375- `wait_before_scraping`: Time to wait in milliseconds before starting the scrape (default: 0)376- `country`: Residential country to load the request from (e.g., US, CA, GB) (optional)377378#### Response:379380```json381{382 "content": [383 {384 "type": "text",385 "text": "# Example Website\n\nThis is the markdown content of the webpage..."386 }387 ]388}389```390391### 8. Get Website URLs (`get_website_urls`)392393Search and retrieve relevant URLs from a website, sorted by relevance to your query.394395```json396{397 "name": "get_website_urls",398 "arguments": {399 "url": "https://example.com",400 "search_query": "your search term"401 }402}403```404405#### Parameters:406407- `url`: The URL of the website to map (required)408- `search_query`: The search query to sort URLs by (required)409410#### Response:411412```json413{414 "content": [415 {416 "type": "text",417 "text": "Found 42 URLs matching your query:\n\nhttps://example.com/page1\nhttps://example.com/page2\n..."418 }419 ]420}421```422423### 9. Web Search (`google_search`)424425Retrieve structured data from web search results.426427```json428{429 "name": "google_search",430 "arguments": {431 "query": "your search query",432 "country": "US"433 }434}435```436437#### Parameters:438439- `query`: The search query to perform (required)440- `country`: Country code for localized results (e.g., US, GB) (default: "US")441442#### Response includes:443444- Organic search results with titles, links, and snippets445- Knowledge graph data when available446- Related questions (People Also Ask)447- Related searches448- Rich snippets and other structured data449450## Error Handling451452The server provides robust error handling:453454- Detailed error messages for API issues455- Network error reporting456- Authentication failure handling457- Rate limit information458459Example error response:460461```json462{463 "isError": true,464 "content": [465 {466 "type": "text",467 "text": "Olostep API Error: 401 Unauthorized. Details: {\"error\":\"Invalid API key\"}"468 }469 ]470}471```472473474## Distribution475476### Docker Images477478The MCP server is available as a Docker image:479480- **Docker Hub:** [`olostep/mcp-server`](https://hub.docker.com/r/olostep/mcp-server)481- **Official Docker MCP Registry:** `mcp/olostep` (coming soon - enhanced security with signatures & SBOMs)482- **GitHub Container Registry:** `ghcr.io/olostep/olostep-mcp-server`483484### Docker Desktop MCP Toolkit485486The Olostep MCP Server is being added to Docker Desktop's official MCP Toolkit, which means users will be able to:487- Discover it in Docker Desktop's MCP Toolkit UI488- Install it with one click489- Configure it visually490- Use it with any MCP-compatible client (Claude Desktop, Cursor, etc.)491492**Status**: Submission in progress to [Docker MCP Registry](https://github.com/docker/mcp-registry)493494### Supported Platforms495496- `linux/amd64`497- `linux/arm64`498499### Building Locally500501```bash502# Clone the repository503git clone https://github.com/olostep/olostep-mcp-server.git504cd olostep-mcp-server505506# Build the image507npm install508npm run build509docker build -t olostep/mcp-server .510511# Run locally512docker run -i --rm -e OLOSTEP_API_KEY="your-key" olostep/mcp-server513```514515## License516517ISC License518
Full transparency — inspect the skill content before installing.