Web Analyzer MCP

1# 🔍 Web Analyzer MCP
2 
3<a href="https://glama.ai/mcp/servers/@kimdonghwi94/web-analyzer-mcp">
4  <img width="380" height="200" src="https://glama.ai/mcp/servers/@kimdonghwi94/web-analyzer-mcp/badge" alt="WebAnalyzer MCP server" />
5</a>
6 
7A powerful MCP (Model Context Protocol) server for intelligent web content analysis and summarization. Built with FastMCP, this server provides smart web scraping, content extraction, and AI-powered question-answering capabilities.
8 
9## ✨ Features
10 
11### 🎯 Core Tools
12 
131. **`url_to_markdown`** - Extract and summarize key web page content
14   - Analyzes content importance using custom algorithms
15   - Removes ads, navigation, and irrelevant content
16   - Keeps only essential information (tables, images, key text)
17   - Outputs structured markdown optimized for analysis
18 
192. **`web_content_qna`** - AI-powered Q&A about web content
20   - Extracts relevant content sections from web pages
21   - Uses intelligent chunking and relevance matching
22   - Answers questions using OpenAI GPT models
23 
24### 🚀 Key Features
25 
26- **Smart Content Ranking**: Algorithm-based content importance scoring
27- **Essential Content Only**: Removes clutter, keeps what matters
28- **Multi-IDE Support**: Works with Claude Desktop, Cursor, VS Code, PyCharm
29- **Flexible Models**: Choose from GPT-3.5, GPT-4, GPT-4 Turbo, or GPT-5
30 
31## 📦 Installation
32 
33### Prerequisites
34- [uv](https://docs.astral.sh/uv/getting-started/installation/) (Python package manager)
35- Chrome/Chromium browser (for Selenium)
36- OpenAI API key (for Q&A functionality)
37 
38### 🚀 Quick Start with uv (Recommended)
39 
40```bash
41# Clone the repository
42git clone https://github.com/kimdonghwi94/web-analyzer-mcp.git
43cd web-analyzer-mcp
44 
45# Run directly with uv (auto-installs dependencies)
46uv run mcp-webanalyzer
47```
48 
49### Installing via Smithery
50 
51To install web-analyzer-mcp for Claude Desktop automatically via [Smithery](https://smithery.ai/server/@kimdonghwi94/web-analyzer-mcp):
52 
53```bash
54npx -y @smithery/cli install @kimdonghwi94/web-analyzer-mcp --client claude
55```
56 
57# IDE/Editor Integration
58 
59<details>
60<summary><b>Install Claude Desktop</b></summary>
61 
62Add to your Claude Desktop_config.json file. See [Claude Desktop MCP documentation](https://modelcontextprotocol.io/quickstart/user) for more details.
63 
64```json
65{
66  "mcpServers": {
67    "web-analyzer": {
68      "command": "uv",
69      "args": [
70        "--directory",
71        "/path/to/web-analyzer-mcp",
72        "run", 
73        "mcp-webanalyzer"
74      ],
75      "env": {
76        "OPENAI_API_KEY": "your_openai_api_key_here",
77        "OPENAI_MODEL": "gpt-4"
78      }
79    }
80  }
81}
82```
83 
84</details>
85 
86<details>
87<summary><b>Install Claude Code (VS Code Extension)</b></summary>
88 
89Add the server using Claude Code CLI:
90 
91```bash
92claude mcp add web-analyzer -e OPENAI_API_KEY=your_api_key_here -e OPENAI_MODEL=gpt-4 -- uv --directory /path/to/web-analyzer-mcp run mcp-webanalyzer
93```
94</details>
95 
96<details>
97<summary><b>Install Cursor IDE</b></summary>
98 
99Add to your Cursor settings (`File > Preferences > Settings > Extensions > MCP`):
100 
101```json
102{
103  "mcpServers": {
104    "web-analyzer": {
105      "command": "uv",
106      "args": [
107        "--directory",
108        "/path/to/web-analyzer-mcp",
109        "run", 
110        "mcp-webanalyzer"
111      ],
112      "env": {
113        "OPENAI_API_KEY": "your_openai_api_key_here",
114        "OPENAI_MODEL": "gpt-4"
115      }
116    }
117  }
118}
119```
120</details>
121 
122<details>
123<summary><b>Install JetBrains AI Assistant</b></summary>
124 
125See [JetBrains AI Assistant Documentation](https://www.jetbrains.com/help/idea/ai-assistant.html) for more details.
126 
1271. In JetBrains IDEs go to **Settings** → **Tools** → **AI Assistant** → **Model Context Protocol (MCP)**
1282. Click **+ Add**
1293. Click on **Command** in the top-left corner of the dialog and select the **As JSON** option from the list
1304. Add this configuration and click **OK**:
131 
132```json
133{
134  "mcpServers": {
135    "web-analyzer": {
136      "command": "uv",
137      "args": [
138        "--directory",
139        "/path/to/web-analyzer-mcp",
140        "run", 
141        "mcp-webanalyzer"
142      ],
143      "env": {
144        "OPENAI_API_KEY": "your_openai_api_key_here",
145        "OPENAI_MODEL": "gpt-4"
146      }
147    }
148  }
149}
150```
151</details>
152 
153## 🎛️ Tool Descriptions
154 
155### `url_to_markdown`
156Converts web pages to clean markdown format with essential content extraction.
157 
158**Parameters:**
159- `url` (string): The web page URL to analyze
160 
161**Returns:** Clean markdown content with structured data preservation
162 
163### `web_content_qna` 
164Answers questions about web page content using intelligent content analysis.
165 
166**Parameters:**
167- `url` (string): The web page URL to analyze
168- `question` (string): Question about the page content
169 
170**Returns:** AI-generated answer based on page content
171 
172## 🏗️ Architecture
173 
174### Content Extraction Pipeline
175 
1761. **URL Validation** - Ensures proper URL format
1772. **HTML Fetching** - Uses Selenium for dynamic content
1783. **Content Parsing** - BeautifulSoup for HTML processing
1794. **Element Scoring** - Custom algorithm ranks content importance
1805. **Content Filtering** - Removes duplicates and low-value content
1816. **Markdown Conversion** - Structured output generation
182 
183### Q&A Processing Pipeline
184 
1851. **Content Chunking** - Intelligent text segmentation
1862. **Relevance Scoring** - Matches content to questions
1873. **Context Selection** - Picks most relevant chunks
1884. **Answer Generation** - OpenAI GPT integration
189 
190## 🏗️ Project Structure
191 
192```
193web-analyzer-mcp/
194├── web_analyzer_mcp/          # Main Python package
195│   ├── __init__.py           # Package initialization
196│   ├── server.py             # FastMCP server with tools
197│   ├── web_extractor.py      # Web content extraction engine
198│   └── rag_processor.py      # RAG-based Q&A processor
199├── scripts/                   # Build and utility scripts
200│   └── build.js              # Node.js build script
201├── README.md                 # English documentation
202├── README.ko.md              # Korean documentation
203├── package.json              # npm configuration and scripts
204├── pyproject.toml            # Python package configuration
205├── .env.example              # Environment variables template
206└── dist-info.json            # Build information (generated)
207```
208 
209## 🛠️ Development
210 
211### Modern Development with uv
212 
213```bash
214# Clone repository
215git clone https://github.com/kimdonghwi94/web-analyzer-mcp.git
216cd web-analyzer-mcp
217 
218# Development commands
219uv run mcp-webanalyzer     # Start development server
220uv run python -m pytest   # Run tests
221uv run ruff check .        # Lint code
222uv run ruff format .       # Format code
223uv sync                    # Sync dependencies
224 
225# Install development dependencies
226uv add --dev pytest ruff mypy
227 
228# Create production build
229npm run build
230```
231 
232### Alternative: Traditional Python Development
233 
234```bash
235# Setup Python environment (if not using uv)
236pip install -e .[dev]
237 
238# Development commands
239python -m web_analyzer_mcp.server  # Start server
240python -m pytest tests/            # Run tests
241python -m ruff check .             # Lint code
242python -m ruff format .            # Format code
243python -m mypy web_analyzer_mcp/   # Type checking
244```
245 
246## 🤝 Contributing
247 
2481. Fork the repository
2492. Create a feature branch (`git checkout -b feature/amazing-feature`)
2503. Commit your changes (`git commit -m 'Add amazing feature'`)
2514. Push to the branch (`git push origin feature/amazing-feature`)
2525. Open a Pull Request
253 
254## 📋 Roadmap
255 
256- [ ] Support for more content types (PDFs, videos)
257- [ ] Multi-language content extraction
258- [ ] Custom extraction rules
259- [ ] Caching for frequently accessed content
260- [ ] Webhook support for real-time updates
261 
262## ⚠️ Limitations
263 
264- Requires Chrome/Chromium for JavaScript-heavy sites
265- OpenAI API key needed for Q&A functionality
266- Rate limited to prevent abuse
267- Some sites may block automated access
268 
269## 📄 License
270 
271This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
272 
273## 🙋‍♂️ Support
274 
275- Create an issue for bug reports or feature requests
276- Contribute to discussions in the GitHub repository
277- Check the [documentation](https://github.com/kimdonghwi94/web-analyzer-mcp) for detailed guides
278 
279## 🌟 Acknowledgments
280 
281- Built with [FastMCP](https://github.com/jlowin/fastmcp) framework
282- Inspired by [HTMLRAG](https://github.com/plageon/HtmlRAG) techniques for web content processing
283- Thanks to the MCP community for feedback and contributions
284 
285---
286 
287**Made with ❤️ for the MCP community**