MCP Codebase Index

1<!-- mcp-name: io.github.MikeRecognex/mcp-codebase-index -->
2# mcp-codebase-index
3 
4[![PyPI version](https://img.shields.io/pypi/v/mcp-codebase-index)](https://pypi.org/project/mcp-codebase-index/)
5[![CI](https://github.com/MikeRecognex/mcp-codebase-index/actions/workflows/ci.yml/badge.svg)](https://github.com/MikeRecognex/mcp-codebase-index/actions/workflows/ci.yml)
6[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
7[![License: AGPL-3.0](https://img.shields.io/badge/License-AGPL--3.0-blue.svg)](https://www.gnu.org/licenses/agpl-3.0)
8[![MCP](https://img.shields.io/badge/MCP-compatible-purple.svg)](https://modelcontextprotocol.io)
9[![Zero Dependencies](https://img.shields.io/badge/dependencies-zero-brightgreen.svg)]()
10 
11A structural codebase indexer with an [MCP](https://modelcontextprotocol.io) server for AI-assisted development. Zero runtime dependencies — uses Python's `ast` module for Python analysis and regex-based parsing for TypeScript/JS, Go, and Rust. Requires Python 3.11+.
12 
13## What It Does
14 
15Indexes codebases by parsing source files into structural metadata -- functions, classes, imports, dependency graphs, and cross-file call chains -- then exposes 18 query tools via the Model Context Protocol, enabling Claude Code and other MCP clients to navigate codebases efficiently without reading entire files.
16 
17**Automatic incremental re-indexing:** In git repositories, the index stays up to date automatically. Before every query, the server checks `git diff` and `git status` (~1-2ms). If files changed, only those files are re-parsed and the dependency graph is rebuilt. No need to manually call `reindex` after edits, branch switches, or pulls.
18 
19**Persistent disk cache:** The index is saved to a pickle cache file (`.codebase-index-cache.pkl`) after every build. On subsequent server starts, the cache is loaded and validated against the current git HEAD — if the ref matches, startup is instant. If a small number of files changed (≤20), the cached index is loaded and incrementally updated instead of rebuilt from scratch. This eliminates the cold-start penalty when restarting Claude Code sessions, restarting the MCP server, or resuming work after context compaction.
20 
21## Language Support
22 
23| Language | Method | Extracts |
24|----------|--------|----------|
25| Python (`.py`) | AST parsing | Functions, classes, methods, imports, dependency graph |
26| TypeScript/JS (`.ts`, `.tsx`, `.js`, `.jsx`) | Regex-based | Functions, arrow functions, classes, interfaces, type aliases, imports |
27| Go (`.go`) | Regex-based | Functions, methods (receiver-based), structs, interfaces, type aliases, imports, doc comments |
28| Rust (`.rs`) | Regex-based | Functions (`pub`/`async`/`const`/`unsafe`), structs, enums, traits, impl blocks, use statements, attributes, doc comments, macro_rules |
29| Markdown/Text (`.md`, `.txt`, `.rst`) | Heading detection | Sections (# headings, underlines, numbered, ALL-CAPS) |
30| Other | Generic | Line counts only |
31 
32## Installation
33 
34```bash
35pip install "mcp-codebase-index[mcp]"
36```
37 
38The `[mcp]` extra includes the MCP server dependency. Omit it if you only need the programmatic API.
39 
40For development (from a local clone):
41 
42```bash
43pip install -e ".[dev,mcp]"
44```
45 
46## MCP Server
47 
48### Running
49 
50```bash
51# As a console script
52PROJECT_ROOT=/path/to/project mcp-codebase-index
53 
54# As a Python module
55PROJECT_ROOT=/path/to/project python -m mcp_codebase_index.server
56```
57 
58`PROJECT_ROOT` specifies which directory to index. Defaults to the current working directory.
59 
60### Persistent Cache
61 
62In git repositories, the server automatically caches the index to `.codebase-index-cache.pkl` in the project root. On startup:
63 
641. **Cache hit (exact match):** If the cached git ref matches the current HEAD, the index loads instantly from disk — no parsing, no file walking.
652. **Cache hit (small changeset):** If ≤20 files changed since the cached ref, the cached index is loaded and incrementally updated on the first query.
663. **Cache miss:** If the changeset is large or no cache exists, a full rebuild runs and saves a new cache.
67 
68Add `.codebase-index-cache.pkl` to your `.gitignore` — it's a local-only build artifact.
69 
70### Configuring with OpenClaw
71 
72Install the package on the machine where OpenClaw is running:
73 
74```bash
75# Local install
76pip install "mcp-codebase-index[mcp]"
77 
78# Or inside a Docker container / remote VPS
79docker exec -it openclaw bash
80pip install "mcp-codebase-index[mcp]"
81```
82 
83Add the MCP server to your OpenClaw agent config (`openclaw.json`):
84 
85```json
86{
87  "agents": {
88    "list": [{
89      "id": "main",
90      "mcp": {
91        "servers": [
92          {
93            "name": "codebase-index",
94            "command": "mcp-codebase-index",
95            "env": {
96              "PROJECT_ROOT": "/path/to/project"
97            }
98          }
99        ]
100      }
101    }]
102  }
103}
104```
105 
106Restart OpenClaw and verify the connection:
107 
108```bash
109openclaw mcp list
110```
111 
112All 18 tools will be available to your agent.
113 
114**Performance note:** The server automatically detects file changes via `git diff` before every query (~1-2ms) and incrementally re-indexes only what changed. However, OpenClaw's default MCP integration via mcporter spawns a fresh server process per tool call, which discards the in-memory index and forces a full rebuild each time (~1-2s for small projects, longer for large ones). With persistent caching, these cold starts are now significantly faster — the server loads from the disk cache instead of re-parsing the entire codebase. For persistent connections (avoiding even the cache load overhead), use the [openclaw-mcp-adapter](https://github.com/androidStern-personal/openclaw-mcp-adapter) plugin, which connects once at startup and keeps the server running:
115 
116```bash
117pip install openclaw-mcp-adapter
118```
119 
120### Configuring with Claude Code
121 
122Add to your project's `.mcp.json`:
123 
124```json
125{
126  "mcpServers": {
127    "codebase-index": {
128      "command": "mcp-codebase-index",
129      "env": {
130        "PROJECT_ROOT": "/path/to/project"
131      }
132    }
133  }
134}
135```
136 
137Or using the Python module directly (useful if installed in a virtualenv):
138 
139```json
140{
141  "mcpServers": {
142    "codebase-index": {
143      "command": "/path/to/.venv/bin/python3",
144      "args": ["-m", "mcp_codebase_index.server"],
145      "env": {
146        "PROJECT_ROOT": "/path/to/project"
147      }
148    }
149  }
150}
151```
152 
153#### Reinforcing Tool Usage with Hooks
154 
155Claude Code tends to default to built-in Glob/Grep/Read tools even when codebase-index is available. In addition to CLAUDE.md instructions (see below), you can add hooks that fire on every prompt to reinforce the behavior. Add this to `.claude/settings.local.json`:
156 
157```json
158{
159  "hooks": {
160    "SessionStart": [
161      {
162        "hooks": [
163          {
164            "type": "command",
165            "command": "echo 'CRITICAL REMINDER: Use codebase-index MCP tools FIRST for ALL code navigation (find_symbol, get_function_source, search_codebase, get_dependencies, etc). Only fall back to Glob/Grep/Read for non-code files.'"
166          }
167        ]
168      }
169    ],
170    "UserPromptSubmit": [
171      {
172        "hooks": [
173          {
174            "type": "command",
175            "command": "echo 'Use codebase-index MCP tools first for code navigation.'"
176          }
177        ]
178      }
179    ]
180  }
181}
182```
183 
184Hook stdout is injected as context Claude sees before responding. `SessionStart` fires on startup, resume, and context compaction. `UserPromptSubmit` fires on every turn.
185 
186### Important: Make the AI Actually Use Indexed Tools
187 
188By default, AI assistants will ignore the indexed tools and fall back to reading entire files with Glob/Grep/Read. Soft language like "prefer" gets rationalized away. Add this to your project's `CLAUDE.md` (or equivalent instructions file) with **mandatory** language:
189 
190```
191## Codebase Navigation — MANDATORY
192 
193You MUST use codebase-index MCP tools FIRST when exploring or navigating the codebase. This is not optional.
194 
195- ALWAYS start with: get_project_summary, find_symbol, get_function_source, get_class_source,
196  get_structure_summary, get_dependencies, get_dependents, get_change_impact, get_call_chain, search_codebase
197- Only fall back to Read/Glob/Grep when codebase-index tools genuinely don't have what you need
198  (e.g. reading non-code files, config, frontmatter)
199- If you catch yourself reaching for Glob/Grep/Read to find or understand code, STOP and use
200  codebase-index instead
201```
202 
203The word "prefer" is too weak — models treat it as a suggestion and default to familiar tools. Mandatory language with explicit fallback criteria is what actually changes behavior.
204 
205### Available Tools (18)
206 
207| Tool | Description |
208|------|-------------|
209| `get_project_summary` | File count, packages, top classes/functions |
210| `list_files` | List indexed files with optional glob filter |
211| `get_structure_summary` | Structure of a file or the whole project |
212| `get_functions` | List functions with name, lines, params |
213| `get_classes` | List classes with name, lines, methods, bases |
214| `get_imports` | List imports with module, names, line |
215| `get_function_source` | Full source of a function/method |
216| `get_class_source` | Full source of a class |
217| `find_symbol` | Find where a symbol is defined (file, line, type) |
218| `get_dependencies` | What a symbol calls/uses |
219| `get_dependents` | What calls/uses a symbol |
220| `get_change_impact` | Direct + transitive dependents |
221| `get_call_chain` | Shortest dependency path (BFS) |
222| `get_file_dependencies` | Files imported by a given file |
223| `get_file_dependents` | Files that import from a given file |
224| `search_codebase` | Regex search across all files (max 100 results) |
225| `reindex` | Force full re-index (rarely needed — incremental updates happen automatically in git repos) |
226| `get_usage_stats` | Session efficiency stats: tool calls, characters returned vs total source, estimated token savings |
227 
228## Benchmarks
229 
230Tested across four real-world projects on an M-series MacBook Pro, from a small project to CPython itself (1.1 million lines):
231 
232### Index Build Performance
233 
234| Project | Files | Lines | Functions | Classes | Index Time | Peak Memory |
235|---------|------:|------:|----------:|--------:|-----------:|------------:|
236| RMLPlus | 36 | 7,762 | 237 | 55 | 0.9s | 2.4 MB |
237| FastAPI | 2,556 | 332,160 | 4,139 | 617 | 5.7s | 55 MB |
238| Django | 3,714 | 707,493 | 29,995 | 7,371 | 36.2s | 126 MB |
239| **CPython** | **2,464** | **1,115,334** | **59,620** | **9,037** | **55.9s** | **197 MB** |
240 
241With persistent caching, subsequent startups bypass the full build entirely. Cache load time is negligible compared to parsing — a cache hit on CPython restores the full index in under a second instead of 56s.
242 
243### Query Response Size vs Total Source
244 
245Querying CPython — 41 million characters of source code:
246 
247| Query | Response | Total Source | Reduction |
248|-------|-------:|------------:|----------:|
249| `find_symbol("TestCase")` | 67 chars | 41,077,561 chars | **99.9998%** |
250| `get_dependencies("compile")` | 115 chars | 41,077,561 chars | **99.9997%** |
251| `get_change_impact("TestCase")` | 16,812 chars | 41,077,561 chars | **99.96%** |
252| `get_function_source("compile")` | 4,531 chars | 41,077,561 chars | **99.99%** |
253| `get_function_source("run_unittest")` | 439 chars | 41,077,561 chars | **99.999%** |
254 
255`find_symbol` returns 54-67 characters regardless of whether the project is 7K lines or 1.1M lines. Response size scales with the answer, not the codebase.
256 
257`get_change_impact("TestCase")` on CPython found **154 direct dependents and 492 transitive dependents** in 0.45ms — the kind of query that's impossible without a dependency graph. Use `max_direct` and `max_transitive` to cap output to your token budget.
258 
259### Query Response Time
260 
261All targeted queries return in sub-millisecond time, even on CPython's 1.1M lines:
262 
263| Query | RMLPlus | FastAPI | Django | CPython |
264|-------|--------:|--------:|-------:|--------:|
265| `find_symbol` | 0.01ms | 0.01ms | 0.03ms | 0.08ms |
266| `get_dependencies` | 0.00ms | 0.00ms | 0.00ms | 0.01ms |
267| `get_change_impact` | 0.02ms | 0.00ms | 2.81ms | 0.45ms |
268| `get_function_source` | 0.01ms | 0.02ms | 0.03ms | 0.10ms |
269 
270Run the benchmarks yourself: `python benchmarks/benchmark.py`
271 
272## How Is This Different from LSP?
273 
274LSP answers "where is this function?" — mcp-codebase-index answers "what happens if I change it?" LSP is point queries: one symbol, one file, one position. It can tell you where `LLMClient` is defined and who references it. But ask "what breaks transitively if I refactor `LLMClient`?" and LSP has nothing. This tool returns 11 direct dependents and 31 transitive impacts in a single call — 204 characters. To get the same answer from LSP, the AI would need to chain dozens of find-reference calls recursively, reading files at every step, burning thousands of tokens to reconstruct what the dependency graph already knows.
275 
276LSP also requires you to install a separate language server for every language in your project — pyright for Python, vtsls for TypeScript, gopls for Go. Each one is a heavyweight binary with its own dependencies and configuration. mcp-codebase-index is zero dependencies, handles Python + TypeScript/JS + Go + Rust + Markdown out of the box, and every response has built-in token budget controls (`max_results`, `max_lines`). LSP was built for IDEs. This was built for AI.
277 
278## Programmatic Usage
279 
280```python
281from mcp_codebase_index.project_indexer import ProjectIndexer
282from mcp_codebase_index.query_api import create_project_query_functions
283 
284indexer = ProjectIndexer("/path/to/project", include_patterns=["**/*.py"])
285index = indexer.index()
286query_funcs = create_project_query_functions(index)
287 
288# Use query functions
289print(query_funcs["get_project_summary"]())
290print(query_funcs["find_symbol"]("MyClass"))
291print(query_funcs["get_change_impact"]("some_function"))
292```
293 
294## Development
295 
296```bash
297pip install -e ".[dev,mcp]"
298pytest tests/ -v
299ruff check src/ tests/
300```
301 
302## References
303 
304The structural indexer was originally developed as part of the [RMLPlus](https://github.com/MikeRecognex/RMLPlus) project, an implementation of the [Recursive Language Models](https://arxiv.org/abs/2512.24601) framework.
305 
306## License
307 
308This project is dual-licensed:
309 
310- **AGPL-3.0** for open-source use — see [LICENSE](LICENSE)
311- **Commercial License** for proprietary use — see [COMMERCIAL-LICENSE.md](COMMERCIAL-LICENSE.md)
312 
313If you're using mcp-codebase-index as a standalone MCP server for development, the AGPL-3.0 license applies at no cost. If you're embedding it in a proprietary product or offering it as part of a hosted service, you'll need a commercial license. See [COMMERCIAL-LICENSE.md](COMMERCIAL-LICENSE.md) for details.
314