Omniparser Autogui MCP

1# omniparser-autogui-mcp
2 
3（[日本語版はこちら](README_ja.md)）
4 
5This is an [MCP server](https://modelcontextprotocol.io/introduction) that analyzes the screen with [OmniParser](https://github.com/microsoft/OmniParser) and automatically operates the GUI.  
6Confirmed on Windows.
7 
8## License notes
9 
10This is MIT license, but Excluding submodules and sub packages.  
11OmniParser's repository is CC-BY-4.0.  
12Each OmniParser model has a different license ([reference](https://github.com/microsoft/OmniParser?tab=readme-ov-file#model-weights-license)).
13 
14## Installation
15 
161. Please do the following:
17 
18```
19git clone --recursive https://github.com/NON906/omniparser-autogui-mcp.git
20cd omniparser-autogui-mcp
21uv sync
22set OCR_LANG=en
23uv run download_models.py
24```
25 
26(Other than Windows, use ``export`` instead of ``set``.)  
27(If you want ``langchain_example.py`` to work, ``uv sync --extra langchain`` instead.)
28 
292. Add this to your ``claude_desktop_config.json``:
30 
31```claude_desktop_config.json
32{
33  "mcpServers": {
34    "omniparser_autogui_mcp": {
35      "command": "uv",
36      "args": [
37        "--directory",
38        "D:\\CLONED_PATH\\omniparser-autogui-mcp",
39        "run",
40        "omniparser-autogui-mcp"
41      ],
42      "env": {
43        "PYTHONIOENCODING": "utf-8",
44        "OCR_LANG": "en"
45      }
46    }
47  }
48}
49```
50 
51(Replace ``D:\\CLONED_PATH\\omniparser-autogui-mcp`` with the directory you cloned.)
52 
53``env`` allows for the following additional configurations:
54 
55- ``OMNI_PARSER_BACKEND_LOAD``  
56If it does not work with other clients (such as [LibreChat](https://github.com/danny-avila/LibreChat)), specify ``1``.
57 
58- ``TARGET_WINDOW_NAME``  
59If you want to specify the window to operate, please specify the window name.  
60If not specified, operates on the entire screen.
61 
62- ``OMNI_PARSER_SERVER``  
63If you want OmniParser processing to be done on another device, specify the server's address and port, such as ``127.0.0.1:8000``.  
64The server can be started with ``uv run omniparserserver``.
65 
66- ``SSE_HOST``, ``SSE_PORT``  
67If specified, communication will be done via SSE instead of stdio.
68 
69- ``SOM_MODEL_PATH``, ``CAPTION_MODEL_NAME``, ``CAPTION_MODEL_PATH``, ``OMNI_PARSER_DEVICE``, ``BOX_TRESHOLD``  
70These are for OmniParser configuration.  
71Usually, they are not necessary.
72 
73## Usage Examples
74 
75- Search for "MCP server" in the on-screen browser.
76 
77etc.