Swift MCP GUI Server

1# Swift MCP GUI Server
2 
3A Model Context Protocol (MCP) server that allows controlling macOS through [SwiftAutoGUI](https://github.com/NakaokaRei/SwiftAutoGUI). This server provides tools for programmatically controlling the mouse and keyboard through MCP clients.
4 
5## Requirements
6 
7- macOS 15.0 or later
8- Swift 6.0 or later
9- Xcode 16.0 or later
10 
11## Installation
12 
131. Clone this repository:
14```bash
15git clone https://github.com/NakaokaRei/swift-mcp-gui.git
16cd swift-mcp-gui
17```
18 
192. Install
20```bash
21swift package experimental-install
22```
23 
243. Add command to your MCP client.
25```json
26{
27  "mcpServers" : {
28    "swift-mcp-gui" : {
29      "command" : "/Users/USERNAME/.swiftpm/bin/swift-mcp-gui"
30    }
31  }
32}
33 
34```
35 
36## Available Tools
37 
38The server provides the following tools for controlling macOS:
39 
40### 1. Mouse Movement
41- Tool name: `moveMouse`
42- Input:
43  - `x`: number (x-coordinate) - accepts integers, doubles, or string representations
44  - `y`: number (y-coordinate) - accepts integers, doubles, or string representations
45- Moves the mouse cursor to the specified coordinates
46 
47### 2. Mouse Clicks
48- Tool name: `mouseClick`
49- Input:
50  - `button`: string ("left" or "right")
51- Performs a mouse click at the current cursor position
52 
53### 3. Keyboard Input
54- Tool name: `sendKeys`
55- Input:
56  - `keys`: array of strings (key names)
57- Sends keyboard shortcuts or key combinations
58- Example keys: "command", "control", "option", "shift", "return", "space", "a", "1", etc.
59 
60### 4. Scrolling
61- Tool name: `scroll`
62- Input:
63  - `direction`: string ("up", "down", "left", "right")
64  - `clicks`: number (number of scroll clicks)
65- Performs scrolling in the specified direction
66 
67### 5. Screen Size
68- Tool name: `getScreenSize`
69- Returns the main screen dimensions (width and height)
70 
71### 6. Pixel Color
72- Tool name: `getPixelColor`
73- Input:
74  - `x`: number (x-coordinate) - accepts integers, doubles, or string representations
75  - `y`: number (y-coordinate) - accepts integers, doubles, or string representations
76- Returns the RGBA color values (0-255) of the pixel at the specified coordinates
77 
78### 7. Capture Screen
79- Tool name: `captureScreen`
80- Input:
81  - `quality`: number (optional, 0.0-1.0, default: 0.5) - JPEG compression quality
82  - `scale`: number (optional, 0.1-1.0, default: 0.25) - Scale factor for image size
83  - `output`: string (optional, "path" or "image", default: "path") - Output format
84- `output: "path"` (default): Saves to a temporary file and returns the file path with dimensions. Reduces token consumption.
85- `output: "image"`: Returns inline image content for AI vision (e.g. Claude)
86 
87### 8. Capture Region
88- Tool name: `captureRegion`
89- Input:
90  - `x`: number (x-coordinate of the region)
91  - `y`: number (y-coordinate of the region)
92  - `width`: number (width of the region)
93  - `height`: number (height of the region)
94  - `quality`: number (optional, 0.0-1.0, default: 0.5) - JPEG compression quality
95  - `scale`: number (optional, 0.1-1.0, default: 0.25) - Scale factor for image size
96  - `output`: string (optional, "path" or "image", default: "path") - Output format
97- `output: "path"` (default): Saves to a temporary file and returns the file path with dimensions. Reduces token consumption.
98- `output: "image"`: Returns inline image content for AI vision (e.g. Claude)
99 
100### 9. Save Screenshot
101- Tool name: `saveScreenshot`
102- Input:
103  - `filename`: string (path to save the screenshot)
104  - `x`: number (optional, x-coordinate of the region)
105  - `y`: number (optional, y-coordinate of the region)
106  - `width`: number (optional, width of the region)
107  - `height`: number (optional, height of the region)
108  - `quality`: number (optional, 0.0-1.0, default: 0.1) - JPEG compression quality
109  - `scale`: number (optional, 0.1-1.0, default: 0.25) - Scale factor for image size
110- Captures the screen or a region and saves it to a file
111- File format is determined by the filename extension (.jpg, .jpeg, .png)
112- Quality parameter only affects JPEG files
113 
114### 10. Execute AppleScript
115- Tool name: `executeAppleScript`
116- Input:
117  - `script`: string (AppleScript code to execute)
118- Executes AppleScript code directly and returns the result
119- Returns "AppleScript Result: <result>" if the script returns a value
120- Returns "AppleScript executed successfully (no result returned)" if the script completes without returning a value
121 
122### 11. Execute AppleScript File
123- Tool name: `executeAppleScriptFile`
124- Input:
125  - `path`: string (path to the AppleScript file)
126- Executes an AppleScript from a file and returns the result
127- Returns "AppleScript Result: <result>" if the script returns a value
128- Returns "AppleScript file executed successfully (no result returned): <path>" if the script completes without returning a value
129 
130## Security Considerations
131 
132This server requires full accessibility permissions in System Preferences to control your mouse and keyboard. Be careful when running it and only connect trusted MCP clients.
133 
134## License
135 
136MIT License 
137