Isaac Sim MCP Extension and MCP Server

1# Isaac Sim MCP Extension and MCP Server
2 
3The MCP Server and its extension leverage the Model Context Protocol (MCP) framework to enable natural language control of NVIDIA Isaac Sim, transforming conversational AI inputs into precise simulation manipulation. This expansion bridges the MCP ecosystem with embodied intelligence applications.
4 
5## Features
6 
7- Natural language control of Isaac Sim
8- Dynamic robot positioning and movement
9- Custom lighting and scene creation
10- Advanced robot simulations with obstacle navigation
11- Interactive code preview before execution
12 
13## Requirements
14 
15- NVIDIA Isaac Sim 4.2.0 or higher
16- Python 3.9+
17- Cursor AI editor for MCP integration
18 
19## **Mandatory** Pre-requisite
20 
21- Install uv/uvx: [https://github.com/astral-sh/uv](https://github.com/astral-sh/uv)
22- Install mcp[cli] to base env: [uv pip install "mcp[cli]"](https://pypi.org/project/mcp/)
23 
24## Installation
25 
26```bash
27cd ~/Documents
28git clone https://github.com/omni-mcp/isaac-sim-mcp
29```
30 
31### Install and Enable Extension
32 
33Isaac Sim extension folder should point to your project folder:
34- Extension location: `~/Documents/isaac-sim-mcp` 
35- Extension ID: `isaac.sim.mcp_extension`
36 
37```bash
38# Enable extension in Isaac Simulation
39# cd to your Isaac Sim installation directory
40# You can change assets root to local with --/persistent/isaac/asset_root/default="<your asset location>"
41# By default it is an AWS bucket, e.g. --/persistent/isaac/asset_root/default="/share/Assets/Isaac/4.2"
42# Setup API KEY for Beaver3d and NVIDIA
43export BEAVER3D_MODEL=<your beaver3d model name>
44export export ARK_API_KEY=<Your Bearver3D API Key>
45export NVIDIA_API_KEY="<your nvidia api key  and apply it from https://ngc.nvidia.com/signout>"
46 
47cd ~/.local/share/ov/pkg/isaac-sim-4.2.0
48./isaac-sim.sh --ext-folder /home/ubuntu/Documents/isaac-sim-mcp/ --enable isaac.sim.mcp_extension 
49```
50 
51Verify the extension starts successfully. The output should look like:
52 
53```
54[7.160s] [ext: isaac.sim.mcp_extension-0.1.0] startup
55trigger  on_startup for:  isaac.sim.mcp_extension-0.1.0
56settings:  {'envPath': '/home/ubuntu/.local/share/ov/data/Kit/Isaac-Sim/4.2/pip3-envs/default', 'archiveDirs': {}, 'installCheckIgnoreVersion': False, 'allowOnlineIndex': True, 'tryUpgradePipOnFirstUse': False}
57Server thread startedIsaac Sim MCP server started on localhost:8766
58```
59 
60The extension should be listening at **localhost:8766** by default.
61 
62 
63 
64### Install MCP Server
65 
661. Go to terminal and run, make sure mcp server could start sucessfully at terminal with base venv.
67   ```
68   uv pip install "mcp[cli]"
69   uv run /home/ubuntu/Documents/isaac-sim-mcp/isaac_mcp/server.py
70   ```
712. Start Cursor and open the folder `~/Documents/isaac-sim-mcp`
723. Go to Cursor preferences, choose MCP and add a global MCP server:
73 
74```json
75{
76    "mcpServers": {
77        "isaac-sim": {
78            "command": "uv run /home/ubuntu/Documents/isaac-sim-mcp/isaac_mcp/server.py"
79        }
80    }
81}
82```
83 
84### Development Mode
85 
86To develop the MCP Server, start the MCP inspector:
87 
88```bash
89uv run mcp dev ~/Documents/isaac-sim-mcp/isaac_mcp/server.py
90```
91 
92You can visit the debug page through http://localhost:5173
93 
94## Example Prompts for Simulation
95Notice: Switch to Agent mode in top left of Chat dialog before you type prompt and choose sonnet 3.7 for better coding.
96 
97### Robot Party
98```
99# Create robots and improve lighting
100create  3x3 frankas robots in these current stage across location [3, 0, 0] and [6, 3, 0]
101always check connection with get_scene_info before execute code.
102add more light in the stage
103 
104 
105# Add specific robots at positions
106create a g1 robot at [3, 9, 0]
107add Go1 robot at location [2, 1, 0]
108move go1 robot to [1, 1, 0]
109```
110 
111### Factory Setup
112```
113# Create multiple robots in a row
114acreate  3x3 frankas robots in these current stage across location [3, 0, 0] and [6, 3, 0]
115always check connection with get_scene_info before execute code.
116add more light in the stage
117 
118 
119```
120### Vibe Coding from scratch
121```
122reference to g1.py to create an new g1 robot simulation and allow robot g1 walk straight  from [0, 0, 0] to [3, 0, 0] and [3, 3, 0]
123create more obstacles in the stage
124 
125```
126### Gen3D with beaver3d model support
127 
128```
129Use following images to generate beaver 3d objects and place them into a grid area across [0, 0, 0] to [40, 40, 0] with scale [3, 3, 3]
130 
131<your image url here, could be multiple images urls>
132```
133 
134### USD search
135```
136search a rusty desk and place it at [0, 5, 0] with scale [3, 3, 3]
137```
138 
139## MCP Tools
140 
141The Isaac Sim MCP Extension provides several specialized tools that can be accessed through natural language in Cursor AI. These tools enable you to control and manipulate NVIDIA Isaac Sim with simple commands:
142 
143### Connection and Scene Management
144 
145- **get_scene_info** - Pings the Isaac Sim Extension Server to verify connection status and retrieve basic scene information. Always use this first to ensure the connection is active.
146 
147### Physics and Environment Creation
148 
149- **create_physics_scene** - Creates a physics scene with configurable parameters:
150  - `objects`: List of objects to create (each with type and position)
151  - `floor`: Whether to create a ground plane (default: true)
152  - `gravity`: Vector defining gravity direction and magnitude (default: [0, -0.981, 0])
153  - `scene_name`: Name for the scene (default: "physics_scene")
154 
155### Robot Creation and Control
156 
157- **create_robot** - Creates a robot in the scene at a specified position:
158  - `robot_type`: Type of robot to create (options: "franka", "jetbot", "carter", "g1", "go1")
159  - `position`: [x, y, z] position coordinates
160 
161### Omniverse Kit and Scripting
162 
163- **omni_kit_command** - Executes an Omni Kit command:
164  - `command`: The Omni Kit command to execute (e.g., "CreatePrim")
165  - `prim_type`: The primitive type for the command (e.g., "Sphere")
166 
167- **execute_script** - Executes arbitrary Python code in Isaac Sim:
168  - `code`: Python code to execute
169 
170### Usage Best Practices
171 
1721. Always check connection with `get_scene_info` before executing any commands
1732. Initialize a physics scene with `create_physics_scene` before adding robots
1743. Use `create_robot` for standard robot placement before trying custom scripts
1754. For complex simulations, use `execute_script` with proper async patterns
1765. Preview code in chat before execution for verification
177 
178## Contributing
179 
180Contributions are welcome! Please feel free to submit a Pull Request.
181 
182## License
183 
184This project is licensed under the MIT License - see the LICENSE file for details.
185 
186## Video Demonstrations
187 
188Below are demonstrations of the Isaac Sim MCP Extension in action:
189 
190### Robot Party Demo
191 
192![Robot Party Demo](media/add_more_robot_into_party.gif)
193 
194*GIF: Adding more robots to the simulation using natural language commands*
195 
196 
197### Video Format (MP4)
198 
199For higher quality video, you can access the MP4 version directly:
200 
201- [Robot Party Demo (MP4)](media/add_more_robot_into_party.mp4)
202 
203When viewing on GitHub, you can click the link above to view or download the MP4 file.
204