Make your meetings accessible to AI Agents 🤖 joinly.ai is a connector middleware designed to enable AI agents to join and actively participate in video calls. Through its MCP server, joinly.ai provides essential meeting tools and resources that can equip any AI agent with the skills to perform tasks and interact with you in real time during your meetings. - Live Interaction: Lets your agents exec
Add this skill
npx mdskills install joinly-ai/joinlyMCP server enabling AI agents to join and interact in video calls across platforms with rich tooling
1<p align="center">2 <a href="https://github.com/joinly-ai/assets">3 <picture>4 <source5 media="(prefers-color-scheme: dark)"6 srcset="https://raw.githubusercontent.com/joinly-ai/assets/main/animations/logo-animations/joinly_logo_black_cropped.gif"7 >8 <img9 alt="Animated joinly.ai logo"10 src="https://raw.githubusercontent.com/joinly-ai/assets/main/animations/logo-animations/joinly_logo_light_cropped.gif"11 >12 </picture>13 </a>14</p>1516[](https://github.com/joinly-ai/joinly/releases)17[](LICENSE)18[](https://github.com/joinly-ai/joinly)19[](https://discord.com/invite/AN5NEBkS4d)20[](https://github.com/joinly-ai/joinly/discussions)21[](https://cloud.joinly.ai)2223<h1 align="center">Make your meetings accessible to AI Agents 🤖</h1>2425**joinly.ai** is a connector middleware designed to enable AI agents to join and actively participate in video calls. Through its MCP server, joinly.ai provides essential [meeting tools](#tools) and [resources](#resources) that can equip any AI agent with the skills to perform tasks and interact with you in real time during your meetings.2627> Want to dive right in? Jump to the [Quickstart](#zap-quickstart)!28> Want to know more? Visit our [website](https://joinly.ai/)!2930> [!IMPORTANT]31> Don't want the hustle of setting everything up? Try our [cloud](https://cloud.joinly.ai) first! ☁️🚀323334# :sparkles: Features35- **Live Interaction**: Lets your agents execute tasks and respond in real-time by voice or chat within your meetings36- **Conversational flow**: Built-in logic that ensures natural conversations by handling interruptions and multi-speaker interactions37- **Cross-platform**: Join Google Meet, Zoom, and Microsoft Teams (or any available over the browser)38- **Bring-your-own-LLM**: Works with all LLM providers (also locally with Ollama)39- **Choose-your-preferred-TTS/STT**: Modular design supports multiple services - Whisper/Deepgram for STT and Kokoro/ElevenLabs/Deepgram for TTS (and more to come...)40- **100% open-source, self-hosted and privacy-first** :rocket:4142# :video_camera: Demos43### GitHub44[](https://youtu.be/XWolVuxw8I8)45> In this demo video, joinly answers the question 'What is Joinly?' by accessing the latest news from the web. It then creates an issue in a GitHub demo repository.46### Notion47[](https://www.youtube.com/watch?v=pvYqZi2KeI0)48> In this demo video, we connect joinly to our notion via MCP and let it edit the content of a page content live in the meeting.4950Any ideas what we should build next? [Write us!](https://discord.com/invite/AN5NEBkS4d) :rocket:5152# :zap: Quickstart53Run joinly via Docker with a basic conversational agent client.5455> [!IMPORTANT]56> **Prerequisites**: [Docker installation](https://docs.docker.com/engine/install/)5758Create a new folder `joinly` or clone this repository (not mandatory for the following steps). In this directory, create a new `.env` file with a valid API key for the LLM provider you want to use, e.g. OpenAI:5960> [!TIP]61> You can find the OpenAI API key [here](https://platform.openai.com/api-keys)6263```Dotenv64# .env65# for OpenAI LLM66# change key and model to your desired one67JOINLY_LLM_MODEL=gpt-4o68JOINLY_LLM_PROVIDER=openai69OPENAI_API_KEY=your-openai-api-key70```7172> [!NOTE]73> See [.env.example](.env.example) for complete configuration options including Anthropic (Claude) and Ollama setups. Replace the placeholder values with your actual API keys and adjust the model name as needed. Delete the placeholder values of the providers you don't use.747576Pull the Docker image (~2.3GB since it packages browser and models):77```bash78docker pull ghcr.io/joinly-ai/joinly:latest79```8081Launch your meeting in [Zoom](https://www.zoom.com), [Google Meet](https://meet.google.com) or Teams and let joinly join the meeting using the meeting link as `<MeetingURL>`. Then, run the following command from the folder where you created the `.env` file:82```bash83docker run --env-file .env ghcr.io/joinly-ai/joinly:latest --client <MeetingURL>84```85> :red_circle: Having trouble getting started? Let's figure it out together on our [discord](https://discord.com/invite/AN5NEBkS4d)!8687# :technologist: Run an external client88In Quickstart, we ran the Docker Container directly as a client using `--client`. But we can also run it as a server and connect to it from outside the container, which allows us to connect other MCP servers. Here, we run an external client using the [joinly-client package](https://pypi.org/project/joinly-client/) and connect it to the joinly MCP server.8990> [!IMPORTANT]91> **Prerequisites**: do the [Quickstart](#zap-quickstart) (except the last command), [install uv](https://github.com/astral-sh/uv), and open two terminals9293Start the joinly server in the first terminal (note, we are not using `--client` here and forward port `8000`):94```bash95docker run -p 8000:8000 ghcr.io/joinly-ai/joinly:latest96```9798While the server is running, start the example client implementation in the second terminal window to connect to it and join a meeting:99```bash100uvx joinly-client --env-file .env <MeetingUrl>101```102103## Add MCP servers to the client104Add the tools of any MCP server to the agent by providing a JSON configuration. The configuration file can contain multiple entries under `"mcpServers"` which will all be available as tools in the meeting (see [fastmcp client docs](https://gofastmcp.com/clients/client) for config syntax):105106```json107{108 "mcpServers": {109 "localServer": {110 "command": "npx",111 "args": ["-y", "package@0.1.0"]112 },113 "remoteServer": {114 "url": "http://mcp.example.com",115 "auth": "oauth"116 }117 }118}119```120121Add for example a [Tavily config](examples/config_tavily.json) for web searching, then run the client using the config file, here named `config.json`:122123```bash124uvx joinly-client --env-file .env --mcp-config config.json <MeetingUrl>125```126127# :wrench: Configurations128129Configurations can be given via env variables and/or command line args. Here is a list of common configuration options, which can be used when starting the docker container:130```bash131docker run --env-file .env -p 8000:8000 ghcr.io/joinly-ai/joinly:latest <MyOptionArgs>132```133134Alternatively, you can pass `--name`, `--lang`, and [provider settings](#providers) as command line arguments using `joinly-client`, which will override settings of the server:135```bash136uvx joinly-client <MyOptionArgs> <MeetingUrl>137```138139## Basic Settings140141In general, the docker image provides an MCP server which is started by default. But to quickly get started, we also include a client implementation that can be used via `--client`. Note, in this case no server is started and no other client can connect to it.142143```bash144# Start directly as client; default is as server, to which an external client can connect145--client <MeetingUrl>146147# Change participant name (default: joinly)148--name "AI Assistant"149150# Change language of TTS/STT (default: en)151# Note, availability depends on the TTS/STT provider152--lang de153154# Change host & port of the joinly MCP server155--host 0.0.0.0 --port 8000156```157158## Providers159160### Text-to-Speech161```bash162# Kokoro (local) TTS (default)163--tts kokoro164--tts-arg voice=<VoiceName> # optionally, set different voice165166# ElevenLabs TTS, include ELEVENLABS_API_KEY in .env167--tts elevenlabs168--tts-arg voice_id=<VoiceID> # optionally, set different voice169170# Deepgram TTS, include DEEPGRAM_API_KEY in .env171--tts deepgram172--tts-arg model_name=<ModelName> # optionally, set different model (voice)173```174175### Transcription176```bash177# Whisper (local) STT (default)178--stt whisper179--stt-arg model_name=<ModelName> # optionally, set different model (default: base), for GPU support see below180181# Deepgram STT, include DEEPGRAM_API_KEY in .env182--stt deepgram183--stt-arg model_name=<ModelName> # optionally, set different model184```185186## Debugging187```bash188# Start browser with a VNC server for debugging;189# forward the port and connect to it using a VNC client190--vnc-server --vnc-server-port 5900191192# Logging193-v # or -vv, -vvv194195# Help196--help197```198199## GPU Support200201We provide a Docker image with CUDA GPU support for running the transcription and TTS models on a GPU. To use it, you need to have the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) installed and `CUDA >= 12.6`. Then pull the CUDA-enabled image:202```bash203docker pull ghcr.io/joinly-ai/joinly:latest-cuda204```205206Run as client or server with the same commands as above, but use the `joinly:{version}-cuda` image and set `--gpus all`:207```bash208# Run as server209docker run --gpus all --env-file .env -p 8000:8000 ghcr.io/joinly-ai/joinly:latest-cuda -v210# Run as client211docker run --gpus all --env-file .env ghcr.io/joinly-ai/joinly:latest-cuda -v --client <MeetingURL>212```213214By default, the `joinly` image uses the Whisper model `base` for transcription, since it still runs reasonably fast on CPU. For `cuda`, it automatically defaults to `distil-large-v3` for significantly better transcription quality. You can change the model by setting `--stt-arg model_name=<model_name>` (e.g., `--stt-arg model_name=large-v3`). However, only the respective default models are packaged in the docker image, so it will start to download the model weights on container start.215216# :test_tube: Create your own agent217218You can also write your own agent and connect it to our joinly MCP server. See the [code examples](https://github.com/joinly-ai/joinly/client/README.md#code-usage) for the joinly-client package or the [client_example.py](examples/client_example.py) if you want a starting point that doesn't depend on our framework.219220The joinly MCP server provides following tools and resources:221222### Tools223224- **`join_meeting`** - Join meeting with URL, participant name, and optional passcode225- **`leave_meeting`** - Leave the current meeting226- **`speak_text`** - Speak text using TTS (requires `text` parameter)227- **`send_chat_message`** - Send chat message (requires `message` parameter)228- **`mute_yourself`** - Mute microphone229- **`unmute_yourself`** - Unmute microphone230- **`get_chat_history`** - Get current meeting chat history in JSON format231- **`get_participants`** - Get current meeting participants in JSON format232- **`get_transcript`** - Get current meeting transcript in JSON format, optionally filtered by minutes233- **`get_video_snapshot`** - Get an image from the current meeting, e.g., view a current screenshare234235### Resources236237- **`transcript://live`** - Live meeting transcript in JSON format, including timestamps and speaker information. Subscribable for real-time updates when new utterances are added.238239# :building_construction: Developing joinly.ai240241For development we recommend using the development container, which installs all necessary dependencies. To get started, install the DevContainer Extension for Visual Studio Code, open the repository and choose **Reopen in Container**.242243<img src="https://raw.githubusercontent.com/joinly-ai/assets/main/images/others/reopen_in_container.png" width="500" alt="Reopen in Devcontainer">244245The installation can take some time, since it downloads all packages as well as models for Whisper/Kokoro and the Chromium browser. At the end, it automatically invokes the [download_assets.py](scripts/download_assets.py) script. If you see errors like `Missing kokoro-v1.0.onnx`, run this script manually using:246```bash247uv run scripts/download_assets.py248```249250We'd love to see what you are using it for or building with it. Showcase your work on our [discord](https://discord.com/invite/AN5NEBkS4d)251# :pencil2: Roadmap252253**Meeting**254- [x] Meeting chat access255- [ ] Camera in video call with status updates256- [ ] Enable screen share during video conferences257- [ ] Participant metadata and joining/leaving258- [ ] Improve browser agent capabilities259260**Conversation**261- [x] Speaker attribute for transcription262- [ ] Improve client memory: reduce token usage, allow persistence across meetings263events264- [ ] Improve End-of-Utterance/turn-taking detection265- [ ] Human approval mechanism from inside the meeting266267**Integrations**268- [ ] Showcase how to add agents using the A2A protocol269- [ ] Add more provider integrations (STT, TTS)270- [ ] Integrate meeting platform SDKs271- [ ] Add alternative open-source meeting provider272- [ ] Add support for Speech2Speech models273274# :busts_in_silhouette: Contributing275Contributions are always welcome! Feel free to open issues for bugs or submit a feature request. We'll do our best to review all contributions promptly and help merge your changes.276277Please check our [Roadmap](#pencil2-roadmap) and don't hesitate to reach out to us!278279# :memo: License280This project is licensed under the MIT License ‒ see the [LICENSE](LICENSE) file for details.281282# :speech_balloon: Getting help283If you have questions or feedback, or if you would like to chat with the maintainers or other community members, please use the following links:284- [Join our Discord](https://discord.com/invite/AN5NEBkS4d)285- [Explore our GitHub Discussions](https://github.com/joinly-ai/joinly/discussions)286287<div align="center">288Made with ❤️ in Osnabrück289 </div>290
Full transparency — inspect the skill content before installing.