mcp-video-analyzer Turn any video — YouTube, Instagram, TikTok, Loom, X, Vimeo, direct links, local files — into transcripts, key frames, OCR text, and metadata for AI agents. No existing video MCP combines transcripts + visual frames + metadata in one tool. This one does — across Loom, the major yt-dlp platforms (YouTube/Vimeo/TikTok/Instagram/X/Twitch/Dailymotion/Facebook), direct video URLs, an
npx mdskills install guimatheus92/mcp-video-analyzerRelated
@guimatheus92? Sign in with GitHub to claim this listing.Comprehensive video analysis with transcripts, frames, OCR, and metadata across major platforms
1<p align="center">2 <img src="assets/icon.svg" width="88" height="88" alt="mcp-video-analyzer" />3</p>45<h1 align="center">mcp-video-analyzer</h1>67<p align="center"><em>Turn any video — YouTube, Instagram, TikTok, Loom, X, Vimeo, direct links, local files — into transcripts, key frames, OCR text, and metadata for AI agents.</em></p>89<p align="center">10 <a href="https://www.npmjs.com/package/mcp-video-analyzer"><img src="https://img.shields.io/npm/v/mcp-video-analyzer?color=e8468f&labelColor=1e1e2e&logo=npm&logoColor=white" alt="npm" /></a>11 <a href="https://github.com/guimatheus92/mcp-video-analyzer/blob/main/LICENSE"><img src="https://img.shields.io/npm/l/mcp-video-analyzer?color=8b5cf6&labelColor=1e1e2e" alt="license" /></a>12 <a href="https://github.com/punkpeye/awesome-mcp-servers#-multimedia-process"><img src="https://img.shields.io/badge/awesome--mcp--servers-listed-ff4d6d?labelColor=1e1e2e" alt="awesome-mcp-servers" /></a>13 <a href="https://mcpservers.org/servers/guimatheus92/mcp-video-analyzer"><img src="https://img.shields.io/badge/mcpservers.org-listed-38bdf8?labelColor=1e1e2e" alt="mcpservers.org" /></a>14</p>1516<p align="center">17 <a href="https://glama.ai/mcp/servers/guimatheus92/mcp-video-analyzer"><img width="360" height="190" src="https://glama.ai/mcp/servers/guimatheus92/mcp-video-analyzer/badge" alt="mcp-video-analyzer MCP server" /></a>18</p>1920No existing video MCP combines **transcripts + visual frames + metadata** in one tool. This one does — across Loom, the major yt-dlp platforms (YouTube/Vimeo/TikTok/Instagram/X/Twitch/Dailymotion/Facebook), direct video URLs, and local files.2122> **Want a full pipeline, not just a tool?** [social-knowledge-base](https://github.com/guimatheus92/social-knowledge-base) is built on top of this server — it downloads whole Instagram creator accounts (reels, stories, highlights), transcribes them, and turns the result into a searchable, RAG-queryable knowledge base with AI-generated notes. Use this MCP when you want per-video analysis inside an agent; use social-knowledge-base when you want to archive and query an entire account.2324## Installation2526### Prerequisites2728- **Node.js 22.12+** — required to run the server via `npx`29- **yt-dlp** — **required** for YouTube/Vimeo/TikTok/Instagram/X/Twitch/Dailymotion/Facebook URLs; optional for everything else (improves Loom download quality). Install with `pip install yt-dlp`30- **Chrome/Chromium** (optional) — fallback for frame extraction if yt-dlp is unavailable3132> Without yt-dlp or Chrome, direct URLs and local files still get frames — the bundled `ffmpeg-static` does the extraction, and Loom falls back to its own CDN download. Platform URLs (YouTube etc.) degrade to a clear "install yt-dlp" warning. Transcripts, metadata, and comments never require either.3334There are three ways in: the **`/video` plugin** (Claude Code — slash command + MCP server auto-configured), a plain **MCP server** config (any MCP client), or the **portable skill + CLI** (Codex, Cursor, Copilot, and any agent with a shell — no MCP required).3536### Claude Code — `/video` plugin (recommended)3738```39/plugin marketplace add guimatheus92/mcp-video-analyzer40/plugin install video@mcp-video-analyzer41```4243This adds the `/video` slash command **and** auto-registers the MCP server — no `claude mcp add` needed:4445```46/video https://youtu.be/jNQXAC9IVRw what happens at 0:10?47/video ~/Movies/screen-recording.mp4 when does the UI break?48```4950### Other agents — Codex, Cursor, Copilot, Gemini CLI, …5152```bash53npx skills add guimatheus92/mcp-video-analyzer54```5556Installs the `video` skill ([Agent Skills](https://github.com/vercel-labs/skills) format) into every agent detected on your machine. Agents without the MCP server configured fall back to the bundled [CLI](#cli-one-shot-no-mcp-client) automatically — zero configuration.5758### Claude Code (MCP only)5960```bash61claude mcp add video-analyzer -- npx mcp-video-analyzer@latest62```6364Then restart Claude Code or start a new conversation.6566### VS Code / Cursor6768Add to your MCP settings file:6970- **VS Code**: `File → Preferences → Settings → search "MCP"` or edit `~/.vscode/mcp.json` / `%APPDATA%\Code\User\mcp.json` (Windows)71- **Cursor**: `Settings → MCP Servers → Add`7273```json74{75 "servers": {76 "mcp-video-analyzer": {77 "type": "stdio",78 "command": "npx",79 "args": ["mcp-video-analyzer@latest"]80 }81 }82}83```8485Then reload the window (`Ctrl+Shift+P` → "Developer: Reload Window").8687### Claude Desktop8889Add to your Claude Desktop config file:9091- **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`92- **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`9394```json95{96 "mcpServers": {97 "video-analyzer": {98 "command": "npx",99 "args": ["mcp-video-analyzer@latest"]100 }101 }102}103```104105Then restart Claude Desktop.106107### CLI (one-shot, no MCP client)108109The same engine is exposed as a one-shot command — this is what the `video` skill uses on agents without MCP, and it works standalone in any terminal:110111```bash112npx -y mcp-video-analyzer@latest analyze "https://youtu.be/jNQXAC9IVRw"113```114115stdout is a single JSON document — `metadata`, `transcript`, `ocrResults`, `timeline`, `warnings`, `frameCount`, and `frames` as `{ time, filePath, mimeType }` entries pointing at JPEG key frames copied to `--out` (default: the per-user cache dir — `%LOCALAPPDATA%` on Windows, `~/Library/Caches` on macOS, `$XDG_CACHE_HOME` or `~/.cache` on Linux — under `mcp-video-analyzer/<url-hash>/`; set `MCP_CACHE_DIR` to an absolute path to relocate it). Unlike the temp dir this used to live in, nothing reaps that location, so frames persist until you delete them — the directories are created `0700`. Progress streams on stderr, so `stdout` can be piped straight into a JSON parser. Partial failures land in `warnings` with exit code 0; only hard failures exit 1.116117| Flag | Description |118|------|-------------|119| `--detail <level>` | `brief` (metadata + transcript, no frames), `standard` (default), `detailed` |120| `--max-frames <n>` | Max key frames, 1–60 (default adapts to duration) |121| `--max-width <px>` | Width cap for emitted frames (default `800`, or `MCP_FRAME_MAX_WIDTH`); `0` keeps the source resolution — see [Frame size](#frame-size-dense-ui-captures) |122| `--fields <list>` | Output filter — comma-separated subset: `metadata,transcript,frames,comments,chapters,ocrResults,timeline,aiSummary`. Filters the emitted JSON only; use `--detail brief` to actually skip download/frame extraction |123| `--force-refresh` | Bypass the cache and re-analyze |124| `--ocr-language <codes>` | Tesseract languages (default `eng+por`) |125| `--model <name>` / `--language <code>` | Whisper overrides for the transcription fallback |126| `--out <dir>` | Where frame images are copied |127128Run with no arguments (`npx mcp-video-analyzer@latest`) to start the MCP stdio server — the CLI is purely additive.129130### Verify it works131132Once installed, ask your AI assistant:133134```135Analyze this video: https://www.youtube.com/watch?v=jNQXAC9IVRw136```137138(also works with an Instagram/TikTok/Loom link, a direct `.mp4` URL, or a local file path). If the server is connected, it will automatically call the `analyze_video` tool.139140## Tools141142Eight tools — the AI picks the cheapest one for the job and calls it automatically. Click any tool to expand its parameters and examples.143144| Tool | What it does |145|------|--------------|146| **`analyze_video`** | Full analysis: transcript + key frames + OCR + timeline + metadata |147| **`analyze_videos`** | Batch version, one structured result per source (resumable) |148| **`get_transcript`** | Transcript only (native captions or Whisper fallback) |149| **`get_metadata`** | Metadata + comments + chapters, no download |150| **`get_frames`** | Key frames only (scene-change or dense 1 fps) |151| **`analyze_moment`** | Deep-dive on a time range (burst frames + transcript + OCR) |152| **`get_frame_at`** | Single frame at a timestamp |153| **`get_frame_burst`** | N frames across a narrow window (motion/animation) |154155<details>156<summary><b><code>analyze_video</code></b> — full video analysis</summary>157158<br>159160Extracts everything from a video URL in one call:161162```163> Analyze this video: https://www.youtube.com/watch?v=abc123...164```165166Returns:167- **Transcript** with timestamps and speakers168- **Key frames** extracted via scene-change detection (automatically deduplicated). For static clips with no scene cuts — e.g. talking-head Reels/Stories where only an on-screen text overlay changes — it automatically falls back to uniform temporal sampling so you still get frames (and OCR) instead of an empty result.169- **OCR text** extracted from frames (code, error messages, UI text, prices/dates/CTAs visible on screen)170- **Annotated timeline** merging transcript + frames + OCR into a unified "what happened when" view171- **Metadata** (title, duration, platform)172- **Comments** from viewers173- **Chapters** and **AI summary** (when available)174175The AI will **automatically** call this tool when it sees a video URL — no need to ask.176177Options:178- `detail` — analysis depth: `"brief"` (metadata + truncated transcript, no frames), `"standard"` (default), `"detailed"` (dense sampling, more frames)179- `fields` — array of specific fields to return, e.g. `["metadata", "transcript"]`. Available: `metadata`, `transcript`, `frames`, `comments`, `chapters`, `ocrResults`, `timeline`, `aiSummary`180- `maxFrames` (1-60) — cap on extracted frames. Default scales with video duration at `standard` detail (~12 for ≤30s up to 60 for >10min); fixed 60 at `detailed`, 0 at `brief`. An explicit value always wins181- `threshold` (0.0-1.0, default 0.1) — scene-change sensitivity182- `forceRefresh` — bypass cache and re-analyze183- `skipFrames` — skip frame extraction for transcript-only analysis184- `model` / `language` / `initialPrompt` — per-call Whisper overrides for the transcription fallback (override `WHISPER_MODEL` / `WHISPER_LANGUAGE` / `WHISPER_PROMPT` for this call only — pick a heavier model or a domain glossary for one hard clip without restarting the server)185186</details>187188<details>189<summary><b><code>analyze_videos</code></b> — batch analysis</summary>190191<br>192193```194> Analyze every .mp4 in this folder195```196197Runs `analyze_video` over a list of `sources` with a `concurrency` limit (default 2), returning one **structured result per source** — counts + warnings on success, or a per-item `error` on failure (one bad file never aborts the batch). Frame images are not inlined and full transcript/OCR/timeline are returned only when `fields` is set; otherwise you get counts. Pair with `MCP_WRITE_SIDECARS=1` (below) so each video's result persists to disk and a re-run resumes instead of recomputing.198199</details>200201<details>202<summary><b><code>get_transcript</code></b> — transcript only</summary>203204<br>205206```207> Get the transcript from this video208```209210Quick transcript extraction. Falls back to Whisper transcription when no native transcript is available. Accepts the same per-call `model` / `language` / `initialPrompt` overrides as `analyze_video`.211212</details>213214<details>215<summary><b><code>get_metadata</code></b> — metadata only</summary>216217<br>218219```220> What's this video about?221```222223Returns metadata, comments, chapters, and AI summary without downloading the video.224225</details>226227<details>228<summary><b><code>get_frames</code></b> — frames only</summary>229230<br>231232```233> Extract frames from this video with dense sampling234```235236Two modes:237- **Scene-change detection** (default) — captures visual transitions238- **Dense sampling** (`dense: true`) — 1 frame/sec for full coverage239240</details>241242<details>243<summary><b><code>analyze_moment</code></b> — deep-dive on a time range</summary>244245<br>246247```248> Analyze what happens between 1:30 and 2:00 in this video249```250251Combines burst frame extraction + filtered transcript + OCR + annotated timeline for a focused segment. Use when you need to understand exactly what happens at a specific moment.252253</details>254255<details>256<summary><b><code>get_frame_at</code></b> — single frame at a timestamp</summary>257258<br>259260```261> Show me the frame at 1:23 in this video262```263264The AI reads the transcript, spots a critical moment, and requests the exact frame to see what's on screen.265266</details>267268<details>269<summary><b><code>get_frame_burst</code></b> — N frames in a time range</summary>270271<br>272273```274> Show me 10 frames between 0:15 and 0:17 of this video275```276277For motion, vibration, animations, or fast scrolling — burst mode captures N frames in a narrow window so the AI can see frame-by-frame changes.278279</details>280281## Detail Levels282283| Level | Frames | Transcript | OCR | Timeline | Use case |284|-------|--------|-----------|-----|----------|----------|285| `brief` | None | First 10 entries | No | No | Quick check — what's this video about? |286| `standard` | Duration-adaptive: ~12 (≤30s) up to 60 (>10min), scene-change | Full | Yes | Yes | Default — full analysis |287| `detailed` | Up to 60 (1fps dense) | Full | Yes | Yes | Deep analysis — every second captured |288289## Caching290291Results are cached in memory for 10 minutes. Subsequent calls with the same URL and options return instantly. Use `forceRefresh: true` to bypass the cache. `skipFrames` is part of the cache and sidecar key, so a transcript-only analysis and a framed one of the same URL never answer for each other.292293### Persistent sidecars (resumable bulk processing)294295The in-memory cache is lost on restart, which makes reprocessing a large local corpus costly. Set `MCP_WRITE_SIDECARS=1` to also persist results **next to each local video** so the work survives restarts and can resume:296297- `<stem>.vtt` — the transcript, **only** when it was generated by the Whisper fallback (an existing `<stem>.vtt` from your own pipeline is never overwritten). A later call reuses it via the normal sidecar reader and skips Whisper entirely.298- `<stem>.analysis.json` + `<stem>.frames/` — the full result (frames + OCR + timeline), keyed by the video's `mtime:size` and the analysis params. On a later call with a matching stamp + params, the result is returned straight from disk (no extraction, no OCR).299300This makes `analyze_videos` over thousands of files resumable, and lets an external GPU transcription pipeline and this MCP share results through the filesystem: the pipeline writes `<stem>.vtt`, and the MCP picks it up instead of running Whisper.301302## Supported Sources303304| Source | Transcript | Metadata | Comments | Frames | Auth |305|--------|:----------:|:--------:|:--------:|:------:|:----:|306| **Loom** | Yes | Yes | Yes | Yes (usually needs yt-dlp — see note) | None |307| **YouTube / Vimeo / TikTok / Instagram / X / Twitch / Dailymotion / Facebook** | Native captions (uploaded > auto-generated) or Whisper fallback | Yes (title, duration, uploader, views, chapters, upload date) | No | Yes (capped at 1080p) | yt-dlp installed; cookies for Instagram / age-restricted (see below) |308| **Direct URL** (.mp4, .mov, .mkv, .webm, …) | No | Duration only | No | Yes | None |309| **Direct URL + TwelveLabs** | Yes (Pegasus, best-effort) | Duration floor + title | No | Yes | `TWELVELABS_API_KEY` |310| **Local file** (absolute path or `file://` URI) | Sidecar `.vtt`/`.srt` or Whisper fallback | Probed via ffmpeg (duration, dims, codec, audio presence) | No | Yes | None |311312> **Loom frames**: transcript, metadata, and comments come straight from Loom's API with no extra tooling. Frame extraction is different — Loom serves most videos as separate DASH video+audio streams, which only [yt-dlp](https://github.com/yt-dlp/yt-dlp) (`pip install yt-dlp`) fetches and merges. Merging uses the bundled `ffmpeg-static`, so no system ffmpeg is required. Without yt-dlp a direct-CDN fallback still covers some videos; when it can't, you get transcript + metadata + comments plus a warning explaining why frames are missing.313>314> **Local files**: pass an absolute path (e.g., `/Users/you/clip.mp4`) or a `file://` URI as the `url` argument to any tool. Relative paths are rejected — the server's working directory is unpredictable from the MCP client. Note that any caller of the MCP server can ask it to read any file the server process has access to. UNC / network share paths (`\\host\share\clip.mp4`) are the exception: they reach the network rather than local disk, so they follow the [network destination](#network-destinations) rules and need `MCP_ALLOW_PRIVATE_URLS=1`.315>316> **Sidecar transcripts**: if a `clip.vtt`, `clip.srt`, `clip.en.vtt`, etc. lives next to `clip.mp4`, it's used as the transcript automatically — no Whisper roundtrip needed. SRT is converted to VTT in-memory.317>318> **Embedded subtitles**: if no sidecar is found and the container has an embedded subtitle stream (common in `.mkv` / `.mov` / `.mp4` from screen recorders), it's transmuxed to VTT via ffmpeg and used as the transcript.319>320> **Recognized extensions** (local files and direct URLs): `.mp4` `.mov` `.mkv` `.webm` `.avi` `.m4v` `.wmv` `.flv` `.mpeg` `.mpg` `.m2ts` `.mts` `.3gp` `.ogv`. The extension only gates routing — ffmpeg does the actual demuxing, so most common containers work. `.ts` is excluded to avoid colliding with TypeScript source files.321322### Network destinations323324Only `http://` and `https://` URLs are fetched, and only to **public** addresses. Requests to loopback (`localhost`, `127.0.0.1`, `::1`), private/LAN ranges, link-local, CGNAT, `.local` mDNS names, and Windows UNC paths are refused, as are non-HTTP schemes like `ftp://` and `data:`.325326This matters because the `url` argument is attacker-reachable in the normal case: an agent driving this server can be steered by the content it reads, so a URL it passes in is not necessarily one the user chose. Without the restriction the server is a proxy into whatever network it happens to sit on.327328The check runs on the resolved address, not just the text, so a public hostname that resolves to `10.0.0.5` is refused too — and **every hop of a redirect chain is re-checked**, since a public URL answering `302 Location: http://127.0.0.1/` would otherwise walk straight past a first-hop-only check.329330```bash331# Serving videos from a NAS or a local dev server? Opt back in:332MCP_ALLOW_PRIVATE_URLS=1333```334335Cloud instance metadata endpoints (`169.254.169.254`, Azure's `168.63.129.16`, and friends) stay blocked **even with that set** — there is no legitimate video there, and they are what an SSRF is usually after.336337Known limitation: the address is checked at resolution time, not at connection time, so DNS rebinding — a domain answering a public address to the check and a private one to the connection — is not covered. Run the server behind an egress proxy if that is in your threat model.338339### Platform URLs via yt-dlp (YouTube, Instagram, TikTok, …)340341Single-video pages on major platforms route through [yt-dlp](https://github.com/yt-dlp/yt-dlp) (`pip install yt-dlp` — required for these URLs). Playlists, channels, and profile pages are rejected by design; pass individual video URLs (batch them with `analyze_videos`).342343- **Transcript**: native captions are preferred and free — uploaded subtitles first, auto-generated captions as fallback (rolling-window duplication is collapsed). `WHISPER_LANGUAGE` (e.g. `pt`) is also used to pick the caption language. Videos with no captions at all fall through to the normal Whisper chain.344- **Metadata**: title, duration, uploader/channel, view count, upload date, and chapters — no download needed.345- **Download**: capped at 1080p (frames/OCR don't need more), live streams are skipped, and DASH audio+video is merged with the bundled `ffmpeg-static` (no system ffmpeg required).346- **Cookies** — Instagram and age-restricted videos usually require a logged-in session:347348| Env var | What it does | Example |349|---------|-------------|---------|350| `YTDLP_COOKIES` | Cookie file (Netscape format), wins when both are set | `C:/secrets/cookies.txt` |351| `YTDLP_COOKIES_FROM_BROWSER` | Extract cookies from an installed browser | `chrome`, `edge`, `firefox` |352353> Browser cookie extraction requires the browser to be **closed** on Windows (the cookie database is locked while it runs). If that's inconvenient, export a `cookies.txt` once (e.g. with a "Get cookies.txt" browser extension) and point `YTDLP_COOKIES` at it. Private/age-restricted videos without valid cookies don't crash the tool — the yt-dlp `ERROR:` line surfaces in `warnings[]`.354355### TwelveLabs Pegasus (optional)356357Set the `TWELVELABS_API_KEY` environment variable to analyze direct video URLs with [TwelveLabs](https://twelvelabs.io) **Pegasus**. Pegasus analyzes the video server-side (visuals **and** its own audio) and returns an **AI-generated, timestamped transcript** plus an AI summary as text — capabilities the `DirectAdapter` can't provide (a raw `.mp4` URL has no transcript or summary on its own), and with **no Whisper key required**.358359The transcript is best-effort LLM output, not a deterministic ASR dump: Pegasus is *prompted* to emit `[MM:SS] line` rows, and lines that don't match that shape are dropped, so wording and exact timestamps depend on the model's prompt adherence. Failures (bad key, timeout, API error) surface in the tool's `warnings[]` rather than silently returning an empty transcript.360361The biggest win is on the text-only paths: `get_transcript` and `get_metadata` return a Pegasus transcript and summary for direct URLs — a few KB of text, no frame images, no per-frame token cost. `analyze_video` at `detail: "standard"`/`"detailed"` still extracts frames in addition (use `detail: "brief"` to stay text-only).362363> **Long videos**: the summary and full transcript share a single capped completion (`max_tokens` = 16384), so for very long videos the transcript may be truncated. For multi-hour content, chunking by time window is the better approach.364365It's fully opt-in and non-breaking: when `TWELVELABS_API_KEY` is set the `TwelveLabsAdapter` handles direct video URLs (it registers the public URL with TwelveLabs — no upload); when it's unset, the `DirectAdapter` handles them exactly as before. Loom URLs are unaffected. Get a key at [playground.twelvelabs.io](https://playground.twelvelabs.io).366367### Transcription (Whisper fallback)368369When a source has no native transcript (no sidecar `.vtt`/`.srt`, no embedded subtitles, no platform captions), the audio track is transcribed with Whisper via a graceful fallback chain (in execution order):370371> **Silent tracks**: before any Whisper run, the audio is probed with ffmpeg `volumedetect` (first 2 minutes). A present-but-mute track — common in muted Reels/Stories — skips transcription entirely and emits a warning that the empty transcript is **expected content, not an error**, saving a pointless Whisper run.3723731. **@huggingface/transformers** (JS-native, zero external deps) — **opt-in only**: this strategy runs *first*, but **only when `WHISPER_HF_MODEL` is explicitly set**. When it's unset (the default) the strategy is skipped entirely, so the CLI below wins and its `WHISPER_MODEL`/`WHISPER_LANGUAGE` settings are never silently overridden.3742. **`whisper` CLI** — used when a `whisper` executable is found (`pip install -U openai-whisper`). Point `WHISPER_BIN` at the executable if it isn't on `PATH`. Model via `WHISPER_MODEL`, language via `WHISPER_LANGUAGE`. The bundled `ffmpeg-static` is put on the CLI's `PATH` automatically, so no system ffmpeg is required.3753. **OpenAI Whisper API** — used when `OPENAI_API_KEY` is set.376377> **No backend configured?** If none of the three is available (no `whisper` on `PATH`/`WHISPER_BIN`, no `OPENAI_API_KEY`, no `WHISPER_HF_MODEL`), transcription tools return an empty transcript **with a warning telling you how to enable one** — rather than a silent "no transcript". Install `openai-whisper` or set one of the keys above. (The CLI is spawned with `PYTHONUTF8=1` so non-English/CJK transcripts don't crash the Python process on Windows.)378379| Env var | Applies to | Default | Example |380|---------|-----------|---------|---------|381| `WHISPER_MODEL` | `whisper` CLI | `tiny` | `small`, `medium` |382| `WHISPER_LANGUAGE` | `whisper` CLI / OpenAI API | auto-detect | `pt`, `en`, `es` |383| `WHISPER_PROMPT` | `whisper` CLI / OpenAI API | — | `Doha, Smiles, Livelo, Latam, milheiro` |384| `WHISPER_BIN` | `whisper` CLI | `whisper` (on PATH) | `C:/.../Scripts/whisper.exe` |385| `WHISPER_DEVICE` | `whisper` CLI (sent only if set) | — | `cuda`, `cpu` |386| `WHISPER_COMPUTE` | `whisper-ctranslate2` only | — | `float16`, `int8_float16`, `int8` |387| `WHISPER_BEAM_SIZE` | `whisper` CLI (sent only if set) | — | `5` |388| `WHISPER_WORD_TIMESTAMPS` | `whisper` CLI (sent only if set) | off | `1` |389| `WHISPER_HF_MODEL` | HF transformers (opt-in) | — (strategy off) | `Xenova/whisper-small` |390| `OPENAI_API_KEY` | OpenAI API | — | `sk-…` |391392> The default `tiny` model is fast but weak for non-English audio. For Portuguese (or other non-English) sources, install the CLI and set `WHISPER_MODEL=small` (or `medium`) + `WHISPER_LANGUAGE=pt` for much better accuracy. Add `WHISPER_PROMPT` with a domain glossary (brand/place names) to fix proper nouns. You can also override `model`/`language`/`initialPrompt` **per call** on `analyze_video` / `get_transcript` / `analyze_videos` — no restart needed.393>394> **GPU (faster-whisper):** `whisper-ctranslate2` (`pip install -U whisper-ctranslate2`) is a drop-in CLI with the same flags plus `--device cuda` / `--compute_type` / `--beam_size`. Point `WHISPER_BIN` at it and set `WHISPER_DEVICE=cuda` (+ optionally `WHISPER_COMPUTE=float16`). These GPU flags are **env-gated** — they're only passed when set, so plain `openai-whisper` (which rejects `--compute_type`) keeps working when they're unset.395>396> **Windows note:** pip installs `whisper.exe` into the Python `Scripts/` dir, which is often **not** on the `PATH` that GUI-launched MCP clients inherit. If transcripts come back empty, set `WHISPER_BIN` to the full path of `whisper.exe`.397398### Frame Extraction Strategies399400Frame extraction uses a two-strategy fallback chain — no single dependency is required:401402| Strategy | How it works | Speed | Requirements |403|----------|-------------|-------|-------------|404| **yt-dlp + ffmpeg** (primary) | Downloads video, extracts frames via scene detection | Fast, precise | [yt-dlp](https://github.com/yt-dlp/yt-dlp) (`pip install yt-dlp`) |405| **Browser** (fallback) | Opens video in headless Chrome, seeks to timestamps, takes screenshots | Slower, no download needed | Chrome or Chromium installed |406407The fallback is automatic — if yt-dlp is not available, the server tries browser-based extraction via `puppeteer-core`. If neither is available, analysis still returns transcript + metadata + comments, just no frames.408409### Post-Processing Pipeline410411After frame extraction, the pipeline automatically applies:412413| Step | What it does | Why |414|------|-------------|-----|415| **Frame deduplication** | Removes near-identical consecutive frames using perceptual hashing (dHash + Hamming distance) | Screencasts often have long static moments — dedup removes redundant frames, saving tokens |416| **OCR** | Extracts text visible on screen from each frame (via tesseract.js). Each frame is first preprocessed — grayscale + 2× upscale + contrast normalization + sharpen — which materially improves accuracy on stylized overlays (prices, dates, coupons, CTAs). | Captures code, error messages, terminal output, UI text that the transcript doesn't cover |417| **Annotated timeline** | Merges transcript timestamps + frame timestamps + OCR text into a single chronological view | Gives the AI a unified "what was said, what changed visually, and what text appeared" at each moment |418419The OCR step requires `tesseract.js` (included as a dependency). If it fails to load, analysis continues without OCR — no frames or transcript are lost. OCR preprocessing is on by default; set `MCP_OCR_PREPROCESS=0` to OCR the raw frames instead.420421OCR always reads the **full-resolution** frame, not the copy emitted to the client. The two have different jobs: the emitted frame is capped for token cost, while recognition needs every pixel it can get.422423### Frame size (dense UI captures)424425Emitted frames are capped at 800 px wide, which suits the common case — talking-head clips, Reels, bug repros — where the subject fills the frame.426427It is the wrong size for a **dense UI capture**: a terminal, dashboard, IDE or spreadsheet recording, where the meaning lives in small text. An unscaled 1920×1080 screen recording lands at 800×450, and a 15 px UI font drops below what a vision model can resolve.428429Pass `maxWidth` per call to keep more (or all) of the source resolution — `0` disables the cap:430431```jsonc432get_frames(url, { maxFrames: 8, maxWidth: 0 }) // source resolution433get_frame_at(url, "2:14", { maxWidth: 1920 })434analyze_video(url, { detail: "standard", maxWidth: 1568 })435```436437Supported on `analyze_video`, `analyze_videos`, `analyze_moment`, `get_frames`, `get_frame_at` and `get_frame_burst`, and on the CLI as `--max-width <px>`.438439Native frames cost several times more context than the default, so raise the cap deliberately — `get_frames` returns up to 20 frames and `analyze_video` at `detailed` up to 60.440441| Variable | Applies to | Default | Notes |442|---|---|---|---|443| `MCP_FRAME_MAX_WIDTH` | Emitted frame width, in px | `800` | `0` (or `native`/`full`/`original`) disables the cap. A per-call `maxWidth` wins over it |444| `MCP_FRAME_JPEG_QUALITY` | Emitted frame JPEG quality | `70` | Raise it when thin glyphs matter; env only, there is no per-call quality parameter. Values outside 1–100 fall back |445| `MCP_CACHE_DIR` | Root for the tessdata cache and the CLI's default `--out` | per-user cache dir | Absolute paths only (a relative value is ignored). Use it when `$HOME` is read-only or absent — a hardened container, `ProtectHome=`, a quota'd home. The published Docker image sets it to `/tmp/mcp-video-analyzer-cache` so `--read-only --tmpfs /tmp` works out of the box |446| `MCP_ALLOW_PRIVATE_URLS` | Reaching private/loopback network addresses | off | `1` allows `localhost`, LAN addresses (`192.168.x`, `10.x`, …), `.local` names and UNC paths. Off by default — see [Network destinations](#network-destinations) below. **Cloud metadata endpoints stay blocked either way** |447448A value either variable can't use — `1e3`, `1920px`, a quality of `150` — is rejected with a one-time warning on stderr and the default applies. It is not silently accepted: the whole point of the setting is to escape a downscale that otherwise looks like a normal result.449450Prefer the per-call parameter: the server starts once per session, so an environment variable cannot differ between an overview of a YouTube clip and a close read of a screen recording. The width a call actually uses is part of the cache and sidecar key, so analyzing the same video at 800 px and then at `maxWidth: 0` re-runs the pipeline instead of returning the first result twice.451452## Complementary Tools453454### Chrome DevTools MCP455456For **live web debugging** alongside video analysis, pair this server with the [Chrome DevTools MCP](https://github.com/anthropics/anthropic-quickstarts/tree/main/mcp-devtools):457458```bash459claude mcp add chrome-devtools npx @anthropic-ai/mcp-devtools@latest460```461462**When to use each:**463464| Scenario | Tool |465|----------|------|466| Bug report recorded as a Loom video | `mcp-video-analyzer` — extract transcript, frames, and error text from the recording |467| Live debugging a web page | Chrome DevTools MCP — inspect DOM, console, network, take screenshots |468| Video shows UI issue, need to reproduce it | Use both: analyze the video first, then open the page in Chrome DevTools to reproduce |469470The two MCPs complement each other: video analyzer understands **recorded** content, DevTools interacts with **live** pages.471472## Example Output473474The [`examples/loom-demo/`](examples/loom-demo/) folder contains **real outputs** from analyzing a public Loom video ([Boost In-App Demo Video](https://www.loom.com/share/bdebdfe44b294225ac718bad241a94fe), 2:55).475476| File | What it shows |477|------|--------------|478| [`metadata.json`](examples/loom-demo/metadata.json) | Title, duration, platform |479| [`transcript.json`](examples/loom-demo/transcript.json) | 42 timestamped entries with speaker IDs |480| [`timeline.json`](examples/loom-demo/timeline.json) | Unified chronological view (transcript + frames merged) |481| [`moment-transcript-0m30s-0m45s.json`](examples/loom-demo/moment-transcript-0m30s-0m45s.json) | Filtered transcript for `analyze_moment` (0:30–0:45) |482| [`full-analysis.json`](examples/loom-demo/full-analysis.json) | Complete `analyze_video` output |483484**Frame images** (19 total in [`examples/loom-demo/frames/`](examples/loom-demo/frames/)):485- `scene_*.jpg` — scene-change detection (key visual transitions)486- `dense_*.jpg` — 1fps dense sampling (every 10th frame saved as sample)487- `burst_*.jpg` — burst extraction for moment analysis (0:30–0:45)488489> **Regenerate after changes:** `npx tsx examples/generate.ts` — requires yt-dlp + network access.490491## Development492493```bash494# Install dependencies495npm install496497# Run all checks (format, lint, typecheck, knip, tests)498npm run check499500# Audit dependencies. `security` covers what the published package ships and501# is a blocking CI job; `security:all` adds devDependencies. Both also run on502# a weekly cron, because npm audit reads a live advisory database.503npm run security504505# Build506npm run build507508# Run E2E tests (requires network; add WHISPER_E2E=1 to include the509# transcription outcome test — needs a whisper CLI installed)510npm run test:e2e511512# Just the video-format matrix: a real clip per container/codec513# (mp4/h264+hevc+av1, webm, mkv, mov, avi, m4v, mpeg, mpg, m2ts, mts,514# 3gp, ogv, flv, wmv) decoded end to end. ~15s on a warm cache; the515# first run fetches ~7MB of tesseract traineddata.516npm run test:formats517518# Build + boot the real MCP server/CLI (seconds)519npm run test:smoke520521# Everything: check → e2e → smoke → verify-package522npm run verify-all523524# Open MCP Inspector for manual testing525npm run inspect526```527528## Architecture529530```531src/532├── index.ts # Entry point (shebang + stdio)533├── server.ts # FastMCP server + tool registration534├── tools/ # MCP tool definitions (7 tools)535│ ├── analyze-video.ts # Full analysis with detail levels + caching536│ ├── analyze-moment.ts # Deep-dive on a time range537│ ├── get-transcript.ts # Transcript-only with Whisper fallback538│ ├── get-metadata.ts # Metadata + comments + chapters539│ ├── get-frames.ts # Frames-only (scene-change or dense)540│ ├── get-frame-at.ts # Single frame at timestamp541│ └── get-frame-burst.ts # N frames in a time range542├── adapters/ # Source-specific logic543│ ├── adapter.interface.ts # IVideoAdapter interface + registry544│ ├── loom.adapter.ts # Loom: authless GraphQL545│ ├── local-file.adapter.ts # Local files: absolute path or file:// URI546│ ├── twelvelabs.adapter.ts # TwelveLabs Pegasus: transcript + AI summary (opt-in)547│ └── direct.adapter.ts # Direct URL: any mp4/webm link548├── processors/ # Shared processing549│ ├── frame-extractor.ts # ffmpeg scene detection + dense + burst extraction550│ ├── browser-frame-extractor.ts # Headless Chrome fallback for frames551│ ├── audio-transcriber.ts # Whisper fallback (HF transformers → CLI → OpenAI)552│ ├── image-optimizer.ts # sharp resize/compress553│ ├── frame-dedup.ts # Perceptual dedup (dHash + Hamming distance)554│ ├── frame-ocr.ts # OCR text extraction (tesseract.js)555│ └── annotated-timeline.ts # Unified timeline (transcript + frames + OCR)556├── config/557│ └── detail-levels.ts # brief / standard / detailed config558├── utils/559│ ├── cache.ts # In-memory TTL cache with LRU eviction560│ ├── field-filter.ts # Selective field filtering for responses561│ ├── url-detector.ts # Platform detection from URL562│ ├── vtt-parser.ts # WebVTT → transcript entries563│ └── temp-files.ts # Temp directory management564└── types.ts # Shared TypeScript interfaces565```566567### Burst timing568569Burst extraction accepts fractional timestamps (for example, `0:01.1`). The570FFmpeg path selects up to the requested count of existing source frames within571`[from, to)`, without duplicating frames to satisfy an excessive sampling rate.572Processor results retain each selected frame's video-relative presentation time,573including fractional seconds. Browser fallback preserves fractional seek targets574and separate screenshot files; its times are requested seeks, not measured source575frame PTS. Browser seeking is not a frame-accurate substitute for local decoding.576577`get_frame_burst` returns a JSON text block containing578`{ frameCount, from, to, warnings, frames: [{ time, timingOrigin }] }`, followed579by images in exactly the same order as `frames`. `timingOrigin: "source-pts"`580means measured video-relative FFmpeg presentation time; `"seek-target"` means581requested browser seek time. Empty/degraded results contain `frames: []`.582583## License584585MIT586
Full transparency — inspect the skill content before installing.