Creating Replay Vision Scanners

1---
2name: creating-replay-vision-scanners
3description: "Guides agents through creating and safely sizing a Replay Vision scanner: choosing the scanner type (monitor/classifier/scorer/summarizer), shaping the RecordingsQuery that selects sessions, and — crucially — estimating observation volume and checking the org's monthly quota before creating, so a broad scanner doesn't exhaust the budget on its first scheduled sweep.\nTRIGGER when: user asks to create, set up, or configure a Replay Vision scanner, OR when you are about to call vision-scanners-create, OR when widening an existing scanner's query or sampling_rate via vision-scanners-update.\nDO NOT TRIGGER when: only reading scanners or observations, deleting a scanner, or running an existing scanner against a single session on demand (vision-scanners-scan-session)."
4---
5 
6# Creating Replay Vision scanners
7 
8A scanner is a standing LLM probe over session recordings. Once created and enabled, it runs on a
9**Temporal schedule that sweeps every 5 minutes**, applying its prompt to each new matching recording and
10recording the result as an observation (a queryable `$recording_observed` event). Each observation counts
11against a **monthly org quota** (a fixed number of observations per calendar month).
12 
13That schedule is exactly why creation needs a gut-check: a scanner with a permissive query and full sampling
14starts consuming quota automatically and can drain the whole month's budget within its first few sweeps.
15Creation itself does **not** check quota — that protection only kicks in at observation time, by which point
16the budget may already be gone.
17 
18## Core principle: size before you ship
19 
20Never create an enabled scanner blind. Estimate its volume, check remaining quota, and — when the projected
21volume is a meaningful fraction of what's left — show the user the numbers and get confirmation before
22creating. This is the heart of the skill; the rest is supporting detail.
23 
24## The flow
25 
26### Step 1: What should the scanner do?
27 
28Pick a `scanner_type` and write its `scanner_config`. Every type needs a `prompt`; the rest is type-specific:
29 
30| Type         | What it produces                                                  | `scanner_config` shape                                                                                                                  |
31| ------------ | ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
32| `monitor`    | Open-ended observation against a prompt (e.g. "flag rage clicks") | `{"prompt": "..."}`                                                                                                                     |
33| `classifier` | Assigns tags from a fixed label set                               | `{"prompt": "...", "tags": ["tag-a", "tag-b"]}` — `tags` needs ≥1 entry; optional `"multi_label": true`, `"allow_freeform_tags": false` |
34| `scorer`     | Numeric score on a rubric                                         | `{"prompt": "...", "scale": {"min": 1, "max": 5, "label": "frustration"}}` — `min` < `max`; `label` optional                            |
35| `summarizer` | Free-text summary; optional facet embeddings for search           | `{"prompt": "..."}`; optional `"length": "short" \| "medium" \| "long"` (default `"medium"`), `"emits_embeddings": false`               |
36 
37`scanner_type` is **locked after creation** — to change it you delete and recreate, so confirm the type is
38right up front, and get the `scanner_config` shape right (a wrong shape is a create error, not a silent
39default).
40 
41If the user's intent makes the type and prompt obvious, just proceed — don't interrogate them.
42 
43### Step 2: Which sessions?
44 
45The `query` is a `RecordingsQuery` shape that selects which recordings the scanner watches. `date_from` and
46`date_to` are **ignored** (the schedule controls time), so don't bother setting them. Narrow the query to the
47sessions that actually matter — by event, URL, person property, duration, etc. A narrow query is the single
48biggest lever on cost.
49 
50`sampling_rate` (0..1, default 1.0) is a random downsample applied _after_ the query matches. Lower it to
51trade coverage for budget.
52 
53### Step 3: Size it — the gut-check (do not skip)
54 
55Before creating, run both checks and reason about them together:
56 
571. **Estimate volume** — call `vision-scanners-estimate-create` with the proposed `query` + `sampling_rate`.
58   It returns `matched_sessions_in_window`, the `window_days` measured, and
59   `estimated_observations_per_month`.
602. **Check budget** — call `vision-quota-retrieve` for `remaining` and `exhausted` against the org's monthly
61   `monthly_quota`.
62 
63Then decide:
64 
65- If `estimated_observations_per_month` comfortably fits within `remaining`, proceed.
66- If it's a large fraction of (or exceeds) `remaining`, **stop and tell the user the concrete numbers**
67  — e.g. "This scanner is projected to produce ~X observations/month; you have Y of Z left this month." —
68  and confirm before creating, or suggest tightening the `query` or lowering `sampling_rate` first.
69- If the org is already `exhausted`, say so — a new enabled scanner won't produce anything until the quota
70  resets, and its observations will be silently skipped.
71 
72Confirmation here is a conversation step, not an API capability — surface the trade-off and let the user
73choose. When the projected volume is clearly small relative to the budget, you don't need to ask.
74 
75### Step 4: Create
76 
77Call `vision-scanners-create`. Minimal example:
78 
79```json
80{
81  "name": "Rage click monitor",
82  "scanner_type": "monitor",
83  "scanner_config": { "prompt": "Flag sessions where the user repeatedly clicks the same element in frustration." },
84  "query": { "kind": "RecordingsQuery", "events": [{ "id": "$rageclick", "type": "events" }] },
85  "sampling_rate": 1.0,
86  "model": "gemini-3-flash-preview",
87  "enabled": true
88}
89```
90 
91`name` must be unique within the team. Set `enabled: false` if the user wants to create it paused (no
92schedule, no quota consumption) and turn it on later.
93 
94## After creation
95 
96- Show the scanner's PostHog URL from the response so the user can review it in the UI.
97- Results take a few minutes to appear (rasterizing the recording to video + the LLM call are slow). Inspect
98  them with `vision-scanners-observations-list` for one scanner over time, or `vision-observations-list`
99  (requires `session_id`) for every scanner's findings on a single session. To dig into a recording, hand off
100  to the `investigating-replay` skill.
101 
102## Updating an existing scanner
103 
104`vision-scanners-update` is a partial update — send only changed fields. **Re-run the Step 3 gut-check
105whenever you widen scope**: a broader `query` or a higher `sampling_rate` raises the sweep volume just like a
106fresh broad scanner would. Toggling `enabled`, tweaking the prompt, or narrowing the query don't need a
107re-estimate. Editing config bumps `scanner_version`; past observations keep a snapshot of the old config.
108 
109## Gotchas
110 
111- **One observation per (scanner, session).** Re-running a scanner on a session it already observed — even a
112  failed or ineligible one — is a no-op and won't produce a fresh scan.
113- **Ineligible ≠ failed.** Observations can land `ineligible` (e.g. `too_short`, `no_recording`) — a terminal
114  non-error outcome. Check `error_reason` when triaging why a scanner produced nothing.
115- **Provider/model are Google/Gemini only** in the current version.
116