Triaging Visual Review Runs

1---
2name: triaging-visual-review-runs
3description: >
4  Inspects PostHog Visual Review (VR) runs that gate PR merges with screenshot regression checks.
5  Use when the user mentions "visual review", "VR", "snapshot diff", "screenshot test", "storybook regression",
6  "playwright snapshot", asks why a PR is blocked or what changed visually, wants to triage the VR backlog,
7  decide whether a snapshot diff is real vs flaky, or check whether a story has been changing across runs.
8  Also invoke when a PR has a failing `visual-review` status check, when a PR comment mentions "Visual review",
9  or when the user is on a branch with an open VR run.
10---
11 
12# Triaging visual review runs
13 
14Visual Review is PostHog's screenshot-regression product: CI captures storybook + playwright screenshots,
15diffs them against committed baseline hashes, and gates the PR until a human approves the visible changes.
16A PR with visual changes carries a `visual-review` GitHub status check that stays red until each diffed
17snapshot is approved or tolerated in the [VR UI](https://us.posthog.com/project/2/visual_review).
18 
19This skill teaches an agent how to answer the questions a human reviewer would actually ask, by chaining
20the VR MCP tools — instead of reaching for `gh pr view` and tab-hopping to the VR web UI. The read tools
21cover status / scope / history / triage. Two are reversible DB-only triage marks (`approve-create`,
22`tolerate-create`); one ships the change (`finalize-create`) — it commits the baseline and greens the gate,
23and only that one needs explicit per-run human confirmation.
24 
25## When this skill applies
26 
27Trigger this skill on any of:
28 
29- A PR number, branch name, or commit SHA paired with words like _visual review_, _VR_, _snapshot_, _screenshot_,
30  _storybook diff_, _playwright snapshot_, _baseline_, _approve_, _tolerated_, _quarantine_.
31- Questions about why a PR is blocked, what visually changed, or whether a diff is real.
32- "Is my run done?" / "What's left to review?" / "Has this story flaked recently?"
33- A failing `visual-review` GitHub check or a PR comment from the `posthog-bot` mentioning visual review.
34 
35When the user asks for the rendered diff image itself, the [VR web UI](https://us.posthog.com/project/2/visual_review)
36is faster — direct them there. This skill is for everything around the diff: status, scope, history, triage.
37 
38## Tools
39 
40Read tools (safe to call freely):
41 
42| Tool                                               | Purpose                                                                                                                                                                                                                                  |
43| -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
44| `posthog:visual-review-runs-list`                  | List runs, filter by `pr_number` / `commit_sha` / `branch` / `review_state`. Start here.                                                                                                                                                 |
45| `posthog:visual-review-runs-retrieve`              | Full detail for a single run (status, summary counts, supersession).                                                                                                                                                                     |
46| `posthog:visual-review-runs-snapshots-list`        | Per-snapshot results inside a run: identifier, `result`, diff %, classification, baseline + current artifact URLs. Quarantined snapshots are excluded by default (see `quarantined_count`); pass `include_quarantined=true` to see them. |
47| `posthog:visual-review-runs-snapshot-history-list` | A single story's last N runs across master/PRs — the flake check.                                                                                                                                                                        |
48| `posthog:visual-review-runs-counts-retrieve`       | Aggregate counts for queue triage (how many runs in `needs_review`, etc.).                                                                                                                                                               |
49| `posthog:visual-review-runs-tolerated-hashes-list` | Hashes the team has explicitly accepted as "known flake / acceptable variation".                                                                                                                                                         |
50| `posthog:visual-review-repos-list`                 | Repos (one per GitHub repo) — usually only one matters; useful for filtering.                                                                                                                                                            |
51| `posthog:visual-review-repos-retrieve`             | Repo metadata: baseline file paths, PR-comment configuration.                                                                                                                                                                            |
52 
53Triage tools (reversible, DB-only — they record a review decision but do NOT change the baseline or the gate):
54 
55| Tool                                         | Purpose                                                                                                                |
56| -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
57| `posthog:visual-review-runs-approve-create`  | Mark `changed` / `new` snapshots reviewed (approved) in the DB. Does NOT commit or green the gate — ship via finalize. |
58| `posthog:visual-review-runs-tolerate-create` | Mark a single changed snapshot as a known tolerated alternate. Does NOT change the baseline — use for benign variants. |
59 
60Ship tool (irreversible, outward-facing — requires explicit per-run human confirmation; see [the gate](#the-finalize-gate)):
61 
62| Tool                                         | Purpose                                                                                                          |
63| -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
64| `posthog:visual-review-runs-finalize-create` | Commit the approved baseline to the PR branch and green the GitHub `visual-review` check. This ships the change. |
65 
66Mark-reviewed call shape (`approve-create`):
67 
68- `id` (required) — the run UUID. It's the route parameter, so the call fails without it.
69- `snapshots: [{identifier, new_hash}]` — `new_hash` is the `content_hash` of each snapshot's `current_artifact`. This only records the review in the DB; nothing is committed and the gate stays red until you finalize.
70 
71Toleration call shape — both fields are required:
72 
73- `id` (required) — the run UUID. It's the route parameter, so the call fails without it.
74- `snapshot_id` (required) — the UUID of the individual snapshot to tolerate (from `visual-review-runs-snapshots-list`). This identifies _which_ snapshot inside the run; it does not replace the run `id`.
75 
76Finalize call shape (`finalize-create`) — the all-or-nothing ship action:
77 
78- `id` (required) — the run UUID.
79- `approve_all: true` — approve every still-pending `changed`/`new` snapshot before finalizing (tolerated ones are left alone). Use when you've verified every remaining diff is intended.
80- Omit `approve_all` (default false) to finalize a run you've already reviewed snapshot-by-snapshot. Finalize is all-or-nothing: it fails with `409 not_fully_resolved` (and lists what's left) unless every changed/new snapshot is approved, tolerated, or quarantined.
81- It commits exactly the snapshots approved in the DB — tolerated snapshots keep their baseline and are never overwritten. When a baseline commit is pushed, its SHA comes back on the run's `metadata.baseline_commit_sha`. It's absent when nothing needed committing (everything resolved by toleration/quarantine — the gate still greens) or when the commit was skipped: no PR, or a `409 sha_mismatch` because the PR has newer commits (that one leaves the gate red — re-run CI on the latest commit and finalize again).
82 
83If finalize fails with `409 stale_run`, the run has been superseded — `visual-review-runs-list { pr_number }` and finalize the newest one. A successful finalize often kicks off a fresh CI run, which is normal.
84 
85### The finalize gate
86 
87Finalize is the one irreversible, outward-facing action in this skill: it rewrites the baseline committed to the
88PR and greens the merge gate. Treat it like pushing to someone's branch — never automatic.
89 
90Before _any_ `finalize-create` call, all of these must hold:
91 
921. **You verified the diffs.** You pulled the current (and, for `changed`, baseline) PNGs and looked at them, ran
93   the flake check on anything suspect, and reached a per-snapshot verdict. Metadata alone is never enough.
942. **You presented the verdict and waited.** Show the user, per snapshot, what changed and your recommendation, then stop.
953. **The user explicitly approved _this_ run.** A broad "get the gate green" / "fix the PR" is permission to
96   investigate and recommend — NOT to finalize. When the task implies finalizing but the human hasn't said it for
97   this specific run, ask.
98 
99`approve-create` and `tolerate-create` are reversible triage and don't need this gate — but they don't ship anything
100either. The moment you're about to `finalize-create` and can't point to a specific human "yes" for this run, stop and ask.
101 
102## Vocabulary cheat sheet
103 
104These appear in tool output and matter for interpretation:
105 
106- **Run `review_state`**: `needs_review` (open, awaiting human), `clean` (zero diffs), `processing` (CI still uploading),
107  `stale` (a newer run on the same PR has superseded this one — check `superseded_by_id`).
108- **Run `run_type`**: `storybook` (component snapshots) or `playwright` (full-page e2e snapshots).
109- **Snapshot `result`**: `unchanged`, `changed` (real diff), `new` (no baseline yet), `removed`.
110- **Snapshot `classification_reason`**: `tolerated_hash` (matches a known-tolerated hash, no action needed),
111  `below_threshold` (under the noise floor), `exact` (byte-identical), `""` (real diff requiring review).
112- **Snapshot `review_state`**: `pending` or `approved`.
113- **Run `summary`**: `total / changed / new / removed / unchanged / unresolved / tolerated_matched` —
114  `unresolved` is what's actually blocking review.
115 
116## Workflows
117 
118### "What's the VR status of this PR?"
119 
120The single most common job. Map a PR number to its run state in two calls.
121 
1221. `posthog:visual-review-runs-list { pr_number: <n>, limit: 5 }` — sort by `created_at` desc, take the latest non-stale one.
1232. If the run has `summary.changed > 0` or `summary.unresolved > 0`, drill in:
124   `posthog:visual-review-runs-snapshots-list { id: <run_id> }` and report the `changed` snapshots.
125 
126Report back: PR number, run UUID, `review_state`, summary counts, and the `_posthogUrl` deep link so the
127user can click straight to the diff viewer.
128 
129### "Is the diff real or unrelated?"
130 
131The most useful judgment a code-aware agent can add. Combine three signals: **scope match**, **flake history**,
132and **the actual rendered images**. The agent should look at the screenshots — not just describe metadata.
133 
1341. **Scope check** — `git diff master...HEAD --stat` (or against the PR's base branch) → list of touched paths.
135   Cross-reference with `posthog:visual-review-runs-snapshots-list { id }` filtered to `result: changed` → story identifiers.
136   Stories are namespaced like `<area>-<scene>--<story>--<theme>`; e.g. `scenes-app-settings-user--settings-user-profile--dark`
137   maps to `frontend/src/scenes/settings/user/...`. Use this to translate story id → likely source path.
138 
1392. **Visual inspection** — for each `changed` snapshot, the tool result contains `current_artifact.download_url`
140   and `baseline_artifact.download_url`. These are pre-signed S3 URLs to PNG files; pull them and look:
141 
142   ```bash
143   curl -s -o /tmp/vr-baseline.png "<baseline_artifact.download_url>"
144   curl -s -o /tmp/vr-current.png "<current_artifact.download_url>"
145   ```
146 
147   Then `Read` both files (the Read tool renders images visually) and compare. Things to call out:
148   - The actual visible delta (text changed, button moved, layout shift, color drift, missing element).
149   - Whether the change is consistent with the diff_pixel_count and diff_percentage in the metadata
150     (e.g. 54% diff but the images look near-identical → screenshot framing changed, not the UI).
151   - Whether the baseline and current have different dimensions (`width` / `height` fields). Mismatched
152     dimensions usually mean the story rendered to a different viewport or didn't fully render before
153     screenshot — a flake signal, not a regression.
154 
1553. **Flake history** — run the flake check below for any story that looks suspect.
156 
1574. **Verdict** — combine all three:
158   - Scope plausible + visible regression matches the code change → real diff, recommend approval.
159   - Scope mismatch + dimensions mismatch + frequent prior changes → flake, recommend tolerating the hash.
160   - Scope plausible + visible regression looks unintended → push a fix; do not approve.
161 
162Always include a one-line description of what you saw in the images — the user uses this to decide whether to
163trust your verdict without opening the VR UI themselves.
164 
165### Flake check: "Has this story been changing?"
166 
167Once you have a suspect snapshot identifier:
168 
169`posthog:visual-review-runs-snapshot-history-list { id: <snapshot_id> }` → returns prior outcomes for the same story.
170 
171Verdicts:
172 
173- Mostly `unchanged` and this run's diff is the outlier → likely a real regression caused by this PR.
174- Frequent `changed` across unrelated branches/master → flaky story; recommend tolerating the hash via the UI.
175- Recent `removed` or large-jump dimension change → baseline likely stale; recommend re-baselining on master.
176 
177### Triaging the queue
178 
179When the user is doing housekeeping rather than asking about a specific PR:
180 
1811. `posthog:visual-review-runs-counts-retrieve` → total queue size.
1822. `posthog:visual-review-runs-list { review_state: needs_review, limit: 50 }` (paginate if needed).
1833. Group by `branch` author or `run_type` to surface clusters (e.g., "12 PRs blocked on the same shared
184   component change" usually means a single underlying root cause to address).
1854. Prefer surfacing runs whose `summary.changed > 0` over runs that are only `new` — `new` means no baseline
186   yet, which is usually trivial to approve; `changed` is the real review work.
187 
188## Output expectations
189 
190For PR-status questions, lead with the verdict in one line, then 2-4 bullets of supporting context. Always
191include the `_posthogUrl` deep link to the run — humans need to see the rendered images to make the call,
192the agent can only describe the metadata.
193 
194For triage / aggregate questions, a short table beats prose. Group by what the user is going to act on.
195 
196## What NOT to do
197 
198- Do not approve or tolerate without explicit user confirmation. The verdict is yours to recommend; the
199  decision to ship belongs to the user. Once they say "approve those" / "tolerate that", call the tool.
200- Do not assume the failing GitHub check on a PR is unrelated to VR — if a `visual-review` check is red on
201  a PR you're working on, that's the trigger to run this skill.
202- Do not declare a verdict from metadata alone when `result: changed`. Pull the baseline and current PNGs
203  and look at them; metadata can only say "something changed", not whether the change is intended.
204