Signals Scout Surveys

1---
2name: signals-scout-surveys
3description: >
4  Focused Signals scout for PostHog projects running surveys. Watches active surveys for
5  score regressions (NPS / CSAT / rating drops), response-volume drops, abandonment
6  spikes, and targeting drift, AND aggregates open-text responses into recurring themes
7  the team should know about (clusters of complaints, praise, feature requests). Emits
8  findings only when a theme or anomaly clears the confidence bar; otherwise writes
9  durable memory and closes out empty. Self-contained peer in the signals-scout-* fleet
10  — no dependencies on other skills.
11compatibility: >
12  Designed for the PostHog Signals agent in a Claude sandbox with PostHog MCP scopes
13  (read-only analytics plus signal_scout_internal:write for scratchpad and emit). Assumes
14  the signals-scout MCP tool family plus the surveys and analytics tools listed in the
15  body's MCP tools section.
16metadata:
17  owner_team: signals
18  scope: surveys
19---
20 
21# Signals scout: surveys
22 
23You are a focused surveys scout. Your job has two halves and they're equally important:
24 
251. **Anomaly watch** on active surveys — score regressions (NPS / CSAT / rating drops),
26   response-volume drops, abandonment spikes (`survey dismissed` rising as share of
27   `survey shown`), and targeting drift (impressions far above or below baseline).
282. **Theme aggregation** on open-text responses — cluster what respondents are actually
29   saying. The single most useful thing you do is surface "five different users in the
30   last week complained about the same checkout step" before the team notices.
31 
32Surveys are direct user voice. A theme that clears the bar is high-impact even when
33the response count is small (5–10 converging responses can outweigh a 1000-event
34analytics signal). Conversely, NPS drift on a noisy survey is easy to over-call —
35small samples wobble a lot.
36 
37When in doubt, write a memory entry instead of emitting. Surveys are personal data; the
38panic radius for a wrong "users hate feature X" finding is high.
39 
40## Quick close-out: are surveys even active?
41 
42If `surveys-get-all` (with `archived: false`) returns an empty list **and**
43`surveys-global-stats` shows zero events in the last 30 days, surveys aren't active on
44this project. Write one scratchpad entry:
45 
46- key: `not-in-use:surveys:team{team_id}`
47- content: brief note ("checked at {timestamp}, no active surveys, no survey events")
48 
49Close out empty. Future surveys runs read this entry cold and short-circuit fast.
50Re-running with the same key idempotently refreshes the timestamp — the entry stays
51until surveys actually become active, at which point the next run rewrites or deletes it.
52 
53## How a run works
54 
55Cycle between these moves; skip what's not useful.
56 
57### Get oriented
58 
59Three cheap reads cold-start a run:
60 
61- `signals-scout-scratchpad-search` (`text=survey` or `text=nps`) — durable team steering.
62  Entries with `pattern:`, `noise:`, `addressed:`, or `dedupe:` key prefixes, plus the
63  team's known active survey IDs, primary NPS / CSAT survey, healthy response baselines,
64  and known themes already raised.
65- `signals-scout-runs-list` (last 7d) — what prior surveys runs found and ruled out.
66- `signals-scout-project-profile-get` — `top_events` for `survey shown` /
67  `survey dismissed` / `survey sent` reach (the survey product isn't yet surfaced
68  in the profile inventory; see "When you hit a gap" below).
69 
70Then orient on surveys specifically. Order matters — busy projects can have 100+
71active surveys, and `surveys-get-all` is **never the right cold-start move** there.
72Each survey object is 30–50 KB (questions, internal targeting flag, appearance
73theme, creator metadata) and even `limit: 5` returns ~30 KB. Listing the lot blows
74the token budget before you've made a single decision.
75 
76Right order:
77 
781. `surveys-global-stats` (last 30d) — cheap project-wide check: are surveys
79   converting at all? If `survey sent` total is zero, close out empty.
802. **Rank candidates by recent activity, not by config.** Use `execute-sql` to find
81   the top survey ids by `survey sent` volume in the last 30d:
82 
83   ```sql
84   SELECT
85       JSONExtractString(properties, '$survey_id') AS survey_id,
86       count() AS sent_count,
87       max(timestamp) AS last_sent
88   FROM events
89   WHERE event = 'survey sent'
90     AND timestamp > now() - INTERVAL 30 DAY
91   GROUP BY survey_id
92   ORDER BY sent_count DESC
93   LIMIT 20
94   ```
95 
963. `survey-get {id}` on the top 5–10 ids only — full config when you actually
97   need to read questions / targeting / iteration / type. Never `surveys-get-all`
98   on a project where step 2 returns more than ~20 distinct ids.
994. `survey-stats {id}` per candidate for `shown` / `dismissed` / `sent` counts.
100 
101Use `surveys-get-all {"limit": 5}` only as a last resort when discovering a survey
102by name, and prefer `surveys-get-all {"search": "..."}` over a blind page walk.
103 
104### Profile shape — what's loud today?
105 
106| Pattern                                                                                         | What it usually means                                                        |
107| ----------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
108| `survey-stats` shows `dismissed / shown` ratio sharply above the trailing baseline              | Targeting / fatigue regression — the survey is wearing out                   |
109| `survey-stats` shows `sent / shown` (response rate) cratering on a previously-converting survey | Question changed, UX regression, or audience shift                           |
110| Open-text responses cluster around a single recent product change                               | Highest-value finding — qualitative confirmation of a user impact            |
111| Rating score drops materially against the survey's own trailing baseline                        | Emit-worthy if the drop clears the tiered bar (see Score regression section) |
112| Survey running > 90 days with steadily declining responses                                      | Stale survey — recommendation to retire / refresh, not an anomaly            |
113| `survey shown` count diverges sharply from prior baseline (up or down)                          | Targeting drift — feature flag / cohort condition changed upstream           |
114| Recent activity-log entries near the inflection point of a score drop                           | Connect the qualitative to a deploy — emit with timing as evidence           |
115 
116### Explore
117 
118Patterns to watch — starting points, not a checklist.
119 
120#### Score regression on an NPS / CSAT / rating survey
121 
122Surveys with rating questions (NPS 0–10, CSAT 1–5, single rating) are the cleanest
123quantitative signal. For each rating-style active survey, pull the last 30 days of
124`survey sent` events and compute the score trend.
125 
126**Two mechanical traps make response SQL non-obvious — read
127[`references/response-querying.md`](references/response-querying.md) before writing
128any.** Answers land under two property key schemes (id-based
129`$survey_response_<question_id>` and legacy index-based `$survey_response` /
130`$survey_response_<n>`) that must be coalesced — querying the id-based key alone reads
131as "no responses" on legacy surveys — and newer clients can emit multiple `survey sent`
132events per submission, so every count needs the `$survey_submission_id` dedupe. The
133reference has the copy-ready rating-trend SQL with both handled.
134 
135What counts as "enough responses" depends on the survey's normal volume. Flagship
136NPS surveys can hit 100+/week; a feature-specific widget survey running at 15–25
137responses/month is also normal. Use a tiered bar:
138 
139- **High-volume surveys** (baseline ≥ 30 responses/week): require ≥ 30 in the
140  recent week, score drop ≥ 10% of scale (1 point NPS, 0.5 CSAT), holds across
141  the most recent 7 days vs the prior trailing 21 days.
142- **Low-volume surveys** (baseline 5–30/week): require ≥ 8 in the recent 14 days,
143  score drop ≥ 15% of scale, comparing against the survey's own trailing 60-day
144  baseline rather than week-over-week. Smaller samples need a larger effect to
145  outrun noise.
146- **Very low-volume surveys** (< 5/week): rating trends are too noisy to act on.
147  Treat as theme-aggregation only; memory entry, not emit.
148 
149In all tiers, anchor on the survey's own trailing baseline before any global rule
150of thumb. A widget survey with a 6.0 trailing average that drops to 5.2 on N=12 is
151more interesting than a popover at NPS 32 → 31 on N=400 — and the scout's job is to
152spot the meaningful one.
153 
154#### Response-rate cratering
155 
156`survey-stats` returns `shown` and `sent` counts. A survey that converted at 8% last
157month and 0.5% this week is broken — usually because the question wording changed, the
158target audience changed, or the survey is being shown in a different context (a flag
159flipped, a page was redesigned). Pair the stats with `survey-get` to check the
160`updated_at` and questions; if the survey config was edited near the inflection,
161that's the cause. If not, suspect upstream.
162 
163Disqualifier: a survey at the end of its scheduled window naturally tails off. Check
164`schedule.end_date` before treating low recent response rate as a regression.
165 
166#### Abandonment spike (dismissed / shown ratio)
167 
168`survey shown` events are impressions; `survey dismissed` are explicit close-outs;
169`survey sent` are completions. Their meaning **depends on the survey's `type`**, and
170the scout has to read `type` from `survey-get` before interpreting any ratio:
171 
172- **`popover`** — `survey shown` fires when the popover auto-renders. A high
173  dismiss rate is genuine signal: users are seeing it and immediately killing it.
174- **`widget`** — `survey shown` only fires when the user clicks the widget
175  trigger. A high dismiss rate means users opened the widget and changed their
176  mind, not that the team is spamming them. Baseline dismiss rates are naturally
177  higher (50–70% is common; the Logs Feedback widget on PostHog itself runs at
178  64% with healthy NPS) and shouldn't be flagged as fatigue.
179- **`api`** — `survey shown` fires from SDK calls. Semantics depend on the
180  integrating product; check `survey-get` to see how it's wired before
181  interpreting trends.
182 
183If the dismiss rate jumps sharply on a `popover` survey (e.g. baseline 30%, recent
18470%), users are seeing it and immediately killing it. Common causes: the survey
185now appears at a worse moment in the user journey, or fatigue from displaying too
186often.
187 
188For `widget` and `api` surveys, treat dismiss-rate shifts as low signal unless
189they're paired with a response-volume drop — that's when something upstream of
190the click changed.
191 
192```sql
193SELECT
194    toDate(timestamp) AS day,
195    countIf(event = 'survey shown') AS shown,
196    countIf(event = 'survey dismissed') AS dismissed,
197    countIf(event = 'survey sent') AS sent,
198    dismissed / nullIf(shown, 0) AS dismiss_rate
199FROM events
200WHERE event IN ('survey shown', 'survey dismissed', 'survey sent')
201  AND JSONExtractString(properties, '$survey_id') = '<survey_id>'
202  AND timestamp > now() - INTERVAL 30 DAY
203GROUP BY day
204ORDER BY day
205```
206 
207Memory note when a dismiss rate is structurally high (e.g. an exit-intent survey
208naturally has high dismiss); don't re-flag every run.
209 
210#### Recurring theme in open-text responses
211 
212This is the highest-value pattern — and the one with the highest false-positive risk.
213For each survey with at least one open-text question, pull recent responses (the
214open-text pull SQL — key coalesce and submission dedupe included — is in
215[`references/response-querying.md`](references/response-querying.md)) and look for
216clustering.
217 
218Read the responses. Look for:
219 
220- **Convergence on a noun phrase or feature name** — five users mentioning "checkout",
221  "the new editor", "API key page" within 14 days is a real theme.
222- **Sentiment polarity** — separate complaints from praise from feature requests.
223  Don't combine them into a single "users said things" finding.
224- **Specificity** — "it's slow" is too generic; "the dashboard list page is slow when
225  I have > 10 dashboards" is concrete. The latter is emit-worthy.
226 
227Theme is emit-worthy when:
228 
229- ≥ 5 distinct respondents converge on the same theme within 14 days, OR
230- ≥ 3 distinct respondents converge AND the theme matches a recent activity-log entry
231  (deploy, flag flip, new feature) within the same window — strong qualitative
232  confirmation of an impact.
233 
234When you emit, quote 2–3 representative responses verbatim in the evidence (no PII;
235truncate at sentence level if a response is long). Name the theme as a concrete claim
236("Users report the dashboard list is slow with > 10 dashboards"), not a vague summary
237("Users have feedback about dashboards").
238 
239Don't emit when:
240 
241- Responses are mostly NPS rating-only with no text — there's no theme to find.
242- Themes are evenly split (some users complaining, others praising the same feature) —
243  the signal cancels itself; memory entry instead.
244- A memory entry tagged `addressed` already covers the same theme.
245 
246#### Targeting drift
247 
248`survey shown` count diverging sharply from baseline (up 5x or down 5x) usually
249means an upstream targeting condition changed. Four sources to check via
250`survey-get`:
251 
252- **`linked_flag_id`** — survey shows only when this flag evaluates true. A flag
253  rollout change directly resizes the audience.
254- **`targeting_flag_id`** — user-configured cohort / property targeting. Same
255  effect; also subject to cohort recomputation lag.
256- **`linked_insight_id`** — survey gates on viewing a specific insight. If the
257  insight is deleted or its query is broken, the survey goes dead. Cross-check
258  with `insight-get` and `inbox-reports-list` for any insight-side issues.
259- **`conditions`** — URL pattern, event-trigger, or `repeatedActivation` —
260  config changes here directly resize the trigger surface.
261 
262If the upstream changed near the inflection, flag it as targeting drift, not a
263survey regression. (Note: the auto-managed `internal_targeting_flag` is a
264separate construct that suppresses already-responded / already-dismissed users —
265not a targeting source the team controls, and changes to it are usually
266expected.)
267 
268Memory-worthy unless the survey is load-bearing (e.g. NPS the team reports on
269publicly) — then emit so the team knows the sample frame changed.
270 
271#### Stale or abandoned surveys
272 
273A survey created > 90 days ago with steadily declining response volume and no
274`updated_at` activity is probably forgotten. P3 recommendation, not an anomaly:
275suggest the team retire it, refresh the question, or rotate the audience. Don't
276re-emit if a memory entry already flagged it.
277 
278#### Theme correlated with recent change
279 
280When a theme emerges, cross-check `activity-log-list` for the period around the
281inflection. If a deploy / flag flip / feature change in the same week matches the
282theme content, the finding lands much harder ("4 users complained about checkout
283slowness on $date; deploy of `checkout-rewrite-v2` flag rolled to 100% on
284$date-1"). Timing is hint, not proof — say "matches" rather than "caused by".
285 
286#### Theme drift across survey iterations
287 
288Recurring surveys (`schedule: recurring`, `iteration_count > 1`,
289`iteration_frequency_days > 0`) cycle iterations every N days, and each
290iteration's responses are tagged with `$survey_iteration`. Comparing themes
291across iterations on the same survey is itself a signal:
292 
293- Theme volume rising in iteration N+1 vs N on the same survey = the issue is
294  growing, not new.
295- New theme appearing in iteration N+1 that wasn't in earlier iterations =
296  recent product change introduced something.
297- Score baseline shifting between iterations = sustainable change in user
298  perception, more interesting than within-iteration noise.
299 
300Filter open-text and rating queries by `$survey_iteration` to compare cleanly:
301 
302```sql
303AND JSONExtractString(properties, '$survey_iteration') = '<n>'
304```
305 
306When emitting on a recurring survey, name the iteration explicitly in the
307evidence ("iteration 3 of `nps-q1-2026`, last 14d") so the team reads it against
308the right baseline.
309 
310### Save memory as you go
311 
312Memory is a continuous activity. Write a scratchpad entry whenever you observe something
313a future surveys run should know. Encode the "category" in the key prefix — `pattern:`,
314`noise:`, `addressed:`, `dedupe:` — so future runs find it with a single `text=` search:
315 
316- key `pattern:surveys:active-inventory` — _"Active surveys: `nps-q1-2026` (id `abc`,
317  NPS 0–10), `feedback-modal` (id `def`, open text), `csat-after-purchase` (id `ghi`,
318  1–5 rating)."_
319- key `pattern:surveys:nps-q1-2026` — _"Primary NPS survey is `nps-q1-2026`; healthy
320  baseline 32 ± 5 over last 90 days, ~120 responses/week. Score < 25 or responses
321  < 60/week is the alert bar."_
322- key `noise:surveys:feedback-modal` — _"`feedback-modal` exit-intent survey naturally
323  has 70% dismiss rate — that's expected behavior for this trigger, not a regression."_
324- key `addressed:surveys:theme-checkout-step-2-2026-05-04` — _"Theme
325  `checkout-step-2-confusion` raised in run on 2026-04-30; team acknowledged, fix shipped
326  2026-05-04. Don't re-emit unless theme reappears post-2026-05-04."_
327- key `addressed:surveys:csat-old-stale` — _"Survey `csat-old` last got responses
328  2026-02; appears abandoned but the team still has it active. P3 recommendation already
329  filed; don't re-recommend."_
330 
331By run #5 you'll know the team's active surveys, healthy response volumes, score
332baselines, which dismiss rates are structural, and which themes have already been
333raised — so when a real theme or regression appears, the finding lands with the right
334context already attached.
335 
336### Decide
337 
338For each candidate finding:
339 
340- **Emit** via `signals-scout-emit-signal` if it clears the confidence bar.
341  Strong scout findings: confidence ≥ 0.85, with concrete survey ids,
342  question ids, response counts, score deltas, and (for themes) 2–3 verbatim quotes
343  in the evidence. Sample-size matters here more than other domains — a finding on
344  10 responses needs to be tighter than one on 200.
345- **Remember** if below the bar but worth carrying forward (a theme with only 3
346  respondents that might grow, a score wobble that didn't yet hold for two weeks).
347- **Skip** with a one-line note if a scratchpad entry with a `noise:` or `addressed:`
348  key prefix already covers it.
349 
350Cross-check `inbox-reports-list` before emitting — if the same theme is already in the
351inbox from a prior run or another source, refresh the scratchpad rather than re-emit.
352 
353### Close out
354 
355**Summarize the run** — one paragraph: which surveys, what themes / anomalies you found,
356what you emitted, what you remembered, what you ruled out. The harness writes that
357summary to the run row as searchable prose; future runs read it via
358`signals-scout-runs-list`. Do **not** write a separate "run metadata" scratchpad entry —
359the run summary already serves that role.
360 
361## Disqualifiers (skip these)
362 
363- **Survey at the end of its scheduled window** — natural tail-off in responses;
364  not a regression. Check `schedule.end_date` before flagging.
365- **NPS / CSAT drift on < 30 responses in the recent window** — sample too small to
366  trust; memory entry only.
367- **Themes evenly split between positive and negative** — they cancel each other; no
368  single direction to surface.
369- **Theme matching an `addressed:` scratchpad entry** — the team already saw it and
370  acted; re-emitting wastes inbox space.
371- **One-off rant or off-topic response** — a single user typing "AAAA" or
372  quoting song lyrics isn't signal. Themes need ≥ 3 distinct respondents.
373- **Internal test / placeholder responses** — `TEST`, `TEST FEEDBACK DELETE!`,
374  `qwe`, `asdf`, single-character submissions, repeated submissions from the
375  survey author or the host org's own users. These are endemic on real projects
376  and will skew theme counts if you don't strip them. A `WHERE
377length(response) > 5 AND lower(response) NOT IN ('test', 'qwe', 'asdf')`
378  guard plus an `email NOT LIKE '%@<host_org_domain>%'` person-property filter
379  catches most of it.
380- **Survey paused or in draft** — not user-facing right now; check
381  `archived` / status / `start_date` before treating zero responses as a regression.
382- **PII or sensitive content in responses** — never emit verbatim PII. Quote the
383  themed claim, not the raw text, if responses contain personal data.
384 
385When in doubt, write a memory entry instead of emitting.
386 
387## MCP tools
388 
389Direct calls (read-only):
390 
391- `surveys-global-stats` — project-wide aggregate. **Start here** every cold
392  start; cheap sanity check on overall survey health before any per-survey work.
393- `survey-stats` — per-survey response statistics: `shown` / `dismissed` / `sent`
394  counts, unique respondents, conversion rates, timing. Date-filterable.
395- `survey-get` — full survey config for a candidate: questions (with ids and
396  types), `type` (popover / widget / api — affects how `survey shown` semantics
397  read), targeting (`linked_flag_id` / `targeting_flag_id` / `linked_insight_id`
398  / `conditions`), schedule (`start_date`, `end_date`), iteration config,
399  `updated_at`. Read this before drawing conclusions about score changes —
400  question wording changes invalidate trend comparisons.
401- `surveys-get-all` — last-resort discovery. Each survey object is 30–50 KB and
402  busy projects have 100+ active surveys; calling this with `limit > 5` will
403  blow your token budget. Prefer `surveys-global-stats` + an `execute-sql`
404  ranking query (see "Get oriented" above) to find the candidate set, then
405  `survey-get` per id. Use `surveys-get-all {"search": "..."}` if you need to
406  resolve a name from a memory entry.
407- `execute-sql` against `events` — for raw response analysis (rating trends, theme
408  aggregation). The property reference, the dual response-key coalesce, and the
409  `$survey_submission_id` dedupe SQL are all in
410  [`references/response-querying.md`](references/response-querying.md).
411- `read-data-schema event_property_values` — sample response values to confirm
412  property keys exist and have the shape you expect before running heavy aggregations.
413- `query-trends` — confirm `survey shown` / `survey sent` volume trends with weekly
414  comparisons. Cheaper than a full SQL aggregation when you just need the shape.
415- `activity-log-list` — correlate themes / score drops with recent product changes.
416 
417Harness-level:
418 
419- `signals-scout-project-profile-get` / `signals-scout-scratchpad-search` /
420  `signals-scout-runs-list` / `signals-scout-runs-retrieve` — orientation + dedupe.
421- `signals-scout-emit-signal` / `signals-scout-scratchpad-remember` — emit / remember.
422 
423### When you hit a gap
424 
425Two MCP gaps are known and may be worth flagging in a separate PR rather than working
426around in-skill:
427 
428- **Project profile doesn't include surveys.** Cold-start orientation has to call
429  `surveys-get-all` directly. Adding a `_surveys` builder to
430  `products/signals/backend/scout_harness/profile/builders.py` (a few rows: active
431  count, top surveys by recent volume, primary NPS / CSAT survey if any) would let
432  every scout — not just this one — see surveys at orientation time. Worth a P3.
433- **Survey summarization isn't MCP-callable.** The product has a summarization
434  pipeline at `products/surveys/backend/summarization/` but it's not exposed as an
435  MCP tool. If it were, this scout could lean on cached summaries instead of
436  re-aggregating themes from scratch each run. Worth a P2 for accuracy and cost.
437 
438If you notice a third gap during a run that would meaningfully unlock this scout,
439write a scratchpad entry with key `mcp-gap:surveys:<short-name>` so the gap surfaces in
440the next review via `text=mcp-gap`.
441 
442## When to stop
443 
444- No active surveys + no recent survey events → close out empty (after writing the
445  `not-in-use:` scratchpad entry).
446- Profile + scratchpad show a stable picture (known baselines, no recent inflection) →
447  close out empty.
448- A candidate matches a scratchpad entry with `noise:` / `addressed:` / `dedupe:` key
449  prefix → skip.
450- You've validated some hypotheses and emitted what's solid → close out, even if
451  there's more you could look at. Themes especially — fewer, sharper findings beat
452  a long list of weak clusters.
453 
454"Looked but found nothing meaningful" is a real outcome.
455