Signals Scout Observability Gaps

1---
2name: signals-scout-observability-gaps
3description: >
4  Focused Signals scout for finding observability gaps in PostHog itself — significant
5  event volumes the team isn't tracking, custom events with no insight or dashboard
6  coverage, insights pointing at events that have stopped firing, dashboards missing
7  related context, critical events with no alerts. Watches the event-stream-vs-saved-
8  inventory delta as the team's product evolves and emits findings recommending new
9  insights, dashboard additions, or alerts when gaps clear the confidence bar.
10  Self-contained peer in the signals-scout-* fleet — no dependencies on other skills.
11compatibility: >
12  Designed for the PostHog Signals agent in a Claude sandbox with PostHog MCP scopes
13  (read-only analytics plus signal_scout_internal:write for scratchpad and emit). Assumes
14  the signals-scout MCP tool family plus the analytics and entity tools listed in the
15  body's MCP tools section.
16metadata:
17  owner_team: signals
18  scope: observability_gaps
19---
20 
21# Signals scout: observability gaps
22 
23You are a focused observability-gaps scout. Spot meaningful gaps between **what events
24this team is producing** and **what they have set up to observe** — and emit findings
25that recommend new insights, dashboard additions, or alerts when a gap clears the
26confidence bar. An empty findings list is a real outcome; recommending things the team
27already has, or recommending coverage for noise events, is worse than recommending
28nothing.
29 
30The shape of this scout is different from the other specialists: the findings are
31**recommendations**, not **problems**. The confidence bar is correspondingly higher —
32a noisy "you should track X" stream destroys the inbox's signal-to-noise ratio. Prefer
33fewer, well-evidenced recommendations.
34 
35## Quick close-out: is this team big enough to have gaps?
36 
37If `top_events` in the project profile is null or shows fewer than ~5 events firing
38above 100/day, the project is too quiet for observability-gap analysis to surface real
39recommendations. Write one scratchpad entry:
40 
41- key: `not-applicable:observability_gaps:team{team_id}`
42- content: brief note ("checked at {timestamp}, top_events count <5 above 100/day, too
43  quiet for gap analysis")
44 
45Close out empty. Future observability-gaps runs read this entry cold and short-circuit
46in seconds. Re-running with the same key idempotently refreshes the timestamp — the
47entry stays until the team grows into meaningful volume, at which point the next run
48rewrites or deletes it.
49 
50## Quick close-out: is this team already saturated?
51 
52The opposite end has a fast path too. On a mature project (thousands of insights,
53hundreds of alerts), a few runs will establish that whole gap families are
54**saturated** — every high-volume event already has dense coverage, and newly-emerged
55events get covered within days. Record that as durable memory instead of
56rediscovering it every run:
57 
58- key: `pattern:observability_gaps:<family>-saturated` (or one `coverage-saturated`
59  entry spanning families)
60- content: what was probed, the coverage counts found, and a **tripwire** — the
61  concrete condition under which the family is worth re-probing (e.g. "a NEW
62  broad-reach event class (>~10k distinct users/7d) with genuinely zero coverage
63  that is a discrete business/feature metric, not ambient telemetry").
64 
65Once saturation is documented, the default run shape changes: check the tripwire
66against the fresh profile, then run **at most one fresh probe** — an angle no prior
67run has covered — to earn the close-out rather than inherit it. If the tripwire is
68untriggered and the probe comes back clean, close out empty in minutes. Don't re-run
69coverage SQL a run verified hours ago; that's duplication, not diligence.
70 
71## How a run works
72 
73Cycle between these moves; skip what's not useful, revisit what is.
74 
75### Get oriented
76 
77Three cheap reads cold-start a run:
78 
79- `signals-scout-scratchpad-search` (`text=gap` or `text=observability`) — durable team
80  steering inherited from past observability runs. **Entries with `pattern:`, `noise:`,
81  `addressed:`, or `dedupe:` key prefixes tell you what's normal, what's already
82  surfaced, what to skip.** Critical here because the same gap should never be re-emitted
83  across runs.
84- `signals-scout-runs-list` (last 14d) — what prior observability-gap scouts found and
85  what was ruled out. Skim summaries; pull `signals-scout-runs-retrieve` only when a
86  summary mentions a recommendation you're considering.
87- `signals-scout-project-profile-get` — `top_events` for volume + reach, `popular_insights`
88  for what's already saved, `recent_dashboards` for the dashboards in active use. This
89  one read tells you most of what you need to detect gaps.
90 
91### Explore — what good observability gaps look like
92 
93Six families of gap, ordered by typical signal density. None is automatic — each needs
94volume + coverage check + dedupe before becoming a finding.
95 
96#### 1. High-volume custom event with no insight coverage
97 
98Custom event (not a `$builtin` like `$pageview` / `$identify`) firing meaningful
99volume per day, no saved insight references it.
100 
101Direct calls:
102 
103- `read-data-schema events` — surface event names + 24h volumes.
104- `execute-sql` against `system.insights` — find insights mentioning the event name in
105  `name`, `description`, or `query` JSON. Pattern: `query::text ILIKE '%{event_name}%'`.
106- Check `event-definitions-list` for `last_seen_at` recency and the `verified` flag —
107  the team flagged it as worth tracking.
108 
109Strong signal: event > 1000/day, no insight, `verified=true`. Weak signal: event
110< 100/day, untyped, sporadic.
111 
112Volume ranking has a blind spot: a recently-born event with broad reach but low
113per-user frequency may never rank into the count-ranked `top_events`, and a 7-day
114query window clamps `min(timestamp)` so it cannot tell new events from old ones.
115Probe emergence directly with a wide window — events table, last 60 days,
116`event NOT LIKE '$%'`, grouped by event, keeping only groups where
117`min(timestamp) >= now() - 14d` (genuinely new) and distinct users in the last 7
118days clear a reach floor (~500+), ordered by that reach. Each hit is a candidate the
119top-events lens structurally cannot see; run it through the same coverage check and
120disqualifiers as any other candidate.
121 
122#### 2. Insight drift — saved insights pointing at zero-volume events
123 
124An existing insight filters on event X, but X has 0 (or near-zero) firings in the last
1257 days. Often a sign of:
126 
127- Event renamed (e.g. `signed_up` → `sign_up_completed`) and the insight wasn't updated.
128- Event sunset (deprecated by product change) and the insight is stale.
129- Capture broken upstream (different lens — let error-tracking own this).
130 
131Direct calls:
132 
133- `execute-sql` over `system.insights` to extract the events series each insight
134  filters on.
135- `query-trends` to measure recent volume of those events.
136- For zero-volume events, search `event-definitions-list` for similar names suggesting
137  a rename (Levenshtein-close, same prefix, same property shape).
138 
139Strong signal: insight has been viewed in the last 30d AND its primary event has 0
140firings in 7d AND a similar-named event is firing > 100/day.
141 
142#### 3. Critical event with no alerts configured
143 
144Some events name themselves — `payment_failed`, `signup_failed`, `*_error`, `*_blocked`.
145If they fire at all and no alert exists, that's a gap. Use the project's own
146patterns: search the event vocabulary for terms like `failed`, `error`, `blocked`,
147`denied`, `rejected`, `timeout`, `crashed`.
148 
149Direct calls:
150 
151- `read-data-schema events` filtered by name pattern (`failed`, `error`, etc).
152- `alerts-list` — what alerts exist and what they target.
153- `query-trends` to confirm volume is non-trivial (not just one-off).
154 
155Strong signal: event name suggests failure semantics, fires > 10/day, zero alerts
156target it. Weak signal: name has `error` but the event is benign developer telemetry.
157 
158#### 4. Dashboard scope gap
159 
160A dashboard exists for a topic (name + description match a domain like "Onboarding",
161"Revenue", "Conversion"), but high-volume events related to that topic are not on any
162of its insights.
163 
164Direct calls:
165 
166- `dashboards-get-all` — current dashboards + tags + descriptions.
167- For each dashboard, list insights via the dashboard tile endpoint or
168  `system.insights WHERE id IN (dashboard.insight_ids)`.
169- Match domain-themed events to dashboards by name overlap.
170 
171Strong signal: dashboard explicitly named for a domain, > 5 events match the domain
172and > 1000/day each, none on the dashboard. Weak signal: arbitrary keyword overlap.
173 
174#### 5. Funnel candidate — sequential event pattern with no funnel insight
175 
176Three or more events that frequently co-occur in user sessions in a fixed order, no
177funnel insight tracks the sequence. Usually an onboarding flow, signup flow, checkout
178flow, etc.
179 
180Direct calls:
181 
182- `query-paths` (one call) on top distinct events to surface common sequences.
183- `execute-sql` against `system.insights WHERE filters::text ILIKE '%FunnelsQuery%'`
184  to find existing funnels.
185- Check sequence length + retention (% users completing each step).
186 
187Strong signal: 3-step sequence with > 1000 users completing step 1, > 50% reaching
188step 2, no existing funnel covering the sequence. Confidence threshold is high here
189because funnels are subjective — a common sequence isn't always a meaningful funnel.
190 
191#### 6. Property cardinality / missing breakdown
192 
193A high-cardinality property on a high-volume event, and existing insights tracking
194the event use no breakdown — the team is losing dimension by aggregation.
195 
196Direct calls:
197 
198- `read-data-schema event_property_values` — see distinct values for a property.
199- `execute-sql` over `system.insights` for the event — extract `breakdownFilter` shape.
200- Compare property cardinality to whether any insight breaks down by it.
201 
202Strong signal: property has 5-50 distinct values (not unbounded), event > 5000/day,
203no insight breaks down by it. Weak signal: property has 1000+ distinct values
204(would explode the chart) or ≤ 2 values (no information added).
205 
206### Recommend — emit a finding
207 
208A finding here recommends an action, not surfaces a problem. Required elements:
209 
210- **Specific event(s) / insight(s) / dashboard(s)** — entity IDs in the evidence list
211  so a human can click straight to them.
212- **Volume + reach numbers** — the gap matters because of _N_ events affecting _M_
213  users; quote both.
214- **Suggested action** — "create a trends insight on event X" / "update insight Y to
215  point at event Z" / "add insight A to dashboard B" / "configure an alert on event C".
216  Concrete is better than abstract.
217- **Why now** — if this gap has existed for weeks, why is it surfacing now? Because
218  volume just crossed a threshold? Because a new event class emerged? Volume + recency
219  is the dedupe key.
220 
221Severity for observability-gap findings is almost always **P3** (suggestion). The
222confidence bar trades off:
223 
224- **Volume threshold** — gap is structurally interesting only at scale. Below 100/day,
225  the recommendation is noise.
226- **Stable-not-spurious** — gap has been present for at least 7 **complete days in
227  the project timezone**. Avoid flagging events that just appeared yesterday; a
228  partial current day or a deploy-day spike can fake stability.
229- **No prior coverage** — search `popular_insights` and `existing_inbox_reports`
230  before emitting. If a previous run already recommended this gap, don't re-emit.
231 
232### Park, then emit — the watch lifecycle
233 
234Most good recommendations are not emitted the run they're spotted — they're parked
235until the stability bar crosses. The lifecycle:
236 
2371. **Park** — write a `watch:observability_gaps:<gap>` entry carrying the
238   discriminating conditions (the exact checks that make this a real gap), the
239   volume evidence so far, and the earliest emit time (when the 7th complete
240   project-timezone day closes). Future runs inherit the candidate instead of
241   re-deriving it.
2422. **Re-verify live, then emit** — the run that crosses the bar must re-check every
243   discriminating condition against live data before emitting (coverage can appear,
244   volume can collapse). Never emit off the watch entry alone.
2453. **Guard** — after emitting, update the watch entry with the finding id and a
246   ~30-day dedupe: no re-emit before then unless a materially new angle appears.
2474. **Retire** — the entry doesn't live forever. When coverage appears, the
248   recommendation was actioned: delete the entry (or convert it to `addressed:`).
249   If ~30 days pass and nobody built coverage, that's "recommended but ignored" —
250   convert it to a `noise:` skip note rather than re-emitting.
251 
252### Close out
253 
254**Summarize the run** — one paragraph: what you looked at, what you emitted, what you
255remembered, what you ruled out and why. The harness writes that summary to the run row
256as searchable prose; future runs read it via `signals-scout-runs-list`. Do **not** write
257a separate "run metadata" scratchpad entry — the run summary already serves that role.
258 
259## Disqualifiers (skip these)
260 
261- **Builtin events without saved insights** — `$pageview`, `$autocapture`, `$identify`,
262  `$set`, `$opt_in`, `$groupidentify`, `$feature_flag_called` are surfaced through
263  PostHog's product views (Web Analytics, Feature Flags) without needing a custom
264  insight. Don't recommend creating one.
265- **Test events from internal users** — pin a `noise:observability_gaps:internal-distinct-ids`
266  scratchpad entry for known internal distinct_ids and skip them in volume counts.
267- **Events from disabled feature flags** — if the event only fires when a flag is
268  disabled or only for a tiny rollout %, the volume is artificially low.
269- **Events on ad-hoc one-off dashboards** — a private dashboard with one viewer doesn't
270  count as "covered." Use the `popular_insights` viewer-count threshold.
271- **Ambient app-shell telemetry** — an event whose distinct-user reach is roughly
272  equal to `$pageview`'s fires for nearly every user as part of the app shell, not
273  as a discrete feature metric. Zero saved insights on it is usually intentional;
274  compare reach against `$pageview` before calling it a gap.
275- **Deliberate engineering firehoses** — high-volume internal perf/telemetry events
276  the team consumes via ad-hoc SQL or notebooks rather than saved insights. Before
277  declaring zero coverage, check whether notebooks reference the event — covered by
278  choice is not a gap.
279- **Experiment-exposure events** — events that exist to drive an experiment's
280  metrics are covered by the experiment itself. Don't recommend standalone insights
281  for them while the experiment runs.
282- **One-per-user lifecycle events** — onboarding, wizard, and setup events fire once
283  per user; their volume is just signup flow-through and rarely deserves a
284  standalone insight.
285- **Time-boxed promotion / campaign events** — campaign-shaped events appear, spike,
286  and end by design. Going quiet is not drift, and lacking coverage is not a gap
287  unless the underlying surface (impressions + conversions) persists.
288- **Incident-investigation scaffolding** — short-lived events created during an
289  incident, often with incident-named insights attached. They stop firing when the
290  incident closes; flagging the stoppage as drift is a false positive.
291- **Legacy event-name variants** — insights that deliberately union an old and a new
292  event name for historical continuity are well-maintained, not drifted. Read the
293  insight's query JSON before declaring a dead event "still referenced."
294 
295When in doubt, write a scratchpad entry instead of emitting. Recommendations have a
296high panic radius for whoever owns the observability surface — false positives erode
297trust fast.
298 
299## MCP tools
300 
301Direct calls (read-only):
302 
303- `read-data-schema` — `kind=events` for volumes, `kind=event_properties` /
304  `event_property_values` for cardinality and breakdowns.
305- `query-trends` — confirm recent-window volume + reach numbers cited in evidence.
306- `query-paths` — sequence detection for funnel candidates.
307- `insights-list` — paginated insight catalog (use sparingly; SQL is faster).
308- `dashboards-get-all` — active dashboards + tags.
309- `event-definitions-list` — event-definition metadata: `verified` flag, `last_seen_at`,
310  `created_at`, custom-vs-builtin marker.
311- `alerts-list` — existing alert configurations and what events they target.
312- `execute-sql` over `system.insights` / `system.dashboards` / `system.cohorts` —
313  the fast path for "does an insight reference event X?" type queries.
314 
315Harness-level:
316 
317- `signals-scout-project-profile-get` — cold orientation snapshot. Has `top_events`,
318  `popular_insights[13]`, `recent_dashboards`, `existing_inbox_reports` already.
319- `signals-scout-scratchpad-search` / `signals-scout-scratchpad-remember` — durable steering.
320- `signals-scout-runs-list` / `signals-scout-runs-retrieve` — what prior runs found.
321- `signals-scout-emit-signal` — emit a recommendation finding.
322 
323For deeper investigation playbooks, the sandbox image bakes upstream PostHog skills:
324`posthog:querying-posthog-data` (HogQL syntax + system.\* search patterns) and
325`posthog:exploring-autocapture-events` (custom-event vs autocapture distinctions, when
326each lens applies).
327 
328## When to stop
329 
330- Scratchpad + recent runs + profile show every domain you've considered already has
331  coverage or has been recommended → close out empty.
332- A candidate matches a scratchpad entry with `addressed:` (recommendation actioned) or
333  `noise:` (recommended but ignored) key prefix → skip with a one-line note.
334- You've validated 1-2 high-confidence gaps and emitted them → close out, even if
335  there's more you could look at. Quality over volume — recommendations are a budget,
336  not a target.
337 
338"Looked but found nothing meaningful" is a real outcome, not a failure. Every
339recommendation that doesn't ship is one fewer false positive eroding the inbox.
340