Authoring Signals Scouts

1---
2name: authoring-signals-scouts
3description: >
4  How to author, edit, and adapt PostHog Signals scouts — the scheduled agents that
5  scan a project and emit findings into the Signals inbox. Use when a user wants to
6  customize a canonical scout for their own setup (narrow its scope, retune its
7  thresholds, add disqualifiers), tweak a scout's schedule or dry-run posture, or
8  write a brand-new scout from scratch for a specific use case (a custom event, a
9  product surface no canonical scout covers). Covers the scout SKILL.md anatomy, the
10  emit contract, the dedupe + scratchpad-memory conventions, the per-team skills-store
11  path vs the canonical in-repo path, and the emit-and-inspect test loop (with dry-run as an
12  optional safety net). Trigger on
13  "write/edit/customize a signals scout", "new scout for X", "tune my scout schedule",
14  "make a scout that watches <event>".
15metadata:
16  owner_team: signals
17---
18 
19# Authoring Signals scouts
20 
21A **scout** is a scheduled agent that wakes on its own interval, looks at one PostHog
22project, decides what's genuinely worth surfacing, and emits it as a **finding** into
23the Signals inbox — or closes out empty, which is a real outcome. PostHog ships a fleet
24of **canonical scouts** (a cross-product generalist plus per-surface specialists). This
25skill helps you and your agent **adapt those canonical scouts to a specific project**, or
26**author new scouts from scratch** for a use case the fleet doesn't cover.
27 
28A scout is just an `LLMSkill` whose name starts with `signals-scout-`. The harness
29discovers scouts by globbing `signals-scout-*` over the project's skills, loads the body
30**verbatim** as the agent's system prompt, and progressively reads any bundled reference
31files on demand. **The `signals-scout-` name prefix is load-bearing: a skill named
32anything else will never run as a scout.**
33 
34## The job before the writing
35 
36Don't write a scout in the abstract. Ground it in the target project first — a scout is
37only as good as its fit to the data it watches.
38 
391. **Read the project.** `posthog:signals-scout-project-profile-get` returns the
40   deterministic snapshot the scout itself cold-starts from: products in use, top events
41   with reach/burst metrics, integrations, existing inbox counts. If the scout watches a
42   specific event, confirm it exists and check its shape with `posthog:read-data-schema`.
43   A scout for an event the project doesn't capture is dead on arrival.
442. **See what already runs.** `posthog:signals-scout-config-list` lists every existing
45   scout on the project with its schedule, `enabled`, and `emit` posture, plus each scout's
46   `description` (pulled from the skill's frontmatter) so you can tell what a scout watches
47   without loading its body. Don't duplicate a surface a canonical scout already covers —
48   adapt that one instead.
493. **Read the closest canonical scout.** It's your template and your reference shape. Pull
50   it with `posthog:llma-skill-get {"skill_name": "signals-scout-<x>"}` (per-team rows) or
51   read it from the repo at `products/signals/skills/signals-scout-*/`. The generalist
52   (`signals-scout-general`) is the broad template; if your scope is domain-tight, pick
53   the specialist closest to your surface — list the live roster with
54   `posthog:llma-skill-list {"search": "signals-scout"}` (specialists exist for most
55   product surfaces: error tracking, logs, AI observability, experiments, feature flags,
56   session replay, web analytics, surveys, and more).
574. **Skim the inbox.** `posthog:inbox-reports-list` shows what findings are actually
58   landing — calibrate so your scout adds signal, not noise.
59 
60## Choose the path
61 
62There are two independent decisions: **what** you're building, and **where** it lives.
63 
64### What
65 
66| Situation                                                                                      | Approach                                                                                                                           |
67| ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
68| A canonical scout is close but too broad / too noisy / missing a disqualifier for this project | **Adapt** it — narrow the scope, add disqualifiers, retune thresholds.                                                             |
69| You want a surface no canonical scout covers (a custom event, a product-specific funnel)       | **New scout from scratch** — copy the closest canonical scout as scaffolding, replace the domain discriminator + explore patterns. |
70| You only want to change _when_ / _whether_ a scout runs                                        | **No authoring** — just tune the config (see Run posture).                                                                         |
71 
72### Where
73 
74| Path                                 | Mechanism                                                                                                                                                                                                                  | Use when                                                                                                                              |
75| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
76| **Per-team** (the common user path)  | Create/edit a `signals-scout-*` `LLMSkill` row in the project's skills store via `posthog:llma-skill-create` / `-update` / `-file-create`, then register its config immediately via `posthog:signals-scout-config-create`. | Customizing for one project. The harness globs the row in on the next tick; canonical sync leaves your edited ("diverged") row alone. |
77| **Canonical** (PostHog contributors) | Edit disk under `products/signals/skills/signals-scout-*/`, lint/build, open a PR.                                                                                                                                         | Improving a scout for _every_ enrolled project. `lazy_seed` mirrors it onto all enrolled teams on the next tick.                      |
78 
79**Adapting-in-place tradeoff:** editing a canonical scout's row for your team marks it
80**diverged** — you stop receiving upstream improvements to that scout. If you only need an
81_additional_ behavior, prefer authoring a **new, differently-named** scout
82(`signals-scout-<your-scope>`) and leaving the canonical one intact.
83 
84See [`references/lifecycle-and-testing.md`](references/lifecycle-and-testing.md) for the
85exact skills-store calls, the build/lint commands, and how seeding works.
86 
87## Write the scout
88 
89First pick the **shape**. [`references/scout-patterns.md`](references/scout-patterns.md) is a
90cookbook of the reference architectures scouts fall into — anomaly watcher, watchlist
91explore/exploit, cross-product correlation, recommendation/gap, warehouse-backed source,
92custom single-event, open-text theme, external-tool/code — each mapped to a canonical scout
93you can copy as scaffolding. It also makes the key point that **a scout can watch any source
94PostHog ingests into the data warehouse, not just analytics events** (a Slack channel sync, a
95billing system, a CRM, a support inbox), plus external systems reachable from the sandbox.
96Find the closest pattern, then write the body.
97 
98Follow [`references/scout-anatomy.md`](references/scout-anatomy.md) — it has the frontmatter
99schema, the canonical body structure (quick close-out → orient → domain discriminator →
100explore patterns → save-memory → decide → disqualifiers → close-out), the lean-body rule,
101and copy-ready skeleton templates for both a specialist and the generalist.
102 
103Two craft references the whole fleet reasons in terms of — a good scout's **Decide** and
104**memory** sections are built on them, so read them before writing those sections:
105 
106- [`references/emit-contract.md`](references/emit-contract.md) — what `emit-signal` takes,
107  the confidence rubric, severity, dedupe keys, `finding_id`, the description
108  prose contract, and a worked example. This is how your scout decides _what clears the
109  bar_ and _how to write the finding_.
110- [`references/dedupe-and-memory.md`](references/dedupe-and-memory.md) — the four-states
111  classifier (net-new / material-update / already-covered / addressed-or-noise), the
112  scratchpad key-prefix vocabulary, and the cross-project noise patterns. This is how your
113  scout avoids re-emitting and learns across runs.
114 
115The single most important design decision in any scout is its **signal-vs-noise
116discriminator** — the cheap profile-shape read that separates "worth investigating" from
117"baseline". For error tracking it's the `count` vs `distinct_users` ratio; for CSP it's
118reach over raw count. Your new scout needs its own. Name it explicitly near the top of the
119body so every run anchors on it.
120 
121## Run posture (config)
122 
123A scout's schedule and emit behavior live on its `SignalScoutConfig`, separate from the
124skill body. For a **brand-new scout**, register the config immediately after creating the
125skill with `posthog:signals-scout-config-create {"skill_name": "signals-scout-<scope>", ...}`,
126setting any of the fields below in the same call — including creating it disabled or in
127dry-run **before it ever runs**. (It's an upsert: if the coordinator already auto-registered
128the row, your fields are applied to it.) Otherwise the coordinator auto-registers an enabled
129hourly default on its next tick (up to ~30 min). For an **existing scout**, tune with
130`posthog:signals-scout-config-update` (find the `id` via `-config-list`):
131 
132- `run_interval_minutes` — 10 to 43200. Default 60 (hourly). Slow a chatty or expensive
133  scout by raising this.
134- `enabled` — `false` pauses the scout entirely (coordinator skips it).
135- `emit` — defaults to **`true`**: the scout writes its findings straight to the inbox. The
136  standard flow is to make a scout and let it emit — seeing what actually lands is the
137  fastest way to calibrate it. Set **`emit=false` (dry-run)** only when you want to be extra
138  careful: the scout still runs and logs its reasoning but writes nothing to the inbox.
139  Reach for dry-run on a scout you expect to be chatty, expensive, or high-stakes; for most
140  scouts, just emitting and watching the inbox is the better loop.
141 
142## Test loop
143 
144You can't force a synchronous run as a user — scouts fire on their schedule. The standard
145loop is **emit + inspect**: ship the scout live, let it emit, and calibrate against what
146actually lands.
147 
1481. Ship the scout (the default `emit=true`) with a short `run_interval_minutes` so it fires
149   soon — set it at creation via
150   `posthog:signals-scout-config-create {"skill_name": ..., "run_interval_minutes": 10}`
151   right after `llma-skill-create`, rather than waiting for the coordinator to
152   auto-register an hourly default.
1532. After a tick, read what it did: `posthog:inbox-reports-list` (the findings it actually
154   emitted), `posthog:signals-scout-runs-list` (run summaries), `-runs-retrieve` (full
155   reasoning for one run), and `-scratchpad-search` (the durable memory it wrote).
1563. Refine the body — tighten the discriminator, add disqualifiers for whatever it
157   false-positived on, fix the emit calibration.
1584. Once it's landing the right findings, restore the interval to something sustainable
159   (hourly+).
160 
161**Want to be extra careful?** Set `emit=false` to dry-run first — create the config with
162`emit=false` via `-config-create` so the scout never has a live first run; it runs and logs
163what it _would_ have emitted (visible via `-runs-list` / `-runs-retrieve`) without writing to
164the inbox. Inspect, refine, then flip `emit=true`. Worth it for a scout you expect to be
165chatty, expensive, or high-stakes; otherwise just emitting and watching the inbox is the
166faster path to a calibrated scout.
167 
168Repo contributors get a faster loop — `hogli sync:skill` and the harness's local run path;
169see [`references/lifecycle-and-testing.md`](references/lifecycle-and-testing.md).
170 
171To **read** what your scouts are doing rather than change them — surveying the fleet, inspecting
172individual runs, the scratchpad memory, and assessing performance — use the read-only companion
173skill `exploring-signals-scouts`. Keep the two in sync when the scout config / run / scratchpad
174surfaces change.
175 
176## Quality bar for a v1 scout
177 
178- A named, cheap **signal-vs-noise discriminator** anchored near the top.
179- A **quick close-out** so a quiet run is cheap (don't pay for deep exploration when the
180  watched surface is at baseline or absent).
181- 2–4 concrete **explore patterns** with the actual queries/tools to run — starting
182  points, not a rigid checklist.
183- **Disqualifiers** listing this project's known noise (single-user quirks, dev-env
184  bursts, allowlisted entities).
185- A **Decide** section calibrated against the emit contract (confidence ≥ 0.65 to emit;
186  below that, write memory).
187- **Save-memory** guidance using the scratchpad prefixes so the scout gets smarter each run.
188- A lean body (push depth into `references/`) — every line is a recurring token cost on
189  every run.
190