Authoring Log Alerts

1---
2name: authoring-log-alerts
3description: >
4  Author useful, low-noise log alerts on services in a PostHog project. Use when the user asks to set up
5  alerts for their logs, suggest alerts they should add, or evaluate whether a service is worth monitoring.
6  Covers service triage, baseline characterisation, threshold drafting, back-testing via simulate, and
7  shipping with a notification destination.
8---
9 
10# Authoring log alerts
11 
12Authoring an alert is a _measurement_ problem, not a guessing problem. You are not trying to be exhaustive — you
13are trying to land thresholds that fire 0–3 times per week on real production patterns, on services that matter.
14 
15## When to use this skill
16 
17- The user asks to "set up alerts" / "suggest alerts" for their project.
18- The user wants to evaluate whether a service is producing alertable signal.
19- The user has just enabled log alerting and wants a starter set.
20 
21## When _not_ to use this skill
22 
23- Tuning an alert that already exists — that's a different job (use `posthog:logs-alerts-events-list` to inspect
24  fire/resolve cadence and `posthog:logs-alerts-partial-update` to adjust).
25- Investigating an active incident — pull rows with `posthog:query-logs`, don't author an alert mid-incident.
26 
27## Tools
28 
29| Tool                                                                  | Job                                                                           | Where it fits      |
30| --------------------------------------------------------------------- | ----------------------------------------------------------------------------- | ------------------ |
31| `posthog:logs-services`                                               | Top-25 services in window with log_count, error_count, error_rate, sparkline. | Step 1 — triage.   |
32| `posthog:logs-attributes-list` / `posthog:logs-attribute-values-list` | Discover keys/values for narrower filters.                                    | Step 2, optional.  |
33| `posthog:logs-count-ranges`                                           | Adaptive time-bucketed counts for a filter.                                   | Step 3 — baseline. |
34| `posthog:logs-alerts-simulate-create`                                 | Replay a draft config against `-7d` history with full state machine.          | Step 4 — validate. |
35| `posthog:logs-alerts-create`                                          | Persist the alert.                                                            | Step 5 — ship.     |
36| `posthog:logs-alerts-destinations-create`                             | Wire the alert to Slack or webhook.                                           | Step 5 — ship.     |
37 
38Do **not** call `posthog:query-logs` during authoring. You need distributions, not rows. Reserve `posthog:query-logs` for
39the very end if the user asks "show me a sample of what would have fired" — `limit: 10` is plenty.
40 
41## Workflow
42 
43### 1. Triage — pick candidate services
44 
45Call `posthog:logs-services` for the last 24h with no filters. The response is capped at 25 services and includes a
46sparkline, so it is small and bounded.
47 
48A service is a candidate when **both** are true:
49 
50- `log_count` is non-trivial (≥ ~1k in 24h — quieter services produce too little signal to alert on).
51- `error_rate` is non-zero, **or** the user has named the service explicitly.
52 
53Skip services with high volume but `error_rate == 0` unless the user wants a volume-shape alert (e.g. "warn me
54if api-gateway suddenly stops producing logs"). Volume-floor alerts use `threshold_operator: below` and need
55different reasoning — see [references/volume-floor-alerts.md](./references/volume-floor-alerts.md).
56 
57If the user names a service, treat it as a candidate even without error signal.
58 
59### 2. (Optional) Narrow the filter
60 
61If a service has many error sub-types, an alert on "all errors" is usually too broad. Use
62`posthog:logs-attributes-list` (try `attribute_type: log`) and `posthog:logs-attribute-values-list` to find a discriminator —
63common ones are `http.status_code`, `error.type`, `k8s.container.name`. Add the narrowing filter to your draft.
64 
65Keep it simple: one severity filter + one or two attribute filters is plenty. Multi-clause filters are
66harder to reason about and rarely improve precision.
67 
68### 3. Baseline — characterise the candidate over 7 days
69 
70Call `posthog:logs-count-ranges` with the candidate's filters, `dateRange: { date_from: "-7d" }`, and
71`targetBuckets: 24` (one bucket ≈ 7h). The response gives you bucket counts.
72 
73**Do not eyeball the percentiles or scale the threshold to the alert window manually.** Pipe the
74count-ranges response into the helper script:
75 
76```bash
77echo '<count-ranges JSON>' | python3 scripts/baseline_stats.py --window-minutes 5
78```
79 
80The script returns:
81 
82```json
83{
84  "n_buckets": 12,
85  "bucket_minutes": 420.0,
86  "alert_window_minutes": 5,
87  "stats": { "p50": 12.0, "p95": 71.25, "p99": 126.25, "max": 140 },
88  "suggested_threshold_count": 5,
89  "rationale": "max(p99=126.25, median*3=36.0, floor=5) scaled from 420m bucket to 5m window",
90  "health": []
91}
92```
93 
94Use `suggested_threshold_count` as your starting threshold. Read `health`:
95 
96| `health` flag           | What it means                                | What to do                                                                                                 |
97| ----------------------- | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
98| `sparse:N_of_M_buckets` | Too few non-empty buckets for a 7d baseline. | Widen filter, extend to `-30d`, or skip.                                                                   |
99| `empty`                 | All buckets are zero.                        | Skip — no signal.                                                                                          |
100| `spiky`                 | `max` is 10×+ `p95`.                         | Count-threshold alerts work well. Proceed.                                                                 |
101| `flat`                  | `p95` ≈ `p50`.                               | Be cautious — either no incidents in lookback, or the metric is too smooth. Try a longer lookback or skip. |
102| `[]` (empty)            | Healthy distribution.                        | Proceed.                                                                                                   |
103 
104### 4. Draft and simulate
105 
106Pick a starter draft from these defaults — see [references/threshold-defaults.md](./references/threshold-defaults.md)
107for the reasoning:
108 
109| Setting               | Default                                     | Notes                                                                 |
110| --------------------- | ------------------------------------------- | --------------------------------------------------------------------- |
111| `threshold_count`     | `suggested_threshold_count` from the script | Already scaled to the alert window.                                   |
112| `threshold_operator`  | `above`                                     | Use `below` only for volume-floor alerts.                             |
113| `window_minutes`      | `5`                                         | Allowed: 5, 10, 15, 30, 60. Must match what you passed to the script. |
114| `evaluation_periods`  | `3`                                         | M in N-of-M.                                                          |
115| `datapoints_to_alarm` | `2`                                         | N in N-of-M. 2-of-3 reduces flap from a single noisy bucket.          |
116| `cooldown_minutes`    | `30`                                        | Minimum time between repeat fires.                                    |
117 
118Call `posthog:logs-alerts-simulate-create` with these settings and `date_from: "-7d"`. The response gives you `fire_count`
119and `resolve_count`.
120 
121### 5. Iterate — three rounds, then ship or skip
122 
123Target: `fire_count` between 0 and ~3 over `-7d`. If outside the band:
124 
125| Outcome                                                         | Adjustment                                                                                           |
126| --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
127| `fire_count` = 0 over 7d _and_ the baseline was spiky           | Lower `threshold_count` toward `stats.p95` from the script, or drop to 1-of-2.                       |
128| `fire_count` = 0 _and_ the baseline was flat                    | The service has no alertable signal. Skip it; log why.                                               |
129| `fire_count` > 5                                                | Raise `threshold_count` toward `stats.max` from the script, or move to 3-of-5 for a smoother window. |
130| `fire_count` is fine but resolve_count never matches fire_count | Cooldown is too long, or the underlying state is genuinely sticky. Acceptable for now.               |
131 
132When adjusting the threshold, **read values from the script's `stats` block — never recompute percentiles
133by hand.**
134 
135Cap iteration at **3 simulate calls per candidate**. If you can't land in the band in 3 rounds, the metric
136is wrong — either the filter is too broad, the window is wrong, or the service genuinely doesn't have a
137threshold-shape signal. Note it and move on.
138 
139### 6. Ship — create + attach destination
140 
141Once a draft simulates cleanly:
142 
1431. Call `posthog:logs-alerts-create` with the validated config. Use a name like `<service> error rate (auto)` so the
144   user can see at a glance which alerts came from this skill.
1452. Call `posthog:logs-alerts-destinations-create` to wire it to a notification target. **An alert with no destination
146   is silent.** Always confirm the channel name or webhook URL with the user before attaching — never wire
147   an auto-generated alert to a production channel without explicit confirmation. If the user is unsure,
148   suggest a low-traffic testing channel for the first few alerts.
149 
150If the user wants alerts created in `enabled: false` state for review-then-flip, pass `enabled: false` to
151`-create` and tell them how many drafts you produced.
152 
153## Filter shape — required
154 
155The `filters` field on `posthog:logs-alerts-create` takes a subset of `LogsViewerFilters` and **must contain at
156least one of**:
157 
158- `severityLevels` — list of `["trace","debug","info","warn","error","fatal"]`
159- `serviceNames` — list of service name strings
160- `filterGroup` — property filter group
161 
162The same shape goes into `posthog:logs-alerts-simulate-create`'s `filters` field. Match the simulate filters to the alert filters
163exactly — otherwise the simulation is testing a different alert than the one you ship.
164 
165Example minimum:
166 
167```json
168{
169  "severityLevels": ["error", "fatal"],
170  "serviceNames": ["api-gateway"]
171}
172```
173 
174## Token-economy rules
175 
176- One `posthog:logs-services` call at the start, not per-candidate.
177- One `posthog:logs-count-ranges` call per candidate at `targetBuckets: 24`. Don't go above 30 during authoring.
178- ≤ 3 `posthog:logs-alerts-simulate-create` calls per candidate.
179- Zero `posthog:query-logs` calls during the authoring loop.
180- Prefer reporting a small set of well-validated alerts over a long list of unvalidated drafts.
181 
182## Output
183 
184Report what you did, in this shape:
185 
186- For each shipped alert: name, filters, threshold, simulated fire_count over 7d, destination.
187- For each skipped candidate: service name + why (flat baseline, can't land threshold, low volume).
188- Total simulate calls made, total alerts created.
189 
190The user should be able to read this and decide whether to disable any drafts before they go live.
191