Investigating Error Issue

1---
2name: investigating-error-issue
3description: >
4  Investigates a single PostHog error tracking issue end-to-end. Use when
5  the user provides an issue ID or pastes an issue URL
6  (`/error_tracking/<id>`) and wants to understand the error — who it
7  affects, what triggers it, when it started, whether it correlates with
8  a release, browser, OS, or feature flag, and what the next step should
9  be. Pulls aggregated metrics, sample exception events, segment
10  breakdowns, linked replays, and synthesizes a hypothesis-grade summary
11  in one pass.
12---
13 
14# Investigating an error tracking issue
15 
16When a user asks "what's going on with this error?" or pastes an issue URL, gather
17the context they would otherwise have to assemble manually: who is hitting it, what
18changed, where it happens, and whether a replay shows the cause.
19 
20## Available tools
21 
22| Tool                                        | Purpose                                                                                     |
23| ------------------------------------------- | ------------------------------------------------------------------------------------------- |
24| `posthog:query-error-tracking-issue`        | Compact issue details (status, assignee, top frame, release, aggregates)                    |
25| `posthog:query-error-tracking-issue-events` | Sampled `$exception` events with stack, URL, browser, `$session_id`                         |
26| `posthog:execute-sql`                       | Breakdowns, release / flag correlations, surrounding events + console logs around the error |
27| `posthog:query-logs`                        | OTEL log entries around the error timestamp for server-side issues                          |
28| `posthog:query-session-recordings-list`     | Linked replays (delegate ranking to `finding-replay-for-issue`)                             |
29| `posthog:read-data-schema`                  | Confirm property keys before filtering on them                                              |
30 
31## Workflow
32 
33### Step 1 — Establish the issue baseline
34 
35Fetch the issue record with its compact aggregates and a sparkline:
36 
37```json
38posthog:query-error-tracking-issue
39{
40  "issueId": "<issue_id>",
41  "dateRange": { "date_from": "-30d" },
42  "includeSparkline": true,
43  "volumeResolution": 12
44}
45```
46 
47Capture: `name`, `description`, `status`, `first_seen`, `last_seen`, `assignee`,
48total `occurrences` / `users` / `sessions`, top in-app frame, latest release
49metadata, and the volume buckets.
50 
51The sparkline tells you the shape — flat, spike, ramp, or recurring — and that
52shape drives the rest of the investigation. If the user only asked a status
53question, skip `includeSparkline` to save tokens.
54 
55### Step 2 — Pull a sample exception event
56 
57A captured event has the stack frames, URL, browser, and properties needed to
58reason about cause. Pull a recent sample first, then an early one to compare.
59 
60```json
61posthog:query-error-tracking-issue-events
62{
63  "issueId": "<issue_id>",
64  "limit": 1,
65  "verbosity": "stack"
66}
67```
68 
69Use `verbosity: "raw"` only if the truncated stack hides the answer. The tool
70defaults to `onlyAppFrames: true`, which strips vendor frames; flip to `false`
71when the bug appears to live in a third-party library — or when the response
72comes back with `stacktrace.type: "resolved"` but no frames at all (common for
73minified bundles where every frame looks vendor-y to the resolver, e.g. React
74production builds).
75 
76For the earliest sample, narrow `dateRange` to a tight window around the
77issue's `first_seen` (e.g. set `date_from` slightly before and `date_to`
78slightly after) and pass `orderDirection: "ASC"` so you get the earliest
79event in the window rather than the latest — the tool defaults to `DESC`,
80which would return a recent event and silently duplicate the first call.
81If recent and earliest events look materially different — different stack
82root, different URL pattern — the issue may be a grouping mistake. Flag for
83`grouping-noisy-errors` instead of continuing as if it were one bug.
84 
85### Step 3 — Run breakdowns to isolate the cause
86 
87Breakdowns aren't a typed tool — drop into `execute-sql`. Run only the
88breakdowns the issue's shape suggests; each one costs a query and clutters the
89synthesis.
90 
91| Sparkline shape   | First breakdown to try                                                   |
92| ----------------- | ------------------------------------------------------------------------ |
93| Spike from zero   | By app version / release — almost always a deploy regression (see below) |
94| Steady-state high | By browser / OS — rendering or platform-specific bug                     |
95| Ramp              | By geography or feature flag — gradual rollout exposure                  |
96| Bursts then quiet | By time of day or `$current_url` — scheduled job or specific page        |
97 
98#### Picking the right version property
99 
100PostHog emits three version-shaped fields. They mean different things and only
101one of them answers "what version of the user's app introduced this?":
102 
103| Property              | What it is                                                | Auto-captured by                                                                   | Use for                                                                  |
104| --------------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
105| `$exception_releases` | Cymbal-managed release map, keyed by release ID           | Only when SDK publishes release metadata (e.g. sourcemap upload tied to a release) | Most precise release attribution **when present**                        |
106| `$app_version`        | The user's deployed app version                           | iOS (`CFBundleShortVersionString`), React Native (Expo / react-native-device-info) | "What deploy of my app introduced this?" — the question users care about |
107| `$lib_version`        | The PostHog SDK library version (e.g. posthog-js 1.298.0) | Every SDK on every event                                                           | The narrow "did upgrading the PostHog SDK introduce this?" question      |
108 
109`$lib_version` is on virtually every event, which makes it tempting — but it's
110the PostHog library version, not the user's app version. A constant
111`$lib_version` paired with a spike means the user shipped a regression in
112their own code with the SDK unchanged, which is the common case. Reach for
113`$lib_version` only when nothing else is populated and you're explicitly
114asking "did upgrading PostHog cause this?".
115 
116Web / server / Node / Java / Python projects do **not** auto-capture
117`$app_version` — the customer has to set it (via `register`, a context
118provider, or `before_send`). If the breakdown comes back with one
119`$app_version` row of all-NULL, say so explicitly in the synthesis and
120suggest the customer wire it up; falling back to `$exception_releases` or to
121a per-day timeline by `first_seen` keeps the investigation moving.
122 
123Example (`$app_version` — populated automatically on mobile, manually on
124web / server):
125 
126```sql
127posthog:execute-sql
128SELECT
129    properties.$app_version AS app_version,
130    count() AS occurrences,
131    uniq(person_id) AS users,
132    min(timestamp) AS first_seen,
133    max(timestamp) AS last_seen
134FROM events
135WHERE event = '$exception'
136    AND (issue_id = '<issue_id>' OR properties.$exception_issue_id = '<issue_id>')
137    AND timestamp > now() - INTERVAL 30 DAY
138GROUP BY app_version
139ORDER BY occurrences DESC
140LIMIT 20
141```
142 
143The `(issue_id = ... OR properties.$exception_issue_id = ...)` pattern
144mirrors the canonical `build_issue_where` clause from
145`products/error_tracking/backend/api/query_utils.py`. `issue_id` is the
146resolved virtual field on `events` (it follows fingerprint overrides so
147merged/split issues route correctly); `properties.$exception_issue_id` is
148the raw event property captured at ingestion. Filtering on only the property
149silently undercounts events for issues that have been merged or split.
150 
151If `first_seen` for one `app_version` is much later than the issue's overall
152`first_seen`, that release introduced or worsened the bug — strong root-cause
153signal. If every row is `NULL`, the SDK isn't reporting an app version on
154this project (common on web / server) — switch to `$exception_releases` if
155the customer ships releases, or fall back to a `toDate(timestamp)` timeline.
156 
157When `$exception_releases` is populated, it's a JSON dict keyed by release
158ID. There is no top-level `$release` property; query `$exception_releases`
159directly when you need release attribution and the customer has it wired up.
160 
161Repeat with `properties.$browser`, `properties.$os`, `properties.$current_url`,
162or any feature flag the project tags errors with.
163 
164### Step 4 — Check feature flag exposure
165 
166If the user suspects an experiment or rollout, check whether affected users had
167a flag enabled when the error fired.
168 
169To enumerate which flags were evaluated on affected users, parse the
170`$active_feature_flags` property — it is materialized as a JSON-encoded string in
171ClickHouse, so `arrayJoin(properties.$active_feature_flags)` directly will fail;
172`JSONExtract` is the working pattern:
173 
174```sql
175posthog:execute-sql
176SELECT
177    arrayJoin(JSONExtract(toString(properties.$active_feature_flags), 'Array(String)')) AS flag,
178    count() AS occurrences,
179    uniq(person_id) AS users
180FROM events
181WHERE event = '$exception'
182    AND (issue_id = '<issue_id>' OR properties.$exception_issue_id = '<issue_id>')
183    AND timestamp > now() - INTERVAL 14 DAY
184    AND notEmpty(toString(properties.$active_feature_flags))
185GROUP BY flag
186ORDER BY occurrences DESC
187LIMIT 20
188```
189 
190Caveat: every event captures every evaluated flag key, so this enumeration often
191returns identical counts across flags and **doesn't tell you which flag
192correlates with the error** — only which were on the user. To actually test a
193hypothesis, query the per-flag value column `properties.$feature/<flag-key>`,
194which carries the evaluated value (`true`/`false`/variant name):
195 
196```sql
197posthog:execute-sql
198SELECT
199    properties.`$feature/my-flag-key` AS variant,
200    count() AS occurrences,
201    uniq(person_id) AS users
202FROM events
203WHERE event = '$exception'
204    AND (issue_id = '<issue_id>' OR properties.$exception_issue_id = '<issue_id>')
205    AND timestamp > now() - INTERVAL 14 DAY
206GROUP BY variant
207ORDER BY occurrences DESC
208```
209 
210Compare the variant split here to the project's overall exposure on the same
211flag in the same window. Disproportionate representation of one variant
212suggests the flag is involved in the cause — not a guarantee, but a strong
213hypothesis.
214 
215### Step 5 — Reconstruct what happened around the error
216 
217Use the `$session_id` from the sample event in step 2 to pull the activity
218surrounding the exception. Three sources stack on each other; run the ones
219that make sense for the SDK that captured the error.
220 
221#### 5a. Surrounding events (client SDKs by `$session_id`)
222 
223Mirrors the ET frontend session timeline. Pulls custom events, page views,
224and other exceptions captured under the same session within a ±1h window:
225 
226```sql
227posthog:execute-sql
228SELECT
229    uuid,
230    event,
231    timestamp,
232    properties.$lib AS lib,
233    properties.$current_url AS url
234FROM events
235WHERE $session_id = '<session_id_from_step_2>'
236    AND (event = '$exception' OR event = '$pageview' OR left(event, 1) != '$')
237    AND timestamp >= toDateTime('<error_timestamp>', 'UTC') - INTERVAL 1 HOUR
238    AND timestamp <= toDateTime('<error_timestamp>', 'UTC') + INTERVAL 1 HOUR
239ORDER BY timestamp ASC
240LIMIT 100
241```
242 
243The `left(event, 1) != '$'` clause drops PostHog autocapture / system events
244while keeping every custom event. The `OR event = '$pageview'`/`'$exception'`
245exceptions re-add the two system events worth seeing on the timeline. This is
246the same filter the ET UI uses.
247 
248Mixed `$lib` values in the output are a feature, not noise. When a server SDK
249propagates `$session_id` from the client request (PostHog's own backend does
250this), the timeline shows server-side activity inline with the browser side —
251"both SDKs when available" for free. Skim the lib column to see how each row
252was produced.
253 
254The skill defaults to a ±1h window because that's what the UI uses; widen it
255when an issue's actions are slow (long batch jobs, background workers) or
256tighten it when only the seconds right before the throw matter.
257 
258#### 5b. Console logs (web / React Native session replay)
259 
260When session replay is enabled, the replay pipeline emits `console.*` calls
261into the `log_entries` table tagged with the same session id. Pull them with
262the matching window:
263 
264```sql
265posthog:execute-sql
266SELECT timestamp, level, message
267FROM log_entries
268WHERE log_source = 'session_replay'
269    AND log_source_id = '<session_id_from_step_2>'
270    AND timestamp >= toDateTime('<error_timestamp>', 'UTC') - INTERVAL 1 HOUR
271    AND timestamp <= toDateTime('<error_timestamp>', 'UTC') + INTERVAL 1 HOUR
272ORDER BY timestamp ASC
273LIMIT 200
274```
275 
276`log_source = 'session_replay'` is the discriminator — `log_entries` is shared
277with other sources. Empty results are common: either replay isn't enabled, or
278this specific session wasn't recorded. Mention that in the synthesis rather
279than treating it as a failure.
280 
281#### 5c. Server logs around the error (OTEL via `query-logs`)
282 
283For server-side exceptions, correlate the exception timestamp with OTEL log
284entries the customer ingests. Many projects don't ingest logs at all — if
285`query-logs` returns nothing or errors, say so and move on. Discover available
286services first with `logs-attribute-values-list` when you don't know which
287service produced the error.
288 
289```json
290posthog:query-logs
291{
292  "query": {
293    "dateRange": {
294      "date_from": "<error_timestamp minus 5 minutes>",
295      "date_to":   "<error_timestamp plus 5 minutes>"
296    },
297    "severityLevels": ["error", "warn"],
298    "serviceNames": ["<service.name if known>"],
299    "limit": 50,
300    "orderBy": "earliest"
301  }
302}
303```
304 
305Caveats worth knowing before relying on this output:
306 
307- Logs are ingested separately from events and typically have shorter retention.
308  Old exceptions may return empty even though the issue is still active.
309- `trace_id` / `span_id` come back zero-padded (`"00000000..."`) when not set.
310  Trace-based correlation only works for explicitly instrumented requests, not
311  for every event.
312- `service.name` is a resource attribute. Narrow with `serviceNames` rather
313  than a free-text `searchTerm` when you know the producer.
314 
315#### 5d. Find a representative replay
316 
317Hand off to `finding-replay-for-issue` when picking the _best_ session matters —
318popular issues link hundreds of recordings, mostly short crash fragments or
319idle-tab sessions, and that skill applies the duration / active-time / recency
320ranking that finds the one most likely to show the cause. Hand off too when the
321user asks for "a replay" without specifying which.
322 
323Skip the hand-off and pull a recording inline via `query-session-recordings-list`
324with `session_ids` from the sample exception events you already fetched in step 2
325when only a handful of sessions are linked, the user already named a specific
326session, or any working example will do (e.g. proving the error reproduces).
327 
328If neither path returns a recording, mention that session replay may not be
329enabled for the affected users — useful context, not a failure.
330 
331### Step 6 — Synthesize
332 
333Present in this order:
334 
3351. **What it is** — type, message, where in the stack
3362. **Who it affects** — total users, sessions, and any segment breakdown that
337   stood out
3383. **When it started** — `first_seen`, plus the release / version that
339   introduced it if a breakdown found one
3404. **Likely cause** — one or two hypotheses backed by the breakdowns above
3415. **Next step** — a concrete action: investigate the suspected release, watch
342   the linked replay, ping the assignee, or escalate
343 
344Keep the synthesis tight. The user wants the answer, not a tour of the data.
345 
346## Tips
347 
348- The canonical join key from events to an issue is the resolved `issue_id`
349  virtual field, with `properties.$exception_issue_id` as fallback — see Step 3
350  for the reason and the `build_issue_where` pattern.
351- For a "what version introduced this?" breakdown, prefer `$app_version` (the
352  user's deployed app version, auto-captured on iOS / React Native and
353  manually set on web / server) or `$exception_releases` when populated. Avoid
354  `$lib_version` for this question — it's the PostHog SDK library version, not
355  the user's app. See the "Picking the right version property" subsection in
356  Step 3.
357- If the issue spans more than 30 days, widen the date range explicitly.
358  Defaults often truncate the original `first_seen` event off the breakdown.
359- Don't propose a fix in the synthesis unless the cause is obvious from the
360  sample stack. Hypotheses backed by data are more useful than confident
361  guesses.
362- If `query-error-tracking-issue` returns an `external_issues` array, the issue
363  is already linked to a Linear / Jira / GitHub ticket. Mention the link in the
364  synthesis so the user doesn't open a duplicate.
365