Exploring Apm Traces

1---
2name: exploring-apm-traces
3description: >
4  Investigates distributed application performance using PostHog APM (OpenTelemetry span) data via MCP.
5  Use when the user asks about service traces, slow HTTP/database spans, error spans, trace IDs, or span
6  attributes — not AI observability traces or product logs. Uses posthog:query-apm-spans, posthog:apm-trace-get,
7  posthog:apm-services-list, posthog:apm-attributes-list, and posthog:apm-attribute-values-list.
8---
9 
10# Exploring APM traces (OpenTelemetry spans)
11 
12PostHog captures distributed traces from OpenTelemetry. Each trace is a tree of spans representing a request's path through services.
13 
14**Disambiguation:** This skill is for **APM / OpenTelemetry traces**. Do not confuse with **AI observability traces** (agent/model `$ai_*` events) or **logs** (`posthog:query-logs`, `posthog:logs-*`).
15 
16## Available tools
17 
18| Tool                                | Purpose                                           |
19| ----------------------------------- | ------------------------------------------------- |
20| `posthog:query-apm-spans`           | Search and filter spans (compact list view)       |
21| `posthog:apm-trace-get`             | Get the full span list for one hex `trace_id`     |
22| `posthog:apm-spans-aggregate`       | Per-operation aggregates (count, p50/p95, errors) |
23| `posthog:apm-spans-tree`            | Call-tree aggregates per `(parent, child)` edge   |
24| `posthog:apm-services-list`         | List distinct service names                       |
25| `posthog:apm-attributes-list`       | List span or resource attribute keys              |
26| `posthog:apm-attribute-values-list` | List values for a specific attribute key          |
27 
28See [references/spans-and-fields.md](./references/spans-and-fields.md) for the response schema and the `kind`/`status_code` enums.
29 
30## Workflow: debug a trace from a URL
31 
32### Step 1 — Fetch the trace
33 
34```json
35posthog:apm-trace-get
36{
37  "trace_id": "<hex_trace_id>"
38}
39```
40 
41The response is `{ results: [span, span, …] }` — a flat list of every span in the trace.
42The list can be very large for fan-out request flows; when it exceeds the inline limit, Claude Code auto-persists it to a file.
43 
44From the result you get:
45 
46- Every span with `name`, `service_name`, `kind`, `status_code`, `parent_span_id`, `duration_nano`, `is_root_span`
47- The `_posthogUrl` — **always include this in your response** so the user can click through to the UI
48 
49### Step 2 — Parse large results with scripts
50 
51When the result is persisted to a file (traces with hundreds of spans across services), use the [parsing scripts](./scripts/) to explore it.
52 
53**Start with the summary** to get the full picture, then drill into specifics:
54 
55```bash
56# 1. Overview: services, span count, slowest spans, errors
57python3 scripts/print_summary.py /path/to/persisted-file.json
58 
59# 2. Indented chronological tree (DFS by parent_span_id)
60python3 scripts/print_timeline.py /path/to/persisted-file.json
61 
62# 3. Drill into a specific span by name
63SPAN="HTTP GET /api/users" python3 scripts/extract_span.py /path/to/persisted-file.json
64 
65# 4. Search for a keyword across span names, services, IDs
66SEARCH="keyword" python3 scripts/search_spans.py /path/to/persisted-file.json
67 
68# 5. When the JSON shape looks unfamiliar
69python3 scripts/show_structure.py /path/to/persisted-file.json
70```
71 
72All scripts support `MAX_LEN=N` env var to control truncation (`0` = unlimited).
73 
74## Tree reconstruction (parent_span_id → span_id)
75 
76The flat span list is a tree. Each span carries:
77 
78- `trace_id` — same on every span in the trace
79- `span_id` — this span's unique hex ID
80- `parent_span_id` — points to the parent's `span_id` (zero-padded hex `000…000` for the root)
81- `is_root_span` — convenience flag for the trace entry
82 
83To rebuild the tree:
84 
851. Spans where `is_root_span` is true (or `parent_span_id == "00000000…"`) are **root spans**.
862. Every other span is a child of the span whose `span_id` matches its `parent_span_id`.
873. Group by `parent_span_id`, walk from each root downward.
88 
89`scripts/print_timeline.py` does this for you and prints a DFS-indented tree.
90 
91## Investigation patterns
92 
93### "Where is time going?"
94 
951. Run `print_summary.py` — it surfaces the top-5 slowest spans by `duration_nano`.
962. For a noisy trace, run `print_timeline.py` and scan the indented durations — you can see whether time is dominated by one child span or fan-out across many.
973. To dig into one slow span, `SPAN="<name>" python3 scripts/extract_span.py FILE`.
98 
99### "Where did the error happen?"
100 
1011. `print_summary.py` lists every span with `status_code == 2` (Error). Each entry shows service, span name, and parent context.
1022. Walk up the tree from an error span via `parent_span_id` to see what request path led there.
1033. Error detail lives in each span's `attributes` map (e.g. `exception.message`, `exception.type`), which **is** returned in the trace payload — read it directly off the error span. `apm-attribute-values-list` is for discovering values across spans, not a prerequisite for reading one span's attributes.
104 
105### "Did the request hit service X?"
106 
1071. Run `print_summary.py` — it prints the set of services involved in the trace.
1082. If service X is missing, the request never reached it (or instrumentation is missing — check `apm-services-list` to confirm X has emitted spans recently at all).
109 
110### "Did the fan-out look right?"
111 
1121. `print_timeline.py` shows the indentation — wide trees mean parallel calls, deep trees mean sequential dependencies.
1132. Look for spans of kind `Client` (3) followed by matching `Server` (2) spans on the called service — that's a synchronous downstream call.
114 
115### Searching by attribute (e.g. `http.method=POST`)
116 
117Each span carries an `attributes` map (span-level OTel attributes like `http.method`, `db.statement`) **in the payload** — so for a span you already have, just read it. **Resource** attributes (k8s labels, `service.version`) are not in the payload. To filter the whole dataset by an attribute:
118 
1191. Use `apm-attributes-list` / `apm-attribute-values-list` to discover keys and values (resource attributes especially).
1202. Re-issue `query-apm-spans` with a `filterGroup` entry of type `span_attribute` or `span_resource_attribute`.
121 
122## Constructing UI links
123 
124`apm-trace-get` and `query-apm-spans` return `_posthogUrl` — **always surface this to the user** so they can verify in the PostHog UI.
125 
126When presenting findings, include the relevant PostHog URL.
127 
128## Finding traces
129 
130Use `posthog:query-apm-spans` to search and filter spans. Note this returns spans, not a tree — pass `query.traceId` or grab a `trace_id` from the results and feed it to `apm-trace-get` for the tree.
131 
132### Discover before filtering
133 
134Before constructing filters, discover what's actually in the project:
135 
1361. **Confirm services exist** — call `apm-services-list` to see which services have emitted spans.
1372. **Find filterable attributes** — call `apm-attributes-list` with `attribute_type: "span"` or `"resource"`.
1383. **Get actual values** — call `apm-attribute-values-list` with a key to see the real values in use.
139 
140Only then construct `query-apm-spans` filters. Custom attributes vary per project and cannot be guessed.
141 
142### By filters
143 
144```json
145posthog:query-apm-spans
146{
147  "query": {
148    "serviceNames": ["api-gateway"],
149    "dateRange": {"date_from": "-1h"},
150    "filterGroup": [
151      {"key": "http.status_code", "operator": "gt", "type": "span_attribute", "value": "499"}
152    ]
153  }
154}
155```
156 
157### By trace ID (when known)
158 
159```json
160posthog:apm-trace-get
161{
162  "trace_id": "0123456789abcdef0123456789abcdef"
163}
164```
165 
166### Common gotchas
167 
168- **Durations are nanoseconds.** 1 second = `1_000_000_000`. Filter values in `query-apm-spans` for `duration` are also nanoseconds.
169- **`status_code == 2` is Error.** `0` is Unset, `1` is OK. Use `OK` to match `{0, 1}` in the UI filter.
170- **`kind`** is an integer 0–5: 0 Unspecified, 1 Internal, 2 Server, 3 Client, 4 Producer, 5 Consumer.
171- **`parent_span_id` of a root span** is `"0000000000000000"` (16 zero hex chars, matching the 8-byte span ID width — _not_ the 16-byte trace ID width), not null.
172 
173## Parsing large trace results
174 
175Trace tool results are JSON. When too large to read inline, Claude Code persists them to a file.
176 
177### Persisted file format
178 
179```json
180[{ "type": "text", "text": "{\"results\": [...], \"_posthogUrl\": \"...\"}" }]
181```
182 
183Every script in `scripts/` unwraps this envelope before parsing.
184 
185### Trace JSON structure
186 
187```text
188results (array of span dicts)
189  └── each span:
190        ├── uuid, trace_id, span_id, parent_span_id (hex strings)
191        ├── name, kind (int 0–5), service_name
192        ├── status_code (int 0–2), is_root_span (bool)
193        ├── timestamp, end_time (ISO 8601)
194        ├── duration_nano (int, nanoseconds)
195        ├── attributes (map of span-level OTel attributes, e.g. db.statement, http.url)
196        └── matched_filter (0/1 — 1 if this span matched the query-apm-spans filter, 0 if it
197            only shares a trace with a match; always present, only meaningful from query-apm-spans)
198```
199 
200### Available scripts
201 
202| Script                                             | Purpose                                              | Usage                                              |
203| -------------------------------------------------- | ---------------------------------------------------- | -------------------------------------------------- |
204| [`print_summary.py`](./scripts/print_summary.py)   | Trace metadata, services, slowest spans, errors      | `python3 scripts/print_summary.py FILE`            |
205| [`print_timeline.py`](./scripts/print_timeline.py) | DFS-indented tree from `parent_span_id` walk         | `python3 scripts/print_timeline.py FILE`           |
206| [`extract_span.py`](./scripts/extract_span.py)     | Full row + parent/children for spans matching a name | `SPAN="name" python3 scripts/extract_span.py FILE` |
207| [`search_spans.py`](./scripts/search_spans.py)     | Find a keyword across name, service_name, IDs        | `SEARCH="kw" python3 scripts/search_spans.py FILE` |
208| [`show_structure.py`](./scripts/show_structure.py) | Show JSON keys and types without values              | `python3 scripts/show_structure.py FILE`           |
209 
210## Tips
211 
212- Always set `dateRange` on `query-apm-spans` — queries without a time range are slow. Default is `-1h`; widen only when needed.
213- Always include the `_posthogUrl` in your response so the user can click through.
214- Span-level attributes **are** in the `apm-trace-get` / `query-apm-spans` payload (each span's `attributes` map). Resource attributes are not — use `apm-attributes-list` (type `resource`) and `apm-attribute-values-list` for those.
215- `is_root_span` is the cheap way to find the trace entry — don't string-match `00000000…`.
216- For aggregates (p95 by operation, slowest children of a span), use `apm-spans-aggregate` for a flat view or `apm-spans-tree` for parent→child edges — don't reach for SQL.
217