A collection of PostHog skills for enhancing AI-assisted workflows. Add this repo as a Claude Code plugin marketplace to get access to all PostHog skills: Then install individual plugins: Or browse available plugins: Copy any skill directory to .claude/skills/ in your project: Any directory under skills/ that contains a .claude-plugin/plugin.json is automatically discovered and added to the market
Add this skill
npx mdskills install PostHog/exploring-llm-traces@PostHog? Sign in with GitHub to claim this listing.Comprehensive trace debugging workflow with clear tool usage, parsing scripts, and investigation patterns
1---2name: exploring-llm-traces3description: >4 ABSOLUTE MUST to debug and inspect LLM/AI agent traces using PostHog's MCP tools.5 Use when the user pastes a trace or session URL (e.g. /ai-observability/traces/<id> or /ai-observability/sessions/<id>),6 asks to debug a trace, figure out what went wrong, check if an agent used a tool correctly,7 verify context/files were surfaced, inspect subagent behavior, investigate LLM decisions,8 or analyze token usage and costs. Also use when raw SQL/HogQL against9 `events.properties.$ai_input` / `$ai_output_choices` returns empty — message content lives only10 on the dedicated `posthog.ai_events` table.11---1213# Exploring LLM traces with MCP tools1415PostHog captures LLM/AI agent activity as traces. Each trace is a tree of events representing16a single AI interaction — from the top-level agent invocation down to individual LLM API calls.1718## Available tools1920| Tool | Purpose |21| ------------------------------- | ------------------------------------------------------------- |22| `posthog:query-llm-traces-list` | Search and list traces; can return large multi-trace payloads |23| `posthog:query-llm-trace` | Get a single trace by ID with full event tree |24| `posthog:read-data-schema` | Discover custom event/person properties before filtering |25| `posthog:execute-sql` | Ad-hoc SQL for complex trace analysis |2627## Event hierarchy2829See the [event reference](./references/events-and-properties.md) for the full schema.3031```text32$ai_trace (top-level container)33 └── $ai_span (logical groupings, e.g. "RAG retrieval", "tool execution")34 ├── $ai_generation (individual LLM API call)35 └── $ai_embedding (embedding creation)36```3738Events are linked via `$ai_parent_id` → parent's `$ai_span_id` or `$ai_trace_id`.3940## Workflow: debug a trace or session from a URL4142### Step 1 — Classify the URL4344First inspect the path. Do not treat every UUID-looking value as a trace ID.4546- `/ai-observability/traces/<trace_id>` or legacy `/llm-analytics/traces/<trace_id>` / `/llm-observability/traces/<trace_id>` is a single trace. Fetch it with `posthog:query-llm-trace`.47- `/ai-observability/sessions/<session_id>` or legacy `/llm-analytics/sessions/<session_id>` is an AI session, not a trace. Fetch traces with `posthog:query-llm-traces-list` filtered by event property `$ai_session_id`.4849Preserve `date_from` / `date_to` query parameters from the URL when present.50If none are present but the URL has a `timestamp` query parameter, use that timestamp as the anchor and query an absolute window around it, for example `timestamp - 36h` to `timestamp + 36h`.51This handles exact session links whose UI timestamp may be offset from the stored event timestamps while keeping the query bounded.52If the URL has neither explicit dates nor `timestamp`, use a safe default like `{"date_from": "-7d"}`.5354For exact trace and session URLs, skip schema discovery for the standard `$ai_*` fields used below. These are AI observability built-ins, not project-specific custom properties.5556### Step 2 — Fetch trace data5758For a trace URL, call `posthog:query-llm-trace` with:5960```json61{62 "traceId": "<trace_id>",63 "dateRange": { "date_from": "-7d" }64}65```6667For a session URL, call `posthog:query-llm-traces-list` with:6869```json70{71 "dateRange": { "date_from": "<timestamp_minus_36h>", "date_to": "<timestamp_plus_36h>" },72 "filterTestAccounts": false,73 "limit": 20,74 "properties": [{ "type": "event", "key": "$ai_session_id", "value": ["<session_id>"], "operator": "exact" }]75}76```7778Use the URL's `date_from` / `date_to` values in the session query if present.79If the URL only has `timestamp`, calculate the absolute date range from that timestamp instead of using a relative range like `-1h`.80Set `filterTestAccounts: false` for an exact URL so the requested trace is not hidden by account filters.8182The result contains the event tree with all properties.83The response may be large — when it exceeds the inline limit, Claude Code auto-persists it to a file.8485From the result you get:8687- Every event with its type (`$ai_span`, `$ai_generation`, etc.)88- Span names (`$ai_span_name`) — these are the tool/step names89- Latency, error flags, models used90- Parent-child relationships via `$ai_parent_id`91- `_posthogUrl` — **always include this in your response** so the user can click through to the UI9293### Step 3 — Parse large results with scripts9495When the result is persisted to a file (large traces with full `$ai_input`/`$ai_output_choices`),96use the [parsing scripts](./scripts/) to explore it.9798**Start with the summary** to get the full picture, then drill into specifics:99100```bash101# 1. Overview: metadata, tool calls, final output, errors102python3 scripts/print_summary.py /path/to/persisted-file.json103104# 2. Timeline: chronological event list with truncated I/O105python3 scripts/print_timeline.py /path/to/persisted-file.json106107# 3. Drill into a specific span's full input/output108SPAN="tool_name" python3 scripts/extract_span.py /path/to/persisted-file.json109110# 4. Full conversation with thinking blocks and tool calls111python3 scripts/extract_conversation.py /path/to/persisted-file.json112113# 5. Search for a keyword across all properties114SEARCH="keyword" python3 scripts/search_traces.py /path/to/persisted-file.json115```116117All scripts support `MAX_LEN=N` env var to control truncation (0 = unlimited).118119## Investigation patterns120121### "Did the agent use the tool correctly?"1221231. Find the `$ai_span` for the tool call (look at `$ai_span_name`)1242. Check `$ai_input_state` — what arguments were passed to the tool?1253. Check `$ai_output_state` — what did the tool return?1264. Check `$ai_is_error` — did the tool call fail?127128### "Was the context correct?" / "Were the right files surfaced?"1291301. Find the `$ai_generation` event where the LLM made the decision1312. Check `$ai_input` — this is the full message history the LLM saw1323. Look at preceding `$ai_span` events for retrieval/search steps1334. Check their `$ai_output_state` — what content was retrieved and fed to the LLM?134135### "Did the subagent work?"1361371. In the structural overview, find spans that are children of other spans (via `$ai_parent_id`)1382. The parent span is the orchestrator; child spans are subagent steps1393. Check each child's `$ai_output_state` and `$ai_is_error`1404. If a child span contains `$ai_generation` events, those are the subagent's LLM calls141142### "Why did the LLM say X?"1431441. Use `search_traces.py` to find where the text appears: `SEARCH="the text" python3 scripts/search_traces.py FILE`1452. This shows which event and property path contains it1463. Check the `$ai_input` of that generation to see what the LLM was told before it said X147148## Constructing UI links149150The trace tools return `_posthogUrl` — always surface this to the user.151152You can also construct links manually:153154- **Trace detail**: `https://app.posthog.com/ai-observability/traces/<trace_id>?timestamp=<url_encoded_timestamp>&event=<optional_event_id>`155- **Traces list with filters**: returned in `_posthogUrl` from `query-llm-traces-list`156157The `timestamp` query param is **required** — use the `createdAt` of the earliest event in the trace, URL-encoded (e.g. `timestamp=2026-04-01T19%3A39%3A20Z`).158159When presenting findings, always include the relevant PostHog URL so the user can verify.160161## Finding traces162163Use `posthog:query-llm-traces-list` to search and filter traces.164165**CRITICAL: Never assume event names, property names, or property values from training data.**166Every project instruments different custom properties. For open-ended searches and custom filters, call167`posthog:read-data-schema` first to discover what properties and values actually exist in the project's168data before constructing filters.169170The exception is exact AI observability trace/session URLs: use the built-in `$ai_trace_id` / `$ai_session_id`171fields directly and skip schema discovery.172173### Discovering the schema first174175Before filtering traces, discover what's available:1761771. **Confirm AI events exist** — call `posthog:read-data-schema` with `kind: "events"` and look for `$ai_*` events1782. **Find filterable properties** — call `posthog:read-data-schema` with `kind: "event_properties"` and `event_name: "$ai_generation"` (or another AI event) to see what properties are captured1793. **Get actual values** — call `posthog:read-data-schema` with `kind: "event_property_values"`, `event_name: "$ai_generation"`, and `property_name: "$ai_model"` to see real model names in use180181Only then construct the `query-llm-traces-list` call with property filters.182183This is especially important for custom properties like `project_id`, `conversation_id`, `user_tier`, etc. — these vary per project and cannot be guessed.184185Do not confirm `$ai_*` properties, but confirm any other like `email` of a person.186187### By filters188189```json190posthog:query-llm-traces-list191{192 "dateRange": {"date_from": "-1h"},193 "filterTestAccounts": true,194 "limit": 20,195 "properties": [196 {"type": "event", "key": "$ai_model", "value": "gpt-4o", "operator": "exact"}197 ]198}199```200201Multiple filters are AND-ed together:202203```json204posthog:query-llm-traces-list205{206 "dateRange": {"date_from": "-1h"},207 "filterTestAccounts": true,208 "properties": [209 {"type": "event", "key": "$ai_provider", "value": "anthropic", "operator": "exact"},210 {"type": "event", "key": "$ai_is_error", "value": ["true"], "operator": "exact"}211 ]212}213```214215You can also filter by person properties (discover them via `read-data-schema` with `kind: "entity_properties"` and `entity: "person"`):216217```json218posthog:query-llm-traces-list219{220 "dateRange": {"date_from": "-1h"},221 "filterTestAccounts": true,222 "properties": [223 {"type": "person", "key": "email", "value": "@company.com", "operator": "icontains"}224 ]225}226```227228### By external identifiers229230Customers often store their own IDs as event or person properties.231Use `posthog:read-data-schema` to discover what custom properties exist, then filter:2322331. Call `posthog:read-data-schema` with `kind: "event_properties"` and `event_name: "$ai_trace"` to find custom properties2342. Review the returned properties and their sample values2353. Construct the filter using the discovered property key and a known value236237```json238posthog:query-llm-traces-list239{240 "dateRange": {"date_from": "-7d"},241 "properties": [242 {"type": "event", "key": "project_id", "value": "proj_abc123", "operator": "exact"}243 ]244}245```246247For more complex SQL patterns, read these references:248249- [Single trace retrieval](./references/example-llm-trace.md.j2) — fetches a single trace by ID with all events and properties (renders the `TraceQuery` HogQL)250- [Traces list with aggregated metrics](./references/example-llm-traces-list.md) — two-phase query: find trace IDs first, then fetch aggregated latency, tokens, costs, and error counts251252## Parsing large trace results253254Trace tool results are JSON. When too large to read inline, Claude Code persists them to a file.255256### Persisted file format257258```json259[{ "type": "text", "text": "{\"results\": [...], \"_posthogUrl\": \"...\"}" }]260```261262### Trace JSON structure263264```text265results (array for list, object for single trace)266 ├── id, traceName, createdAt, totalLatency, totalCost267 ├── inputState, outputState (trace-level state)268 └── events[]269 ├── event ($ai_span | $ai_generation | $ai_embedding | $ai_metric | $ai_feedback)270 ├── id, createdAt271 └── properties272 ├── $ai_span_name, $ai_latency, $ai_is_error273 ├── $ai_input_state, $ai_output_state (span tool I/O)274 ├── $ai_input, $ai_output_choices (generation messages)275 ├── $ai_model, $ai_provider276 └── $ai_input_tokens, $ai_output_tokens, $ai_total_cost_usd277```278279### Available scripts280281| Script | Purpose | Usage |282| -------------------------------------------------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------- |283| [`print_summary.py`](./scripts/print_summary.py) | Aggregate list/session totals, trace metadata, tool calls, errors, and final LLM output | `python3 scripts/print_summary.py FILE` |284| [`print_timeline.py`](./scripts/print_timeline.py) | Chronological event timeline with I/O summaries | `python3 scripts/print_timeline.py FILE` |285| [`extract_span.py`](./scripts/extract_span.py) | Full input/output of a specific span by name | `SPAN="name" python3 scripts/extract_span.py FILE` |286| [`extract_conversation.py`](./scripts/extract_conversation.py) | LLM messages with thinking blocks and tool calls | `python3 scripts/extract_conversation.py FILE` |287| [`search_traces.py`](./scripts/search_traces.py) | Find a keyword across all event properties | `SEARCH="keyword" python3 scripts/search_traces.py FILE` |288| [`show_structure.py`](./scripts/show_structure.py) | Show JSON keys and types without values | `cat blob.json \| python3 scripts/show_structure.py` |289290## Tips291292- Always set `dateRange` — queries without a time range are slow. Use narrow windows (`-30m`, `-1h`) for broad listing queries; wider windows (`-7d`, `-30d`) are fine for narrow queries filtered by trace ID or specific property values293- Always include the `_posthogUrl` in your response so the user can click through294- `$ai_input_state` / `$ai_output_state` on spans contain tool call inputs and outputs295- `$ai_input` / `$ai_output_choices` on generations contain the full LLM conversation — can be megabytes; when the result is persisted to a file, use the parsing scripts296- In raw SQL, heavy content (`$ai_input` / `$ai_output` / `$ai_output_choices` / `$ai_input_state` / `$ai_output_state` / `$ai_tools`) lives only on the `posthog.ai_events` table, not `events.properties` — see the [event reference](./references/events-and-properties.md) for the column mapping and trace-id-anchored query patterns297- Use `filterTestAccounts: true` to exclude internal/test traffic when searching298- `$ai_trace` events are NOT in the `events` array — their data is surfaced via trace-level `inputState`, `outputState`, and `traceName`299
Full transparency — inspect the skill content before installing.