A collection of PostHog skills for enhancing AI-assisted workflows. Add this repo as a Claude Code plugin marketplace to get access to all PostHog skills: Then install individual plugins: Or browse available plugins: Copy any skill directory to .claude/skills/ in your project: Any directory under skills/ that contains a .claude-plugin/plugin.json is automatically discovered and added to the market
Add this skill
npx mdskills install PostHog/exploring-llm-evaluations@PostHog? Sign in with GitHub to claim this listing.Comprehensive guide to both deterministic and LLM-based evaluation workflows with clear tool usage patterns
1---2name: exploring-llm-evaluations3description: >4 Investigate AI observability evaluations of both types — `hog` (deterministic5 code-based) and `llm_judge` (LLM-prompt-based). Find existing evaluations,6 inspect their configuration, run them against specific generations, query7 individual pass/fail results, and generate AI-powered summaries of patterns8 across many runs. Use when the user asks to debug why an evaluation is9 failing, surface common failure modes, compare results across filters,10 dry-run a Hog evaluator, prototype a new LLM-judge prompt, or manage the11 evaluation lifecycle (create, update, enable/disable, delete).12---1314# Exploring LLM evaluations1516PostHog evaluations score `$ai_generation` events. Each evaluation is one of two types,17both first-class:1819- **`hog`** — deterministic Hog code that returns `true`/`false` (and optionally N/A).20 Best for objective rule-based checks: format validation (JSON parses, schema matches),21 length limits, keyword presence/absence, regex patterns, structural assertions, latency22 thresholds, cost guards. Cheap, fast, reproducible — no LLM call per run. Prefer this23 when the criterion can be expressed as code.24- **`llm_judge`** — an LLM scores generations against a prompt you write. Best for25 subjective or fuzzy checks: tone, helpfulness, hallucination detection, off-topic26 drift, instruction-following. Costs an LLM call per run and requires AI data27 processing approval at the org level.2829Results from both types land in ClickHouse as `$ai_evaluation` events with the same30schema, so the read/query/summary workflows are identical regardless of evaluator type —31the only thing that changes is whether `$ai_evaluation_reasoning` was written by Hog32code or by an LLM.3334This skill covers the full lifecycle: list/inspect/manage evaluation configs (Hog or35LLM judge), run them on specific generations, query individual results, and get an36AI-generated summary of pass/fail/N/A patterns across many runs.3738## Tools3940| Tool | Purpose |41| ---------------------------------------- | -------------------------------------------------------------- |42| `posthog:llma-evaluation-list` | List/search evaluation configs (filter by name, enabled flag) |43| `posthog:llma-evaluation-get` | Get a single evaluation config by UUID |44| `posthog:llma-evaluation-create` | Create a new `llm_judge` or `hog` evaluation |45| `posthog:llma-evaluation-update` | Update an existing evaluation (name, prompt, enabled, …) |46| `posthog:llma-evaluation-delete` | Soft-delete an evaluation |47| `posthog:llma-evaluation-run` | Run an evaluation against a specific `$ai_generation` event |48| `posthog:llma-evaluation-test-hog` | Dry-run Hog source against recent generations (no save) |49| `posthog:llma-evaluation-summary-create` | AI-powered summary of pass/fail/N/A patterns across runs |50| `posthog:execute-sql` | Ad-hoc HogQL over `$ai_evaluation` events |51| `posthog:query-llm-trace` | Drill into the underlying generation that an evaluation scored |5253All `llma-evaluation-*` tools are defined in `products/ai_observability/mcp/tools.yaml`.5455## Event schema5657Every run of an evaluation emits an `$ai_evaluation` event. Key properties:5859| Property | Meaning |60| --------------------------- | -------------------------------------------------------- |61| `$ai_evaluation_id` | UUID of the evaluation config |62| `$ai_evaluation_name` | Human-readable name |63| `$ai_target_event_id` | UUID of the `$ai_generation` event being scored |64| `$ai_trace_id` | Parent trace ID (for jumping to the trace UI) |65| `$ai_evaluation_result` | `true` = pass, `false` = fail |66| `$ai_evaluation_reasoning` | Free-text explanation (set by the LLM judge or Hog code) |67| `$ai_evaluation_applicable` | `false` when the evaluator decided the generation is N/A |6869When `$ai_evaluation_applicable = false`, the run counts as N/A regardless of `$ai_evaluation_result`.70For evaluations that don't support N/A, this property may be `null` — treat null as "applicable".7172## Workflow: investigate why an evaluation is failing7374Works the same way for `llm_judge` and `hog` evaluations — the differences only matter75when you eventually go to fix the evaluator (edit the prompt vs. edit the Hog source).7677### Step 1 — Find the evaluation7879```json80posthog:llma-evaluation-list81{ "search": "hallucination", "enabled": true }82```8384Look at the returned `id`, `name`, `evaluation_type`, and either:8586- `evaluation_config.prompt` for an `llm_judge`87- `evaluation_config.source` for a `hog` evaluator8889The Hog source is the ground truth for why a hog evaluator passes or fails — read it90before assuming the failure is in the generation.9192### Step 2 — Get the AI-generated summary9394```json95posthog:llma-evaluation-summary-create96{97 "evaluation_id": "<uuid>",98 "filter": "fail"99}100```101102Returns:103104- `overall_assessment` — natural-language summary105- `fail_patterns` — grouped patterns with `title`, `description`, `frequency`, and `example_generation_ids`106- `pass_patterns` and `na_patterns` — same shape, populated when `filter` includes them107- `recommendations` — actionable next steps108- `statistics` — `total_analyzed`, `pass_count`, `fail_count`, `na_count`109110The endpoint analyses the most recent ~250 runs (`EVALUATION_SUMMARY_MAX_RUNS`).111Results are cached for one hour per `(evaluation_id, filter, set_of_generation_ids)`.112Pass `force_refresh: true` to recompute.113114**Compare filters in two calls** to spot what's distinctive about failures vs passes:115116```json117posthog:llma-evaluation-summary-create118{ "evaluation_id": "<uuid>", "filter": "pass" }119```120121Then diff the `pass_patterns` against the `fail_patterns` from Step 2.122123### Step 3 — Drill into example failing runs124125Each pattern surfaces `example_generation_ids`. Pull the underlying trace for the most126representative example:127128```json129posthog:query-llm-trace130{ "traceId": "<trace_id>", "dateRange": {"date_from": "-30d"} }131```132133(If you only have a generation ID, query for it via `execute-sql` first to find the134parent trace ID — see below.)135136### Step 4 — Verify the pattern with raw SQL137138The summary is LLM-generated and should be verified. Use `execute-sql` to count and139spot-check:140141```sql142posthog:execute-sql143SELECT144 properties.$ai_target_event_id AS generation_id,145 properties.$ai_trace_id AS trace_id,146 properties.$ai_evaluation_reasoning AS reasoning,147 timestamp148FROM events149WHERE event = '$ai_evaluation'150 AND properties.$ai_evaluation_id = '<evaluation_uuid>'151 AND properties.$ai_evaluation_result = false152 AND (153 properties.$ai_evaluation_applicable IS NULL154 OR properties.$ai_evaluation_applicable != false155 )156 AND timestamp >= now() - INTERVAL 7 DAY157ORDER BY timestamp DESC158LIMIT 25159```160161The N/A guard (`IS NULL OR != false`) is important — it matches the same logic the162backend uses to bucket runs.163164## Workflow: run an evaluation against a specific generation165166Use this when the user pastes a trace/generation URL and asks "what would evaluation X167say about this?".168169```json170posthog:llma-evaluation-run171{172 "evaluationId": "<eval_uuid>",173 "target_event_id": "<generation_event_uuid>",174 "timestamp": "2026-04-01T19:39:20Z",175 "event": "$ai_generation"176}177```178179The `timestamp` is required for an efficient ClickHouse lookup of the target event.180Pass `distinct_id` if you have it — it speeds up the lookup further.181182## Workflow: build and test a new evaluator183184### Hog evaluator (deterministic, code-based)185186Reach for this first when the criterion is rule-based — it's cheaper, faster, and187reproducible. Prototype with `llma-evaluation-test-hog` (no save):188189```json190posthog:llma-evaluation-test-hog191{192 "source": "return event.properties.$ai_output_choices[1].content contains 'sorry';",193 "sample_count": 5,194 "allows_na": false195}196```197198The handler returns the boolean result for each of the most recent N `$ai_generation`199events. Iterate on the source until it behaves as expected, then promote it via200`llma-evaluation-create`:201202```json203posthog:llma-evaluation-create204{205 "name": "Output is valid JSON",206 "description": "Fails when the assistant message can't be parsed as JSON",207 "evaluation_type": "hog",208 "evaluation_config": {209 "source": "let raw := event.properties.$ai_output_choices[1].content; try { jsonParseStr(raw); return true; } catch { return false; }"210 },211 "output_type": "boolean",212 "enabled": true213}214```215216Hog evaluators have full access to the event and its properties — common patterns217include schema validation, length/token limits, regex matches, and tool-call shape218checks. Because they're deterministic, results are reproducible across reruns and219trivially diff-able.220221### LLM-judge evaluator (subjective, prompt-based)222223Use this when the criterion is fuzzy and a code rule would be brittle (tone, factuality,224helpfulness, on-topic-ness). There's no equivalent of `llma-evaluation-test-hog` for LLM225judges — the typical loop is to create the evaluator with `enabled: false`, run it226manually against a handful of representative generations via `llma-evaluation-run`, inspect227the results, refine the prompt with `llma-evaluation-update`, and then flip `enabled: true`228when you're satisfied:229230```json231posthog:llma-evaluation-create232{233 "name": "Response stays on-topic",234 "description": "LLM judge — fails if the assistant changes topic from the user's question",235 "evaluation_type": "llm_judge",236 "evaluation_config": {237 "prompt": "You are evaluating whether the assistant's reply stays on-topic relative to the user's most recent question. Return true if it does, false if the assistant changed the subject. Return N/A if the user did not actually ask a question."238 },239 "output_type": "boolean",240 "output_config": { "allows_na": true },241 "model_configuration": {242 "provider": "openai",243 "model": "gpt-5-mini"244 },245 "enabled": false246}247```248249Then dry-run against a known-good and a known-bad generation:250251```json252posthog:llma-evaluation-run253{254 "evaluationId": "<new_eval_uuid>",255 "target_event_id": "<generation_uuid>",256 "timestamp": "2026-04-01T19:39:20Z"257}258```259260LLM judges require organisation AI data processing approval. Hog evaluators do not.261262## Workflow: manage the evaluation lifecycle263264| Action | Tool |265| -------------------------- | --------------------------------------------------------------------------------------------------------------------- |266| Add a Hog evaluator | `llma-evaluation-create` with `evaluation_type: "hog"` and `evaluation_config.source` |267| Add an LLM-judge evaluator | `llma-evaluation-create` with `evaluation_type: "llm_judge"`, `evaluation_config.prompt`, and a `model_configuration` |268| Tweak the source or prompt | `llma-evaluation-update` (edits `evaluation_config.source` for Hog, `evaluation_config.prompt` for LLM judge) |269| Toggle N/A handling | `llma-evaluation-update` with `output_config.allows_na` |270| Disable temporarily | `llma-evaluation-update` with `enabled: false` |271| Remove | `llma-evaluation-delete` (soft-delete via PATCH `{deleted: true}`) |272273`llm_judge` evaluations require AI data processing approval at the org level274(`is_ai_data_processing_approved`). The same gate applies to275`llma-evaluation-summary-create`. Hog evaluations do **not** require this gate276— they run as plain code on the ingestion pipeline.277278## When to use Hog vs LLM judge279280Reach for **Hog** by default. Switch to LLM judge only when the criterion can't be281expressed as code.282283| Use Hog when… | Use LLM judge when… |284| ----------------------------------------------------- | ------------------------------------------------------- |285| The check is structural (JSON parses, schema matches) | The check is about meaning (on-topic, helpful, factual) |286| You need a deterministic, reproducible result | A small amount of judgement variability is acceptable |287| The criterion is cheap to compute | The criterion requires reading and understanding text |288| You can't get AI data processing approval | You have approval and the criterion is genuinely fuzzy |289| You need to enforce a hard limit (length, cost, etc.) | You need to rate a quality dimension |290| You want sub-millisecond evaluation | A few hundred milliseconds + LLM cost are acceptable |291292A common pattern is to **layer them**: a Hog evaluator gates obvious format/length293violations cheaply, and an LLM-judge evaluator only fires on the generations that pass294the Hog gate (via `conditions`).295296## Investigation patterns297298The summarisation tool works the same way regardless of whether the evaluator is `hog`299or `llm_judge` — it analyses the resulting `$ai_evaluation` events, not the evaluator300itself. The fix path differs (edit Hog source vs. edit prompt) but the diagnosis is301identical.302303### "Why is evaluation X suddenly failing more?"3043051. `llma-evaluation-list` — confirm the evaluation is still enabled and unchanged306 (compare `evaluation_config.source` or `evaluation_config.prompt` to the version you307 expect)3082. `llma-evaluation-summary-create` with `filter: "fail"` — get the dominant309 failure patterns and example IDs3103. SQL count of fails per day to confirm the regression window:311312 ```sql313 SELECT toDate(timestamp) AS day, count() AS fails314 FROM events315 WHERE event = '$ai_evaluation'316 AND properties.$ai_evaluation_id = '<uuid>'317 AND properties.$ai_evaluation_result = false318 AND timestamp >= now() - INTERVAL 30 DAY319 GROUP BY day320 ORDER BY day321 ```3223234. Drill into a representative trace per pattern via `query-llm-trace`324325### "Are passes and fails caused by the same root content?"3263271. Generate two summaries: one with `filter: "pass"`, one with `filter: "fail"`3282. If `pass_patterns` and `fail_patterns` describe similar content:329 - For an `llm_judge`: the prompt or rubric is probably ambiguous — reword330 `evaluation_config.prompt` and use `llma-evaluation-update`331 - For a `hog` evaluator: the rule is probably under- or over-matching — read the332 source via `llma-evaluation-get`, narrow the predicate, and retest with333 `llma-evaluation-test-hog` before pushing the fix via `llma-evaluation-update`334335### "Did a Hog evaluator regression after a code change?"336337Hog evaluators are reproducible — if the source hasn't changed, identical inputs should338yield identical outputs. When fail rates jump for a Hog evaluator:3393401. `llma-evaluation-get` — note the current source and `updated_at`3412. Spot-check the latest failing runs with the SQL query from Step 4 above3423. Re-run the source against those exact generations using `llma-evaluation-test-hog` with a343 modified `conditions` filter that targets them3444. If the test results match the live results, the change is in the _generations_, not345 the evaluator (a model upgrade, prompt change upstream, etc.) — investigate the346 producer3475. If they diverge, the evaluator was edited; check git history of the source field via348 the activity log349350### "What kinds of generations does this evaluator skip as N/A?"351352```json353posthog:llma-evaluation-summary-create354{ "evaluation_id": "<uuid>", "filter": "na" }355```356357Inspect `na_patterns` to see whether the N/A logic is doing the right thing. If a358pattern in `na_patterns` looks like something that should have been scored:359360- For an `llm_judge`: the applicability instruction in the prompt is too broad — narrow361 it362- For a `hog` evaluator with `output_config.allows_na: true`: the source is returning363 `null` (or whatever the N/A signal is) too eagerly — tighten the precondition364365### "Score this single generation right now"366367`llma-evaluation-run` with the trace's generation ID and timestamp. Useful for spot-checking368or wiring evaluations into a larger agent loop.369370## Constructing UI links371372- **Evaluations list**: `https://app.posthog.com/ai-evals/evaluations`373- **Single evaluation**: `https://app.posthog.com/ai-evals/evaluations/<evaluation_id>`374- **Underlying generation/trace**: see the `exploring-llm-traces` skill's URL conventions375376Always surface the relevant link so the user can verify in the UI.377378## Tips379380- The summary tool is **rate-limited** (burst, sustained, daily) and **caches results381 for one hour** — repeated calls with the same `(evaluation_id, filter)` are cheap; use382 `force_refresh: true` only when you genuinely need fresh analysis383- Pass `generation_ids: [...]` to scope a summary to a specific cohort of runs (max 250)384- The `statistics` block in the summary response is computed from raw data, not the LLM385 — trust those counts even if a pattern's `frequency` field is qualitative386- For rich filtering not supported by `llma-evaluation-list` (e.g. by author or model387 configuration), fall back to `execute-sql` against the `evaluations` Postgres table or388 the `$ai_evaluation` ClickHouse events389- When showing failure patterns to the user, always include 1-2 example trace links so390 they can validate the pattern visually391- `llma-evaluation-*` tools use `evaluation:read` for read tools and `evaluation:write` for392 mutating tools; `llma-evaluation-summary-create` uses `llm_analytics:write`393- Hog evaluators are reproducible — if you suspect a regression, `llma-evaluation-test-hog`394 with the suspect source against the failing generations is the fastest way to bisect395 whether the change is in the evaluator or in the producer of the generations396- LLM-judge evaluators are non-deterministic across reruns; expect 1-5% noise even with397 a fixed prompt and model. If you're chasing a small regression in fail rate, prefer398 Hog or pin a deterministic provider/seed in the `model_configuration`399
Full transparency — inspect the skill content before installing.