Feature Usage Feed

1---
2name: feature-usage-feed
3description: >
4  Set up an LLM-judge evaluation that extracts canonical use cases for a
5  PostHog feature at scale and streams the results to a Slack channel as a
6  live feed. Use when someone wants to understand how users are actually
7  using a specific AI/LLM-powered feature in production — what they're
8  investigating, what questions they're trying to answer, and what
9  patterns surface — without manually reading hundreds of traces. Assumes
10  the feature emits `$ai_generation` and `$ai_evaluation` events with
11  `$session_id` linkage to the trigger user's recording (the standard
12  setup post the session-summary linkage PRs).
13---
14 
15# Building a feature usage feed via LLM evals
16 
17Some PostHog features (group session summaries, single session summaries, replay AI search, error tracking AI debug, etc.) generate hundreds or thousands of LLM traces per week. Reading them by hand is not feasible. This skill covers the end-to-end pattern for turning that trace volume into a live Slack feed of canonical use cases — what users are actually doing with the feature.
18 
19The workflow is **mixed, and leans UI**. Trace inspection and filter discovery (steps 1-2) are MCP-driven. Eval creation, dry-running, and enabling (steps 4-5) are MCP-driven _when_ `posthog:llma-evaluation-*` tools are exposed to your agent — but they often aren't, in which case fall back to the UI (Data pipeline → destinations for the alert is always UI). Each step flags its UI fallback. Expect to finish in the UI even when you start from chat.
20 
21## When to use
22 
23- "How are people actually using [feature X] in production?"
24- "Can we identify the canonical use cases for [feature X] so we can write better docs / prioritize improvements?"
25- "I want a Slack feed of representative usage examples without manually skimming traces."
26- "Set up a feed of use cases for [feature X] in #team-[area]-usage."
27 
28If the user just wants to debug a single trace or tune an existing eval, redirect to `exploring-llm-traces` or `exploring-llm-evaluations` instead.
29 
30## Two filter patterns
31 
32This skill supports two different ways to scope an eval to "the feature you care about":
33 
34**Pattern A — Feature-native trace_id prefix.** For standalone features that emit their own `$ai_trace_id` pattern (e.g. `session-summary:group:`, `replay-search:`, error-tracking-specific flows). Filter on the prefix.
35 
36**Pattern B — PostHog AI agent mode.** For features the user interacts with _via_ PostHog AI in a specific agent mode (error tracking, product analytics, session replay, SQL, flags, surveys, AI observability). Filter on `ai_product = 'posthog_ai' AND agent_mode = '<mode>'`. This requires PR #55160 (merged April 2026) to be deployed, which threads `agent_mode` and `supermode` onto every `$ai_generation` emitted by the chat agent loop. A useful ergonomic side-effect: `agent_mode IS NOT NULL` is a reliable "user-facing chat turn" filter — batch jobs and tool-internal LLM calls go through different code paths and have `agent_mode=null`, so they're excluded for free.
37 
38If the user asks "what are users trying to DO in [ET / replay / SQL / flags / surveys] mode of PostHog AI", that's Pattern B. If they ask "what use cases does [standalone feature] cover", that's Pattern A. Pick the pattern first — the prompt, filter, and Slack channel naming all follow from it.
39 
40## Prerequisites
41 
42| Requirement                                                                                              | How to verify                                                                                                                                                                                                             |
43| -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
44| (Pattern A) Feature emits `$ai_generation` events with a stable `$ai_trace_id` pattern                   | `posthog:execute-sql` for distinct `$ai_trace_id` prefixes                                                                                                                                                                |
45| (Pattern B) `agent_mode` property is present on recent `$ai_generation` events                           | `posthog:execute-sql` group-by `properties.agent_mode` on recent `ai_product='posthog_ai'` events. Null bucket is normal (batch jobs + tool-internal calls) — you want non-null coverage across the modes you care about. |
46| `$session_id` is attached to the `$ai_generation` events (links trace to trigger session)                | `posthog:execute-sql` for `countIf($session_id IS NOT NULL) / count()`                                                                                                                                                    |
47| `$session_id` is also attached to the `$ai_evaluation` events (lets the Slack alert link to the session) | Same query but on `$ai_evaluation` events after the eval has run once                                                                                                                                                     |
48| User has organisation-level AI data processing approval                                                  | Required for `llm_judge` evaluations and the eval summary tool                                                                                                                                                            |
49 
50If `$session_id` is missing on either event type, file a backend fix before continuing — there is no UI workaround. The session-summary feature has a worked example of the threading pattern in PR #54952. For Pattern B, the agent-mode threading pattern is in PR #55160.
51 
52## Tools
53 
54| Tool                                               | Purpose                                                                                                                                                                                                                                                           |
55| -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
56| `posthog:query-llm-traces-list`                    | Find sample traces matching the feature's `$ai_trace_id` pattern                                                                                                                                                                                                  |
57| `posthog:query-llm-trace`                          | Inspect a specific trace's contents end-to-end                                                                                                                                                                                                                    |
58| `posthog:execute-sql`                              | Verify trace volume, session_id coverage, eval result distributions                                                                                                                                                                                               |
59| `posthog:llma-evaluation-create`                   | (**often unexposed** — UI fallback: AI observability → Evaluations → New) Create the LLM-judge eval (disabled at first)                                                                                                                                           |
60| `posthog:llma-evaluation-run`                      | (**often unexposed** — UI fallback: the eval's detail page has a "Run on event" button) Dry-run the eval against specific generations during prompt iteration                                                                                                     |
61| `posthog:llma-evaluation-update`                   | (**often unexposed** — UI fallback: edit the eval in AI observability → Evaluations) Tweak the prompt / enable when ready                                                                                                                                         |
62| `posthog:llma-evaluation-summary-create`           | (**often unexposed** — UI fallback: the eval detail page has a "Summarize results" button) After the feed is running, get an AI summary of pass/N/A patterns to validate signal quality                                                                           |
63| `posthog:workflows-list` / `posthog:workflows-get` | (**often unexposed** — UI: Data pipeline → Workflows) Browse existing workflow configs — useful for cloning an existing feed's structure when setting up a new one. Read-only; no create/update tool is exposed yet, so step 6's Slack workflow setup is UI-only. |
64 
65Before starting, **check which of the `posthog:llma-evaluation-*` tools are actually exposed in your agent's MCP tool set.** If they aren't loaded, treat steps 4-5 as UI walkthroughs rather than tool calls.
66 
67## Workflow
68 
69### Step 1 — Identify the filter
70 
71**Pattern A (feature-native trace_id prefix):** find the prefix that maps to your feature.
72 
73```sql
74SELECT
75    splitByChar(':', coalesce(properties.$ai_trace_id, ''))[1] AS root,
76    splitByChar(':', coalesce(properties.$ai_trace_id, ''))[2] AS subtype,
77    count() AS events
78FROM events
79WHERE timestamp > now() - INTERVAL 3 DAY
80    AND event = '$ai_generation'
81    AND properties.$ai_trace_id IS NOT NULL
82GROUP BY root, subtype
83ORDER BY events DESC
84LIMIT 25
85```
86 
87Note: `coalesce(..., '')` is load-bearing — `splitByChar` on a nullable column errors out in HogQL otherwise.
88 
89**Pattern B (PostHog AI agent mode):** verify coverage and volume for the mode you're targeting.
90 
91```sql
92SELECT
93    properties.agent_mode AS agent_mode,
94    properties.supermode AS supermode,
95    count() AS events,
96    count(DISTINCT properties.$ai_trace_id) AS traces
97FROM events
98WHERE timestamp > now() - INTERVAL 3 DAY
99    AND event = '$ai_generation'
100    AND properties.ai_product = 'posthog_ai'
101GROUP BY agent_mode, supermode
102ORDER BY events DESC
103LIMIT 20
104```
105 
106Expected values for `agent_mode`: `error_tracking`, `product_analytics`, `sql`, `session_replay`, `flags`, `survey`, `llm_analytics`, `null`. Null ≈ batch jobs + tool-internal calls (not user chat). `supermode='plan'` splits planning turns from execution turns — worth calling out separately if your feed is about plan-mode specifically.
107 
108Record the mode + rough volume. Low-volume modes (<100 events/day) will produce a trickle-feed that's hard to validate early; high-volume modes (>1k/day) may need sampling to avoid Slack flooding. See the "Tips" section on sampling.
109 
110### Step 2 — Pull a handful of sample traces
111 
112Use these for prompt iteration in step 4.
113 
114**Pattern A:**
115 
116```json
117posthog:query-llm-traces-list
118{
119  "properties": [
120    { "type": "event", "key": "$ai_trace_id", "operator": "icontains", "value": "<your-prefix-here>" }
121  ],
122  "limit": 10,
123  "dateRange": { "date_from": "-2d" },
124  "randomOrder": true
125}
126```
127 
128**Pattern B:**
129 
130```json
131posthog:query-llm-traces-list
132{
133  "properties": [
134    { "type": "event", "key": "ai_product", "operator": "exact", "value": "posthog_ai" },
135    { "type": "event", "key": "agent_mode", "operator": "exact", "value": "<mode-here>" }
136  ],
137  "limit": 10,
138  "dateRange": { "date_from": "-2d" },
139  "randomOrder": true
140}
141```
142 
143`randomOrder: true` matters — recency bias produces a non-representative sample. Pick 5-10 traces to test against.
144 
145**Output size warning:** `query-llm-traces-list` with `limit: 10` routinely returns 3-6MB of JSON (full input/output per generation). This will blow your context window. **Immediately delegate the summarization to a subagent** the moment you see the "result exceeds maximum allowed tokens" error — ask the subagent to extract, per trace: the trace id, the first user message (truncated to ~300 chars), the sampled `$current_url`, and a one-sentence description of what the conversation was about. Don't try to read the raw file in-line.
146 
147**Watch for topic drift in Pattern B samples.** The `agent_mode` tag reflects the user's mode selection at the time of the turn — but chat state retains the mode even if the user drifts off-topic within the same conversation (e.g. user selected "error tracking" mode, then asked an unrelated pricing question three turns later). Your eval prompt's classification step needs to be permissive about topic-drift: PASS should mean "user is doing something recognizably in-scope for this mode", FAIL should catch the off-topic drift. If you don't, your feed will include irrelevant PASS entries that happen to carry the mode tag.
148 
149### Step 3 — Draft the LLM-judge prompt
150 
151The prompt has two responsibilities: (a) classify the trace as relevant or not, (b) produce reasoning text that is **directly postable to Slack** (no preamble, no meta-description). The reasoning field becomes the Slack message body.
152 
153Template:
154 
155```text
156You are analyzing a PostHog [FEATURE NAME] trace to extract its real use case.
157Your reasoning text will be posted directly to a Slack channel as a notification.
158Write it as a short, ready-to-post message — no preamble, no meta-description.
159 
160Step 1 — Classification:
161- PASS = this trace is the [feature kind] you care about
162- FAIL = a different LLM call or a false match
163- N/A = ambiguous from the trace alone
164 
165Step 2 — Reasoning (only matters if PASS). Write 2-3 sentences in this exact format:
166 
167"[OPENER] [what they targeted/filtered for]. They were
168trying to [understand X / debug Y / find Z]. The result surfaced [key pattern
169or finding]."
170 
171Your output MUST start with the exact phrase "[OPENER]". No other opening is allowed.
172 
173Rules:
174- No "This is a [feature]..." or "The input contains..." preamble
175- No JSON, field names, system-prompt references, or meta-description
176- Concrete > generic. "users hitting error tracking for the first time" beats "user behavior"
177- If you cannot infer one of the three pieces from the trace, write "(unclear from trace)" in that slot — do not guess
178```
179 
180**Pick an `[OPENER]` that matches how users actually interact with the feature.** The forced opener is load-bearing (it prevents the model from drifting into "this trace is a..." meta-description), but the exact verb has to fit the interaction:
181 
182| Feature / mode                    | OPENER                                     |
183| --------------------------------- | ------------------------------------------ |
184| Session summary (group / single)  | `A user ran a summary on`                  |
185| Replay AI search                  | `A user searched replays for`              |
186| PostHog AI in error tracking mode | `A user asked PostHog AI about`            |
187| PostHog AI in session replay mode | `A user asked PostHog AI about`            |
188| PostHog AI in SQL mode            | `A user asked PostHog AI to write SQL for` |
189 
190Note: `supermode='plan'` is a sub-filter that layers _on top of_ an `agent_mode` row — it's not its own row. If you want plan-mode-only, filter `agent_mode='<mode>' AND supermode='plan'` and pick an opener like `"A user asked PostHog AI to plan"`.
191 
192If you force `"A user ran"` on a chat-based feature, the model will produce awkward contortions ("A user ran a question about...") that read wrong in Slack. The forced-opener pattern is the mechanism — the specific phrase is per-feature.
193 
194The negative example list ("No 'This is a...' preamble", etc.) is load-bearing regardless of opener. Don't remove it.
195 
196### Step 4 — Create the eval (disabled), test, iterate
197 
198Create with `enabled: false` so it doesn't immediately fan out to all traces.
199 
200**If `posthog:llma-evaluation-create` is exposed**, use this payload:
201 
202```json
203posthog:llma-evaluation-create
204{
205  "name": "[feature] use case feed",
206  "description": "Extracts canonical use cases for [feature] for the #team-[area]-usage Slack feed",
207  "evaluation_type": "llm_judge",
208  "evaluation_config": {
209    "prompt": "<full prompt from step 3>"
210  },
211  "output_type": "boolean",
212  "output_config": { "allows_na": true },
213  "model_configuration": {
214    "provider": "<provider>",
215    "model": "<model>"
216  },
217  "enabled": false,
218  "conditions": {
219    "filters": [
220      // Pattern A — feature-native trace_id prefix:
221      { "key": "$ai_trace_id", "operator": "icontains", "value": "<your-prefix>" }
222 
223      // Pattern B — PostHog AI agent mode (use these INSTEAD of the trace_id filter):
224      // { "key": "ai_product", "operator": "exact", "value": "posthog_ai" },
225      // { "key": "agent_mode", "operator": "exact", "value": "<mode>" }
226    ]
227  }
228}
229```
230 
231Leave model choice to the user — LLM-judge cost scales linearly with event volume, and cheap-vs-capable is a real tradeoff they should make based on their own spend tolerance and signal-quality requirements. Don't pick for them.
232 
233**UI fallback** (when `llma-evaluation-create` isn't exposed): AI observability → Evaluations → New evaluation. Type = `LLM judge`, output = boolean + allow N/A, filters as above, enabled = off. Paste the prompt from step 3.
234 
235Then dry-run against your sample traces.
236 
237**If `posthog:llma-evaluation-run` is exposed:**
238 
239```json
240posthog:llma-evaluation-run
241{
242  "evaluationId": "<uuid from create>",
243  "target_event_id": "<a $ai_generation event id from step 2>",
244  "timestamp": "<ISO timestamp of that event>"
245}
246```
247 
248**UI fallback:** on the eval detail page, use the "Run on event" button with the trace sample's event id.
249 
250Look at the returned `$ai_evaluation_reasoning`. If it preambles, drifts, or describes the input, fix the prompt (via `llma-evaluation-update` or by editing in the UI) and re-run. Iterate on 3-5 traces before enabling.
251 
252Common failure modes during iteration:
253 
254| Symptom                                                    | Fix                                                                        |
255| ---------------------------------------------------------- | -------------------------------------------------------------------------- |
256| Reasoning starts with "This is a..."                       | Strengthen the forced opener instruction; add a counter-example            |
257| Reasoning is generic ("user behavior", "various patterns") | Add positive examples of concrete phrasing in the prompt                   |
258| Model classifies everything as PASS                        | Tighten the FAIL definition; add an example of what a non-match looks like |
259| Reasoning is too long for Slack                            | Add a hard sentence cap ("MAX 3 sentences, hard limit")                    |
260 
261### Step 5 — Enable the eval
262 
263Once 3-5 sample runs produce clean Slack-ready output.
264 
265**If `posthog:llma-evaluation-update` is exposed:**
266 
267```json
268posthog:llma-evaluation-update
269{
270  "evaluationId": "<uuid>",
271  "enabled": true
272}
273```
274 
275**UI fallback:** AI observability → Evaluations → open the eval → toggle enabled.
276 
277The eval will now run on every new matching `$ai_generation` event.
278 
279### Step 6 — Build the workflow (UI only)
280 
281Workflow setup is not MCP-accessible for writes (`posthog:workflows-list` / `posthog:workflows-get` are read-only). The steps below are a UI walkthrough.
282 
283**Prereq:** before you start, invite the PostHog Slack bot to your target channel (`/invite @PostHog` in the Slack channel). Without this, the Slack dispatch step will fail with an opaque permission error at send time, not at save time — easy to miss.
284 
285#### 6.1 Create the workflow
286 
287Data pipeline → Workflows → New workflow. Name it `<feature> use case feed` to match the eval name from step 4.
288 
289#### 6.2 Trigger step
290 
291- **Event:** `AI evaluation (LLM)` — i.e. `$ai_evaluation`. This is the event emitted when an eval runs, and it's the only event that carries `$ai_evaluation_*` properties. The original `$ai_generation` event is **not** enriched with eval results, so filtering on `$ai_generation` here matches nothing.
292- **Property filters (both required):**
293  - `AI Evaluation Name (LLM)` equals `<your eval name from step 4>`
294  - `AI Evaluation Result (LLM)` equals `true`
295 
296**⚠️ LOAD-BEARING:** the stored values for `$ai_evaluation_result` are the strings `'True'` / `'False'` / `'None'` — NOT `'PASS'` / `'FAIL'` / `'N/A'` (despite what the prompt template calls them internally). The Workflows UI property filter normalizes `true` → `'True'`, so selecting `equals true` from the dropdown works. But if you were wiring this in raw SQL somewhere else (say a hog function), you'd need the string literal. Verify the stored distribution before saving:
297 
298```sql
299SELECT DISTINCT toString(properties.$ai_evaluation_result) AS result, count() AS n
300FROM events
301WHERE event = '$ai_evaluation'
302  AND properties.$ai_evaluation_name = '<your eval name>'
303  AND timestamp > now() - INTERVAL 1 HOUR
304GROUP BY result
305```
306 
307If the only values are `True`/`False`/`None` and `True` dominates, the UI `equals true` filter will match. If you see anything else, adjust accordingly.
308 
309#### 6.3 Slack dispatch step
310 
311- **Add step → Slack dispatch**
312- **Channel:** `#<your-team>-usage-feed`
313- **Sender / bot display name:** something that reads well in the channel (e.g. `PostHog Usage Feed`)
314- **Blocks (Slack block-kit JSON)** — paste this and replace `<project_id>` with your actual numeric project ID (e.g. `2`):
315 
316```json
317[
318  {
319    "text": {
320      "text": "<emoji> *{event.properties.$ai_evaluation_name}* triggered by *{person.name}*",
321      "type": "mrkdwn"
322    },
323    "type": "section"
324  },
325  {
326    "text": {
327      "text": "{event.properties.$ai_evaluation_reasoning}",
328      "type": "mrkdwn"
329    },
330    "type": "section"
331  },
332  {
333    "type": "actions",
334    "elements": [
335      {
336        "url": "https://us.posthog.com/project/<project_id>/ai-observability/traces/{event.properties.$ai_trace_id}?event={event.properties.$ai_target_event_id}",
337        "text": { "text": "View Trace", "type": "plain_text" },
338        "type": "button"
339      },
340      {
341        "url": "https://us.posthog.com/project/<project_id>/replay/{event.properties.$session_id}",
342        "text": { "text": "View Trigger Session", "type": "plain_text" },
343        "type": "button"
344      },
345      {
346        "url": "{person.url}",
347        "text": { "text": "View Person", "type": "plain_text" },
348        "type": "button"
349      }
350    ]
351  }
352]
353```
354 
355Pick an `<emoji>` that matches the feature's shape: 📊 product analytics, 🐛 error tracking, 🎬 session replay, 🔎 search/AI search, 🧪 experiments, 🚩 flags, 📋 surveys, 🧠 generic AI.
356 
357The `{event.properties.X}` and `{person.X}` placeholders are valid PostHog template syntax and resolve at send time.
358 
359#### 6.4 Test before enabling
360 
361The Workflows Test panel has two modes — this matters because naively hitting "Test" can look like a broken integration when it isn't:
362 
363- **Synthetic event** (default) — the Test panel fabricates an `$ai_evaluation` payload and runs the flow without hitting Slack's real API. Useful as a dry-run of the block template, but `{event.properties.$ai_*}` placeholders may resolve to `null` and Slack's block validator will reject the payload with `invalid_blocks`. That's a test-harness artifact, not a real bug — don't chase it.
364- **"Make real HTTPS requests"** — flip this toggle on. Workflows then pulls a recent real `$ai_evaluation` event matching your filters and runs the flow end-to-end, including the actual Slack post. This is the test that tells you "it works" for real. If no matching real event exists yet (common if the eval was just enabled), trigger the feature yourself, wait ~1 minute, and retry.
365 
366Recommended flow: synthetic → sanity-check the block template renders → flip real-requests on → confirm an actual post lands in the channel → save + enable the workflow.
367 
368### Step 7 — End-to-end verify in production
369 
370Once the workflow is enabled, trigger the feature yourself. Within a minute or two:
371 
3721. The `$ai_generation` event should appear in AI observability
3732. The eval should auto-run and emit an `$ai_evaluation` event
3743. The workflow should fire and the Slack post should land in the configured channel
3754. Click "View Trigger Session" — should land on the recording of you using the feature, not the replay homepage
376 
377If "View Trigger Session" lands on the replay homepage, `$session_id` is missing on the `$ai_evaluation` event (which is separate from the `$ai_generation` event — threading is independent for the two). Backend fix needed — see prerequisites.
378 
379## Worked example A (Pattern A): group session summary use cases
380 
381Pattern: a `group_summary_use_case_feed` eval streaming to a `#<team>-usage-feed` channel. Trace prefix: `session-summary:group:`. Opener: `"A user ran a group summary on"`. Slack channel showed e.g.:
382 
383> 📊 _group_summary_use_case_feed_ triggered by _some user_
384> "A user ran a group summary on a company's onboarding sessions from the last 7 days. They were trying to understand why account activation rates are low. The summary surfaced that most users abandon at the company onboarding wizard after creating accounts."
385> [View Trace] [View Trigger Session] [View Person]
386 
387The PRs that made this work (linked here as worked examples of the session_id threading pattern, not as steps in the skill itself):
388 
389- PostHog/posthog#54952 — threads `trigger_session_id` through to `$ai_generation` events on the session summary backend
390- (Followup PR — threads `$session_id` onto `$ai_evaluation` events specifically)
391 
392## Worked example B (Pattern B): PostHog AI in error tracking mode
393 
394Pattern: an `agent_mode = 'error_tracking'` scoped feed streaming to a `#<team>-usage-feed` channel, answering "what are users actually trying to DO when they chat with PostHog AI in error tracking mode?" Mode sizing varies by an order of magnitude or more across agent modes — spot-check volume per §Step 1 before wiring, because a high-volume mode can flood a channel. Opener: `"A user asked PostHog AI about"`.
395 
396Enabling PR: PostHog/posthog#55160 — threads `agent_mode` and `supermode` onto every `$ai_generation` emitted by the chat agent loop. Wiring lives in `ee/hogai/core/agent_modes/executables.py` (`AgentExecutable._get_model`) and passes the dict through the existing `posthog_properties` field on `MaxChatMixin` in `ee/hogai/llm.py`. Before this PR, scoping a PostHog AI eval to a specific mode wasn't possible — you'd end up evaluating every PostHog AI generation, which produced noisy feeds with low single-digit PASS rates.
397 
398Key observation from setup: the `agent_mode` tag reflects the mode at turn-time, but chat state retains mode selection even when users drift off-topic mid-conversation. Spot-check: a random `agent_mode=error_tracking` sample included a conversation that ended up being about session replay pricing. The eval prompt's classification must be permissive about topic drift — PASS only when the turn is recognizably in-scope for the mode, FAIL when the conversation has drifted to something else entirely.
399 
400## Validating signal quality after launch
401 
402Once the feed has been running for a day or two, sanity-check the eval output at scale.
403 
404**If `posthog:llma-evaluation-summary-create` is exposed:**
405 
406```json
407posthog:llma-evaluation-summary-create
408{
409  "evaluation_id": "<uuid>",
410  "filter": "fail"
411}
412```
413 
414**UI fallback:** open the eval in AI observability → Evaluations → "Summarize results" button, filter = fail.
415 
416If the FAIL bucket is large, the classification step is too strict — relax it. If the PASS bucket has lots of generic reasonings, iterate on the prompt to enforce concreteness. The summary tool gives a quick read on this without you having to scroll through individual events.
417 
418Spot-check raw events when needed (note: the stored result value is `'True'`, not `'PASS'` — see step 6):
419 
420```sql
421SELECT
422    properties.$ai_evaluation_reasoning AS reasoning,
423    properties.$ai_trace_id AS trace_id,
424    timestamp
425FROM events
426WHERE event = '$ai_evaluation'
427    AND properties.$ai_evaluation_name = '<your eval name>'
428    AND properties.$ai_evaluation_result = 'True'
429    AND timestamp > now() - INTERVAL 1 DAY
430ORDER BY timestamp DESC
431LIMIT 25
432```
433 
434## Tips
435 
436- The reasoning field IS the Slack message — design the prompt for that, not for "chain of thought before classification." Models can produce structured Slack-ready text in one pass.
437- LLM judges are non-deterministic across reruns. Expect 1-5% noise even with a fixed prompt and model. If you need reproducibility, pin a deterministic provider/seed in `model_configuration`.
438- Keep the eval scoped tightly via `conditions.filters` on `$ai_trace_id` prefix. Otherwise it fans out to every `$ai_generation` event in the project and burns LLM cost.
439- For high-volume features (>10k traces/week), consider sampling — set the eval to run on a percentage of matching events rather than all of them. Slack flooding is a real failure mode.
440- The "View Trigger Session" button is the highest-value link in the alert. Without it, the feed is just text — you can't watch what the user was actually doing. Verify it works in step 7 before considering the feed shipped.
441- Once the feed is live, periodically re-run the eval summary tool with `filter: "pass"` to surface the dominant use case clusters. That's how you turn the feed into actual product insights instead of just a notification stream.
442