Exploring LLM Clusters

1---
2name: exploring-llm-clusters
3description: 'Investigate AI observability clusters — understand usage patterns in AI/LLM traffic, compare cluster behavior, compute cost/latency metrics, and drill into individual traces within clusters.'
4---
5 
6# Exploring LLM clusters
7 
8Use this skill when investigating AI observability clusters —
9understanding what patterns exist in your AI/LLM traffic,
10comparing cluster behavior, and drilling into individual clusters.
11 
12## Tools
13 
14| Tool                               | Purpose                                         |
15| ---------------------------------- | ----------------------------------------------- |
16| `posthog:llma-clustering-job-list` | List clustering job configurations for the team |
17| `posthog:llma-clustering-job-get`  | Get a specific clustering job by ID             |
18| `posthog:execute-sql`              | Query cluster run events and compute metrics    |
19| `posthog:query-llm-traces-list`    | Find traces belonging to a cluster              |
20| `posthog:query-llm-trace`          | Inspect a specific trace in detail              |
21 
22## How clustering works
23 
24PostHog clusters LLM traces (or individual generations) by embedding similarity.
25A Temporal workflow runs periodically or on-demand, producing cluster events stored as
26`$ai_trace_clusters` (trace-level) or `$ai_generation_clusters` (generation-level).
27 
28Each cluster event contains:
29 
30- `$ai_clustering_run_id` — unique run identifier (format: `<team_id>_<level>_<YYYYMMDD>_<HHMMSS>[_<job_id>]`)
31- `$ai_clustering_level` — `"trace"` or `"generation"`
32- `$ai_window_start` / `$ai_window_end` — time window analyzed
33- `$ai_total_items_analyzed` — number of traces/generations processed
34- `$ai_clusters` — JSON array of cluster objects
35- `$ai_clustering_params` — algorithm parameters used
36 
37### Cluster object shape (inside `$ai_clusters`)
38 
39```json
40{
41  "cluster_id": 0,
42  "size": 42,
43  "title": "User authentication flows",
44  "description": "Traces involving login, signup, and token refresh operations",
45  "traces": {
46    "<trace_or_generation_id>": {
47      "distance_to_centroid": 0.123,
48      "rank": 0,
49      "x": -2.34,
50      "y": 1.56,
51      "timestamp": "2026-03-28T10:00:00Z",
52      "trace_id": "abc-123",
53      "generation_id": "gen-456"
54    }
55  },
56  "centroid_x": -2.1,
57  "centroid_y": 1.4
58}
59```
60 
61- `cluster_id: -1` is the **noise/outlier** cluster (items that didn't fit any cluster)
62- Items in `traces` are keyed by trace ID (trace-level) or generation event UUID (generation-level)
63- `rank` orders items by proximity to centroid (0 = closest)
64- `x`, `y` are 2D coordinates for visualization (UMAP/PCA/t-SNE reduced)
65 
66## Clustering jobs
67 
68Each team can have up to 5 clustering jobs. A job defines:
69 
70- **name** — human-readable label
71- **analysis_level** — `"trace"` or `"generation"`
72- **event_filters** — property filters scoping which traces are included
73- **enabled** — whether the job runs on schedule
74 
75Default jobs named `"Default - trace"` and `"Default - generation"` are auto-created
76and disabled when a custom job is created for the same level.
77 
78## Workflow: explore clusters
79 
80### Step 1 — List recent clustering runs
81 
82```sql
83posthog:execute-sql
84SELECT
85    properties.$ai_clustering_run_id as run_id,
86    properties.$ai_clustering_level as level,
87    properties.$ai_window_start as window_start,
88    properties.$ai_window_end as window_end,
89    toInt(properties.$ai_total_items_analyzed) as total_items,
90    timestamp
91FROM events
92WHERE event IN ('$ai_trace_clusters', '$ai_generation_clusters')
93    AND timestamp >= now() - INTERVAL 7 DAY
94ORDER BY timestamp DESC
95LIMIT 10
96```
97 
98### Step 2 — Get clusters from a specific run
99 
100```sql
101posthog:execute-sql
102SELECT
103    properties.$ai_clustering_run_id as run_id,
104    properties.$ai_clustering_level as level,
105    properties.$ai_clustering_job_id as job_id,
106    properties.$ai_clustering_job_name as job_name,
107    properties.$ai_window_start as window_start,
108    properties.$ai_window_end as window_end,
109    toInt(properties.$ai_total_items_analyzed) as total_items,
110    properties.$ai_clusters as clusters,
111    properties.$ai_clustering_params as params
112FROM events
113WHERE event IN ('$ai_trace_clusters', '$ai_generation_clusters')
114    AND properties.$ai_clustering_run_id = '<run_id>'
115LIMIT 1
116```
117 
118The `clusters` field is a JSON array. Parse it to see cluster titles, sizes, and descriptions.
119 
120**Important:** The clusters JSON can be very large (thousands of trace IDs with coordinates).
121When the result is too large for inline display, it auto-persists to a file.
122Use `print_clusters.py` from [scripts/](./scripts/) to get a readable summary.
123 
124### Step 3 — Compute metrics for clusters
125 
126For trace-level clusters, compute cost/latency/token metrics:
127 
128```sql
129posthog:execute-sql
130SELECT
131    properties.$ai_trace_id as trace_id,
132    sum(toFloat(properties.$ai_total_cost_usd)) as total_cost,
133    max(toFloat(properties.$ai_latency)) as latency,
134    sum(toInt(properties.$ai_input_tokens)) as input_tokens,
135    sum(toInt(properties.$ai_output_tokens)) as output_tokens,
136    countIf(properties.$ai_is_error = 'true') as error_count
137FROM events
138WHERE event IN ('$ai_generation', '$ai_embedding', '$ai_span')
139    AND timestamp >= parseDateTimeBestEffort('<window_start>')
140    AND timestamp <= parseDateTimeBestEffort('<window_end>')
141    AND properties.$ai_trace_id IN ('<trace_id_1>', '<trace_id_2>', ...)
142GROUP BY trace_id
143```
144 
145For generation-level clusters, match by event UUID:
146 
147```sql
148posthog:execute-sql
149SELECT
150    toString(uuid) as generation_id,
151    toFloat(properties.$ai_total_cost_usd) as cost,
152    toFloat(properties.$ai_latency) as latency,
153    toInt(properties.$ai_input_tokens) as input_tokens,
154    toInt(properties.$ai_output_tokens) as output_tokens,
155    if(properties.$ai_is_error = 'true', 1, 0) as is_error
156FROM events
157WHERE event = '$ai_generation'
158    AND timestamp >= parseDateTimeBestEffort('<window_start>')
159    AND timestamp <= parseDateTimeBestEffort('<window_end>')
160    AND uuid IN ('<gen_uuid_1>', '<gen_uuid_2>', ...)
161```
162 
163### Step 4 — Drill into specific traces
164 
165Once you've identified interesting clusters, use the trace tools to inspect individual traces:
166 
167```json
168posthog:query-llm-trace
169{
170  "traceId": "<trace_id_from_cluster>",
171  "dateRange": {"date_from": "<window_start>", "date_to": "<window_end>"}
172}
173```
174 
175## Investigation patterns
176 
177### "What kinds of LLM usage do we have?"
178 
1791. List recent clustering runs (Step 1)
1802. Load the latest run's clusters (Step 2)
1813. Review cluster titles and descriptions — each represents a distinct usage pattern
1824. Compare cluster sizes to understand traffic distribution
183 
184### "Which cluster is most expensive / slowest?"
185 
1861. Load clusters from a run (Step 2)
1872. Extract trace IDs from each cluster
1883. Compute metrics per cluster (Step 3)
1894. Aggregate: `avg(cost)`, `avg(latency)`, `sum(cost)` per cluster
1905. Compare across clusters
191 
192### "What's in this cluster?"
193 
1941. Load the cluster's traces (from the `traces` field)
1952. Sort by `rank` (closest to centroid = most representative)
1963. Inspect the top 3-5 traces via `query-llm-trace` to understand the pattern
1974. Check the cluster `title` and `description` for the AI-generated summary
198 
199### "Are there error-heavy clusters?"
200 
2011. Compute metrics (Step 3) with `error_count`
2022. Calculate error rate per cluster: `items_with_errors / total_items`
2033. Focus on clusters with high error rates
2044. Drill into errored traces to find root causes
205 
206### "How do clusters compare across runs?"
207 
2081. List multiple runs (Step 1)
2092. Load clusters from each run
2103. Compare cluster titles — similar titles across runs indicate stable patterns
2114. Track cluster size changes to detect shifts in traffic patterns
212 
213## Constructing UI links
214 
215- **Clusters overview**: `https://app.posthog.com/ai-observability/clusters`
216- **Specific run**: `https://app.posthog.com/ai-observability/clusters/<url_encoded_run_id>`
217- **Cluster detail**: `https://app.posthog.com/ai-observability/clusters/<url_encoded_run_id>/<cluster_id>`
218 
219Always surface these links so the user can verify visually in the PostHog UI.
220 
221## Tips
222 
223- Always set a time range in SQL queries — cluster events without time bounds are slow
224- Start with run listing to orient, then drill into specific clusters
225- Cluster titles and descriptions are AI-generated summaries — verify by inspecting traces
226- The noise cluster (`cluster_id: -1`) contains outliers that didn't fit any pattern
227- Use `llma-clustering-job-list` to understand what clustering configs are active
228- Trace IDs in clusters can be used directly with `query-llm-trace` for deep inspection
229- For large clusters, inspect the top-ranked traces (closest to centroid) for representative examples
230