Analyzing Experiment Session Replays

1---
2name: analyzing-experiment-session-replays
3description: 'Analyze session replay patterns across experiment variants to understand user behavior differences. Use when the user wants to see how users interact with different experiment variants, identify usability issues, compare behavior patterns between control and test groups, or get qualitative insights to complement quantitative experiment results.'
4---
5 
6# Analyzing experiment session replays
7 
8This skill guides you through analyzing session recordings for experiment variants to understand behavioral differences between control and test groups.
9 
10## When to use this skill
11 
12Use this skill when:
13 
14- The user asks to analyze session replays for an experiment
15- The user wants to understand how users behave differently across experiment variants
16- The user asks to compare user behavior between control and test variants
17- The user wants qualitative insights to complement experiment metrics
18- The user asks questions like "How are users behaving in my experiment?" or "Show me session replays for variant X"
19 
20## Prerequisites
21 
22Before analyzing session replays:
23 
241. The experiment must be **launched** (not in draft state)
252. Session replay must be enabled for the project
263. Users must have been exposed to the experiment variants
274. The experiment must have a start date
28 
29## Workflow
30 
31### 1. Get experiment details and feature flag variants
32 
33First, retrieve the experiment information and the feature flag variants (source of truth).
34 
35**Step 1a: Get experiment metadata**
36 
37You can either:
38 
39- **Option A**: Use the `experiment-get` tool if you already have the experiment ID from context
40- **Option B**: Query the experiments table via HogQL:
41 
42```sql
43SELECT
44    e.id,
45    e.name,
46    f.key AS feature_flag_key,
47    e.start_date,
48    e.end_date
49FROM system.experiments e
50JOIN system.feature_flags f ON f.id = e.feature_flag_id
51WHERE e.id = <experiment_id>
52```
53 
54From the experiment data, extract:
55 
56- `feature_flag_key`: The feature flag controlling the experiment
57- `start_date` and `end_date`: The experiment's time range
58 
59**Step 1b: Get variants from the feature flag**
60 
61**IMPORTANT**: Always get variants from the feature flag, NOT from `experiment.parameters.feature_flag_variants`.
62The parameters can be out of sync or deprecated. The feature flag is the source of truth.
63 
64Query the feature flag to get the current variants:
65 
66```sql
67SELECT filters.multivariate.variants AS variants
68FROM system.feature_flags
69WHERE key = '<feature_flag_key>'
70```
71 
72Select the variants path directly — selecting the whole `filters` object gets truncated in results for flags with large targeting configs.
73Example structure: `[{"key": "control", "name": "Control", "rollout_percentage": 50}, {"key": "test", ...}]`
74 
75The variant `key` values (e.g., "control", "test", "variant_a") are what you'll use to filter session recordings.
76 
77### 2. Build session recording filters for each variant
78 
79For each variant in the experiment, construct recording filters that match users exposed to that variant.
80 
81**Filter structure for a variant** (input to `query-session-recordings-list`):
82 
83```json
84{
85  "date_from": "<experiment.start_date>",
86  "date_to": "<experiment.end_date or current time>",
87  "filter_test_accounts": true,
88  "properties": [
89    {
90      "type": "event",
91      "key": "$feature/<feature_flag_key>",
92      "operator": "exact",
93      "value": ["<variant_key>"]
94    }
95  ]
96}
97```
98 
99**Key points:**
100 
101- The `$feature/<flag_key>` event property records which variant the user saw — filtering on it matches recordings containing at least one event from that variant
102- `value` is an array of variant key strings (e.g. `["control"]`); for boolean flags use `["true"]` or `["false"]`
103- Avoid the `type: "flag"` / `flag_evaluates_to` property filter for variant scoping — the recordings query accepts it but silently ignores it, returning unfiltered results (last verified 2026-06-10). If you want to try it anyway, verify it actually filters first: a query with a nonexistent flag key should return zero recordings
104- Set the date range to the experiment's start and end dates
105- Enable `filter_test_accounts: true` to exclude test users
106 
107### 3. Retrieve recordings for each variant
108 
109Use the `query-session-recordings-list` tool with the filters constructed in step 2.
110 
111Call the tool once per variant to get recordings for each group:
112 
113- Variant "control" → recordings for control group
114- Variant "test" → recordings for test variant
115- Additional variants if the experiment has more than 2
116 
117The tool returns a list of recordings with metadata including:
118 
119- `distinct_id` — the person's distinct ID
120- `recording_duration`, `active_seconds`, `inactive_seconds`
121- `click_count`, `keypress_count`, `mouse_activity_count`
122- `console_log_count`, `console_warn_count`, `console_error_count`
123- `start_url` — first page URL visited
124- `start_time` / `end_time`, `activity_score`
125 
126### 4. Compare and analyze
127 
128Compare the recordings between variants by looking for:
129 
130**Quantitative patterns:**
131 
132- Session duration differences
133- Activity levels (clicks, keypresses)
134- Console error rates
135- Bounce rates
136 
137**Qualitative insights:**
138 
139- User confusion or frustration indicators
140- Different navigation paths
141- Feature discovery patterns
142- Error recovery behavior
143 
144### 5. Present findings
145 
146Summarize the behavioral differences between variants, highlighting:
147 
148- Total recordings per variant
149- Notable behavior patterns unique to each variant
150- Usability issues or friction points observed
151- Recommendations based on the qualitative data
152 
153## Example interaction
154 
155```text
156User: "How are users behaving in my checkout experiment?"
157Agent steps:
1581. Query experiment details (ID: 123, feature_flag_key: "checkout-flow-test", date range: 2025-01-01 to 2025-01-31)
1592. Query feature flag "checkout-flow-test" to get variants from filters.multivariate.variants
1603. Extract variant keys: "control" and "new-checkout"
1614. Build filters for control variant:
162   - Property filter: { type: "event", key: "$feature/checkout-flow-test", operator: "exact", value: ["control"] }
163   - Date range: 2025-01-01 to 2025-01-31
1645. Call query-session-recordings-list with control filters → 147 recordings found
1656. Build filters for new-checkout variant and call query-session-recordings-list → 152 recordings found
1667. Compare patterns:
167   - Control: Average 3m 45s session duration, 12% console errors
168   - New-checkout: Average 2m 30s session duration, 5% console errors
1698. Present findings:
170   "I analyzed session replays for your checkout experiment. The new checkout flow shows:
171   - 33% faster completion (2m 30s vs 3m 45s)
172   - 58% fewer console errors (5% vs 12%)
173   - Users in the new variant navigate directly to payment, while control users often backtrack to review cart
174   - Recommendation: The new checkout flow reduces friction and errors"
175```
176 
177## Important notes
178 
179**Do not make assumptions:**
180 
181- Always verify the experiment has recordings before analyzing
182- Check that the experiment is launched (has a start_date)
183- If no recordings are found, inform the user clearly
184 
185**Filter construction:**
186 
187- The `$feature/<flag_key>` event property is how you scope recordings to a variant
188- One filter per variant — call the tool once per variant with its own filter
189- For boolean flags, use `["true"]`/`["false"]` as the value instead of a variant key
190 
191**Error handling:**
192 
193- If the experiment is in draft state, tell the user it hasn't started yet
194- If no recordings exist, suggest enabling session replay or waiting for user traffic
195- If the variant count is unexpected, double-check the experiment configuration
196 
197## Related tools
198 
199- `query-session-recordings-list`: Core tool for retrieving session recordings with filters
200- `experiment-get`: Get experiment metadata; `experiment-results-get` for statistical results
201- `execute-sql`: Query experiments table for details via HogQL
202