Diagnosing Experiment Results

1---
2name: diagnosing-experiment-results
3description: "Diagnoses bias, anomalies, and strange-looking results on a specific PostHog experiment. Covers empty / 0-exposure experiments, sample ratio mismatch, identity fragmentation, multi-variant exposure, uneven-split exclusion bias, significance traps (peeking, A/A, Bayesian vs Frequentist), PostHog-vs-SQL discrepancies, and surprises after mid-run edits. Symptom-driven dispatch to the right diagnostic.\nTRIGGER when: user asks 'is my experiment biased?' or 'why 0 exposures?', references the bias banner, says a variant looks strange / wrong / off, sees significance flipping, notices PostHog numbers disagreeing with their SQL, sees an A/A test showing significance, or reports surprises after mid-run edits.\nDO NOT TRIGGER when: creating a new experiment (use creating-experiments), only configuring rollout (use configuring-experiment-rollout) or metrics (use configuring-experiment-analytics), or only asking lifecycle questions (use managing-experiment-lifecycle)."
4---
5 
6# Diagnosing experiment results
7 
8This skill answers: **My PostHog experiment results look wrong, biased, or empty — what's going on?**
9 
10Match the user's complaint in the dispatch table, then read the matching reference file for the
11diagnostic.
12 
13Each diagnostic in the reference files is tagged `[HIGH]`, `[MEDIUM]`, or `[LOW]` based on how
14strongly it's verified — `[HIGH]` is verified directly in PostHog code, `[MEDIUM]` is partially or
15team-source verified, `[LOW]` describes SDK/external behavior that wasn't verified here. Treat `[LOW]`
16items as hypotheses to test, not facts to assert.
17 
18## Step 1 — Resolve the experiment
19 
20If the user refers to an experiment by name or description, load the `finding-experiments` skill first to
21resolve it to a concrete ID.
22 
23Call `experiment-get` and pull these fields. They are inputs for almost every diagnostic:
24 
25- `parameters.feature_flag_variants[].rollout_percentage` — the variant split
26- `parameters.rollout_percentage` — the overall rollout (% of users entering the experiment)
27- `exposure_criteria.multiple_variant_handling` — defaults to `"exclude"` if absent
28- `exposure_criteria.exposure_event` — `null` means default `$feature_flag_called`
29- `exposure_criteria.filterTestAccounts` — defaults to `true`
30- `feature_flag.active`, status (`draft` / `running` / `paused` / `stopped`), `start_date`, `end_date`
31- `feature_flag.filters.groups[].variant` — any non-null value is a forced-variant override on the
32  matched cohort (release-condition assignment, not randomized). Surfaces A7 by default.
33- `stats_config` — Bayesian (default) or Frequentist
34 
35## Step 1.5 — Pull a diagnostic snapshot (verify before asking)
36 
37Before asking the user clarifying questions, pull the diagnostic snapshot in
38[references/diagnostic-snapshot.md](references/diagnostic-snapshot.md). Most diagnostics in this skill
39can be confirmed or ruled out from that data without an interview.
40 
41## Step 2 — Match symptom to diagnostic
42 
43| User says...                                                                               | Diagnostic group                             |
44| ------------------------------------------------------------------------------------------ | -------------------------------------------- |
45| "Smaller variant looks biased" / banner says bias                                          | A — bias & skew                              |
46| "Variant ratio doesn't match my split" / SRM warning                                       | A — bias & skew                              |
47| "Why isn't it 50/50?" / "users in both groups"                                             | A — bias & skew                              |
48| "Users in both control and test" / high `$multiple` %                                      | A — bias & skew                              |
49| Multi-variant exposure on a server-rendered app                                            | A — bias & skew                              |
50| Banner about feature-flag/experiment state mismatch                                        | A — bias & skew                              |
51| "Migrating distinct_id" / "switching from anonymous to user_id" mid-run                    | A — bias & skew                              |
52| Metric count is much smaller than exposures (e.g. 10× or 100× gap)                         | A — bias & skew (route here before D)        |
53| "Experiment shows 0 / not enough data" / empty                                             | B — empty experiment                         |
54| "Variant always undefined / false"                                                         | B — empty experiment                         |
55| "$feature_flag_called fires but no exposures show up"                                      | B — empty experiment                         |
56| "Experiment says running but exposures haven't moved in weeks/months"                      | B — empty experiment                         |
57| "Significance keeps flipping as we run longer"                                             | C — interpretation traps                     |
58| "Significance was declared, then it wasn't significant anymore"                            | C — interpretation traps                     |
59| "30/16 split at 46 exposures, is this broken?"                                             | C — interpretation traps                     |
60| "A/A test is showing significant results"                                                  | C — interpretation traps                     |
61| "Many metrics — some significant, some not"                                                | C — interpretation traps                     |
62| "Bayesian says 96% chance to win — should we ship?"                                        | C — interpretation traps                     |
63| "Confidence intervals overlap — does that mean not significant?"                           | C — interpretation traps                     |
64| "An external tool (significance calculator or AI agent) disagrees with PostHog"            | C — interpretation traps                     |
65| "Should I ship? Primary is up but a secondary is down"                                     | C — interpretation traps                     |
66| "PostHog numbers ≠ my SQL count"                                                           | D — numbers vs SQL                           |
67| "Funnel says X% but my raw event count says Y"                                             | D — numbers vs SQL                           |
68| "Sum of revenue looks wrong" / "breakdown shows 'none'"                                    | D — numbers vs SQL                           |
69| "Recordings panel doesn't match the stats"                                                 | D — numbers vs SQL                           |
70| "I applied a filter but the user count didn't change"                                      | D — numbers vs SQL                           |
71| "I want to slice results by current person properties (as of now, not as of exposure)"     | D — numbers vs SQL                           |
72| "Changed split / rollout / metric / criteria mid-run, now odd"                             | E — mid-run changes                          |
73| "Ended/shipped — flag now flipped to 0/100 unexpectedly"                                   | E — mid-run changes                          |
74| "Long-term metric moves opposite from primary"                                             | E — mid-run changes                          |
75| "Retention metric counts users I didn't expect"                                            | E — mid-run changes                          |
76| "Can't convert the feature flag back to a simple (boolean) flag after the experiment ends" | E — mid-run changes                          |
77| "How do I restart an experiment with new variants?"                                        | E — mid-run changes                          |
78| Metric line is rendered but the result block is empty / no chance-to-win or significance   | E — mid-run changes (E13 legacy methodology) |
79 
80If the symptom is unclear, ask one clarifying question before picking. Most diagnostics have different fixes
81— do not guess.
82 
83## Step 3 — Surface every diagnostic the evidence supports
84 
85After matching the symptom in Step 2 and reading the relevant reference file(s), list each diagnostic
86that applies before recommending an action.
87 
88Surface co-occurring mechanisms independently — even when one is more salient, don't collapse them
89into a single "wait" or "fix" recommendation. Different mechanisms have different fixes: a
90_systematic_ bias (e.g. uneven-split + Exclude) doesn't resolve by waiting; a _statistical_ pattern
91(e.g. small-sample variance) does. Bundling them leaves the bias in place after the user follows the
92bundled advice.
93 
94Only list mechanisms that have a path to verification in the project state — config (from
95`experiment-get`), snapshot data, activity log, or repo source. Config-derived mechanisms count: an
9680/20 split with default `multiple_variant_handling="exclude"` is visible in `experiment-get` and is
97therefore enumerable. Naming a mechanism with no source (e.g. SRM when the snapshot shows a clean
98variant ratio) is not.
99 
100## Diagnostic groups
101 
102### A — Bias & skew
103 
104Variants don't look balanced, one variant looks biased, the in-app warning banner appeared, or users are
105showing up under multiple variants. Covers the uneven-split + Exclude interaction, SRM, identity
106fragmentation, bootstrap × `/decide` mismatch, and flag/experiment state inconsistency.
107 
108→ See [references/bias-and-skew.md](references/bias-and-skew.md)
109 
110### B — Empty experiment / 0 exposures / "not enough data"
111 
112A frequent pain point. Covers SDK call (wrong evaluation method, `identify()` timing, dedup),
113exposure capture (custom event missing variant property, required properties, ad-blockers), and
114exposure-criteria match (test-account filter, eligibility ordering, events firing before exposure).
115 
116→ See [references/empty-experiment.md](references/empty-experiment.md)
117 
118### C — Significance / interpretation traps
119 
120Significance flipping, A/A test showing significance, Bayesian vs Frequentist confusion, multiple
121comparisons, low-volume variance, peeking / early stopping. Includes the legacy stats issue (A/A tests
122historically over-fired before the new Bayesian module) and how the win-probability methodology changed in
123Jan 2025 (single test vs control, not control vs all variants).
124 
125→ See [references/interpretation.md](references/interpretation.md)
126 
127### D — Numbers don't match (PostHog vs the user's SQL / raw count)
128 
129The experiment page applies an exposure scope, `$multiple` exclusion, test-account filter, and date range
130that ad-hoc SQL almost never replicates. Covers funnel attribution (only first→last step counts for stats),
131breakdowns (read from the exposure event, not the metric event), the "sum of revenue" mean-of-per-user
132confusion, and the recordings-panel-vs-stats divergence.
133 
134→ See [references/numbers-vs-sql.md](references/numbers-vs-sql.md)
135 
136### E — Surprises after mid-run changes (incl. lifecycle and retention quirks)
137 
138Increasing rollout is safe; decreasing is caution; changing the variant split is an anti-pattern; adding
139metrics mid-run is p-hacking; ship-variant can rewrite the flag in surprising ways; reset clears
140results not the flag. Also covers retention-metric quirks (first-event-must-be-after-exposure design),
141"matured users" filtering, and long-term vs short-term metric divergence.
142 
143→ See [references/mid-run-changes.md](references/mid-run-changes.md)
144 
145## Step 4 — Calibrate recommendations to experiment state
146 
147Surface diagnostics first (Step 3). Then recommend — but scope what you recommend to what the
148experiment's current state permits.
149 
150- **Draft** — config changes are free; recommend and apply.
151- **Running** — every change has a tradeoff. Explain the mid-run impact (anti-pattern? safe?
152  user-visible?) before recommending. See `configuring-experiment-rollout` and its reference file
153  `references/changing-distribution-after-launch.md` for the mid-run rules.
154- **Stopped / archived** — the experiment AND its feature flag represent the documented outcome of
155  the run. Recommendations are scoped to (a) interpretation of the existing data, (b) what to do for
156  the _next_ experiment, or (c) explaining what happened.
157 
158On a stopped or archived experiment, don't preemptively offer reversal of a state mutation
159(ship-variant flag rewrite, manual flag edit, reset, archive). If the user asks "why did X happen?",
160explain X — don't append a "here's how to undo it" coda. That pattern assumes intent the user didn't
161signal. Conditional offers like _"if this wasn't intended, you could…"_ or _"want me to revert it?"_
162count as preemptive too — only the user explicitly naming the reversal action ("how do I undo this?",
163"can I roll back ship-variant?", "how do I get the 50/50 split back?") is a request to surface
164reversal mechanics.
165 
166Use consistent terminology: variant _split_ (between variants) is distinct from _rollout_ (overall %
167entering); the `$feature_flag_called` exposure event is distinct from a _custom exposure event_; the
168_Exclude_ / _First seen_ options control multivariate handling, not exposure.
169