Configuring Experiment Rollout

1---
2name: configuring-experiment-rollout
3description: Configures the rollout shape of a PostHog experiment — the variant split (50/50, 80/20, A/B/C ratios), the overall rollout percentage that gates how many users enter the experiment, and the disambiguation when a percentage like "roll out to 25%" could mean either. Use when the user mentions a rollout percentage, variant split, or traffic distribution; gives a ratio like 60/40, 70/30, or 80/20; asks "who sees the test variant?"; wants to increase, decrease, or change the rollout or split on a draft or running experiment; weighs equal vs uneven splits; or proposes a mid-experiment split change (often an anti-pattern that needs reset or end-and-restart).
4---
5 
6# Configuring experiment rollout
7 
8This skill answers: **Who sees what variant?**
9 
10## Recommended approach: equal split + adjust rollout percentage
11 
12In most cases, experiments work best with an equal split. If you want to limit exposure to the test variant, adjust the rollout percentage instead.
13 
14Why equal splits are better:
15 
16- Equal splits maximize statistical power — each variant has the same sample size
17- Equal splits balance traffic and thus reach significance faster
18- Increasing user exposure throughout the experiment through increasing rollout is clean (changing split mid-experiment can cause users to switch variants, which is bad for user experience and data quality)
19 
20Always default to an equal split unless the user explicitly requests otherwise.
21 
22## When an uneven split is required
23 
24Uneven splits combined with the default "Exclude multivariate users" handling can introduce bias.
25If the experiment observes multi-variant users (users exposed to more than one variant) then those are
26dropped asymmetrically — the smaller variant loses a larger fraction of its assignments. If those users
27behave differently from the rest, the smaller variant's metrics will be skewed.
28 
29The right mitigation depends on experiment state:
30 
311. **Pre-launch, or live but with few exposures so far — use an equal split and reduce the overall
32   rollout.** Achieves the same test-variant exposure without the bias and preserves statistical
33   power. See the disambiguation question below.
342. **Live experiment with significant exposures — switch multivariate handling to "First seen
35   variant".** Changing the split mid-run reassigns users across variants (anti-pattern; see
36   "Changing rollout on a running experiment" below). Switching handling instead keeps everyone in
37   their original variant and avoids the asymmetric exclusion. See `configuring-experiment-analytics`
38   for how to set this. Note that "first seen" handling can introduce other biases, but it's
39   preferable to mid-run reassignment.
40 
41## The two rollout controls
42 
43There are two separate controls that determine who sees what. Both are set via `parameters`.
44 
45### 1. Variant split (`parameters.feature_flag_variants`)
46 
47How users **inside** the experiment are distributed across variants.
48 
49- Array of `{key, name, split_percent}` — percentages must sum to 100
50- First variant must have key `"control"` — this is the baseline
51- Minimum 2 variants, maximum 20
52- Default: control 50% / test 50%
53 
54If the user says "A/B/C test", map the baseline to `"control"` and create additional variants for the others.
55 
56### 2. Overall rollout (`parameters.rollout_percentage`)
57 
58What percentage of **all** users enter the experiment at all. Default: 100%.
59 
60Users not included are excluded entirely — they don't see any variant and are **not part of the analysis**.
61 
62### How they interact
63 
64These two controls multiply:
65 
66| Overall rollout | Variant split      | % seeing test | % in analysis |
67| --------------- | ------------------ | ------------- | ------------- |
68| 100%            | 50/50              | 50%           | 100%          |
69| 100%            | 75/25 control/test | 25%           | 100%          |
70| 50%             | 50/50              | 25%           | 50%           |
71| 25%             | 50/50              | 12.5%         | 25%           |
72 
73## The disambiguation question
74 
75**CRITICAL**: If the user requests an uneven variant split (e.g. "60/40", "70/20/10") or mentions a
76specific percentage that could refer to either the split or the rollout (e.g. "roll out to 25%"), you
77MUST clarify before proceeding. This covers two cases:
78 
79### Case 1: Single percentage ("25%", "roll out to 40%")
80 
81The percentage is ambiguous — it could mean a variant split or a rollout change. Ask:
82 
83> There are two ways to get 25% of users seeing the test variant:
84>
85> 1. **Reduced rollout with equal split** (recommended): reduce the overall rollout and split
86>    variants equally. Only a subset of users enter the experiment, and of those, each variant
87>    gets the same share.
88>    Equal splits maximize statistical power and avoid bias.
89> 2. **Asymmetric split**: keep 100% rollout but give the test variant only 25%.
90>    All users enter the experiment, but the uneven split reduces power on the smaller variant
91>    and risks bias.
92>
93> Which approach do you prefer?
94 
95Adjust the numbers to match whatever percentage the user requested.
96 
97### Case 2: Uneven ratio ("60/40", "70/30", "80/20", etc.)
98 
99The ratio looks like an explicit variant split, but a reduced rollout with an equal split is almost
100always better. Explain the trade-off and recommend the alternative:
101 
102> An uneven variant split works, but an equal split with reduced rollout is recommended:
103>
104> 1. **Equal split + reduced rollout** (recommended): reduce the overall rollout so that the same
105>    fraction of users sees the test variant, but split variants equally within the experiment.
106>    Equal splits maximize statistical power and avoid bias from asymmetric multivariate exclusion.
107> 2. **Uneven split**.
108>    Achieves the same user-facing outcome, but reduces power on the smaller variant and risks bias.
109>
110> Would you like the equal split approach, or do you have a specific reason for the uneven split?
111 
112Adjust the numbers to match the ratio. For experiments with more than two variants, "equal" means
113each variant gets the same share (e.g. 34/33/33 for three variants). If the user confirms they want
114the uneven split after seeing the trade-off, proceed — but DO NOT skip the next section.
115 
116### After the user picks the uneven split
117 
118If the user proceeds with an uneven split (option 2 in either case above), you MUST surface the
119multivariate-handling implication BEFORE creating or updating the experiment. The user has chosen
120the riskier rollout path and needs to make an informed choice about how to mitigate.
121 
122Ask:
123 
124> One more thing — with an uneven split, the default "Exclude multivariate users" handling drops
125> users exposed to multiple variants asymmetrically. The smaller variant loses a larger fraction of
126> its assignments, which can skew its metrics if those users behave differently from the rest.
127>
128> Two options:
129>
130> 1. **Switch multivariate handling to "First seen variant"** (recommended for uneven splits) —
131>    keeps all users in the analysis and avoids asymmetric exclusion. Has its own caveats (other
132>    biases can creep in) but is preferable to the default for uneven splits.
133> 2. **Keep the default "Exclude" handling** and accept the bias risk.
134>
135> Which would you like?
136 
137See `configuring-experiment-analytics` for how to set the multivariate handling. Apply the choice
138as part of the same operation (creation or update) — do not leave the user with an uneven split
139under default handling without an explicit, informed decision.
140 
141## Persist flag across authentication steps
142 
143This option (`ensure_experience_continuity` on the feature flag) is only relevant when:
144 
145- The feature flag is shown to **both** logged-out AND logged-in users
146- You need the same variant assignment before and after login
147 
148This is not compatible with all setups. Learn more: https://posthog.com/docs/feature-flags/creating-feature-flags#persisting-feature-flags-across-authentication-steps
149 
150Only mention this to the user if their use case involves pre/post-authentication experiences.
151 
152## Resolving experiments
153 
154Rollout changes require an experiment ID. If the user refers to an experiment by name
155or description (e.g. "change rollout on my signup test"), load the `finding-experiments`
156skill to resolve it to a concrete ID before proceeding.
157 
158## Changing rollout on a running experiment
159 
160**Any change to rollout or variant split on a running experiment affects both user experience and statistical validity.**
161You MUST warn the user and get explicit confirmation before making the change.
162 
163Do NOT silently apply the change — even if the user asked for it directly.
164Present the warning covering both perspectives:
165 
1661. **Who sees what variant?** — will users switch variants or lose a feature?
1672. **Who is in my analysis?** — how does this affect data quality?
168 
169**Exception**: Increasing rollout (without changing the split) is generally safe — no users switch variants, more users are added cleanly.
170 
171**Mid-experiment fix for uneven-split bias**: switching multivariate handling from "Exclude" to "First
172seen variant" is the recommended mitigation for already-launched experiments — no users switch variants
173and all collected data stays in the analysis. Changing the split to be even is an anti-pattern mid-run
174(typically requires resetting or ending the experiment) and is only preferred if the experiment hasn't
175been exposed to many users yet. See `configuring-experiment-analytics` for how to change the handling.
176 
177See `references/changing-distribution-after-launch.md` for detailed warnings, what to tell the user, and when to recommend alternatives.
178