Statistical Method
Sample Size & Duration: Planning Your Experiment Timeline
How many users you need and how long to run the experiment: the calculations that ensure your results will be statistically reliable.
For proportion metrics (e.g., conversion rate, click-through rate)
n per group = (Zα/2 + Zβ)² × (p₁(1−p₁) + p₂(1−p₂)) / (p₁ − p₂)²
Where Zα/2 = 1.96 (for α = 0.05, two-tailed) and Zβ = 0.84 (for 80% power). p₁ is the baseline rate; p₂ is the rate you expect after applying the MDE.
For continuous metrics (e.g., revenue per user, session duration)
n per group = (Zα/2 + Zβ)² × 2σ² / δ²
Where σ is the standard deviation of the metric and δ is the minimum detectable difference in absolute terms.
Calculating experiment duration
Duration (days) = (2 × n per group) / eligible users per day
The factor of 2 accounts for both the control and treatment groups needing the full sample size.
Minimum duration
14 days
Captures at least 2 full weekday/weekend cycles
Maximum duration
8 weeks
Longer runs risk contamination from external factors
Best practice
Always round your calculated duration up to complete full weeks, as partial weeks introduce day-of-week bias. Also account for holidays and extend if a major one falls within your experiment window.
Beyond the theory
If you've got the theory down, see how it plays out in the simulator.
See the simulator