Statistical Method
Significance Level (α): Controlling False Positives
The significance level sets your tolerance for false positives. How willing are you to declare a winner when there's actually no real difference?
The significance level (α, pronounced "alpha") is the maximum probability you're willing to accept of making a Type I error: concluding that your treatment had an effect when it actually didn't. In other words, it's your false positive rate.
α = 0.05 (industry standard)
You accept a 5% risk of declaring a winner when there's no real effect
Adjusting α based on risk
High-stakes changes
α = 0.01
Use for pricing, onboarding, or core flows where a wrong decision is costly
Low-stakes changes
α = 0.10
Acceptable for button colors, copy tweaks, or easily reversible changes
Best practice
Default to two-tailed tests, which check for effects in both directions (increase or decrease). Only use a one-tailed test if you have a strong, pre-registered justification for expecting the effect in only one direction.
Beyond the theory
If you've got the theory down, see how it plays out in the simulator.
See the simulator