Statistical Method
Minimum Detectable Effect (MDE): Setting the Sensitivity Bar
The MDE defines the smallest effect size your experiment is designed to detect, and it should be driven by business impact, not by what you hope to see.
The MDE is not the effect you hope to see or expect to see. It's the smallest change in your primary metric that would be meaningful enough to justify shipping the feature. Setting the right MDE is a business decision as much as a statistical one; it directly determines how many users you need and how long the experiment runs.
How to determine your MDE
- Ask: "What's the smallest improvement that would justify the cost of building, shipping, and maintaining this change?"
- High implementation cost (e.g., backend rewrite) → set a larger MDE, because small lifts won't recoup the investment
- Low implementation cost (e.g., copy or UI tweak) → set a smaller MDE, because even modest improvements deliver positive ROI
Typical MDE ranges by metric type
Conversion metrics (e.g., sign-up rate, purchase rate)1–3% relative
Engagement metrics (e.g., sessions per user, DAU/MAU)2–5% relative
Retention metrics (e.g., D7, D30 retention)0.5–2 percentage points
Beyond the theory
If you've got the theory down, see how it plays out in the simulator.
See the simulator