What is Statistical Significance and A/B test sample size calculation in product analytics?

Statistical significance quantifies the probability that an observed difference between variants is not due to random variation, and sample size calculation determines how many users you need to detect a target effect with desired confidence and power.

Step‑by‑step workflow

1. Define business goal – e.g., increase conversion from 5% to 5.5% (10% relative lift).
2. Set statistical parameters – α = 0.05 (two‑tailed), power = 0.80, baseline conversion = 0.05.
3. Compute required sample size per variant.

from statsmodels.stats.power import NormalIndPower
power_analysis = NormalIndPower()
required_n = power_analysis.solve_power(effect_size=None,
                                         nobs1=None,
                                         alpha=0.05,
                                         power=0.8,
                                         ratio=1,
                                         alternative='two-sided',
                                         prop1=0.05,
                                         prop2=0.055)
print(f"Required N per group: {int(required_n)}")

4. Randomly assign users ensuring equal allocation (or pre‑defined ratio). Use feature‑flag tools like LaunchDarkly (variation: "control" / "treatment").
5. Collect data for the planned duration or until the calculated N is reached.
6. Run hypothesis test – chi‑square or two‑proportion z‑test.

SELECT
    variant,
    COUNT(*) AS users,
    SUM(conversion) AS conversions,
    AVG(conversion) AS cr
FROM ab_events
GROUP BY variant;

Then in Python:

from statsmodels.stats.proportion import proportions_ztest
import numpy as np
count = np.array([conv_control, conv_treatment])
nobs = np.array([n_control, n_treatment])
stat, pval = proportions_ztest(count, nobs, alternative='two-sided')

7. Interpret – if p‑value < α, reject null and declare the lift statistically significant; otherwise, treat as inconclusive.

Quick reference table

| Metric | Typical threshold |
|-----------------------|-------------------|
| Significance level α | 0.05 (two‑tailed) |
| Statistical power | 0.80 – 0.90 |
| Minimum detectable effect (MDE) | 5 % – 20 % relative lift |
| Allocation ratio | 1:1 (or as needed) |

Follow this checklist to avoid under‑powered tests and false positives.