Effect Size, Statistical Uncertainty & Significance

A result can be statistically detectable and practically trivial.

A result can also be important while remaining statistically uncertain because the study is small.

Those statements are not contradictory.

They describe different quantities.

Effect size asks: how much?

An effect size describes the magnitude of a difference or association.

Examples include:

• difference in means;
• risk difference;
• risk ratio;
• odds ratio;
• correlation;
• standardized effect size.

The appropriate measure depends on the question.

An effect size tells us more than whether the result crossed a significance threshold.

A p-value asks a narrower question

A p-value is calculated under a statistical model and a null hypothesis.

Roughly, it describes how incompatible the observed data, or more extreme data, would be with that model if the null hypothesis were true.

It does not tell us:

• the probability that the null hypothesis is true;
• the probability that the alternative is true;
• the size of the effect;
• whether the effect matters;
• whether the study was well designed.

The ASA has emphasized these distinctions because p-values are routinely overinterpreted.

Statistical significance depends on sample size

With a very large sample, tiny effects can produce small p-values.

With a very small sample, substantial effects can produce wide uncertainty and fail to reach a conventional threshold.

That means “statistically significant” is not synonymous with “important,” and “not significant” is not synonymous with “no effect.”

Confidence intervals show precision

A confidence interval summarizes uncertainty around an estimate under the assumptions of the analysis.

A narrow interval suggests greater precision than a wide one.

The interval can also show which effect sizes remain compatible with the data.

For example, a result may be inconsistent with a large benefit but still compatible with a small benefit, no meaningful effect or a small harm.

That is more informative than a binary significant/not-significant label.

Practical importance depends on scale

Suppose an intervention reduces an outcome from 10.0% to 9.8%.

A large study might detect the difference precisely.

Whether that difference matters depends on:

• severity of the outcome;
• treatment cost;
• side effects;
• population size;
• alternatives;
• patient preferences.

Statistical evidence and decision value are related but not identical.

Relative effects can look larger than absolute effects

A change from 2 in 10,000 to 4 in 10,000 is a 100% relative increase.

The absolute increase is 2 in 10,000.

Both are mathematically correct.

They communicate different aspects of the same effect.

This is why baseline risk belongs in risk interpretation.

Do not worship confidence intervals either

Confidence intervals depend on model assumptions, sampling assumptions and the chosen analysis.

A precise estimate can still be biased by poor measurement, confounding, selective reporting or design flaws.

Precision is not validity.

Better result reading

When evaluating a study, ask:

1. What is the estimated effect?
2. On what scale?
3. What is the confidence interval?
4. Which effect sizes remain plausible?
5. Is the magnitude practically meaningful?
6. How large is the baseline risk?
7. Are the design assumptions credible?

The useful summary

A strong interpretation describes:

direction + magnitude + uncertainty + practical context

rather than stopping at:

p < 0.05.