Measurement, Validity & Reliability

A scientific result is only as meaningful as the measurement that produced it.

Before asking whether two groups differ, whether a treatment works or whether an exposure is dangerous, we need to know what was actually measured and how well that measurement represents the thing the claim is about.

Measurement error is unavoidable

Every measurement contains some uncertainty.

A thermometer has finite precision. A laboratory assay varies. A questionnaire depends on wording and interpretation. A human observer may score the same event differently on different occasions.

Measurement error does not make data useless.

The question is whether the error is small enough, understood well enough and distributed in a way that still allows the intended inference.

Reliability asks about consistency

A reliable instrument produces sufficiently consistent results under appropriate repeated conditions.

But reliability is not the same as perfect agreement.

COSMIN distinguishes reliability from measurement error because they answer different questions.

A measurement can have small absolute error but poor ability to distinguish individuals in a very homogeneous population.

Conversely, an instrument can rank individuals consistently while still having non-trivial absolute error.

Validity asks whether we measured the right thing

A measure can be reliable and invalid.

A scale that is always five kilograms too high may still rank people consistently.

A questionnaire can produce stable scores while measuring a different construct from the one intended.

Validity therefore asks whether the interpretation attached to the measurement is justified.

Depending on the field, this can involve:

content validity;
construct validity;
criterion validity;
structural validity;
predictive validity;
other domain-specific validation.

Proxies need justification

Scientific studies often use proxy measurements because the target outcome is difficult to observe.

Examples include biomarkers, surrogate endpoints, self-reported behavior and model-derived estimates.

A proxy is useful only if the relationship between the proxy and the target is strong enough for the intended conclusion.

Detecting a biomarker change does not automatically establish improvement in health.

Measuring emissions does not automatically establish personal exposure.

Measuring exposure does not automatically establish clinical harm.

Each inferential step needs support.

Misclassification can create or hide associations

If people are placed into the wrong exposure or outcome categories, estimated relationships can change.

Some forms of measurement error dilute associations. Others can create systematic bias.

That is why measurement quality is not merely “noise around the edges.” It can alter the substantive conclusion.

More decimals do not mean more validity

A measurement printed to six decimal places can still be based on a weak construct.

Precision of display should not be confused with precision of measurement, and neither should be confused with validity.

This matters whenever an impressive technical output creates more confidence than the underlying measurement process deserves.

How to evaluate a measure

Ask:

What exactly is the instrument measuring?
How was it calibrated or validated?
Against what reference, if any?
How large is measurement error?
Is reliability adequate in this population?
Does the measure capture the outcome the claim actually concerns?
Are thresholds pre-specified or chosen after seeing the data?

The methodological connection

Testability tells us what needs to be observed.

Measurement quality determines whether the observation can carry the inference.

A precisely stated claim with a poor measurement is still poorly tested.