Evidence Quality, Risk of Bias & Certainty

Evidence strength is not determined by whether a paper exists.

A study can report a dramatic result while leaving substantial uncertainty.

A modest result can be supported by a large, consistent and low-bias evidence base.

Certainty asks how confident we should be that the estimated effect is close enough to the truth for the conclusion being made.

Risk of bias asks whether the result may be systematically distorted

Bias is different from random error.

Random error produces imprecision.

Bias pushes results systematically away from the quantity we want to estimate.

Potential sources include:

• problems with randomization;
• unblinded outcome assessment;
• selective dropout;
• missing data;
• selective outcome reporting;
• confounding;
• measurement bias;
• publication bias.

Different study designs have different bias structures.

Study design is not the final quality score

Randomized trials often begin with stronger causal identification for interventions because randomization can reduce confounding.

Observational studies can still provide strong evidence, especially for harms, prognosis, rare outcomes and exposures that cannot be randomized.

A design label tells us what questions to ask.

It does not replace appraisal.

GRADE evaluates a body of evidence

GRADE formalizes certainty judgments across domains such as:

• risk of bias;
• inconsistency;
• indirectness;
• imprecision;
• publication bias.

Under some circumstances, evidence can also become more convincing because of features such as large effects or dose-response patterns.

The exact rules depend on the question and framework.

The important principle is broader: certainty is multi-dimensional.

Inconsistency can mean more than noise

If studies produce very different effects, possible explanations include:

• chance;
• population differences;
• dose differences;
• different measurements;
• different implementation;
• bias;
• genuine effect modification.

A responsible synthesis investigates those possibilities rather than averaging away the disagreement.

Indirect evidence may still be useful

Evidence can be indirect because it studies:

• another population;
• another dose;
• another intervention;
• a surrogate outcome;
• another comparator.

Indirect evidence can be the best available evidence.

Its limitation is that an extra inferential step separates the study from the exact claim.

That extra step should remain visible.

Imprecision limits what we can conclude

A study may be compatible with substantial benefit and substantial harm.

In that case the correct conclusion is uncertainty, even if the point estimate looks impressive.

Larger samples often narrow uncertainty, but sample size alone does not remove bias.

Publication and reporting matter

A literature where negative studies disappear can make an intervention look more effective than it is.

Likewise, reporting only favorable outcomes from a trial changes the visible evidence base.

Evidence appraisal therefore extends beyond the numbers printed in one paper.

Quality is not the same as status

Claim by Claim separates:

Evidence quality: how reliable and informative is the evidence base?

from:

Evidence status: what does that evidence support about the claim?

A high-quality evidence base can show that an effect is tiny.

A low-quality evidence base can suggest a huge effect without justifying confidence.

The practical certainty question

Ask:

If another careful team investigated the same claim with better data, how surprised would we be if the conclusion changed materially?

The lower the certainty, the more room the answer should leave for revision.