Module 2 · Bias, Chance & Confounding

One wrong answer. Three completely different reasons.

A study reports that a vitamin prevents heart attacks. It's wrong — the vitamin does nothing. But why it's wrong could be one of three completely different things:

• The study was tiny, and the result was a fluke.
• The study only surveyed health-magazine subscribers, who were healthier to begin with.
• Vitamin-takers also exercise more — and it's the exercise protecting their hearts.

Three wrong answers. Three different culprits. Same innocent-looking conclusion.

Test 1: would a bigger study fix it?

Here's the single fastest way to start sorting the three. Ask one question:

Would running the study on far more people make the problem go away?

For one of the three enemies — chance — the answer is yes. Luck washes out as numbers grow. Flip a coin ten times and you might see eight heads; flip it ten thousand times and you'll sit close to half. More data drowns out flukes.

For the other two — bias and confounding — the answer is a flat no. Make a biased study a hundred times bigger and you simply get a bigger, more confident wrong answer. Size never fixes a tilt; it just sharpens it.

That one test splits the field in half: chance on one side, the two "systematic" enemies on the other.

Chance, up close

Chance is pure random noise — the luck of who happened to end up in your sample.

Its fingerprints:

• It has no direction. It's as likely to make a useless drug look good as to make a good drug look useless.
• It shrinks with size. Big numbers tame it; small studies are riddled with it.
• It's the enemy statistics was built to handle — measuring exactly how much "could just be luck" is the whole job of the tools you'll meet in M3.

So when you see a dramatic result from a handful of patients, your first reflex should be: is this just chance? You're not doubting anyone's honesty — you're respecting how loudly small numbers lie. The cure is simple to name and hard to do: study more people.

Test 2: where does the tilt come from?

Now the harder split — between the two enemies that a bigger study can't fix. They feel similar (both push the answer in a consistent direction) but they come from opposite places.

Bias comes from how you ran the study. Confounding comes from the world itself.

• Bias is a flaw you introduced — in who you let into the study, how you measured them, or who you let drop out. Fix the design, and the bias is gone.
• Confounding is already out there before you start — a real third factor that genuinely affects both things you're studying, creating a true-but-misleading link. No measurement was botched; reality is just tangled.

That's why they need different cures. You'll see. Bias first.

Bias and its four faces

Bias is a systematic tilt — an error that pushes the result the same way every time, built into the study itself. It's almost never dishonesty; it's design. And it sneaks in at four different points in a study's life:

Four faces, one family: a systematic tilt, and never fixed by a bigger sample.

These four are the workhorses, but they're not the whole zoo — there are subtler relatives, like bias in how a treatment is delivered or how outcomes are detected. You'll meet those when we get to formally grading study quality in M4.

Each study below is tilted. Tap where the tilt got in.

Confounding, up close

Now the subtle one — and the one people most often misfile as "just another bias." It isn't.

Confounding is a real third factor that independently affects both the supposed cause and the effect — creating a link that's genuine in the data but misleading about cause.

Take the joggers who live longer. Jogging didn't add the years — being the kind of person who jogs (younger, richer, eats better, doesn't smoke) did. Those traits sit at the top of the triangle, driving both the jogging and the long life. The link between jogging and longevity is real in the numbers — and wrong about cause.

Why this isn't bias: nothing in the study was mis-built. Nobody was wrongly selected, mis-measured, or lost. The tangle was in the world before the study began. And that's the clue to its cure: since the problem is that the groups differ in some hidden trait, the fix is to make the groups identical — which is exactly what randomisation does, and why an RCT shuts confounding out where an observational study can't. When you can't randomise, you try to adjust for the confounder afterward (a tool you'll meet properly in M11).

A note on naming: some textbooks file confounding under the broad umbrella of "systematic error," alongside bias. We deliberately keep them apart, because their cures differ — bias is fixed by repairing the study, confounding by randomising or adjusting. Same family of "non-chance error," but worth separating in your head.

Bias is a broken study. Confounding is an honest study fooled by a tangled world.

Full diagnosis now. For each suspicious result, name the enemy. Two questions guide you: would more data fix it? and where does the tilt come from?

Three enemies, three cures

Diagnosis done. Here's the pay-off — because each enemy has its own cure, naming the enemy tells you exactly what to demand:

A caveat worth keeping: in the real world these three often turn up together, tangled in the same study. Here we trained on clean, single-enemy cases so you'd know each signature. Reading a real submission means spotting all three at once — and there are subtler cousins still (publication bias waits for you in M4). But the three questions you now carry — would size fix it? where's the tilt from? is a third factor at work? — will take you a remarkably long way.

Why this matters for HTA

On your desk, this is not academic. Every weakness in a manufacturer's evidence is one of these three enemies wearing a disguise — and your job is to name it, because the name dictates the remedy you can demand:

• An effect resting on a handful of patients? Suspect chance — ask for more evidence before you believe it.
• A comparison where the groups were measured or selected differently? Suspect bias — ask why, and how it was prevented.
• An observational claim that drug-takers did better? Suspect confounding — ask what made those patients different in the first place, and whether it was adjusted for.

An assessor who can't tell the three apart accepts a confounded result as proof, or dismisses a solid one as a fluke. Naming the enemy is where critical appraisal actually begins.

The three enemies, told apart

• Three ways a result goes wrong — chance, bias, confounding — with different causes and different cures.
• Test 1: would more data fix it? Only chance says yes.
• Test 2: where's the tilt from? Bias is a flaw in the study; confounding is a tangle in the world.
• Bias wears four faces: selection, information, recall, attrition — sorted by where in the study it sneaks in.
• Confounding isn't a bias — it's a real third factor driving both things, which is why randomisation is its cure.

Don't ask "is this result wrong?" Ask "which of the three would make it wrong?" — and you'll know exactly what to demand next.

You can now name the enemies. Next, we meet their most powerful cure head-on: how randomisation and blinding are engineered to shut bias and confounding out — the machinery that makes the RCT the strongest design there is.