Module 3 · Variation & Uncertainty

Two identical studies. Two different answers. Nobody lied.

Two research teams test the very same drug, the same way, with the same flawless methods. One finds it helps 35% of patients. The other finds 25%. Neither made a mistake. Neither cheated.

So which number is the truth?

A sample is a lottery

Picture the full truth you can never see: every patient who could ever take this drug, and the real proportion it helps. Call that the population, and its real answer the truth. It's fixed — but hidden.

A study can't reach the whole population. It grabs a sample — a few hundred patients — and measures them. That sample's answer is your estimate of the truth.

But here's the catch: which patients happen to land in your sample is down to chance. A few extra responders by luck, and your number drifts up. A few extra non-responders, and it drifts down. Same drug, same truth — different draw, different answer.

The number a study reports is not the truth. It's one sample's guess at the truth — and the guess wobbles. The only question that matters is: how much?

Feel it yourself

Let's stop describing it and let you do it. Below is a drug whose real effect is fixed — it truly helps 30% of patients (the dashed line). But you don't get to see that truth directly. You only get to run a study and see what your sample says.

Set the sample size, then run the study — again and again. Watch where the results land.

Try this: set N small (say 10) and run it ten times. Then drag N up to 1000 and run it ten more.

Where does that average come from?

The simulator hands you a single number — the sample mean. But it's nothing mystical. You add the readings up and divide by how many there are:

The law you just discovered

You've just uncovered one of the most important facts in all of statistics:

A result from a small sample bounces around a lot. A result from a large sample sits close to the truth. Uncertainty shrinks as the sample grows.

This wobble has a name: sampling variation — the variation in your answer caused purely by which individuals happened to be sampled. It isn't a mistake. It's built into the act of sampling itself.

And it shrinks in a specific, reliable way as the sample grows — a relationship precise enough to put a number on (that's the next few lessons). For now, hold the shape of it: bigger sample, steadier answer.

This is "chance," made formal

Remember the three enemies from M2 — chance, bias, confounding? You're now looking the first one square in the face.

Sampling variation is chance. It's the reason a small study can show a striking effect that simply evaporates when the study is repeated larger. And notice what's special about it: of the three enemies, chance is the only one a bigger sample can cure. Double a biased study and you get a bigger biased answer; double a small study and the chance genuinely shrinks.

That's why "how many patients?" is the very first question to ask of any result. Not to be pedantic — because the answer tells you how much of what you're seeing might just be the luck of the draw.

Which result would you trust?

Quick gut-check. Two studies report the same result — a drug helping 40% of patients — but at very different sizes:

• Study A: 40% — out of 20 patients.
• Study B: 40% — out of 1,000 patients.

If each were run a second time, one result would barely move and the other could swing wildly.

So a single number is never enough

If the truth is hidden and your sample only estimates it, then reporting one bare number is dishonest by omission. It hides the wobble.

The honest move is to report not just your best guess, but a range of values the truth could plausibly take — wide when you're unsure, narrow when you're confident.

That range is what statisticians call a confidence interval, and giving it properly is the subject of a lesson very soon. For now, just carry the instinct: a result without a sense of its uncertainty is only half-reported.

Why this matters for HTA

This is where it bites on your desk. Back in the very first lesson, one of the statements you sorted was: "The trial included only 38 patients, so the result could be down to chance." Now you know exactly what that means — and why it's often the first thing an assessor checks.

A manufacturer can show you a striking number from a small study. Your reflex is now automatic: how big was the sample, and how much could this be the luck of the draw? A dramatic effect in 38 patients and the same effect in 3,800 are not the same evidence — even if the percentage is identical.

Never read a result as a fact. Read it as an estimate with a wobble — and let the sample size tell you how big the wobble is.

Variation & uncertainty, in one breath

• A study measures a sample, not the whole truth — and which individuals get sampled is down to chance.
• So the same drug, studied flawlessly twice, gives two different numbers: sampling variation.
• The mean is nothing mystical: sum the values, divide by n — one number standing in for the whole sample.
• Small samples bounce a lot; large samples sit close to the truth. Uncertainty shrinks as the sample grows.
• This is chance from M2 — the one enemy a bigger sample actually cures.
• A single number, with no sense of its uncertainty, is only half-reported.

A result is not a fact. It's one sample's estimate of a hidden truth — and the smaller the sample, the more it lies.

You can now feel uncertainty. Next, we learn to describe its shape — why so many measurements pile up into the same bell-shaped curve, and how that curve becomes the ruler we measure uncertainty with.