M4 · EVIDENCE SYNTHESIS — LESSON 2
Where did every study go?
A systematic review promises an auditable denominator. PRISMA is how it shows the receipts — and, for an assessor, how you check that none went missing.
Last lesson ended on a promise: a systematic review fixes its set of included studies before the results are known, so the denominator is auditable.
But "auditable" is just a promise until someone actually audits it. If a review tells you it included twelve studies, how do you know it didn't quietly drop a thirteenth that pointed the wrong way?
You demand the receipts. You make the review account for every single record it ever touched — from the first database hit to the final included study — and show you where each one went, and why.
That accounting has a standard format. It's called PRISMA, and learning to read it is one of the most practical skills an assessor has.
What PRISMA is, and what it isn't
PRISMA stands for Preferred Reporting Items for Systematic reviews and Meta-Analyses. It comes in two parts:
- a checklist of 27 items — did the review report its question, its search, its methods, its funding, and so on?
- a flow diagram — where did every study go, from search to inclusion?
Before either of those, PRISMA asks the most revealing question of all: did you write down a protocol before you started, and register it — for example on PROSPERO? A protocol pre-declares the question, the search, and the inclusion rules. Registering it timestamps the plan, so a reader can check that the rules weren't quietly rewritten once the results came in. It's the same logic as last lesson — the rule comes first — made into a public record.
The trap
PRISMA is a reporting standard, not a quality standard. It tells you whether a review is transparent, not whether it is good. A review can follow PRISMA to the letter and still be built on a biased search; a genuinely excellent review can be written up sloppily. Transparency is not rigour. What PRISMA buys you is the ability to check — and that turns out to be worth a great deal.
The flow diagram is a funnel
The flow diagram tracks a search as it narrows from everything found to the few studies that survive. Here is a complete one.
Notice three different words. Records are raw database hits — citations, with plenty of duplicates. Reports are the actual documents you retrieve and read. Studies are the research projects themselves. They matter because one study can produce several reports — a main paper, a follow-up, a conference abstract. So the reports→studies step is the one place a count can legitimately shrink without anything being "excluded": three reports of the same trial collapse into one study. Everywhere else, a drop means an exclusion — and an exclusion needs a reason.
The hidden rule
Once you see it, you can't unsee it: a flow diagram is a ledger.
records in = records excluded + records carried forward
Nothing appears from nowhere; nothing vanishes without a reason. The whole chain has to close.
Here is a different review. Fill the gap using the rule.
How many records moved on to screening?
860 − 110 = ?
Right — 750 records survive de-duplication and go on to screening. Every record is accounted for: 110 removed, 750 carried forward, 860 total. The ledger closes.
Close the chain
The same review continues. This time walk the ledger rule through more than one step.
How many studies were included in the review?
Start from the 60 reports sought, then subtract what dropped out.
60 − 4 − 47 = ?
Nine. Sixty reports were sought; four couldn't be retrieved and forty-seven were excluded after full-text review, each with a reason. The chain closes on nine included studies — and a reader can verify every single number.
When the numbers lie
Now the assessor's move. Here is a manufacturer's submission flow diagram. It looks tidy. But the ledger rule lets you check it in seconds. Tap the stage where the chain breaks.
Run the rule: in = excluded + carried forward. (a) 1,500 − 300 = 1,200 ✓. (d) 200 − 188 = 12 ✓. Keep checking.
Found it. After screening, 1,200 − 1,050 = 150 reports should have been sought — but the diagram claims 200. Fifty reports walked in from nowhere. Every number below inherits the error, so the tidy-looking "12 included" rests on a count that doesn't add up. You don't need to read a single trial to know this submission has a question to answer.
Why this matters in an HTA
A systematic review in an HTA dossier is the foundation of everything else. If the denominator is wrong, the pooled effect is wrong, the cost-effectiveness model is wrong, and the reimbursement decision is wrong. PRISMA is how you kick the tyres before you trust the foundation.
- Completeness check: if the flow diagram says 1,200 records screened but only 800 are accounted for as excluded or carried forward, where did the other 400 go? That single arithmetic check can surface a cherry-picked sample before you read a single study.
- Protocol vs report: ask for the registered protocol and lay it alongside the published review. Any change in the inclusion criteria, outcome, or analytic approach that appeared after the results were known needs an explanation — or it's a red flag.
- Exclusion reasons matter: a flow diagram that lumps everything into "not relevant" is hiding information. PRISMA requires reasons at the full-text stage. When reasons are missing, so is accountability.
You don't audit a review by reading it carefully. You audit it by doing the arithmetic.
PRISMA, in one breath
- A PRISMA flow diagram accounts for every record from the first database hit to the final included study — nothing can disappear without a reason.
- The ledger rule — in = excluded + carried forward — closes at every stage. Any stage that doesn't close is an error or an omission worth flagging.
- PRISMA is a reporting standard, not a quality standard. Transparency is a necessary condition for trustworthiness, not a sufficient one.
- A registered protocol timestamps the inclusion rules before the results are known. Its absence doesn't disqualify a review, but its presence raises confidence.
One flow diagram, checked with arithmetic, tells you more than three hours of reading the methods section.
Next, we look at what happens once the included studies are assembled: how to pool results statistically, when a meta-analysis is appropriate, and what a forest plot is telling you. The question-framing skills from PICO come back here — the included population, the comparator, and the outcome all determine whether pooling is even appropriate.