M14 · THE PRACTITIONER'S WORKSHOP
Something's off here.
An experienced assessor is handed a 200-page manufacturer's dossier on a Monday morning. By the time coffee is made, they've flipped through perhaps a fifth of it — and they look up. "Something's off here," they say. "Check the comparator, and look hard at the tail on that survival curve." They haven't read all two hundred pages. They haven't run a single calculation. And yet they've already pointed, correctly, at the two things in the dossier that matter most.
That isn't magic, and it isn't "experience" used as a cop-out for reasoning too vague to spell out. It's compressed intuition — fourteen modules of a course folded so tightly that they fire in seconds, as a feeling, arriving well before the conscious mind can explain why. This final lesson is about that compression. It's about the reflexes practitioners call red flags: what they really are, what they are not, and how the entire rest of this course is quietly present every time one fires.
A red flag is a signature, not the problem.
Start by being precise about what a red flag actually is, because getting this wrong is the most common way to misuse one. A red flag is not the problem itself. It's the problem's signature — the trace a problem leaves on the surface of a dossier, the way it shows up in how something is written, before you've proven anything about whether it's true. Experienced assessors read how something is presented, not only what it says: a comparator chosen a little too conveniently; an endpoint that's a little too flattering; a survival tail that's a little too optimistic; a sensitivity analysis that's a little too calm; a section that's a little too absent. None of these is the problem. Each is the shape the problem makes as it reaches the page.
Saying "something's off" hasn't proven anything yet — it has detected a signature. A red flag is a pointer, generated by pattern-matching the dossier in front of you against everything you already know a dossier can go wrong in. And "everything you already know" is a specific, answerable question: matching against what? The honest answer is the whole course. Every flag you'll meet in this lesson is really a compressed pointer back to something you've already spent real time studying.
Every flag is a shortcut to a module.
Once you see red flags this way, a pattern falls out immediately: each one indexes a module you've already studied. Run through five of the most common:
- "Compared to placebo, not standard of care." → Module 1, PICO and the comparator.
- "Only a surrogate endpoint." → Module 6, hard outcomes vs the surrogate endpoint.
- "An optimistic tail on the survival curve." → Module 8, modelling and extrapolation.
- "The sensitivity analysis barely moves." → Module 9, uncertainty and sensitivity analysis.
- "Results from a subgroup not in the protocol." → Module 2, internal validity and data dredging.
Running your eye down a dossier catching flags is, quite literally, re-walking the entire course at the speed of a glance. And that's why flags are only available to someone who actually has the modules to compress — a beginner can't shortcut what they never built in full. Pattern-recognition isn't a substitute for the knowledge; it is the knowledge, running at the speed of intuition rather than the speed of a first read.
Signal, not verdict.
There's a specific danger that appears right after someone first learns to spot flags, and it's worth naming before you fall into it. You spot the placebo comparator, feel the little jolt of recognition, and think: gotcha. You reject the dossier on the strength of the flag alone. That's mistaking the signal for the verdict.
A flag says "look here." It does not say "this is wrong." It's a signal, not a verdict — a hypothesis worth checking, not a conclusion you've already reached. And every flag you'll meet has an innocent explanation sitting right next to the guilty one. Placebo is the correct comparator when no standard of care exists. A surrogate endpoint is the honest choice when the hard outcome would take twenty years to mature. An optimistic extrapolation is right when the biological mechanism genuinely justifies a durable effect. The skill isn't spotting the flag — that part is fast, almost automatic. The skill is adjudicating it: is this instance innocent, or guilty? And that's exactly the slow, careful critical-appraisal work from earlier in this module, now applied to a signal your intuition handed you rather than to the whole document from scratch.
The expert's response to a flag is "interesting — why?" The novice's response is "caught you." One is inquiry; the other closes the case before it's opened. Curiosity keeps you honest; conviction is how flags turn into prejudice.
Scan the dossier.
Here are six elements pulled from a manufacturer's dossier. For each, decide first whether it raises a red flag — then, for the ones that do, decide whether it's a real problem or a signal that turns out to be justified once you check why. Remember: a flag says "look here," not "this is wrong." (Watch closely for the two that share a signature but not a verdict.)
Compared against placebo. An established standard-of-care treatment is in routine use for this condition.
Real problem, or justified once you check why?
Compared against placebo. No treatment currently exists for this ultra-rare condition.
Real problem, or justified once you check why?
The only endpoint is a surrogate marker; a hard clinical outcome was measurable in the trial's timeframe but isn't reported.
Real problem, or justified once you check why?
Beyond the trial data, the survival curve has a long optimistic tail; alternative fits aren't shown.
Real problem, or justified once you check why?
No subgroup analysis appears anywhere in the dossier, though the overall result looks strong.
Real problem, or justified once you check why?
Compared against the current standard of care, on a hard clinical endpoint, over a justified time horizon.
Cards resolved: 0 of 6 · Flags found: 0 of 5 · Adjudicated correctly: 0 of 6
The placebo pair: same signature, opposite verdict — that's signal-not-verdict in one move.
Notice the two placebo cards. They open with identical text — a placebo comparator — and both correctly trigger the same flag. But they resolve in opposite directions, because the reason behind the choice is different in each: one hides a real standard of care, the other has none to hide. Flagging both is correct recognition. Rejecting both, or accepting both, on the strength of the shared signature alone, is the beginner's trap this lesson exists to close.
Now you.
Each red flag is a shortcut back to a module you've already studied. Given the observation, which module does it point to?
1. Compared to a weaker treatment instead of the current standard of care.
2. Relies on a surrogate marker rather than a hard clinical outcome.
3. Survival extrapolated with an optimistic tail far beyond the data.
4. The sensitivity analysis varies trivia and declares the ICER robust.
5. Results come from a post-hoc subgroup not specified in the protocol.
6. A model built on a favourable, unjustified structural assumption.
The hardest flags are absences.
Every flag you've met so far has been a presence — something written on the page that shouldn't be there, or written in a way that gives itself away. The most dangerous constructions in a dossier don't work that way at all. They work by omission, and omission is a much harder thing to see.
A dossier that never mentions the real standard of care, because the drug would lose against it. A sensitivity analysis with no pessimistic scenario, because a pessimistic scenario would sink the result. Strong overall results with no subgroup breakdown, because half the subgroups don't actually work. A surrogate endpoint with no hard endpoint reported at all, because the hard endpoint came out worse. None of these can be read off the page — each has to be inferred from a gap, by asking a question the dossier itself will never prompt you to ask: what would I expect to see here that isn't here?
These absences are only noticeable if you're carrying a complete map of what an honest dossier should contain — which is, again, the entire course. A beginner working from half the modules sees a full-looking page and moves on. An expert working from all fourteen sees the exact shape of the hole.
Recognition guides; it never replaces.
There's one last misuse of pattern-recognition to close off, and it's the most tempting one precisely because it feels like mastery. It's the thought: "I felt the problem instantly, so I don't need to do the slow work." That's exactly backwards.
Pattern-recognition guides analysis. It never replaces it. A flag tells you where to start digging — which is genuinely valuable, because it saves you from grinding uniformly through two hundred pages looking for a needle that could be anywhere. But then you still have to dig: reconstruct the base case, check which direction each lever was turned, adjudicate whether the flag is innocent or guilty, and be ready to defend that judgement impartially in the room. Rejecting a dossier on a flag alone isn't expertise; it's prejudice wearing expertise's clothes.
Intuition without analysis is bias with good instincts. Analysis without intuition is a slow, unguided slog through material that mostly doesn't matter. Practising HTA well means running both together — the glance that tells you where to look, and the discipline that tells you whether what you found is real.
What should this pattern make you do?
A dossier's sensitivity analysis varies drug price (±5%), administration cost (±10%), the discount rate (±0.5 percentage points), and three minor resource-use assumptions — and nothing else. The ICER stays comfortably under the threshold in every scenario shown. The extrapolation method and the comparator choice are never varied. What should this pattern make you do?
Why this matters for HTA
Here's where this lands on your desk: every dossier you're ever handed will contain some genuine strengths, some genuine weaknesses, and a handful of signatures that could be either. What you do with the first flicker of recognition determines whether the rest of your analysis is efficient or wasted.
- Trust the "something's off" — then earn it. The flicker of recognition is real information, compressed from everything you know; don't dismiss it as unfounded. But it earns nothing on its own — the actual appraisal work is what turns a hunch into a defensible conclusion.
- A flag is a signal, never a verdict. Every recurring pattern in a dossier has an innocent explanation sitting beside the guilty one. Treating a flag as proof, rather than as a question worth asking, converts pattern-recognition into prejudice.
- Learn to read what isn't there. The hardest, most consequential flags in a real dossier are absences — the comparator never mentioned, the scenario never run, the subgroup never shown. Reading them requires holding a complete map of what an honest dossier looks like, which only comes from knowing the whole field.
An expert doesn't read a dossier slower than a beginner — they read it faster, because most of the reading has already happened, once, properly, the first time they learned each of these modules. A red flag is your whole training, folded into a glance, arriving before you've consciously started to look.
Red flags, in one breath.
- A red flag is a signature, not the problem itself — the trace a problem leaves on the surface of a dossier, visible in how something is written before you've proven anything about whether it's true.
- Every flag is a shortcut back to a module you've already studied — spotting one at a glance is really re-walking the whole course at speed, and that shortcut is only available to someone who built the modules it compresses.
- A flag is a signal, never a verdict — every signature has an innocent explanation beside the guilty one, so the fast part is spotting it and the real work is adjudicating why it's there. And the hardest flags of all are absences: the things a dossier carefully never shows you.
- Pattern-recognition guides the analysis; it never replaces it. A flag tells you where to dig, not what you'll find — reject a dossier on a flag alone and you've swapped expertise for prejudice.
You began this course asking what HTA even is. You end it able to pick up a stranger's two-hundred-page dossier and feel, within minutes, where its weaknesses are likely to hide — not because you've stopped needing the analysis, but because the analysis is now fast enough to run as a glance.
That is the last red flag, the last lesson, and the end of the course. You've gone from the first question — what is health technology assessment, and why does it exist? — all the way to the practitioner's glance that reads a dossier's weaknesses before the analysis begins. Everything in between, the fourteen modules of it, now lives in that glance. Go and use it well.