Multimodality in Medical AI: Evaluating and Simulating Multiclinician Workflow for Brain Lesions
A clinical model can post a strong number and still be wrong about what matters at the bedside, so the evaluation has to match how the model would actually be used.
A clinical model is only as trustworthy as the assumption printed beside its headline number.
When a clinical model reports a gain, is it a gain a clinician could obtain in practice, or one that exists only because the evaluation already assumed part of the answer?
The same worry follows me into the clinic, where a number that looks like a gain can point a radiologist at the wrong part of the brain. A clinical model is usually judged by one benchmark number, and that number can be correct while the conclusion drawn from it is wrong. I build and evaluate clinical models against the decision they would actually support at the bedside. The main study asks whether an anatomical localization prior helps a 3D detector recover epileptogenic lesions it would otherwise miss. Restricting the search to the suspected lobe and looking for several candidate peaks recovers 14 of 25 missed lesions, 56%, against 1.4 expected by chance, with P below 0.001. That number is an upper bound reported under oracle localization, which means it assumes the correct lobe is already known, so I report it as a ceiling rather than an achievable EEG-guided result. Two further threads carry the same standard: a routing policy that decides per patient whether to wait, predict, or defer, and a catalogue of seven ways a clinical evaluation can report the right number and still support the wrong conclusion.
01, The Challenge
The slide number and the bedside number are different things
A clinical model is usually judged by a single benchmark number, and that number can be right while the conclusion drawn from it is wrong. A detector can look strong on average and still miss the hard cases a clinician actually needs help with, and a result that quietly assumes the answer is already known can read as if the model found it. In this work I measure a clinical model against the decision it would really support, and say out loud which assumptions each number rests on.
When a clinical model reports a gain, is that a gain a clinician could obtain in practice, or a gain that only exists because the evaluation already assumed part of the answer?
These lesions can be small and low-contrast, so a detector that works on obvious cases can still fail on the subtle ones that bring a patient to imaging in the first place. The subtle ones are what I care about here, because the obvious lesions do not need a model.
When a method is told which lobe to search, its recovery numbers are an upper bound on what a deployable pipeline could achieve, because a deployable pipeline has to find the region first. Reporting the oracle number is useful, as long as it is read as the ceiling it is.
02, The Approach
Report the achievable number, and label the assumptions
I separate what a method could do under ideal information from what it could do in practice, and I say which is which every time a number appears. Where a result depends on a region being known in advance, I report it as an upper bound under oracle localization, and I put the failure cases next to the gains. The standard is the same across all three threads: a number is only as trustworthy as the assumption printed beside it.
03, Results
Three threads, one standard of evidence
Detecting subtle epileptic lesions
Structural-functional integration for enhanced detection of subtle epileptogenic lesions
ECCV AI4M3D 2026OralSole author
I wanted to know whether anatomical localization can help a 3D detector recover epileptogenic lesions it would otherwise miss. The intuition is simple: a detector that scores every voxel with one global threshold either sets the bar high and loses faint lesions, or sets it low and fills up with false positives. If you already suspect which lobe the lesion is in, you can lower the bar there and raise it everywhere else, so a faint lesion in the right place survives while noise elsewhere is suppressed. The method does that, and then searches for several candidate peaks inside the restricted region rather than trusting a single maximum. The recovery numbers below are upper bounds under oracle localization, which means they assume the correct region is already known rather than derived from EEG.
The zones come from clinical annotations and anatomical parcels, so the pipeline is told where to look. It works from those annotations rather than an EEG trace, which is why every recovery number here is an upper bound rather than an achievable EEG-guided result. Naming that gap is the part that matters.
- Zone-conditioned threshold Lower the bar inside the suspected zone, raise it outside. The method adjusts a 3D nnU-Net detection threshold by anatomical zone defined on SynthSeg parcels, lowering it inside the zone and raising it to 0.90 outside, so a faint lesion in the right place is kept while false positives elsewhere are suppressed.
- Lobe-restricted multi-peak search Look for several candidate peaks within the lobe. Inside the restricted lobe, the method searches for multiple peaks rather than a single maximum, which is what recovers lesions that a single-threshold detector places just below its cutoff.
As an upper bound under perfect (oracle) localization, the lobe-restricted multi-peak search recovered 14 of 25 missed lesions, which is 56%, against 1.4 expected by chance, with P < 0.001. This is an oracle ceiling rather than an achievable EEG-guided result.
Finer subregional restriction added no significant improvement over the lobe, so the lobe is the level at which the localization prior helps, and the right granularity for it.
The failure case worth stating is a wrong prior. Pointing the search at the opposite-side lobe drops recovery from 14 of 25 to 1, below the 9 the model recovers with no prior at all, so a wrong localization actively suppresses true lesions. Tightening the zone also costs, since the subregional zone dropped two cases the lobe had recovered. The prior has to be right, and no finer than the lobe.
The recovery of 14 of 25 assumes the correct lobe is already known. A deployable pipeline would have to find the lobe from data first, so the achievable number is lower than this oracle number. That is why I report it as an upper bound every time it appears, and why the next step is removing the assumption rather than tuning it.
This matters because the lesions that reach imaging are the subtle ones, and a prior that recovers 14 of 25 of the misses would change a patient's path only if a deployable pipeline can supply the lobe without being told it. The oracle number shows the ceiling is high enough to aim for, and the next step is reaching it from measured data.
Limitations. The localization zone is simulated from the confirmed lesion lobe, not from a measured EEG report or raw signal, so every recovery number is a ceiling a measured prior would approach. The recovery hyperparameters, the fusion scale, the flood-fill thresholds, the 0.90 out-of-zone threshold, and the 10 mm margin, were fixed a priori rather than tuned, and the recovered count is stable at 14 of 25 across a sweep of them, though they were set on this cohort with no held-out split and there is no external test set yet. The recovery step still needs a specificity check on healthy controls. The next step is a pipeline that takes localization from measured EEG, which removes the oracle assumption that makes these numbers an upper bound.
Routing EEG localization decisions
Wait, predict, or defer? Evaluating an offline sequential routing policy for EEG localization
NeurIPS HAIC 2026Sole author
A model forced to answer every patient will answer the hard ones too, and a confident wrong localization is worse at the bedside than no answer, because it can send attention to the wrong region. So I evaluated a different design on 166 patients. Instead of always predicting, a policy watches the localizer's output as 10-second EEG segments arrive and decides, at each step, whether to report the current hemisphere, wait for another segment, or defer the patient after a fixed observation budget. Nothing is retrained, the policy replays held-out predictions, and I choose every threshold on training patients only. The policies are compared at 40% coverage, which means each answers the 40% of patients it is most willing to commit to and defers the rest.
What surprised me is that the signal that knows when to answer is agreement across segments rather than the model's own confidence. At 40% coverage, a policy that routes on cross-segment self-agreement, how often the separate segments point to the same hemisphere, answers its patients at 88.4% accuracy, against 70.0% for the model's own confidence, 67.3% for random routing, and 68.7% for answering every patient. The bootstrap gap between self-agreement and confidence is 18.7 points [95% CI +7.3, +30.6] over 4,000 resamples. The sequential policy reaches that accuracy on about 7.9 segments per answered patient, and reading all of a patient's segments before deciding gains almost nothing, 89.0%, while spending about three times as many observations. The tradeoff holds across segment orderings and observation budgets.
At 40% coverage, routing on cross-segment self-agreement answers patients at 88.4% accuracy against 70.0% for the model's own confidence and 68.7% for answering everyone, a bootstrap gap of 18.7 points [CI +7.3, +30.6], so the signal that knows when to answer is agreement across segments rather than confidence.
The useful quantity in the clinic is not average accuracy, it is how often the model is confidently wrong on a case a clinician would have caught. A policy that knows when to abstain can be worth more than one that is slightly more accurate on average, and the way to see that is to evaluate the decision, not the raw score.
This matters because it applies the same standard as thread 1 to deployment, that a model should be judged on the action it supports. Here the action is a triage choice, and the question is whether the policy improves that choice rather than whether it posts a higher number.
Limitations. This is an offline replay of a routing policy rather than an observed clinician-agent team, and a deferral has no measured downstream clinician outcome. Segment order is tested against random within-patient reorderings but is not a prospective arrival log, and the policy runs on cached out-of-fold predictions from a single clinical corpus.
Seven failures in clinical ML evaluation
Right number, wrong conclusion: seven failures in clinical ML evaluation
NeurIPS RCMLR 2026OralSole author
This thread is the standard behind the other two, written down. It catalogues seven corrections to my own EEG-localization work where a statistic was computed correctly and still invited a conclusion that was wrong, each with the one worked number that changed and the claim it would have licensed. The oracle upper bound in thread 1 is the same kind of error these are about, a number that is right but assumes part of the answer, which is why that bound is labeled every time it appears. The catalogue gives a checklist a result has to pass before its number means what it seems to, applied first to my own.
- A denominator that flattered every model 2.00× over chance, but a spatial-blind model still clears 1.57×. A density-matched chance baseline corrects for how many channels a window labels, not which are a priori likely, so a read-out with the spatial structure removed still beats it by learning the class-frequency prior. The corrected floor is a model-free prior-only predictor at 1.37×. It would have licensed reading a chance-denominated margin as spatial skill.
- A threshold that moved when I added seeds A 90% prior beat no-prior at 3 seeds, and the effect reverses at 8. At 8 seeds a 90%-accurate prior is indistinguishable from no prior (p = 0.55), so break-even sits at or above 90%, and the derived claim that the localizer clears the bar does not survive. This one had no cheap diagnostic.
- An ablation that could not fail Coordinates looked inert (2.30→2.13×) because identity reached the head anyway. The read-out indexed tokens by channel position, so sensor identity survived however the coordinates were corrupted. With identity removed first, the same corruption costs 1.53→1.22×. Phrased as a mechanism rather than a measurement, this is the one that would have done the most downstream damage.
- A comparison missing one of its controls Pretraining credited the full 1.604, worth +0.072 against its own twin. I compared two pretrained encoders with only one random-init control. Giving EEGPT its own twin shows its pretrained 1.604 sits against a random-init 1.532, so pretraining is worth +0.072, not the whole number.
- Two ablations at incomparable granularities Sensor dominance (0.068 vs 0.80) is a tie once matched (0.300 vs 0.295). The two costs came from different read-outs at different severities, mid-range on one axis against the extreme of the other, so the dominance claim was an artifact of the comparison rather than the representation.
- The previous correction was also wrong The axes cannot be ranked becomes the sensor axis winning by 14.9 sd. Block permutation whitens rather than removing order, its cost collinear with spectral distortion at r = 0.998, and clean order-only and identity-only probes separate the axes decisively. The first and last claims would have read as a stable finding, with no sign that neither analysis behind them held.
- A single-site result I called cohort-wide A 262-case hurt column was 25 cases, all one site. The sweep capped at a sorted-filename prefix, which was one site. Re-run on the full cohort it is 259 of 262 with signs unchanged, but only then is it the multi-site statement the original number asserted.
Across seven self-reported corrections the value was computed correctly each time and the check changed what it licensed. Six were cheap to notice, a single run or count or question, and the seventh, the break-even threshold, had no cheap diagnostic, which is why I report it as an ordering rather than a threshold.
Each failure is a trap the rest of this work is built to avoid, and naming them turns a set of habits into a standard someone else can apply. A reader can take the catalogue to a result, mine included, and ask which of the seven it might be making.
This matters because a correct number with a wrong conclusion is the hardest kind of clinical result to catch, since it passes review on its arithmetic. The catalogue is the review the arithmetic does not get.
Takeaways
What the threads add up to
Limitations
Limitations across the line. The strongest detection result is an oracle upper bound that a deployable pipeline has not yet reached, the routing result is an offline replay rather than a clinician-agent study, and the failure-mode catalogue is a set of self-reported corrections to my own EEG-localization work rather than a general audit of the field. None of these numbers should be quoted outside the papers until the appendices are complete.
Cite