Built by Paige and Microsoft, PRISM2 reads whole-slide images through a perceiver-based encoder trained jointly on tissue tiles and clinical dialogue drawn from pathology reports. The model aggregates thousands of tile embeddings per slide into one representation, then generates text that answers diagnostic questions rather than simply classifying pixels.
Training data spans 2.3 million whole-slide images. The dialogue supervision comes from 685,507 pathology reports that Memorial Sloan Kettering Cancer Center collected during routine care, converted into question-and-answer pairs by GPT-4o.
Architecture and embedding design
The architecture runs in two distinct phases. Stage one trains the slide encoder itself, teaching it to aggregate tile-level features into a single slide-level vector that correlates with report language.
Stage two freezes that encoder entirely and shifts the work onto the language model, fine-tuning it on dialogue so it learns pathology reporting conventions rather than encoder mechanics.
Single-turn dialogue supplies the supervision at this second stage. No multi-turn conversation history enters the training signal, which caps the kind of interactive back-and-forth a deployed system could support without further engineering.
A perceiver-based slide encoder sits at the centre of stage one, aggregating Virchow2 tile embeddings into the slide-level representation. Two loss functions train it simultaneously.
BioGPT text embeddings drive a contrastive objective, pulling slide representations toward matching report language and pushing them from mismatched pairs. Phi-3 Mini runs an autoregressive objective alongside it, forcing the encoder’s output to support direct text generation rather than just similarity scoring.
Contrastive training alone tends to produce embeddings good at retrieval but weak at generation, and autoregressive training alone can overfit to surface text patterns without learning transferable visual features. Running both against the same encoder output is the architectural bet PRISM2 makes.
PRISM2 exposes two separate embeddings rather than one. Base embeddings come straight from the slide encoder and transfer to biomarker prediction tasks. Diagnostic embeddings are pulled from the hidden state of the 4-billion-parameter language model instead, which is extracted only after that model has processed the slide latents alone, using the image without a text prompt.
That distinction shapes where each embedding performs best. Training tunes the diagnostic embedding specifically for cancer detection, subtyping, and detection of benign indications tasks. Paper authors recommend the base embedding for anything sitting outside that diagnosis-focused distribution. Survival tasks get a third, separately fine-tuned embedding on top of the other two.
Performance and benchmark results
PRISM2 matches or exceeds the balanced accuracy of clinical-grade products calibrated for prostate and breast cancer detection, tested on those products’ own evaluation datasets. Breast lymph node classification joins the comparison, and PRISM2 outperforms Paige BLN there without any further training on that specific task.
The earlier foundation models in the comparison don’t hold up as well under contrastive classification: PRISM and TITAN both fall short of product-level performance, and the gap widens on breast lymph node testing specifically.
Pan-cancer detection extends the benchmark further. Diagnostic embeddings reached 0.967 AUC, against 0.956 for the base embedding on the same task. PRISM scored 0.947 in that comparison while TITAN came in at 0.931.
Rare cancer detection dropped the diagnostic embedding’s score from 0.967 down to 0.957 AUC. The paper attributes that drop to sparse training examples for those tissue types.
Linear probing gives a cleaner read on representation quality than end-to-end fine-tuning does, since it holds the encoder fixed and tests only whether a simple classifier can extract the relevant signal.
Under that method, PRISM2 embeddings never statistically underperform prior foundation models across the diagnostic benchmarks tested, at P < 0.05. Results improve most where the pan-cancer training set holds few samples from the tissue types under comparison.
Survival and biomarker tasks follow a related pattern under the same probing method. Researchers gathered over 225,000 cases tracking overall survival across nearly 100,000 patients, then compared a fine-tuned PRISM2 slide encoder directly against a survival specialist model trained from scratch on the identical dataset. PRISM2 embeddings won that comparison.
MSK colorectal cancer recurrence-free survival produced the widest gap: a 0.809 concordance index against 0.773 for the specialist model.
Base embeddings hold up here too, without any survival-specific fine-tuning at all. On biomarker tasks – which sit outside the report-dialogue training distribution – base embeddings actually beat diagnostic ones, averaging 0.854 AUC on MSK tasks and 0.784 on TCGA tasks. COBRA and TITAN track close behind on those same figures.
The ablation study isolates what the dialogue supervision contributes on its own. Adding dialogue templates to the original PRISM starting point lifted prompt-based inference from roughly 0.498 balanced accuracy to 0.653.
The question-answering dataset behind that jump runs 3.5 times larger than the PRISM subset it builds on, and the paper attributes about half of PRISM2’s diagnostic improvement to that scale increase alone rather than architectural changes.
Training records span 200,692 patients in total. GPT-4o generated every dialogue example from the underlying 685,507 reports, a synthetic-data pipeline that itself needed independent quality checking.
Data quality, error rates, and architectural limitations
A pathologist reviewed 50 held-out specimens across 10 tissue types to check both the generated training text and PRISM2’s own outputs.
Ground-truth question errors landed at 3 percent for open-ended and multiple-choice formats combined. Diagnostic summaries ran higher, at an 8 percent error rate. Complementary yes/no questions performed worst of all three categories: 18 percent were irrelevant or inaccurate.
PRISM2’s own question-answering errors sat between 7 percent and 11 percent in that same review. Hallucination and omission were the dominant failure types, not factual contradiction of the source slide.
The model carries no position encoding across tiles, meaning it has no built-in mechanism for reasoning about where structures sit relative to one another on a slide. Every scan in training and testing ran at a single fixed resolution: 0.5 microns per pixel.
Neither gap is trivial for an ML team evaluating the architecture. Spatial reasoning tasks and any need to work across variable magnification remain unaddressed problems rather than solved-but-unoptimised ones.
MSK scanned every slide used in training. Evaluation also drew on non-MSK data. The authors call for external validation before deployment beyond that single-institution scanning pipeline.
Model weights are published on Hugging Face. Training and inference pipelines still depend on proprietary Paige and Microsoft infrastructure, which limits how far an outside team can reproduce the full pipeline rather than just running inference against the released weights.
Teams building on top of PRISM2 should test embedding transfer against their own scanner output before assuming parity with the MSK-trained baseline.
Correction: This article has been updated to clarify that PRISM2’s evaluation data was not limited to MSK-sourced slides, that the diagnostic embedding also supports detection of benign indications, and that diagnostic embeddings are generated from the image alone, without a text prompt.
See also: Why biological data matters more in AI drug discovery
Want to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo taking place in Amsterdam, California, and London. The comprehensive event is part of TechEx and is co-located with other leading technology events including the Cyber Security & Cloud Expo. Click here for more information.
AI News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.
