What if a medical AI system could do more than say “this scan looks abnormal”—and could also point to the medical concepts that led it there?
That question sits at the heart of a new study published in Nature Biomedical Engineering. Researchers have developed ConceptCLIP, an explainable biomedical foundation model designed to combine the broad capabilities of modern vision-language AI with something healthcare desperately needs: interpretability.
The research matters because medical AI has reached an awkward stage of development. Algorithms can perform impressively on many imaging tasks, but high accuracy alone does not solve the trust problem. A clinician needs to know not just what an algorithm predicts, but also why it made that prediction and whether its reasoning is clinically plausible.
ConceptCLIP attempts to address that problem by teaching an AI model to connect medical images not only with text descriptions, but also with explicit biomedical concepts.
The result is an approach that could point toward a different generation of medical foundation models—ones designed around performance and explanation at the same time.

The Medical AI Problem: Accuracy Isn’t Enough
Artificial intelligence has become increasingly capable at analyzing medical images.
Algorithms can assist with:
- chest X-rays;
- CT scans;
- MRI;
- mammography;
- ultrasound;
- retinal images;
- dermatology photographs;
- pathology slides;
- endoscopy images.
The attraction is obvious. A foundation model trained on broad biomedical information could potentially be adapted to many clinical tasks instead of requiring researchers to build a completely new model for every disease or imaging modality.
But medicine is not an ordinary computer-vision problem.
If an image classifier labels a photograph of a cat incorrectly, the consequences are usually trivial.
If an AI system incorrectly interprets a medical image, the consequences can be much more serious.
That makes explainability more than a nice feature.
It becomes part of the safety equation.
The Black-Box Problem
Many deep-learning systems operate as black boxes.
The model receives an image.
It processes millions or billions of numerical relationships.
Then it produces a prediction.
For example:
“Possible pneumonia.”
But what exactly did the model see?
Was it a genuine lung abnormality?
A medical device artifact?
A particular anatomical pattern?
A subtle opacity?
Or something unrelated to the disease that happened to correlate with the training data?
Without an understandable explanation, clinicians have a harder time determining whether the AI is making a sensible inference or simply exploiting a misleading shortcut.
The researchers behind ConceptCLIP argue that existing biomedical foundation models have often prioritized performance without providing explanations that clinicians can readily understand.
What Is ConceptCLIP?
ConceptCLIP is a biomedical vision-language foundation model.
At a basic level, vision-language models learn relationships between images and language.
A general-purpose model might learn that a photograph containing a dog corresponds to the word “dog.”
A biomedical model needs far more specialized relationships.
For example, it may need to associate an image region with concepts such as:
- pleural effusion;
- tumor;
- fracture;
- opacity;
- inflammation;
- anatomical structures;
- pathological patterns.
ConceptCLIP goes one step further by explicitly incorporating biomedical concepts into its pretraining process.
The researchers call this approach large-scale concept-enhanced vision-language pretraining.
The Foundation: MedConcept-23M
One of the most important parts of the project isn’t the model itself.
It is the dataset.
The researchers created MedConcept-23M, a large-scale collection containing approximately 23 million biomedical image-text-concept triplets.
Each training example is designed to connect three things:
Image → Text → Medical Concept
That structure matters.
Traditional image-text learning can teach a model that a particular image is associated with a description.
Concept-enhanced learning tries to establish more precise relationships between parts of the image and specific clinical concepts.
The researchers derived biomedical concepts from the Unified Medical Language System (UMLS) and used automated processing to identify, disambiguate and validate relevant medical terminology.
Why Medical Concepts Matter
Imagine an X-ray accompanied by a report mentioning a pleural effusion.
A conventional image-text model might learn a broad relationship between the X-ray and the report.
ConceptCLIP attempts to establish a more specific relationship:
this region of the image ↔ this medical concept.
That distinction could make the model’s internal representation more clinically meaningful.
Instead of merely learning:
“This image is associated with a report containing the words pleural effusion.”
the system attempts to learn:
“This particular region corresponds to the concept pleural effusion.”
That is a much more useful foundation for explanation.
Two Types of Alignment
The model’s architecture revolves around two complementary forms of alignment.
1. Image-text alignment
This captures the broad relationship between an entire medical image and its associated textual description.
It helps the model understand the overall clinical context.
2. Region-concept alignment
This focuses on relationships between specific image regions and specific medical concepts.
This second component is crucial for explainability.
The research describes ConceptCLIP as jointly learning global image-text representations and fine-grained region-concept associations.
In other words:
the model learns the big picture and the clinically meaningful details.
How the Concept Pipeline Works
Building a dataset like MedConcept-23M is not simply a matter of scraping medical articles and throwing their captions into an AI model.
The researchers developed a concept-extraction pipeline.
It includes three broad stages:
Biomedical entity recognition
The system identifies candidate medical terms in text.
Semantic disambiguation and UMLS linking
Those terms are mapped to standardized biomedical concepts.
Contextual validation
A language model-based textual entailment process checks whether the original caption actually supports the identified concept.
Only validated relationships are retained.
That step is important because medical language is full of ambiguity.
A word appearing in a report does not necessarily mean that the corresponding condition is actually present.
For example, a report might say that a disease is not present, or that it is being considered as a differential diagnosis.
A naive keyword-matching system could easily get this wrong.
Turning Words Into Visual Concepts
The system then creates numerical representations of medical concepts.
The paper describes how the relevant words in a caption can be encoded into a concept representation, which is subsequently compared with features from image regions using cosine similarity.
In plain English:
the model asks how closely a particular area of an image matches a particular medical concept.
That gives the system a mechanism for producing concept-based explanations.
Why This Is Different From a Heatmap
Medical AI systems have used visual explanations before.
One common approach is a heatmap showing areas that influenced a prediction.
That can be useful.
But a glowing red region does not necessarily tell a doctor what the region means.
Concept-based explanation is potentially more informative.
Instead of:
“Look here.”
the system can move toward:
“This region is associated with this clinically meaningful concept.”
That difference may seem small.
It isn’t.
For clinicians, the language of medicine is far more actionable than an abstract mathematical visualization.
ConceptCLIP’s Broad Evaluation
The researchers evaluated ConceptCLIP across a large benchmark involving 78 datasets and 10 imaging modalities.
The datasets span a remarkably broad range of medical applications, including:
- chest X-rays;
- CT;
- MRI;
- ultrasound;
- mammography;
- pathology;
- dermatology;
- ophthalmology;
- endoscopy;
- cardiac imaging.
The breadth is important because foundation models are supposed to generalize beyond one narrowly defined medical task.
A model that works only on one hospital’s chest X-rays isn’t really demonstrating the full promise of a biomedical foundation model.
Stronger Zero-Shot Performance
One of the headline findings is the model’s performance on zero-shot diagnosis.
Zero-shot learning means the model can attempt a task without being specifically fine-tuned for every individual target condition.
According to the study, ConceptCLIP produced absolute AUC improvements ranging from approximately 1.46 to 10.73 percentage points over previous models across the evaluated modalities, with particularly large gains reported for CT and ultrasound.
That is significant because zero-shot capability is one of the major reasons researchers are interested in foundation models in the first place.
Instead of building:
Model A → disease A
Model B → disease B
Model C → disease C
the goal is a general-purpose model that can transfer knowledge across multiple clinical contexts.
The CT and Ultrasound Results Stand Out
The reported gains were especially notable in:
- CT: +10.73 percentage points
- Ultrasound: +10.06 percentage points
according to the study summary.
These improvements don’t mean ConceptCLIP is ready to replace radiologists.
They do, however, suggest that concept-enhanced pretraining may provide useful generalization benefits.
It Doesn’t Stop at Diagnosis
Another interesting aspect of the model is that the researchers evaluated tasks beyond simple classification.
ConceptCLIP was tested for:
- cross-modal image-text retrieval;
- medical visual question answering;
- medical report generation;
- concept annotation;
- disease-concept relationship discovery;
- whole-slide image analysis.
This matters because a medical foundation model is potentially more valuable when it can serve as a common platform for many tasks rather than functioning as a single-purpose classifier.
Medical Image-to-Text Retrieval
One example is cross-modal retrieval.
Imagine a researcher has thousands of medical images and wants to locate the ones most closely associated with a particular textual description.
A strong vision-language model can search the image collection using language.
The study reports an Image-to-Text Recall@1 of 82.85% on the PMC-9K benchmark, outperforming BiomedCLIP by 9.44 percentage points.
That could be useful for research workflows, dataset exploration and clinical information retrieval.
Medical Visual Question Answering
The model was also evaluated on medical visual question answering, or VQA.
This involves asking questions about medical images.
For example:
“Is there evidence of a mass?”
or:
“Which anatomical structure is abnormal?”
The ability to connect visual evidence with medical language is an important step toward interactive medical AI systems.
But again, answering a medical question correctly in a benchmark is not equivalent to being safe for unsupervised clinical use.
Medical Report Generation
Another application is generating medical reports.
A model that understands both medical images and clinical language could potentially help draft reports.
That could reduce repetitive documentation work.
But report generation has an obvious danger:
a fluent report can still be wrong.
This is one reason explainability becomes particularly valuable.
A clinician could potentially inspect the concepts and image regions supporting a generated statement rather than blindly accepting polished AI-generated prose.
Explainability as a Built-In Property
This is perhaps ConceptCLIP’s most interesting philosophical contribution.
Many explainable-AI techniques are added after a model has already been trained.
ConceptCLIP instead tries to make concepts part of the model’s learning objective from the beginning.
The researchers argue that region-concept alignment enables inherent interpretability without sacrificing diagnostic performance.
That is a major idea.
Rather than:
Build powerful model → bolt on explanation
the approach becomes:
Build explanation into the representation itself.

Concept Bottlenecks Could Make AI Easier to Audit
The researchers also explored ConceptCLIP as an inherently interpretable concept bottleneck model.
Concept bottleneck approaches encourage AI systems to make predictions through human-understandable intermediate concepts.
That creates a potential audit trail.
For example:
Image
↓
Detected concepts
↓
Disease prediction
A clinician can potentially inspect the middle layer.
If the system predicts a disease because it detected several medically plausible features, the prediction may be more trustworthy.
If it relies on an irrelevant concept, that could reveal a potential failure.
The Clinician Study Is Especially Important
Benchmark scores tell only part of the story.
The researchers therefore conducted a clinician user study across three imaging modalities.
The study found that concept-based explanations could help clinicians verify predictions and identify potential model errors.
That is arguably more important than another percentage point on an AI leaderboard.
A medical explanation has value when it helps a human make a better decision.
An Explanation Can Reveal an AI Mistake
One of the study’s extended examples illustrates an important scenario.
The model makes an incorrect prediction.
But its explanation highlights an artifact rather than a genuine clinical finding.
A clinician can then recognize the problem and override the AI’s conclusion.
This illustrates an underappreciated benefit of explainability:
The goal isn’t necessarily to make AI infallible.
The goal can be to make AI easier for humans to supervise.
That is a much more realistic objective.
Another Explanation Can Reveal Something Useful
The researchers also describe cases in which a correct explanation identifies a subtle but important finding and provides additional diagnostic insight.
This points toward a future in which AI isn’t simply a replacement for human expertise.
Instead, it becomes a second set of eyes.
And in medicine, a second set of eyes can be extremely valuable.
Why Multimodal AI Matters for Medicine
Medical information is inherently multimodal.
Doctors don’t work from images alone.
They combine:
- images;
- reports;
- symptoms;
- medical history;
- laboratory results;
- demographics;
- previous examinations;
- pathology;
- clinical notes.
Vision-language models represent one step toward AI systems capable of connecting different types of information.
ConceptCLIP focuses specifically on strengthening the relationship between medical images and language while adding a layer of standardized biomedical concepts.
The Model’s Real Potential May Be Its Generality
Perhaps the biggest long-term implication isn’t one particular diagnosis.
It is the possibility of a shared biomedical representation.
Imagine one model that can understand concepts across:
X-ray → CT → MRI → ultrasound → pathology → ophthalmology → dermatology.
That could make it easier to build specialized clinical applications on top of the same foundation.
Researchers could adapt the model rather than starting from scratch every time.
But Generalization Is Not the Same as Clinical Readiness
This is where the hype needs to be restrained.
A strong benchmark result does not automatically translate into safe hospital deployment.
Real-world clinical environments contain challenges that research datasets cannot fully capture.
Images may differ because of:
- scanner manufacturers;
- imaging protocols;
- hospital populations;
- patient demographics;
- disease prevalence;
- image quality;
- clinical workflows.
A model trained on one distribution may behave differently on another.
Dataset Bias Remains a Major Concern
Medical AI inherits biases from its training data.
If some diseases, ethnic groups, age groups or imaging devices are underrepresented, performance may vary.
The researchers explicitly evaluated conceptually novel and underrepresented diseases in additional experiments, which is useful because foundation models should not be judged only on conditions that appeared frequently in training data.
Still, no benchmark can eliminate the broader problem of dataset shift.
Explainability Does Not Guarantee Truth
This distinction is critical.
An AI can produce an explanation that sounds convincing and looks medically reasonable while still being wrong.
Concept-based explanations are therefore not proof that the model’s reasoning is correct.
They are evidence that can help a human evaluate the prediction.
That is a healthier way to think about explainable AI.
Explanation ≠ certainty.
The “Right Reason for the Wrong Prediction” Problem
AI systems can sometimes exploit shortcuts.
For example, a model may associate a particular imaging artifact, scanner characteristic or hospital-specific pattern with a disease because that correlation existed in its training data.
A model could then produce a seemingly reasonable explanation while still relying on a hidden confounder.
This is why prospective validation and external testing remain essential.
What About Patient Privacy?
Large-scale biomedical AI also raises obvious privacy questions.
Medical imaging datasets can contain sensitive patient information.
The ConceptCLIP project uses a mixture of public and restricted datasets. The researchers report that the MedConcept-23M image component comes from the publicly available PMC Open Access Subset, while captions and concepts are provided separately; evaluation also includes private datasets subject to restrictions.
Any future clinical deployment would still require rigorous privacy, governance and security controls.
Open Data Could Accelerate Research
One positive feature of the project is its commitment to reproducibility.
The researchers have made the ConceptCLIP implementation available, along with reconstruction tools for MedConcept-23M, and provide model weights through Hugging Face.
That gives researchers an opportunity to inspect, reproduce and extend the work.
Open research is particularly valuable in medical AI because independent evaluation can reveal weaknesses that the original development team may not encounter.
Why the Timing Matters
The paper arrives during a broader shift in medical AI.
The field is moving away from dozens of isolated models toward increasingly general-purpose multimodal foundation models.
Reviews of medical vision-language research describe this as a transition from task-specific systems toward models capable of zero-shot and few-shot adaptation across clinical domains.
ConceptCLIP adds another requirement to that evolution:
general-purpose AI should also be understandable.
The Bigger Lesson: Medical AI Needs a Human Interface
There is a tendency to think of explainability as something technical.
But ultimately, it is a human-interface problem.
Doctors don’t need to understand every mathematical operation inside a neural network.
They need information that helps them answer questions such as:
- What did the model notice?
- Where did it notice it?
- Which medical concept does it associate with that region?
- Does that concept make clinical sense?
- Could the model be relying on an artifact?
- Should I trust or challenge the prediction?
ConceptCLIP is designed around precisely this kind of interaction.
Could This Reduce Clinician AI Fatigue?
Doctors are increasingly exposed to AI tools.
If every system produces another opaque probability score, clinicians may eventually stop paying attention.
But an AI assistant that provides concise, clinically meaningful evidence may be easier to integrate into workflows.
The ideal future is not:
“AI says 87% probability. Trust it.”
It is closer to:
“AI identifies these clinically relevant regions and concepts. Here is why the prediction was made. Please verify.”
That is a fundamentally different relationship between humans and machines.
The Future Could Be AI as a “Second Reader”
One of the most realistic applications for systems like ConceptCLIP is decision support.
Rather than replacing doctors, an AI could:
- analyze an image;
- identify relevant concepts;
- highlight suspicious regions;
- produce a preliminary interpretation;
- explain its reasoning;
- allow the clinician to accept, reject or investigate the finding.
That human-in-the-loop approach could be more practical than fully autonomous diagnosis.
Where ConceptCLIP Could Go Next
The next generation of research will likely need to move beyond retrospective benchmarks.
Important areas include:
Prospective clinical trials
Does the system actually improve clinical outcomes?
Multi-hospital validation
Does it work outside the institutions represented in training?
Diverse populations
Does performance remain consistent across demographic groups?
Workflow integration
Does it save clinicians time or create more work?
Error analysis
What types of mistakes does the model make?
Calibration
Do its confidence estimates correspond to real-world accuracy?
Longitudinal reasoning
Can the system understand how a patient’s imaging changes over time?
These questions will determine whether foundation models become clinical tools or remain impressive research demonstrations.
The Hardest Problem May Be Trust
The ultimate challenge isn’t building an AI that can recognize disease.
Researchers are already doing that.
The harder challenge is creating systems that clinicians can appropriately trust.
Too little trust means doctors ignore useful AI.
Too much trust means doctors may accept dangerous errors.
The ideal system encourages calibrated trust.
In other words:
Trust the AI when the evidence supports it—and know when to challenge it.
Explainability can help achieve that balance.
ConceptCLIP’s Broader Significance
The most interesting thing about this research may not be that ConceptCLIP is another powerful medical AI model.
It is the direction it suggests.
The next generation of biomedical foundation models may need three characteristics simultaneously:
Generalization.
They should work across many tasks and modalities.
Performance.
They must actually provide useful predictions.
Interpretability.
Humans must be able to understand and challenge those predictions.
Achieving all three is considerably harder than optimizing for benchmark accuracy alone.
Final Thoughts
Medical AI is entering a new phase.
The question is no longer simply:
“Can artificial intelligence recognize disease?”
Increasingly, the question is:
“Can artificial intelligence recognize disease in a way that a clinician can understand, verify and safely use?”
ConceptCLIP represents an ambitious attempt to answer that question.
By combining a large biomedical image-text-concept dataset with joint image-text and region-concept learning, the researchers are trying to build explainability directly into a foundation model rather than treating it as an afterthought. The study evaluates the approach across 78 datasets and 10 imaging modalities and reports improvements in diagnostic, retrieval and other multimodal tasks, alongside clinician-oriented concept explanations.
That does not mean the technology is ready to replace radiologists, pathologists or other medical professionals.
It doesn’t mean AI explanations are automatically correct.
And it certainly doesn’t eliminate the need for external validation, privacy safeguards and clinical trials.
But it points toward a more sensible vision of medical AI.
The best medical AI may not be the machine that makes the final decision.
It may be the machine that says:
“Here is what I found. Here is where I found it. Here is the medical concept behind my prediction. Now you can decide whether I’m right.”
That shift—from black-box prediction to inspectable evidence—could ultimately prove more important than another incremental improvement on an AI benchmark.
5 Frequently Asked Questions
1. What is ConceptCLIP?
ConceptCLIP is an explainable biomedical vision-language foundation model designed to analyze medical images while connecting its predictions to human-interpretable biomedical concepts. It was developed using the MedConcept-23M dataset containing about 23 million image-text-concept triplets.
2. What makes ConceptCLIP different from traditional medical AI?
Traditional medical AI models often focus on a specific task, while many biomedical foundation models can still operate as relatively opaque systems. ConceptCLIP explicitly learns both image-text relationships and relationships between image regions and medical concepts, allowing it to produce concept-based explanations.
3. How accurate is ConceptCLIP?
The study reports improved zero-shot diagnostic performance across its evaluation benchmark, with absolute AUC gains ranging from 1.46 to 10.73 percentage points compared with previous models across 10 imaging modalities. The largest reported gains included CT and ultrasound.
4. Can ConceptCLIP replace doctors?
No. The research demonstrates performance on datasets and a clinician user study, but that is not equivalent to proving that the system can independently diagnose patients safely. Clinical deployment would require additional validation, regulatory assessment, monitoring and human oversight.
5. Why is explainability important in medical AI?
Explainability can help clinicians understand what evidence an AI system used, verify whether the reasoning appears clinically plausible and recognize potential mistakes. In the study’s clinician evaluation, concept-based explanations helped users verify predictions and identify possible AI errors.
Bottom line: ConceptCLIP doesn’t solve every problem facing medical AI, but it tackles one of the biggest: how to make powerful biomedical foundation models more understandable to the humans who are ultimately responsible for patient care.

Sources nature

