A Research Review of Developing Real-World Benchmarks for Healthcare AI Agents

By Nigam Shah et al. (Stanford University and collaborators) | NEJM AI

Published: September 2025 | DOI: 10.1056/AIdbp2500144

Overview

As AI agents begin to enter clinical practice, evaluating their real-world safety, reliability, and effectiveness is paramount. In this paper, researchers at Stanford propose a framework for developing real-world benchmarks for healthcare AI agents. Unlike traditional ML benchmarks (which often rely on static test datasets), this work emphasizes workflow integration, contextual grounding, and longitudinal evaluation in healthcare settings.

Key Focus Areas

1. Motivation

• Traditional benchmarks (e.g., ImageNet, MIMIC-III) assess model accuracy but fail to capture how AI agents perform in dynamic, real-world clinical environments.

• Clinical AI needs benchmarks that go beyond technical accuracy to measure patient safety, fairness, accountability, and workflow alignment.

2. Proposed Benchmarking Framework

• Contextual grounding: Evaluate AI agents on tasks within authentic clinical settings (e.g., discharge planning, triage, medication management).

• Multidimensional metrics: Measure safety, efficiency, interpretability, fairness, and trustworthiness.

• Workflow integration: Test not only outputs but also how agents interact with clinicians, electronic health records (EHRs), and patients.

• Longitudinal evaluation: Assess consistency and impact over time rather than one-off performance snapshots.

3. Case Examples

• The paper outlines use cases such as AI-assisted discharge summaries and medication reconciliation, where success requires both technical accuracy and seamless integration into clinical workflows.

• Benchmarks must capture error cascades (how a small AI mistake can escalate in patient care).

4. Governance and Collaboration

• Argues for a shared benchmarking consortium, involving academia, healthcare providers, regulators, and patients.

• Suggests parallels with FDA drug trials, where benchmarks evolve as standards of care evolve.

Strengths of the Paper

• Problem framing: Clearly identifies the gap between static AI benchmarks and dynamic clinical realities.

• Practicality: Provides concrete healthcare tasks as examples, avoiding abstract or purely theoretical proposals.

• Interdisciplinary vision: Links computer science, medicine, and policy to argue for benchmarks that are both scientifically rigorous and clinically meaningful.

Future Directions

• Global benchmark adoption: Expanding beyond U.S.-centric healthcare systems to low- and middle-income countries. This direction reveals the paper's blind spot—most healthcare AI will be deployed in under-resourced settings where ideal benchmarking conditions are impossible to maintain.

• Patient-centered evaluation: Incorporating patient experience, trust, and outcomes into benchmark metrics. The fact that patient perspectives are relegated to "future directions" rather than being central to the framework reveals a fundamental misunderstanding. An AI system that's technically perfect but erodes physician-patient trust is a net negative, regardless of benchmark scores.

• Regulatory pathways: Alignment with FDA/EMA frameworks to make benchmarks usable for approval and compliance. This assumes regulatory bodies can keep pace with AI development—a questionable assumption given their current struggles with basic AI oversight.

• Continuous updating: Creating "living benchmarks" that evolve with clinical practice and AI system updates. While appealing, this ignores practical constraints. Healthcare systems struggling with basic interoperability aren't equipped for sophisticated performance monitoring.

Context in Ongoing AI Research

• Aligns with calls from NIST AI RMF (2023) for domain-specific measurement frameworks, though NIST's guidance remains largely aspirational without enforcement mechanisms.

• Extends work like MedPerf (MLCommons), which also focuses on real-world clinical benchmarking. However, MedPerf's early results suggest that "real-world" benchmarks often just reveal how context-dependent AI performance really is, rather than providing transferable insights.

• Resonates with broader Responsible AI themes, especially transparency, accountability, and continuous monitoring. Yet the Responsible AI movement has struggled to move beyond high-level principles to actionable practices, and this paper doesn't clearly bridge that gap.

Conclusion

The Stanford-led paper “Developing Real-World Benchmarks for Healthcare AI Agents” marks a critical step toward aligning technical AI evaluation with the realities of clinical care. By emphasizing context, workflows, and longitudinal performance, it moves benchmarking closer to what truly matters: patient safety, clinician trust, and healthcare system efficiency.

For Responsible AI, this paper highlights the urgent need to evolve beyond dataset accuracy and design benchmarks that measure what matters in the real world.

Read full paper: MedAgentBench: A Virtual EHR Environment to Benchmark Medical LLM Agents