MAGIC: How the Global Assessment Is Changing Generative AI in Healthcare

An international group of researchers and healthcare professionals presented MAGIC—short for Medical AI Global Initiative and Collaboration—in a letter published by *Nature Medicine*. The network aims to create a global ecosystem for evaluating large language models and other forms of generative AI applied to healthcare.^[1]

The project brings together healthcare professionals, artificial intelligence researchers, industry representatives, government officials, and local experts. Its website highlights two main deliverables: a global, evolving benchmark for healthcare language models, and a reporting standard designed to clarify the conditions of use, performance, and limitations of the evaluated systems.^[2]

This distinction is important. MAGIC is neither a new medical model nor a comparative study that establishes a ranking. It is a framework for collaboration and a methodological proposal. Its value should be assessed based on the published resources, their diversity, the transparency of the protocols, and their adoption by independent teams.

Medical evaluations of AI often rely on questionnaires, retrospective case records, or simulated scenarios. They measure a useful capability, such as answering a clinical question, summarizing a case, or suggesting a course of action. However, they may overlook variations in clinical practices, available resources, vocabulary, disease prevalence, and cultural expectations.

MAGIC proposes placing representativeness at the heart of the protocol. The announced benchmark is intended to cover three dimensions: languages that are still underrepresented in assessments, regional profiles of diseases and comorbidities, and the sociocultural adaptation of responses. Its modular design is intended to allow for the gradual addition of new resources.^[2]

The novelty, therefore, lies less in the existence of an additional benchmark than in the desire to organize a distributed generation of evidence. Local experts can help define what constitutes a medically relevant response in their own context, rather than automatically applying a standard developed elsewhere.

MSc AI Governance, Risk & Compliance

The Bac+6 program that trains legal professionals to understand AI and Data technologies so they can master their governance, risks and compliance.

12 months — Bac+6 Entry with a Bac+5 Work-study internship, initial or continuing education Paris-Villejuif campus

Discover the program → RNCP Level 7 certification

A global benchmark cannot simply be a translation of an English-language questionnaire. A literal translation may alter a symptom, obscure a local clinical practice, or retain assumptions that do not reflect actual access to tests and treatments. The assessment must therefore take into account language, clinical context, and conditions of use.

The announced reporting standard complements this approach. It must document the intended use, target users, care setting, evaluation data, performance, safety, scope, and limitations. This documentation makes comparisons easier to interpret, without eliminating the need for external validation and real-world testing.

A generalization error does not have the same consequences in all settings. A recommendation may assume that a test is readily available, that a medication is accessible, or that a patient can easily return for a follow-up visit. In another health care system, these assumptions may render the recommendation useless or even delay appropriate care.

Linguistic diversity also poses a security challenge. A model may appear competent in a dominant language but lose accuracy when processing a less-documented language, a mixture of languages, or local expressions. The average performance then masks the groups for which the system performs less well.

A review presented by WHO Europe in March 2026 examined 154 articles on equity in digital health. It concludes that equity is more widely recognized, but that it remains unevenly integrated into regulatory, implementation, and evaluation frameworks.^[5] MAGIC directly addresses this gap by proposing to make diversity measurable and documented.

The publication confirms that an international network has established a common direction and made public the key aspects of its future framework. It also shows that the scientific community now considers external validity—that is, the ability of a finding to hold true in another population or environment—to be a central issue.

It does not yet allow us to conclude that the models evaluated will be safer, that all regions will be represented, or that a single standard will be adopted. No overall score, no comprehensive protocol, and no independent comparison from the MAGIC benchmark have yet been presented in the publicly available information reviewed.

This approach aligns with the Validation Accords, another initiative published in *Nature Medicine* in September 2026. The Validation Accords emphasize that robust validation should progress from an internal dataset to external validation, and then to a real-world clinical setting. The authors point out that this final step is often overlooked.^[3] MAGIC can complement this approach by broadening the diversity of settings, without replacing prospective studies or the assessment of patient benefit.

The WHO recommends that large-scale multimodal health models be regulated throughout their entire lifecycle. Risks include, in particular, incorrect or incomplete responses, bias, privacy, cybersecurity, overreliance on automation, and decision-making accountability.^[4] A credible global assessment must therefore measure more than just the accuracy of a single response.

Data governance is equally critical. Building local resources requires protecting health data, obtaining the necessary authorizations, documenting the origin of cases, and ensuring that certain populations are not merely used as sources of data. Local partners must be involved in defining the research questions, quality criteria, and procedures for sharing results.

In the European Union, regulatory status depends on the system’s intended use. A general-purpose generative AI is not automatically a medical device. When a system becomes a product or a safety component subject to third-party conformity assessment, it may fall under the high-risk systems regime provided for by the AI Act, in conjunction with the regulations applicable to medical devices.^[6] An international benchmark can contribute to the body of evidence, but it does not replace regulatory compliance or post-deployment monitoring.

The first question will concern the network’s actual representativeness. It will be necessary to examine the languages covered, the representation of low- and middle-income countries, the participation of patients and caregivers, and the governance of contributions. An international list of institutions alone does not guarantee a balanced distribution of scientific power.

The second question will concern the protocols. Assessments must disclose the cases used, the scoring criteria, the use of experts, measures of uncertainty, costs, model versions, and access conditions. Without these elements, the results will be difficult to replicate and compare over time.

Finally, we will need to link the scores to observable clinical outcomes: errors avoided, time to treatment, workload, patient understanding, equity, and safety. MAGIC paves an important methodological path, but its impact will depend on its ability to translate a global ambition into local, reproducible evidence that is useful for patient care.