A double-blinded study in which licensed Ethiopian physicians evaluate how two frontier AI models: one from OpenAI and one from Google: write clinical instructions in Amharic: whether the advice is safe, whether the language is usable, and whether it matches Ethiopian national treatment guidance.
Health workers and patients across Africa already ask AI models for medical guidance in their own language. Yet the evaluations that decide whether a model is considered clinically reliable are overwhelmingly English-language and anchored to Western tertiary care. Three failure modes are invisible to those benchmarks.
A model can be correct in English and dangerous in Amharic: a dose, a volume or a warning can shift meaning when it is translated literally.
Advice may follow an international protocol while contradicting Ethiopian standard treatment guidance that clinicians are required to follow.
Models routinely recommend medicines that are not stocked at district level or not listed by the national regulator, leaving the patient with no usable instruction.
The protocol is designed so that the resulting dataset can be inspected, reproduced and defended.
Each case is answered independently by two frontier models: one from OpenAI and one from Google. Reviewers see the two answers side by side in a randomised order, with model identities withheld before, during and after submission.
Every reviewer is a credential-verified clinician practising in Ethiopia. Credentials are checked by Linco staff before a reviewer is admitted to the study cohort.
No model scores another model. Every judgement in the dataset is a human clinical judgement, recorded with the reviewer's specialty and review duration.
Reviewers highlight the exact phrase at issue, tag the error type, explain the clinical problem and write the phrasing that should have been used instead.
Each submission requires a written clinical justification and a corrected patient instruction of at least 20 words. There is no upper limit.
A completed review is locked. Nothing can be edited afterwards, so the record stays audit-ready for clients, funders and reviewers.
Both answers are generated from an identical, neutral prompt and stored word for word, together with the model identifier, the API run identifier and the UTC timestamp. Nothing shown to a reviewer is edited, shortened or rewritten by Linco.
Preregistration
The Wave 1 study design was preregistered with a timestamped, public open-science registry in October 2026, before confirmatory cases were evaluated. The registration fixes the case generation method, blinding procedure, reviewer qualification criteria, rating rubric, minimum and maximum reviews per case, and the statistical analysis plan, including inter-rater reliability. Data collected during the earlier pilot (Wave 0) is reported separately as exploratory and is excluded from confirmatory analysis.
Research institutions, grant reviewers and partners may request the full protocol and registry verification by contacting info@lincomedical.com.
Wave 1 is the first confirmatory phase of a broader evaluation framework. Targets describe the planned scope, not results.
Synthetic clinical directives written around Ethiopian practice, released on a rolling daily schedule.
Each case is reviewed independently by credential-verified physicians, with a minimum before analysis and a cap to prevent any single case from dominating.
Agreement between physicians is measured with established statistics and reported in full, including where agreement is low.
Reviewers rate each answer separately for clinical safety and for Amharic quality, then mark the specific words at fault using a fixed taxonomy. Every mark carries a written explanation and a suggested correction.
Cases are written around the clinical realities of Ethiopian practice and released on a rolling basis. Individual cases, their clinical hypotheses and the model answers are withheld from this page so that reviewers are never primed before they evaluate.
Domain 01
Gestational-age-specific pharmacotherapy, contraindicated medicines in pregnancy, and the warning signs a patient must be told to act on.
Domain 02
Home preparation and administration instructions, weight-based dosing, and the red flags that require immediate referral.
Domain 03
Co-treatment sequencing, drug-drug interactions, adherence counselling, and dose adjustment for organ impairment.
Domain 04
Anticoagulation and other fine-margin regimens, dietary and medicine interactions, monitoring intervals, and haemorrhage or deterioration danger signs.
The full case bank, including the clinical hypothesis behind each case, is available to research funders and partners under review conditions that preserve the blinding of the active cohort.
Funding goes directly to physician compensation, credential verification and analysis of the completed reviews.
Expand to several hundred completed, credential-verified evaluations across multiple specialties and regions.
Release a clinical safety and localisation report covering head-to-head results, the error taxonomy and worked case studies.
Make the protocol, the taxonomy and an anonymised sample of the dataset available so others can reproduce and extend the work.
Results are reported only from reviews that have actually been submitted. Linco publishes no figure, finding or claim that the completed dataset does not support.
We welcome research funders, AI safety teams and health organisations who want access to the methodology, the dataset or a dedicated evaluation cohort. Physicians licensed in Ethiopia can join the reviewing cohort directly.
Or email info@lincomedical.com. General background on our expert network is on the Expert Network page.