HLA-Bench: a benchmark for LLMs on HLA nomenclature and matching

Updated 2026-10-30 · allele examples checked against IPD-IMGT/HLA 3.66.0 on 2026-10-30

HLA-Bench is a benchmark that measures how well language models handle HLA allele nomenclature and donor-recipient match arithmetic. Every task is generated from a pinned IPD-IMGT/HLA release and graded by an executable oracle computed from that release, so there are no human labels and no model judges: an answer is right if it matches what the reference data say.

Its headline findings, as published: every model family tested scored 0 of 30 on two-field ambiguity expansion, and models fabricated allele names at 0.06 to 0.20 per task across nine models on the full suite. HLA-Bench is built and published by the makers of HLA-Verify; read the results with that in mind, and check the transcripts, which are public for the dev split.

What the benchmark tests

The nomenclature suite published on the benchmark page, HLA-Bench-A-v0.1, has 550 generated tasks in 20 subtypes across four tiers, graded against release 3.65.0:

Tier	What it tests
1. Syntax	Reading and writing names correctly
2. Groups	G and P group membership
3. History	What a name meant in a given release, renames and deletions
4. Adversarial	Near-misses, null-allele traps, release drift, chains of renames

A second family covers donor-recipient matching: 6/6 to 12/12 frameworks, null-allele traps and GvH/HvG direction. Its public dev split has 43 of 205 pairs.

How it is graded

Each answer is compared exactly with the oracle's answer. There is no partial credit and no fuzzy matching. Beyond right or wrong, every answer gets an outcome: clean_correct, correct_but_overconfident, wrong_but_overconfident, hallucinated_answer or refused. The number of fabricated allele names per task is reported next to accuracy, because in this domain a confident invention is worse than a refusal.

Two design choices aim to keep scores meaningful:

Sealed splits. Of the 550 nomenclature tasks, 112 are public with full transcripts and 438 are sealed. The dataset card says the scored splits regenerate per release from a tag and a seed.
Post-cutoff tasks. Because tasks are generated from the current release, some concern names that did not exist when a model was trained. The benchmark page reports that 32.7% of Tier-3 and Tier-4 allele tasks concern names that did not exist at release 3.58.0, the assumed training cutoff. New IPD-IMGT/HLA releases arrive every quarter, so the pool keeps moving.

Published results

From the benchmark page, all on the full 550-task suite at release 3.65.0, with 95% confidence intervals:

System	Accuracy	Fabricated names per task
oracle-reference (the HLA-Verify engine)	100%	0.00
ollama/qwen2.5:7b	31% [27-35]	0.09
baseline-naive-string	28% [25-32]	0.01
ollama/llama3.2:3b	15% [12-18]	0.11
baseline-cautious-abstainer	4% [2-6]	0.00

The oracle row is the ceiling by construction: it is the same engine that generated the answers. The baselines are there to calibrate: a naive string-rewriting baseline scores 28%, and a baseline that refuses everything still scores 4%, because refusing is sometimes the correct answer on adversarial tasks.

The dataset card adds results across model families, including Claude, Qwen, Mistral, Llama, Phi and Gemma: all scored 0 of 30 on two-field ambiguity expansion, and the nine models run on the full suite fabricated 0.06 to 0.20 names per task. It notes that the claude-sonnet-4-6 fabrication rate of 0.09 is a lower bound, because 187 of its 550 responses were truncated and graded malformed.

Where models do worst

Two published slices stand out.

Deleted names. On the 100 tasks where a name is no longer current, qwen2.5:7b scored 8% and llama3.2:3b 3%, both below the naive string baseline at 18%. A name that used to be right is the case most likely to survive review, and the case where a successor lookup in the release files is mechanical. The deleted alleles guide shows how.

Match counting. The dataset card reports that on the matching tasks "naive string manipulation collapses to 0% and open models reach 0–14% — they count matched loci instead of chromosomes". The per-chromosome counting guide explains the rule they break.

What the fabrications look like

The published wrong-answer samples include names that parse perfectly and do not exist in any release: DQB1*05:03:26:99, C*04:535:99 and A*29:110:99 (an extra field on a real name), DRB1*11:605 and B*1487, colon-less retired forms such as DQB1*0601 and C*0701 emitted as if current, and P*24:02, a group-like name for a locus that does not exist. The benchmark page groups them as invented fields, legacy resurrection and fabricated groups. None can be caught by a regular expression, and all are caught by checking the name against a release. The guide to fabricated names covers the patterns and the checks.

How to read the numbers

They are per release. Scores were computed at 3.65.0; the tasks and answers move with each release.
They are per suite. A score on this benchmark is evidence about these task types, not a general verdict on a model.
The baselines matter. A model's accuracy means more next to the 28% that string rewriting achieves.
Fabrication rate is a separate risk from accuracy. A model can improve its accuracy and still invent names; in a laboratory, the invented name is the dangerous output.

Running it yourself

The tasks, graders and transcripts are in the public verifiable-science-envs repository, and the dev splits are on Hugging Face. The graders fetch IPD-IMGT/HLA at runtime rather than redistributing it, in line with the database's CC BY-ND licence. Whatever model you test, POST /v1/verify applies the same name check to its output in one call.

Frequently asked questions

What is HLA-Bench?

A benchmark of generated tasks on HLA allele nomenclature and donor-recipient matching, graded by an oracle computed from a pinned IPD-IMGT/HLA release, with no human labels or model judges.

How do language models score on HLA-Bench?

On the published nomenclature suite at release 3.65.0, qwen2.5:7b scored 31% and llama3.2:3b 15%, against 28% for a naive string baseline. The dataset card reports that every model family tested scored 0 of 30 on two-field ambiguity expansion.

How does HLA-Bench avoid memorised answers?

Scored splits are sealed and regenerated from each database release, and a share of tasks concern names newer than the assumed training cutoff: 32.7% of Tier-3 and Tier-4 allele tasks in the published suite.

Who makes HLA-Bench?

It is published by Brelsford Software, the maker of HLA-Verify, with its tasks, graders and dev-split transcripts public for audit.

Sources

HLA-Verify is a research-and-evaluation tool for nomenclature and reference-release validation. It is not a medical device and this guide is not clinical advice. Send allele names, never patient identifiers.

Why language models invent HLA allele names, and how to checkHow LLMs fabricate plausible HLA allele names, what the HLA-Bench benchmark measured, the three failure patterns, and how to verify model output.
HLA match counting: per chromosome, not per locusWhy an HLA match grade counts alleles rather than loci, how to count one or two mismatches or a homozygous side, and the errors that come from counting loci.
HLA-Verify verdicts explained: valid, group, deleted, hallucinatedWhat each HLA-Verify verdict and flag means, with real examples from IPD-IMGT/HLA 3.66.0, how the clean flag is computed, and the action each verdict calls for.
Converting pre-2010 HLA allele names to current nomenclatureHow to convert colon-less HLA names like A*0101, Cw*0702, A*9201 and DPB1*0102 to current names, and the cases where adding colons gives the wrong answer.

All guides