HLA-Bench-A-v0.1@IMGT-3.65.0

Can a model resolve HLA allele names the way a lab must?

550 generated tasks, 20 subtypes in 4 tiers, graded by exact match against IPD-IMGT/HLA release 3.65.0. No fuzzy credit, no partial marks. Fabricated allele names are the headline metric, because in this domain a confident invention is worse than a refusal.

Headline

Model	n	Accuracy	Tasks with fabricated names	Fabricated / task	Most common outcome
`oracle-reference`	550	100% _[99–100]	0	0.00	`clean_correct`
`ollama/qwen2.5:7b`	550	31% _[27–35]	47	0.09	`wrong_but_overconfident`
`baseline-naive-string`	550	28% _[25–32]	8	0.01	`wrong_but_overconfident`
`baseline-confident-guesser`	550	28% _[25–32]	8	0.01	`wrong_but_overconfident`
`ollama/llama3.2:3b`	550	15% _[12–18]	57	0.11	`wrong_but_overconfident`
`baseline-cautious-abstainer`	550	4% _[2–6]	0	0.00	`refused`

All rows are the full 550-task suite. Bracketed figures are 95% confidence intervals. oracle-reference is the HLA-Verify engine itself answering the same tasks — it is the ceiling, and it is the thing the API sells.

A 7B model barely beats string manipulation

qwen2.5:7b scores 31% against a naive-string baseline at 28%. Whatever it knows about HLA nomenclature is worth three points over rewriting the input.

Smaller is worse than the baseline

llama3.2:3b scores 15% — below the baseline — and fabricates more than the 7B. The scale ordering is real, and so is the direction of travel.

Abstention scores 4%

The cautious-abstainer baseline refuses everything and still scores 4%, because on adversarial tasks refusing is sometimes correct. That is the bar a confident wrong answer has to clear.

By tier

Tier 1 is syntax. Tier 2 is group membership. Tier 3 is release history — what a name meant, and when. Tier 4 is adversarial: near-misses, null traps, release drift, chains of renames.

Model	T1 syntax	T2 groups	T3 history	T4 adversarial
`oracle-reference`	100%	100%	100%	100%
`ollama/qwen2.5:7b`	39% _[31–48]	31% _[24–39]	26% _[20–33]	28% _[20–37]
`baseline-naive-string`	63% _[54–71]	26% _[20–34]	19% _[14–26]	6% _[3–12]
`ollama/llama3.2:3b`	22% _[16–31]	15% _[10–22]	13% _[9–19]	6% _[3–12]
`baseline-cautious-abstainer`	0%	0%	0%	20% _[13–29]

The naive-string baseline beats the 7B model on Tier 1 (63% vs 39%): pure string manipulation is better at syntax than a model reasoning about it. The model's advantage only appears in the tiers that need the database — where it is still under a third correct.

By slice — where clinical risk concentrates

Aggregate accuracy hides the distribution. These slices are the ones that would matter in a laboratory.

Slice	n	`qwen2.5:7b`	`llama3.2:3b`	naive-string
deleted_name — the name is no longer current	100	8% _[4–15]	3% _[1–8]	18% _[12–27]
serology_uncertain	15	0% _[0–20]	7% _[1–30]	100% _[80–100]
expression_suffix — N, L, S, Q alleles	7	0% _[0–35]	0% _[0–35]	14% _[3–51]
class_II_secondary_locus — DRB3/4/5, DPA1, DQA1	137	37% _[30–46]	14% _[9–21]	30% _[23–38]
partial_sequence	143	45% _[37–53]	15% _[10–21]	33% _[26–41]
unconfirmed	248	46% _[40–53]	19% _[15–24]	35% _[30–42]
post_cutoff — names newer than the assumed training cutoff	78	59% _[48–69]	29% _[21–40]	22% _[14–32]
null_allele	36	81% _[65–90]	3% _[0–14]	6% _[2–18]

oracle-reference scores 100% on every slice above and is omitted for width. The deleted_name row is the one to read twice: the case where a name used to be right is where models do worst and where a stale answer is most likely to survive review unchallenged — and where a successor lookup is a solved problem.

What the fabrications look like

These are real tokens the models emitted, taken from the published wrong-answer samples. Every one is well-formed. None of them exists.

qwen2.5:7b C*04:535:99 DQB1*05:03:26:99 A*29:110:99 DRB1*11:605 B*1487 DRB5*0212 P*1801 P*24:02 B*1501:05 llama3.2:3b DRB5*0101×3 DQB1*0601×3 DRB4*0101×3 DRB1*04:94:02×2 C*0701

Three distinct failure modes show up. Invented fields — a fourth field appended to a real 3-field name, producing something that parses perfectly. Legacy resurrection — colon-less forms retired decades ago, emitted as if current. Fabricated groups — P*1801 and P*24:02 are not group names at all; the model has invented the concept of a locus called P. A regex cannot catch any of these. A pinned database catches all of them.

Methodology

Generated, not curated

550 tasks generated deterministically from the reference database itself, seeded so the suite is reproducible: 4 Tier-1 subtypes, 5 Tier-2, 6 Tier-3, 5 Tier-4. Regenerated on each IPD-IMGT/HLA release, so the benchmark does not go stale.

Dev split public, sealed split held back

112 tasks are published with full transcripts so anyone can audit the grading. 438 are sealed and never leave the machine, so a published score means something.

Contamination-resistant by construction

32.7% of Tier-3 and Tier-4 allele tasks concern names that did not exist at IMGT 3.58.0, the assumed training cutoff. A model cannot have memorized them, and ~600 more arrive each quarter.

Graded on more than accuracy

Every answer gets an outcome: clean_correct, correct_but_overconfident, wrong_but_overconfident, hallucinated_answer, refused. Fabricated-name count and calibration are reported alongside accuracy, because in this domain they are the metrics that matter.

Every wrong answer in the tables above was audited against a stratified sample of at most three per subtype, to check whether the model was wrong or the grader was too strict. In every case examined so far the model was wrong and the grader stood — including the two most suspicious subtypes, where qwen2.5:7b answers "1" for how many full alleles a 2-field name covers (true counts run from 2 to 389) and copies a formatting example rather than answering the question.

Tasks, graders and transcripts on GitHub

Where this is going

Family A is nomenclature. Family B is imputation, family C is donor–recipient matching — the same deterministic-grading approach applied to the next two things a laboratory actually does. Frontier-model rows are being added as budget allows; the rows above were run on self-hosted hardware at zero cost, which is why they are open models.

Running a model you want to see on this page? The suite exports to verifiers and Inspect AI, and the Harbor task packaging is done. If you want a sealed-split run of your own model with a versioned report, that is a product — get in touch.