Guides › AI and HLA

Why language models invent HLA allele names, and how to check

Updated 2026-10-29 · allele examples checked against IPD-IMGT/HLA 3.66.0 on 2026-10-29

Large language models produce HLA allele names that look exactly right and do not exist. A name like DQB1*05:03:26:99 has a real gene, a real allele group, and four well-formed fields. It is still invented: no IPD-IMGT/HLA release from 1.05.0 to 3.66.0 has ever assigned it.

Because the format is so regular, these inventions survive a casual read and most pattern checks. The only reliable test is to look each name up in a specific release of the reference database.

What the HLA-Bench benchmark measured

HLA-Bench is an open benchmark that grades model answers on HLA tasks with an executable oracle computed from the pinned IPD-IMGT/HLA release: no human labels and no model judges. Its dataset card on Hugging Face reports two headline results:

• Fabrication. Across the nine models run on the full 550-task nomenclature suite, models fabricated allele names at 0.06 to 0.20 per task. The card notes that one model's rate, 0.09, is a lower bound because 187 of its 550 responses were truncated and graded malformed.
• Two-field ambiguity. Every model family tested (Claude, Qwen, Mistral, Llama, Phi and Gemma) scored 0 of 30 on tasks that ask which full-resolution alleles a two-field name covers.

The benchmark's public development splits (112 nomenclature tasks and 43 matching pairs) are on Hugging Face. The scored splits are sealed and regenerate from each database release, so a versioned share of tasks concerns names newer than a model's training data.

Why this is hard for a model

Three properties of HLA nomenclature make it a trap for text prediction.

The space is huge and keeps growing. The September 2026 statistics at hla.alleles.org list 46,146 named HLA alleles, and IPD-IMGT/HLA adds more every quarter. Between releases 3.65.0 and 3.66.0 alone, 772 names were added. A model's training data is a snapshot of a moving list.

Valid-looking is cheap. Any gene, two digits, a colon and two or three more digits produces something that parses. The naming rules assign fields only when needed, so whether A*29:110:99 exists is a fact about the database, not a property of the string.

Names change. Alleles gain fields, gain or lose suffixes and are renamed. A model may repeat a name that was correct years ago. A*02:01:14Q was the current name from 2011 until release 3.66.0 removed the Q.

The three failure patterns

The published wrong-answer samples on the HLA-Bench results page fall into three groups.

None of these can be caught by a regular expression that checks shape. All of them are caught by a lookup against a pinned release.

How to check model output

If an application, an agent or a person is going to rely on HLA names that a model produced, check them first. A workable process:

1. Extract every allele-shaped token from the output, including names inside sentences, tables and GL Strings.
2. Look each one up against one stated release of IPD-IMGT/HLA, and record which release.
3. Classify the result: current name, valid lower-resolution prefix, group name, legacy spelling with a current equivalent, deleted with a successor, or never assigned.
4. Block or annotate anything never assigned, and rewrite or flag anything outdated, before the output is shown or stored.
5. Do not let the model count matches. Match grades are arithmetic over normalised names; compute them with a deterministic tool and show the model's text only as narrative.

The key distinction in step 3 is between "outdated" and "never existed". A*02:01:14Q is a real former name with a successor; B*57:01:99 was never assigned. Lumping them together either hides fabrications or punishes correct historical references.

What a check looks like

Checked against release 3.66.0, this text:

A*02:01:01G covers it. The patient carries B*57:01:99 and DRB1*15:01.
A*02:01:14Q was also reported.

comes back from POST /v1/verify with:

• A*02:01:01G: group, a G group name in this release;
• B*57:01:99: hallucinated, no such name in any release back to 1.05.0;
• DRB1*15:01: valid;
• A*02:01:14Q: deleted, successor A*02:01:14;
• clean: false, because a fabricated name is present.

That example uses an allele string only. Never send patient identifiers to any checking service; de-identify first.

For agent builders

Two patterns follow for tool-using models.

• Give the model a verification tool and an instruction to use it. HLA-Verify's MCP server exposes verify_text, normalize_allele, allele_info and match_score, among others. An agent can call verify_text on its own draft before answering.
• Verify outside the model as well. A tool the model may decide not to call is not a control you can rely on. Run the same check on the final output in your application code, and fail closed when it reports a fabricated name.

The benchmark, graders and engine are open source in the verifiable-science-envs repository, so the checks can also run in your own environment.

Frequently asked questions

Do language models make up HLA allele names?

Yes. The HLA-Bench benchmark measured fabricated allele names at 0.06 to 0.20 per task across nine models on its full 550-task suite.

Can a regular expression catch fabricated HLA names?

No. Fabricated names follow the nomenclature's shape exactly. Only a lookup against a specific IPD-IMGT/HLA release can tell an assigned name from an invented one.

Is a deleted HLA name a hallucination?

Not necessarily. A deleted name was assigned once and may have a successor. A hallucinated name was never assigned in any release. Good checks report the two differently.

Which release should I check against?

The one your application or lab uses, stated explicitly with every result. Names change between releases, so a check without a release is incomplete.