Hugging Face proposes MICs after a UMT5 bug sat in main almost two months
Important
Primary
On November 3, 2026, Hugging Face published Transformers Model Integration Contracts (MICs): small fixtures and expected results meant to catch regressions on cheap CPU runners. They report an UMT5 decoder bug introduced July 30, 2026 that stayed in main until September 28 — and say a 66 MB UMT5 fixture (vs a 1,227 MB source checkpoint) flagged it on every replayed commit.
At a glance
Figures from Hugging Face’s November 3, 2026 Transformers MIC post.
Bug window
July 30–September 28, 2026 (HF)
UMT5 fixture
66 MB vs 1,227 MB source (HF)
CPU tests on main
88,000+ (HF)
Daily GPU tests
200,000+ (HF)
Model types
750+ (HF)
MIC CI (as of Nov 1)
Reports failures; does not block PRs
How to read this claim
Measurement — UMT5 bug in main ~two months (Jul 30–Sep 28); 66 MB fixture vs 1,227 MB checkpoint flagged it — Hugging Face’s figures; MIC CI reports,…
Bill — A score is not a price. If you will call a model next, list input and output per million tokens on the vendor page.
Hugging Face proposed Model Integration Contracts after documenting a two-month UMT5 regression caught by a 66 MB fixture in replay.
What it introduces
Model Integration Contracts (MICs) for Transformers — fixtures plus expected results for cheap regression checks.
Who benefits Teams that treat “88k tests” as proof a checkpoint change is safe, and anyone confusing a forward-pass smoke test with training-time attention contracts.
Try this
When a lab says the suite is large, ask whether a MIC-style fixture exists for the changed model family, which revision pinned the baselines, and whether CI blocks or only reports.
What we do not know
Whether MIC CI will eventually block PRs is an open question in the post.
Coverage across all 750+ model types is incomplete; HF invites contributions for uncovered models.
How often agent-generated contracts need human recipe fixes is not quantified beyond the narrative.
Evidence
PrimaryTransformers Model Integration Contracts. UMT5 dates, fixture sizes, and test counts are from Hugging Face’s November 3, 2026 Transformers Model Integration Contracts post.
Independent Dates, sizes, and test counts are Hugging Face’s. Third-party reruns of the MIC fleet are not established by this post alone.
Hugging Face published Transformers Model Integration Contracts on November 3, 2026. The post says that on July 30, 2026, enabling PyTorch SDPA for T5 let UMT5’s decoder attend to future tokens during training, and the bug stayed in main until a September 28 fix. Transformers supports over 750 model types with 88,000+ CPU tests on main and 200,000+ GPU tests in a daily run. HF’s approach is to publish a shrunk fixture — the experimental UMT5 fixture used 66 MB of weights versus 1,227 MB for the source checkpoint — and run fast integration tests on CPU. transformers-mic packages a Model Integration Contract with three parts (including expected results published under huggingface-mic). Frozen MICs replayed over five months of commits flagged the UMT5 regression from introduction to fix. As of November 1, 2026, the MIC job in Transformers CI reports failures without blocking PRs.
Why it matters
Huge test counts are not the same door as a contract that would have failed this bug. For anyone shipping model libraries, the measurement is whether a named fixture catches the failure mode — not how many GPU jobs run overnight.
HF says fixtures will not replace full integration tests; MICs are a cheaper early signal.
Agents (via tmic) are used to draft contracts across model families; edge cases feed recipe fixes.