A 4-bit quantization experiment, the regression it introduced, and what shipped instead
Data Alias finds person names with a model that runs inside the browser tab. Two of them, in fact — one Korean, one English — and every byte of both crosses the user's connection the first time the feature is switched on. Detection quality is the interesting half of that feature. Download size decides whether anyone waits long enough to see it.
So we tried to make the models smaller. Four-bit quantization is the obvious lever: it is supported in the runtime we already use, and on paper it halves an 8-bit weight. The experiment failed, in a way worth writing down. The straightforward 4-bit export came out larger than the 8-bit model it was meant to replace, and the variant that did shrink cost both accuracy and speed.
Where the bytes actually are
The two models ship as int8 ONNX files, served from this site's own origin:
| Model | Weights (int8) | Served as gzip | Licence |
|---|---|---|---|
Korean (ko-privacy-ner) | 14,560,915 bytes | 10,782,618 bytes | Apache-2.0 |
English (en-ner) | 13,983,047 bytes | 10,236,537 bytes | Apache-2.0 |
Both are quantized conversions of published models, redistributed with their provenance recorded beside them. Neither is the highest-scoring model we evaluated. The best Korean model in our comparison carried no licence tag at all, so it was set aside in favour of an Apache-2.0 one that over-captures honorifics and needs post-processing to clean up — a trade of raw quality for the right to ship it.
The runtime matters as much as the weights. After the one-time, user-initiated download, the model files and the ONNX Runtime WebAssembly binary are both fetched from this origin and cached in the browser; the worker sets allowRemoteModels = false and pins the WASM path, so no request goes to a model host or a CDN during a scan. That constraint is the reason size is a problem at all. A service that runs inference server-side has no download to optimize, and also no claim to make about where the file went. How browser-local file processing works—and how to verify it covers how to check that for yourself.
What 4-bit quantization actually converts
The mental model that makes 4-bit sound like free money — every weight is half as wide, so the file is half as big — does not survive contact with what the tooling converts.
ONNX Runtime's block-wise weight-only quantization documents exactly two operators. Its quantization guide states that "the supported op types are: MatMul" and "Gather," that a MatMul "is quantized only if the input B is constant" and becomes a MatMulNBits node, and that a Gather "is quantized only if the input data is constant" and becomes a GatherBlockQuantized node.
Our export applied the MatMul path only. In a small encoder model, that is the wrong half. The embedding table is a large share of the parameters, and it lives in a Gather. Converting the MatMuls to 4-bit blocks — each block carrying its own scale — while leaving the embeddings untouched produced a file larger than the int8 baseline it was supposed to beat.
That is a result about the export we ran, not a law about 4-bit quantization. The runtime documents a 4-bit path for Gather; we did not test it. Anyone reading this as "4-bit does not work on small encoders" is reading more than the experiment supports.
The hybrid, and what it cost
The obvious repair is to quantize what MatMulNBits left behind. A hybrid pass that also int8-quantized the embeddings produced a genuinely smaller file, and the rest of the trade came with it. Accuracy was measured end to end through the product's own post-processing on a 15-sentence Korean set with one gold name each:
| Variant | Size | Exact matches | Mean person score | Inference, WASM 1 thread |
|---|---|---|---|---|
| int8 (baseline) | 14.6 MB | 12/15 | 0.939 | 7.5 ms/sentence |
| 4-bit, pure | 24.5 MB | 12/15 | 0.921 | not run — larger than the baseline |
| 4-bit + int8 hybrid | 10.9 MB | 11/15 | 0.908 | 16.5 ms/sentence |
The hybrid saves about 3.7 MB and runs 2.2 times slower. Those two numbers are not the same kind of number, which turned out to be the whole decision. The download is paid once per browser, and never again — the model is cached, and a repeat visit costs no network at all. The inference cost is paid on every scan of every file, forever.
A boundary error is not a cosmetic error
The accuracy column understates what actually went wrong. No variant missed a gold name outright; every regression was a false positive or a boundary. One of those boundaries is worse than it looks.
On the hybrid, 김민수와 박지영이 — "Kim Minsu and Park Jiyoung" — produced a person span of 김민수와, with the Korean particle 와 glued to the end of the name.
Aliasing is keyed on the exact detected string. The replacement pass looks for literal occurrences of each detected name, so a span of 김민수와 matches only that string. Every bare 김민수 elsewhere in the same file goes unaliased, unless the model happened to produce a clean span for it somewhere else. A single boundary error at one cell becomes a recall loss across the whole file — and it announces itself nowhere. There is no error, no warning, no count that moves. The file simply comes out with a name still in it.
This is why the shipped model runs through a post-processing pass rather than being trusted raw. Honorifics and titles the model tends to swallow — 고객, 님, 씨, 팀장 — are stripped back off the span, with an attached single-syllable honorific removed only when what remains is still a plausible name. The Apache-2.0 model was chosen knowing it needed that work. The hybrid would have added a new class of boundary error underneath it.
What shipped instead
The size problem was real, and it had a cheaper answer that we had been ignoring: compress the file on the wire.
Model weights are already high-entropy, so .onnx bodies are not obviously compressible, and CDNs generally do not compress them by default. Gzipping these two anyway takes roughly a quarter off each: 25.9% for the Korean model, 26.8% for the English one. Same weights, same accuracy, same inference speed — the decompressed bytes are identical.
Getting that through the loader took one seam. The worker overrides the library's fetch, requests the .gz copy, and pipes the response through DecompressionStream. Progress is counted on the compressed stream, because the decompressed one carries no content-length for a progress bar to read against. If the server already declared content-encoding: gzip, the body is used as-is. Any failure falls through to the plain .onnx, which stays deployed for exactly that reason.
Three of those properties are asserted by an end-to-end test that drives a real browser against the built output: that the gzipped copy is the one fetched and no plain .onnx request is made, that the WebAssembly runtime downloads in parallel with the model rather than after it, and that it is fetched exactly once.
What is measured, and what is not
The per-sentence timings above come from the ONNX Runtime WebAssembly engine running single-threaded under Node — the same engine a browser uses without cross-origin isolation, and the closest proxy we measured. They are not browser measurements. The end-to-end test does load the model in a real browser and confirm the scan completes and the expected names are found, but it records its duration as a diagnostic rather than asserting a threshold, so there is no committed browser latency figure and this post does not give one.
The accuracy column rests on 15 short Korean sentences. That is enough to catch a boundary regression, which is what it was for. It is not a benchmark, and the numbers in it should not be read as a quality score. Noisy free text — mixed languages, typos, romanized names, nicknames — remains unmeasured for these models.
Name detection is also the layer our published detection benchmark cannot see: that harness covers the pattern layer only, so its name rows score zero by construction, and the gap is left visible in the table rather than hidden. What names the models miss is a separate question from what the patterns miss, and What automatic PII detection misses in spreadsheets covers the second. Detected names are suggestions in either case, and the review step before export is where they get confirmed.
The rule we took from it
Optimizing the number that is easiest to see is a reliable way to make a product worse. Download size is visible, quotable, and paid once. Inference speed and detection quality are paid on every file, by every user, for as long as the feature exists — and a recall loss caused by a glued particle is paid silently.
The 3.7 MB was worth about six seconds, once, on a first visit. Gzip returned most of it for nothing. The experiment is in the repository either way: a negative result that stayed unshipped is still the reason the shipped thing looks the way it does.