How well does the detector actually work?
On a seeded set of 306 labeled cells, the Data Alias detector found every API key, JWT, US Social Security number, Korean resident registration number, IPv4 address, phone number, and payment card in the data. It found half the email addresses, two thirds of the IBANs, and — in this harness, which cannot run the on-device name models — none of the person names. Precision is lower than recall for phone numbers, cards, and IPs: it over-flags rather than under-flags.
Data Alias publishes this because a detection tool that only reports its wins is not telling you what you need to know. The numbers below are produced by a script in the repository, from a fixed seed, and the same script fails the build if any score drops. They are also self-authored and measured on synthetic data — the limits are set out in full further down.
- Dataset
- 306 cells
- 252 positive spans
- Seed
- 20260728
- Identical data on every run
- Generated
- 2026-09-03
- Regenerated, never hand-edited
What did it find, category by category?
Recall is the share of real values it found. Precision is the share of its flags that were real. A detector that flags everything scores perfect recall and poor precision; one that flags nothing scores the reverse. Rows are ordered strongest to weakest.
| Category | Values | Recall | Precision | F1 |
|---|---|---|---|---|
| API keys | 18 | 100.0% | 100.0% | 100.0% |
| JWTs | 18 | 100.0% | 100.0% | 100.0% |
| US Social Security numbers | 18 | 100.0% | 100.0% | 100.0% |
| Korean resident registration numbers | 36 | 100.0% | 100.0% | 100.0% |
| IPv4 addressesSix dotted version strings were flagged as IP addresses. | 18 | 100.0% | 75.0% | 85.7% |
| IBANsEvery IBAN it flagged was real; it missed a third of them. | 18 | 66.7% | 100.0% | 80.0% |
| Phone numbersFound every phone number, and also flagged long order numbers and generic 13-digit IDs. | 36 | 100.0% | 64.3% | 78.3% |
| Payment card numbersLuhn-valid digit runs inside other identifiers were flagged as cards. | 18 | 100.0% | 64.3% | 78.3% |
| Email addressesPlain addresses: 18/18. Obfuscated ones ("name at example dot com"): 0/18. That single format is the entire miss. | 36 | 50.0% | 100.0% | 66.7% |
| Korean person namesRegex layer only — names are found by the on-device NER model, which this harness cannot run. | 18 | 0.0% | — | 0.0% |
| English person namesSame as above: measured without the NER layer, by construction. | 18 | 0.0% | — | 0.0% |
Why is email recall only 50%?
Because of one format. The dataset contains two kinds of email address in equal number: ordinary ones, and deliberately obfuscated ones written as “name at example dot com”. Data Alias found 18 of 18 ordinary addresses and 0 of 18 obfuscated ones. The aggregate 50% is that single gap, not a detector that misses half of all normal email addresses. It is still a real miss: an obfuscated address in a free-text column would reach the safe copy unchanged.
Does it suggest the right column types?
Data Alias suggested the expected type for 24 of 27 labeled column headers (88.9%), across English and Korean header names. The three it got wrong were tel, cust_email, and 생년월일 (date of birth), each classified as generic text rather than its specific type. Column types are always suggestions you review, never confirmations — a wrong suggestion costs you a click, not a leak.
Do aliases stay consistent, and do they reverse cleanly?
These two properties are what separates pseudonymization from redaction, so they are measured separately from detection. Across 163 occurrences, every recurring value kept one stable alias (32/32), every value appearing in two different files kept the first file’s alias in the second (15/15), and every alias embedded in a simulated AI answer and result file restored to its original value (15/15).
These measurements feed known values into the aliasing and mapping machinery. They test that the machinery holds; they say nothing about whether the detector would have found those values in the first place.
What this benchmark does not tell you
The data is synthetic
Every value comes from a seeded generator using reserved ranges — fiction phone numbers, RFC 2606 email domains, RFC 5737 IP addresses, and generated digit strings for the rest. No value comes from a real record. Results on real, messy data will differ, and typically for the worse.
Data Alias wrote both the engine and the test
The categories in this benchmark mirror what the engine targets, so it measures regressions and format coverage. It is not an independent evaluation, and it should not be read as one.
The name detector is not measured here
This harness runs the regex and heuristic layer only. The on-device name models cannot run in it, so both name rows score zero by construction. That is a gap in the benchmark, not a claim that Data Alias never finds names.
Detection is incomplete by nature
The misses in the table are real misses on this dataset, and any format not listed is untested. Data Alias reduces exposure risk but cannot catch every sensitive value, which is why you review the result before you share it.
Consistency and reversibility are not detection scores
Those two measurements feed known values into the aliasing and mapping machinery to check it holds. A value the detector misses would never reach them in real use.
Reproducing it
The harness lives in the Data Alias repository and runs with bun scripts/bench/run-bench.ts. It regenerates the full report, including tables not reproduced here, and exits non-zero if any category’s F1 falls below its recorded floor. The repository is not currently public, so this page is a report of that run rather than something you can execute yourself today. The claim you can check yourself is the one about where your file goes.