research · october 2026 · round 2
we don't only grade our own homework.
our evidence page uses documents we wrote ourselves. here we run limenn on four public benchmarks built by others: real court judgments and three large multilingual collections. we show where we fall short as well as where we do well, and we do it all with small models that run on your own computer.
of personal values caught across four public benchmarks, limenn today
81%with our next-generation detector, now in development
91%fewer personal values slipping through with the next generation
55%small models, any computer
the model limenn uses today, running inside your browser
the next-generation model: together still under half a gigabyte, about the size of a phone app
graphics cards or special chips needed: both run on the ordinary processor every computer has
bytes of your documents sent to a server to find personal data. the models run where your data is
the AI assistants limenn protects you from run on data-centre hardware. finding personal data doesn't need that: a few hundred million parameters, compressed to 8 bits, are enough. every number on this page was measured on an ordinary computer processor (Apple M5), with no graphics chip in use. the next-generation model needs about 30 milliseconds for a short prompt in our desktop app; in the browser it prepares your vault documents in the background, so sending them stays instant.
which benchmarks
TAB: real judgments of the European Court of Human Rights, with every identifier marked by legal experts. English.
ai4privacy open: short everyday texts full of names, addresses and IDs. English, German, French and Italian.
Gretel finance: loan forms, statements, contracts and banking messages. English, German, French and Italian.
Nemotron-PII: documents from healthcare, HR, legal and finance, in US and international formats. English.
5,127 documents with 20,236 labelled personal values. Each benchmark labels personal data a little differently, so we mapped their labels onto the 16 kinds of data limenn protects. Values limenn doesn't cover, such as passwords or job titles, are left out of the score. Everything runs on the device; the optional second look in the cloud was switched off. Documents were sampled at random with a fixed seed, so every number here can be reproduced. Settings were tuned on separate tuning data from the same sources; the numbers shown come from one final run on test data limenn was never tuned on.
how we compare
| limenn today | limenn next generation | Presidio, default | Presidio, 4 languages | |
|---|---|---|---|---|
| personal values left in, four public benchmarks | 19.1% | 8.7% | 15.8% | 20.4% |
| replacements that really are personal data | 82.8% | 75.6% | 54.3% | 69.1% |
| personal values left in, Swiss documents (DE/FR/IT/EN) | 13.7% | 9.5% | 28.4% | 32.4% |
| names and case numbers left in court judgments | 1.9% | 0.3% | 36.7% | 36.7% |
| memory while running | 0.75 GB | 1.9 GB | 1.3 GB | 1.9 GB |
| one-time download | 135 MB | 0.5 GB | 0.4 GB | 2.3 GB |
Presidio's default setup lets fewer values through than limenn today on these mostly English benchmarks, but almost every second thing it replaces isn't personal data, and on Swiss documents it misses twice as much. "4 languages" adds Presidio's German, French and Italian models. Memory is the whole process (limenn: our desktop engine; Presidio: Python), measured while redacting.
what others need
| tool | where it runs | hardware, as published by the vendor |
|---|---|---|
| limenn | your computer, in the browser or desktop app | no graphics card · 0.75 GB memory today, 1.9 GB with the next generation |
| Limina (formerly Private AI) | self-hosted server, or their cloud | at least 7.5 GB free memory, 16 GB recommended, 64 GB for all features; GPU version: 28 GB memory and a 16 GB graphics card (docs) |
| Tonic Textual | their cloud, or self-hosted | graphics card with at least 16 GB memory; without one "significantly slower" (docs) |
| Azure AI Language PII | Microsoft cloud, or a container that stays connected for billing | container: at least 2 GB memory, 8 GB recommended (docs) |
| AWS Comprehend, Google Cloud DLP | their cloud only | your text is sent to the provider to be checked |
vendors' own accuracy claims are measured on their own data, with different rules for what counts, so we don't put them side by side with ours. the only fair comparison is the same documents on the same computer, which is what the table above does.
what we learned
01 · catching more, without crying wolf
our first next-generation prototype caught more but also replaced harmless words: in court judgments it treated roles such as "the applicant" or "the Government" as names. we taught it to ignore roles and pronouns and to replace a value only when two independent checks agree or one is very sure. on court judgments, the share of replacements that really are personal data rose from 37% to 84%. averaged over all four benchmarks it is 76%, against 83% for limenn today. part of that gap is the benchmarks themselves (see 05).
02 · public data uses fake numbers
limenn only replaces card numbers and IBANs that pass their built-in check digits, so it never mistakes an order number for a card. most numbers in these public datasets are made up and fail that check: only 41 of 340 card numbers and 34 of 60 IBANs would be valid. on real documents these are caught reliably. that's why we leave them out of the chart above.
03 · postcodes come with a town
in Switzerland a postcode is personal data together with its town (8001 Zürich), and that's what limenn looks for. a bare "40214" or a UK district code on its own isn't matched by design.
04 · health and salaries need their own test
none of the four benchmarks labels diagnoses or salaries, the most sensitive data our users handle. we measure those on our own documents instead, including a set written blind, without access to how limenn works. see evidence.
05 · some "mistakes" aren't
many values limenn replaces that count against it are personal data the benchmark simply didn't label, such as passport numbers, dates in a medical log or loan amounts. the precision shown here is a floor.
want the full write-up, or a test on your own documents? we're happy to share both.
get in touch ↗