Can language models reproduce personal data? Three studies, with caveats

What three extraction studies measured, and the caveats their authors attached

The short version for practitioners

Between 2021 and 2024, three research teams showed that trained language models can reproduce pieces of their training data word for word, including names, phone numbers, and email addresses. For anyone deciding what a spreadsheet should contain before it reaches an AI service, the studies support one working assumption: text that enters a training set can come back out, and the values most likely to come back out are the rare, identifying kind — a name beside a phone number, an account ID, a customer row.

The same studies come with limits their own authors wrote down. Each experiment ran under specific conditions, on specific models, and none of them estimates how likely it is that a particular file, sent to a particular service, will ever be memorized or extracted. An upload decision needs both halves, so this post gives the findings and the caveats equal weight.

Three studies at a glance

Each study attacked a different model generation with a different method, and each found memorized training data in the output. The sections below describe how the attacks worked and the scope limits the authors reported.

What each study measured

Carlini et al. 2021. The team attacked GPT-2 deliberately: its training data was scraped from the public web, so anything the attack surfaced was already public and no one was newly exposed by the experiment. The method had three steps. Sample a large volume of text from the model, rank the samples by statistical signals of unusual model confidence, then check the top candidates against the training data. 604 of the 1,800 candidates they checked turned out to be verbatim training sequences. Among those, 46 contained named individuals and 32 contained contact information; the paper's first figure shows the name, address, and phone number of a real person, recovered from the model's weights, which the authors note appear in just a few documents on the internet. One comparison proved durable: in one test configuration, the 1.5-billion-parameter GPT-2 XL memorized over 18 times as much content as the 124-million-parameter version.

Nasr, Carlini et al. 2023. ChatGPT (gpt-3.5-turbo) had looked resistant. Under existing attacks it appeared, in the authors' measurement, fifty times more private than earlier models, and alignment training was the presumed reason. The team asked the model to repeat a single word forever. It complies for a few hundred repetitions, then diverges into other text, and inside that divergence it emits verbatim training data at 150 times the rate of ordinary querying. The recovered examples were verified by matching outputs against a nine-terabyte merge of public pre-training corpora (The Pile, RefinedWeb, RedPajama, Dolma). If anything the count is an undercount: ChatGPT's actual training set is not public, so only matches against known text could be confirmed. Of the generations tested for personal information, 16.9% contained memorized PII, and the authors' scaling estimate suggested a larger query budget could extract roughly ten times more. They concluded that current alignment techniques had hidden memorization without eliminating it, disclosed the vulnerability to OpenAI on August 30, 2023, and waited the standard 90 days before publishing.

Nakka et al. 2024. This workshop paper asked the targeted version of the question: can an attacker pull a specific person's number, rather than whatever happens to fall out? The subject was GPT-J-6B, whose training set (the Pile) includes the Enron email corpus — real emails and real phone numbers, public since the corpus was released during the Enron investigation. Hand-written prompt templates of the form "the phone number of [name] is" extracted fewer than 0.15% of target numbers in a single query. Grounding the prompt with a genuine snippet of someone else's email from the same domain lifted that to 0.92% in one query, 3.9% across 128 queries, and 6.86% across 2,308 queries — roughly one target in fifteen. The authors state their scope limits directly: one PII type, a base model without instruction tuning, and an attacker who already holds real text from the same domain.

What the studies show

A language model stores no documents. Training adjusts billions of numeric weights so the model gets better at predicting the next token, and for the bulk of a corpus the result is generalization: thousands of documents share the pattern "the capital of France is Paris," and the model learns the pattern rather than any single document. Memorization happens at the other end of the distribution. A string that is rare, unique, and internally patternless — a UUID, an API key, a name sitting beside a phone number — cannot be compressed into a rule shared with other text. For strings like that, the cheapest way for training to reduce prediction error is to encode the string more or less whole.

That mechanism explains what the extraction papers keep finding. Carlini's team defined the extreme case, strings recoverable from the model despite appearing in at most one training document, and then filled a table with real examples of it. A customer row sits close to this worst case: a name beside an email beside a phone number is precisely the rare, high-entropy shape that resists generalization. The scaling observation completes the picture. Memorization grew with model size in every comparison the 2021 team ran, models have grown by orders of magnitude since GPT-2's 1.5 billion parameters, and the 2023 result showed the trend surviving alignment. Across three model generations, the attacks got cheaper and more targeted.

What the studies do not show

Four things these papers did not demonstrate, in roughly the order people assume otherwise.

They did not show that most training data is extractable. GPT-2 trained on roughly 40 GB of web text, and the 2021 attack confirmed 604 memorized sequences from it. The 2023 attack's verified ChatGPT haul was ten thousand examples against a training set assumed to be vastly larger. An average sentence in a training corpus has a very low chance of ever coming back out. The risk is unevenly spread, though: it concentrates on the rare, unique strings that identify people, so a low average and a concentrated worst case are both accurate descriptions of the same data.

They did not show that typing into a chatbot puts text into a model. Memorization is a property of training data, and a chat prompt is not automatically training data. Whether a prompt ever reaches a training run depends on provider policy: which tier trains on conversations by default, what an opt-out covers, how long deleted chats are retained. Those policies differ between providers and shift with terms updates. The papers settle the mechanism side only: if text does end up in training data, memorization is possible, and rare identifying strings are the most likely candidates.

They did not show that today's products fall to yesterday's published attacks. After the 2023 team's disclosure, OpenAI began filtering prompts that repeat a single token. Dropbox engineers found in January 2024 that repeating a two-token sequence slipped past the filter and still triggered divergence — reported on January 24, filtered by January 29. The history shows a running contest in which extraction methods resurface against each new defense. The list of attacks that still work today is not something the papers provide.

And they did not document a breach of anyone's private uploads. GPT-2's web scrape and the Enron emails were public before the experiments touched them, by the researchers' own careful design, and the ChatGPT extraction counted only what it could match to public web text, so nothing non-public was demonstrated there either. The literature shows the machinery exists; it has no incident of private data to report.

One distinction runs through all four points. These studies demonstrate that extraction is possible under the conditions they measured. Estimating the probability that a specific uploaded file will be retained, trained on, memorized, and later extracted is a different exercise, and none of these papers performs it. Every step in that sequence depends on choices — retention policy, training-data selection, deployment defenses — that sit with the provider and were outside the studies' scope. Treating the sequence as a confirmed path from upload to extraction overstates what was measured. At the same time, the memorization mechanism the studies demonstrated is real, and an upload decision that ignores it rests on less evidence than one that accounts for it.

Where this fits in an upload decision

Most of what happens after a file is sent belongs to someone else: the provider's defaults, the wording of an opt-out, next year's terms update, a training pipeline you will never see. The variable entirely in your hands is what the file contains at the moment it leaves. That is the variable this research makes worth controlling, because the strings it identifies as highest-risk — rare, unique, identifying — are the ones you can transform before sending anything.

Replacing identifying values with consistent aliases changes what the worst case yields. If an aliased export were somehow retained, trained on, and memorized, an extraction attack would recover PERSON_001 beside EMAIL_001, and the mapping that connects those codes to people stays on your machine. Because the aliases stay consistent across the file, analysis that depends on repeat values — return contacts, journeys, segments — still works on the copy; Redaction vs. pseudonymization: which keeps your spreadsheet useful? compares the techniques for keeping that structure. Aliasing covers only the values detection found. The common blind spots are documented in What automatic PII detection misses in spreadsheets, and the measured per-pattern scores are published in the detection benchmark.

Data Alias implements this workflow in the browser, with consistent aliases and a review step before export.

Read as a whole, the extraction literature informs one decision well: what a file should contain before it goes anywhere you do not control. It cannot settle the others — which provider retains what, whether a given service trains on uploads, or the odds that any one file is ever memorized. Those are account and policy questions with per-provider answers; Before you upload a customer spreadsheet to ChatGPT collects them into a checklist.

Further reading

• Extracting Training Data from Large Language Models — Carlini et al., USENIX Security 2021. The 604 sequences, the single-document memorization definition, the model-size comparison.
• Scalable Extraction of Training Data from (Production) Language Models — Nasr, Carlini et al., ICLR 2025, published there as "Scalable Extraction of Training Data from Aligned, Production Language Models." The divergence attack, the $200 figure, the alignment finding.
• PII-Compass: Guiding LLM training data extraction prompts towards the target PII via grounding — Nakka et al., PrivateNLP workshop at ACL 2024. Targeted extraction and what grounding does to the success rate.