OpenAI contractors read real ChatGPT chats under secret ‘Project Lily’

Hundreds of outside contractors hired by OpenAI are reading real ChatGPT conversations as part of an internal initiative codenamed "Project Lily," according to an investigation by 404 Media that raises fresh questions about the privacy of more than 900 million ChatGPT users.

The contractors, paid more than $50 an hour through third-party staffing firms, read user prompts, summarize the user's intent, and then rate four ChatGPT-generated responses on a scale of one to seven. The goal is to make the chatbot sound less robotic, less sycophantic, and more natural — but the process means that sensitive personal details shared in what many users treat as private conversations can end up in front of human eyes.404media

What Contractors See

Internal documents and instruction guides obtained by 404 Media reveal that the contractor dashboard sometimes includes a "user memories summary" — an overview of what a user has previously discussed with ChatGPT, which can reveal their approximate location, profession, or personal circumstances. OpenAI says it strips usernames and runs conversations through an automated "Privacy Filter" before they reach reviewers, but the company acknowledges the filter "can miss uncommon identifiers or ambiguous private references."404media

Some prompts seen by contractors showed users asking ChatGPT to "keep this between us," unaware that such instructions do not override the platform's data-collection practices. "No," one contractor told 404 Media when asked whether ChatGPT users realize humans may be reading their chats. "I don't think they would imagine some contractor somewhere is analyzing the conversations."tomsguide+1

Training Rules and Industry Practice

Instruction guides tell reviewers to penalize "AI-speak" such as unnecessary emoji use, to flag responses where ChatGPT claims human experiences, and to mark sycophantic language — a known problem linked to lawsuits alleging that an overly agreeable ChatGPT model contributed to user harm. Reviewers are not expected to fact-check responses but are told to flag factual errors they notice and escalate tasks containing personal information or safety concerns.404media

The practice is not unique to OpenAI. Anthropic confirmed to 404 Media that it uses human review to improve Claude when users leave training enabled, and Google's Alphabet Inc. ↗ 2.82% Gemini displays a disclaimer that "humans review some saved chats."tomsguide+1

How to Opt Out

For free, Plus, and Pro ChatGPT accounts, model improvement is turned on by default. Users can disable it by navigating to Settings, selecting Data Controls, and toggling off "Improve the model for everyone." Enterprise, Business, and Edu accounts have the setting off by default. The opt-out applies only to new conversations — prompts already de-identified and queued remain in the training pipeline.tomsguide+1

After 404 Media contacted the company, OpenAI updated its help page to add more detail on the opt-out process, though it still does not explicitly acknowledge that humans may read users' prompts.404media