Security firm Dragos spent five months hiding trackable secrets inside nearly two million AI prompts, then watched where those secrets surfaced. The results, published November 19, show breached providers, resellers reading customer prompts, and privacy settings that failed when tested.
The experiment used canary tokens, secrets that exist in only one place, so any later use points back to the leak. Each prompt carried a unique web address, password or Amazon Web Services key tied to an empty, fully logged account.
Three Providers Were Already Breached
The most serious finding was that attackers were already inside three AI hosting providers, stealing data as requests arrived. Dragos says none of the three knew about the breach until it notified them. The earliest theft dates to mid-July 2026.
In the largest breach, a couple of hundred Amazon keys were stolen during a few short windows in late October. Every request Dragos sent during those windows was taken, including some the provider rejected before any model ran. The copying happened at the front door.
Once stolen, the keys were tested within hours, then used to list storage buckets, search secret stores and try to switch on hosted models. The pattern echoes the PyPI hijack that hit Microsoft’s agent framework, and the bait keys drove tens of thousands of Amazon API calls.
Resellers Read Prompts Without Hacking Anything
The second finding needed no exploit at all. On inference marketplaces, sellers offer popular models at the lowest price, and the platform’s standard logging lets them read the prompts they handle. Dragos caught individual sellers doing exactly that, working from home connections and personal computers.
Privacy Settings Were Ignored
The third finding is that privacy settings were often ignored. A Chinese AI lab’s systems fetched the secret web addresses from more than half the prompts sent to its models between July and October, including roughly two-thirds marked no data collection or Zero Data Retention.
The same scraper showed up when prompts reached the lab through United States hosting providers reselling its models, so a customer who chose a United States provider to keep data in the country did not get what they paid for.
One secret link from a prompt sent to a United States provider with the strictest privacy flag later appeared in a company chat in China. Within two minutes it was opened from three desktops and three phones in Beijing and Shanghai.
Our Read
The study’s power is that it measures what happens to prompts, not what privacy pages promise. Three breached providers, resellers reading prompts as a business model, and a lab scraping links marked do-not-store all point the same way: once a prompt leaves your systems, every hop can keep a copy.
Dragos names no providers, sellers or labs, which limits how far any customer can act on the findings. Even so, the pattern is hard to dismiss: the cheapest route through the inference market is often the leakiest, and a privacy flag only binds the first hop, as Dragos puts it.
Why It Matters
Prompts now carry source code, contracts, customer lists and credentials, so a leak in the inference chain is a data breach by another name. The study gives every team that routes prompts through cheap resellers a concrete reason to check where their requests actually go, and what each hop keeps.
Background
Dragos built the study on canary tokens and a custom seeding service that ties every secret back to one prompt, one model and one privacy setting. It lands days after a hacked DeepSeek model answered users for three days, another quiet supply-chain failure.
Source: Dragos’ canary-token study


0 Comments