AI prompt theft at inference providers is not a theoretical risk: it has been happening for months. That is the finding of a study by the security company Dragos. Its chief research officer, HD Moore, best known as the creator of Metasploit, sent about two million prompts containing unique, planted secrets to widely used AI services between July and November 2026, then watched where those secrets turned up. The result: attackers sat undetected inside at least three inference providers, sellers on a marketplace read customer requests, and privacy settings such as Zero Data Retention were often ignored. On Sunday, November 22, the AI router OpenRouter responded by removing three providers from its platform.
How the canary study works
A canary token is a secret that exists in only one place. If anyone ever uses it, you know that place has been exposed. Dragos gave every request its own AWS key, DNS name and URL that appeared nowhere else. The AWS keys were real keys tied to empty, fully logged accounts. The team later built a dedicated service that generates unique hostnames, URLs and credentials for each prompt and runs the services they point to: DNS, web, Git, SSH, databases and mail.
According to the report published on November 19, the first tokens fired within days, and the tokens from July are still firing today. Because every token can be traced back to its prompt, model, router and privacy setting, the researchers could reconstruct exactly which request leaked and by which path.
AI prompt theft at three providers
The most serious finding: attackers were actively stealing data from three different hosting providers. The earliest breach dates back to mid-July. None of the three providers knew about it until Dragos told them. Because the bait prompts were processed alongside real customer requests, the researchers assume those were copied too.
The largest case involved a couple of hundred planted AWS keys, each sent in exactly one request to one provider during a few short windows in late October. Every request sent during those windows was stolen, including some the provider had rejected before a model ever ran. That suggests the attacker was copying traffic at the front door. In the other two breaches, some of the stolen requests were flagged for Zero Data Retention. “A privacy flag doesn’t help when the attacker is already inside,” Moore writes.
The attackers followed the same path with every key. They checked whether it worked, in some cases within a few hours of the request. They then searched S3 buckets, Secrets Manager and Parameter Store for data and tried to enable models in AWS Bedrock, create Bedrock API keys and add their own IAM users. In total, the bait keys were used for tens of thousands of AWS API calls. According to The Hacker News, one attacker brought in the AI-driven penetration testing framework ARTEX on November 1 to confirm that its keys were still live. ARTEX already featured in the cyberattack on the European Commission.
Marketplace sellers simply read along
The second finding required no break-in at all. AI routers offer a single API endpoint that forwards requests to many models and providers. Some are open marketplaces where the cheapest seller wins the request. The sellers Dragos caught only had to list a model at the lowest price, enable request logging and wait.
What stood out was how personal it was. These were individuals on home internet connections. In one case, someone opened both secret URLs from two requests in a desktop browser about a day and a half later and started guessing usernames. In another, someone on a Mac on a home broadband line in France tried to log in as root over SSH with passwords taken from two prompts, typing each attempt by hand. Another router was quietly forwarding part of its traffic to the marketplace without disclosing it. Using a kind of traceroute for AI requests, Dragos traced about 2,000 routes. Several hundred passed through systems that could not be identified, and one seller secretly added about 1,000 extra tokens to every prompt.
Zero Data Retention was often ignored
The third finding concerns privacy promises. Nobody used a credential here, but the secret URLs from the prompts were fetched repeatedly by companies that should never have seen the requests. The clearest example is a Chinese AI lab that Dragos does not name. Its systems fetched the secret URLs from more than half of the prompts sent to its models, including roughly two-thirds of the requests marked “no data collection” or Zero Data Retention. That also happened when the requests went to US providers reselling the lab’s models.
Large US providers had issues too. One kept crawling a secret URL more than two months after a single request to its search endpoint. The strangest case involved a secret link that showed up in a chat on the workplace app Feishu 24 days after the request and was opened within two minutes from three desktops and three phones in Beijing and Shanghai. The request had gone to a US provider with the Zero Data Retention flag set.
OpenRouter removes three providers
OpenRouter published an announcement about the incidents on Sunday. The company says it has off-boarded three providers, removed 125 endpoints from Zero Data Retention eligibility and notified all potentially affected customers. The removals cover providers flagged by the researchers plus others cut as a precaution. OpenRouter says it was not breached itself. It plans to adopt the researchers’ testing approach on an ongoing basis and will only restore Zero Data Retention status after validation. How heavily OpenRouter is now used as a gateway to models was recently shown in the US report on Chinese open-weight models.
OpenRouter does not say which three providers it removed, and Dragos does not name the three compromised providers. Three providers have published their own notices, which OpenRouter links to: Phala reports a compromised inference server, Together AI a leak of stored prompt samples from June and July, and Novita a configuration error that forwarded requests to another company.
What organizations should do now
- Rotate credentials sent in prompts: anyone who has sent keys, passwords or internal links to models through routers or resellers should treat them as exposed. AI agents are particularly risky, because they pass on whatever they can read.
- Decide where requests go: according to Moore, a privacy flag only binds the first hop. Router customers should pick the upstream providers themselves and verify that their settings actually change the route. Dragos advises against reseller marketplaces.
- Seed your own canaries: a fake key in the context that agents and pipelines already send works as a tripwire. Any use of it is an incident.
- Enable secret filtering: some providers automatically strip known credential formats from prompts. That helps, but it misses sensitive content that is not shaped like a key.
How quickly stolen credentials from developer environments are abused was also shown recently by the compromised Microsoft Agent Framework packages.
Our take
The study hits a sore spot for the AI industry. Prompts often contain source code, internal communications, customer data and credentials. That makes them more valuable than the compute it takes to process them. At the same time, requests travel through chains of routers, resellers and hosts that customers can barely see. In that chain, privacy settings are only worth as much as the weakest link. The researchers say they are now working with about a dozen providers, but stolen keys and links are still being accessed. Anyone using AI services should therefore not ask whether a provider promises Zero Data Retention, but who can actually see the request on its way there.



