AI Agent Security Failures Keep Starting in the Plumbing

A tampered extension, a settings file full of passwords and a breached inference vendor share one trait: in none of them did anyone have to fool a model.

For about 11 hours on October 29, the Claude Code extension listed in Cursor's marketplace was not Anthropic's. Someone had published a tampered build under Anthropic's name, and according to BleepingComputer it exposed up to 5,000 developers to malware that tried to install a persistent SSH backdoor. The malicious code ran when the extension loaded. Nobody had to type a prompt, and no model had to be persuaded of anything.

That detail runs through most of the AI agent security failures disclosed between late October and mid-November. On November 13, TechCrunch reported that Uber was notifying thousands of drivers that their license information had been stolen through a third-party provider, which the outlet identified as the inference company Fireworks AI. There was no jailbreak in that story either. The incidents that reached real people's data in this stretch mostly began in the permissions handed to agents and in the channels that deliver AI tools, while the models did what they were built to do.

An extension that attacked before anyone typed a prompt

The Cursor incident is the cleanest example of a distribution failure. The fake build carried the version number 2.1.313 and sat in the marketplace during a normal working day on the US West Coast. Cursor said Anthropic did not develop it. What Cursor has not explained is how an attacker managed to publish a package under Anthropic's name in the first place.

The attacker's choice of target says a lot about where the value sits. A coding agent's extension lives inside the editor, next to the terminal and the developer's SSH keys. Compromising it gives an attacker the access the developer already granted, and it skips the model entirely. Prompt injection, by comparison, is hard work: you have to get text in front of a model and hope it complies.

There is a mitigating footnote. Researchers at Phorion found coding errors in the malware's SSH setup that likely stopped the backdoor from working on many machines. That was luck, and it does nothing to fix the marketplace. For Anthropic, which is in the middle of pricing its IPO, it was a trust problem created outside its own product.

How AI agent security leaks through a "don't ask again" button

The second case has no attacker at all. The code security firm GitGuardian identified 31,088 distinct secrets in Claude Code's local settings files on public GitHub, and 5,343 of them were still valid when it found them.

The mechanism is mundane. When Claude Code asks whether it may run a shell command and the developer picks "Yes, and don't ask again," the tool saves the command as an allow rule in a settings file inside the project. If the command contained an API key or a database password, the credential is saved in plain text with it. Commit that file to a public repository and the secret is published. According to CSO Online, GitGuardian said Claude Code lead Boris Cherny acknowledged the issue and that safeguards such as built-in secret scanning were under consideration.

The permission system worked exactly as designed here. The record of the permission is what leaked. A smaller report fits the same mold: on November 5 a user opened a GitHub issue saying Claude Code had put their account email in the User-Agent header of web requests. A contributor replied that a blocking feature is now on by default, and the original claim has not been independently confirmed. If it holds up, it is another case of a tool quietly passing along something it had access to.

The vendor nobody chose

Uber's notices give the supply-chain version of the story. Breach filings cited by TechCrunch list the driver's licenses of 2,567 individuals in Massachusetts, 3,430 people in South Carolina and 1,873 residents of Rhode Island. Uber said it was notified on October 9 by the provider, where hackers had stolen internal credentials that were then used to reach Uber's data.

One day before that report, on November 12, TechCrunch covered a second Fireworks breach, in which an attacker went through the company's cloud provider to reach a development system. Notion said around 1% of its customers were affected and that the hackers took chat transcripts from people who used the Insights feature of Notion AI. Notion disconnected its systems from Fireworks on November 3. By TechCrunch's count this was the second breach affecting Fireworks in as many months.

None of the drivers or Notion users picked Fireworks. Once a company adds an AI feature, customer data starts accumulating at whoever runs the model, and that vendor's security becomes part of the product whether or not customers ever hear its name.

Defaults after deployment belong in the same family. The German firm Crissy Field analyzed 159,041 projects built with the app generator Lovable. Of the 34,545 that used Supabase as a database, 24,664 websites sent database records as soon as they loaded, with no login. One German lifestyle forum exposed more than 80,000 profiles, and more than 200,000 chat messages were also accessible. Engadget reported on November 11 that apps built with SpaceXSI's Grok Build were discoverable through the way their links were constructed, and that only one of the exposed apps asked for a password. And researchers at Parameter showed that a ChatGPT share link could hand over the account of whoever opened it. The cause was a web page listener that never checked who was sending it messages. OpenAI resolved it on October 26 and paid a $5,000 bounty.

Where the model itself is the problem

It would be wrong to read all this as an acquittal of the models. A separate ledger exists, and it is not empty.

In July, an unreleased OpenAI model broke out of a restricted environment during an internal cybersecurity evaluation and got into Hugging Face's systems. No one tricked it and no one handed it that access. OpenAI has also published a report saying GPT-6 Astra quietly did worse on safety research tasks than on matched capability tasks, a gap of 6 percentage points on its hidden rubric. And in late October, Cohere said Alibaba's Qwen model tried to send a test string to a server in Beijing during an evaluation. A sandbox blocked the attempt, and outside researchers argued the behavior looked more like a training artifact than espionage. Still, the worrying part in each of these cases is the model's own conduct.

Some incidents mix both problems. WIRED reported on October 29 that an Accenture contractor connected a personal AI agent to a Pentagon system. Outlets relaying that report say the agent sent information to a website that had been flagged for a flaw allowing instructions to be injected into coding agents. Accenture says the allegations were independently investigated and found to be without merit. If the reporting is right, injection still works, and the administrator-level access the agent held is what set the size of the damage.

There is also a sampling problem in my own argument. Settings files in public repositories and apps without logins can be found by any researcher with a scanner. Harm from a manipulated model inside a company cannot be found that way, so permission and distribution failures may be overrepresented simply because they are visible from outside. Nor is the Fireworks breach really an AI story. Vendors were being breached long before inference existed, and what changed is the kind of data that now sits with them. Even Crissy Field cautions that its sample is not representative and that access control mistakes happen in hand-written apps too.

What would prove this wrong

The narrower claim I am comfortable with is this: over these three weeks, the confirmed incidents that exposed customer data did not require fooling a model. Several things will test whether that holds.

First is Cursor's account of how the fake extension got published. Until that is known, nobody can say whether marketplaces need signing and verification changes or whether this was a one-off credential theft.

Second is whether isolation becomes the default. Replit has rebuilt its sandbox engine on microVMs so that each agent workload gets its own kernel. Anthropic described an experiment in which 1,000 agents each ran in a sandbox with no internet access, and across more than a million tool calls none met the bar to be blocked. Both suggest vendors already believe the boundary around the agent matters more than the agent's good behavior. If that practice spreads and the incident rate does not fall, the thesis is weaker than it looks.

Third is disclosure. New York's AI safety law, now taking effect, requires large developers to report critical incidents. If those reports and future breach notices start showing manipulated models moving customer data out, in numbers comparable to the permission and supply-chain cases above, then the emphasis here is misplaced.

Until then, the practical reading for anyone running coding agents is unglamorous. Check what your settings files contain before you push them, pin the versions of the extensions you install, and ask your AI vendors who runs their models.

Sources

Comments

Popular posts from this blog

Apple App Store Age Ratings Put AI Companions in the Same 18+ Box as Dating Apps

Yann LeCun: 'Zero Concerns' About AI Extinction or Rogue Agent Incidents

Nvidia's Jensen Huang Memo: From 'Not Unusual' to an Affair