Black October: every AI dev platform breach in one list, and what to rotate
Wiz's "Black October" post lists twelve AI developer platforms with publicly reported incidents as of its latest edit (Nov 3, 15:40 UTC): Fireworks AI, Andromeda, Zeabur, Lightning AI, OpenCode, Railway, Warp, CommandCode, Northflank, Brev (NVIDIA), Trigger.dev and Chainstack. Wiz says the reports show attackers "explicitly hunting stored environment variables, with a focus on those granting AI access," with no single vulnerability linking them.
If you stored a key on any of them, rotate it at the provider that issued it. Deleting it from the platform doesn't stop it working. Each row below says what that vendor says was exposed and what it says to rotate, quoted from its own notice where one exists.
The list, in Wiz's order
Dates are the vendor's, or Wiz's where the vendor gave none. "Our note" marks advice that is ours, not the vendor's.
Fireworks AI · Oct 6. Exposed: "environment variables, including API keys and tokens stored as secrets," used in evaluation jobs in "a limited number of customer accounts." Rotate: affected customers were emailed the credentials needing action; replace the Fireworks API keys it revoked, and rotate the third-party credentials named in the notice with their issuers ("Removing a stored secret alone does not invalidate the underlying credential"). No notice means "no incident-specific action is currently required." Fireworks security bulletin.
Andromeda · on or about Oct 9 (found Oct 13). Exposed: "certain Andromeda and customer credentials" in "a limited number of customer accounts." Rotate: Andromeda says it rotated all affected keys and that "If we believe you were affected, you've already heard from us"; it has published no customer rotation list. Questions go to your Andromeda representative. Andromeda security update.
Zeabur · Oct 14–15. Exposed: environment variables in the platform database (3,218,965 records, names and values, confirmed), account data for 321,810 accounts, secrets on the shared cluster in 9 regions, and commands run in 865 database services. Rotate: "Replace every password and key stored on Zeabur, even if you already replaced them after the August incident." Zeabur is shutting down; see Zeabur is shutting down: migrate and rotate. Zeabur preliminary report.
Lightning AI · Oct 15. Exposed: "certain customer credentials were compromised to gain access to environments and connected third-party services" (Studio, Deployments, Pipelines, Drive; not GPU cluster rental, Kubernetes or SLURM). Rotate, for all users whether contacted or not: "Rotate secrets, API keys, and revoke credentials stored in your AI Studios environment such as Github credentials," and update credentials for third-party services your VMs accessed. Lightning revoked the cloud-account keys it identified. Drive is back read-only via the CLI since Oct 28. Lightning AI incident page.
OpenCode · night of Oct 16. Exposed: 1.4% of OpenCode Console users, "a subset of users who had external providers linked." "If you did not get an email you were not impacted." Rotate: the provider keys you linked to Console, revoked at OpenAI, Anthropic or Google. Our note: removing a key from Console doesn't revoke it. OpenCode on X.
Railway · Oct 20–22. Exposed: encrypted environment variables, some decrypted; Railway can't tell which. Rotate: "immediately roll all environment variables, including any API keys, stored in Railway, as well as any private registry credentials, and review logs." See Railway env var disclosure: what to rotate. Railway security disclosure.
Warp · found Friday Oct 23, disclosed Oct 26. Exposed: "a limited amount of agent trace and telemetry data," with "an upper bound of roughly 1% of Warp users." Rotate: nothing published; "Anyone who has been impacted will receive instructions from us." Our note: if an agent session read a .env file or printed a key, treat that key as exposed. Warp CEO on X.
CommandCode · Oct 28. Exposed: CommandCode says "we've found no evidence that any users were impacted." Rotate: it "revoked all API keys"; "log in to Studio again to generate a new key," then update wherever you used the old one. CommandCode on X.
Northflank · Oct 30. Exposed: "Assume all environment variables have been decrypted." Rotate: "all secrets now, both integration secrets and project secrets," starting with AI provider keys, then cloud, payment, private keys, database and other service credentials; sign in again and reconnect integrations. See Northflank breach: what to rotate. Northflank status page.
Brev (NVIDIA) · Oct 4–20, per Wiz. We found no public notice from NVIDIA. The press report Wiz links quotes NVIDIA's email to affected customers: the potentially exposed item is "the SSH key that Brev provisioned for your Brev-provisioned VM," which "has since been rotated." Our note: if you got that email, check each VM's authorized_keys and rotate any credentials stored on the VM. Wiz: Black October.
Chainstack · Oct 26 (dev), Oct 29 (production), cut off Oct 31. Exposed: "organization names, account holder names, the email addresses of your organization's members on Chainstack, and password hashes." The attacker "also had access to systems holding node credentials. We have no evidence they were read, but we are treating them as exposed." Rotate: change your password, "including on any other services where they reused the same password"; rotate node credentials (self-service in the console; per the docs, old and new work together until you click Deactivate old credentials); "Regenerate the platform API keys if they use the platform API." Chainstack incident report, Nov 3.
Outside Wiz's list
Developer-tool incidents from the same weeks that Wiz doesn't count (as of its Nov 3 edit).
GitBook · Nov 2–3. Exposed, per its post-mortem: GitBook API tokens, integration tokens and secrets, session signing secrets, and the GitBook app's access to connected GitHub and GitLab repositories; GitBook treats all customer data as exposed. Rotate: create a new API token (it revoked personal tokens), rotate third-party and published-integration secrets, regenerate the authenticated-access signing key; Git Sync keys were rotated for you. See GitBook breach: what to rotate. GitBook post-mortem.
WorkOS · Oct 30. Exposed: an internal sales tool; WorkOS says the incident "affected only CRM and customer support data" (business contacts, account histories, summaries of sales calls, support and customer Slack conversations, product-usage metadata) and that there is no evidence its application data or production services were accessed. Rotate: nothing published; it warns to watch for phishing that uses those details. WorkOS bulletin, Nov 2.
Appwrite · Nov 1 (notified Nov 4). Exposed: a Frankfurt compute access token that "could have been used to read the environment variables of Functions and Sites" for ~11 hours (03:53–14:46 UTC); Appwrite found no evidence the attacker read any variables, and is notifying 1,022 project owners as a precaution. Rotate (if emailed): secret-named variables set before 14:46 that day on Functions/Sites that ran on that Frankfurt compute — create replacements at each issuer, then revoke the old ones; for Appwrite API keys, create a scoped key, update the variable, redeploy, delete the old key. See Appwrite runtime vulnerability: what to rotate. Appwrite incident report, Nov 4.
Used more than one? One order for all of it
Only Northflank published an order; this one follows it. For each key: pull its usage, revoke it at the issuer, create a replacement, update every platform that holds it, redeploy. A key you rotated after one incident and then stored on another affected platform needs rotating again.
AI provider keys. OpenAI, Anthropic, Google, OpenRouter, Bedrock. Export each key's usage and billing first, while you still know which key was which; the provider will ask for it. Then revoke, replace, and set a spend limit where offered.
Cloud credentials. AWS access keys, GCP service account keys. Create the replacement, deploy, deactivate the old one, and search the cloud audit log (CloudTrail on AWS) for calls made with it.
Payment keys. A Stripe secret key can create charges, issue refunds and read customer data; roll it with the shortest expiry you can deploy within, as in rotating a leaked Stripe key without breaking webhooks. Afterwards, a read-only scan such as Stripe Leak Check flags card-testing bursts, fraud warnings you can still refund and disabled webhook endpoints, and it needs only a read-only restricted key, not the secret key you just replaced.
Private keys and node credentials. SSH and deploy keys, signing keys, RPC node credentials. Install the new one, then remove the old.
Database and service credentials, then everything else. Change the password in the database itself, then the connection string. Then webhook secrets, email and SMS keys, analytics, error tracking, storage.
Platform accounts. Change the password anywhere you reused it (Chainstack exposed password hashes), sign in again, and reconnect GitHub and cloud integrations the platform revoked.
Want us to run it for you? The Ship Check Audit lists every key that ever left your server and confirms each old value now gets a 401 or 403: $149, within 48 hours.
Search provider logs for the IP addresses in Wiz's IOC table (23.106.46.239, 38.246.114.118, 104.243.32.50, 31.57.201.201). Wiz says they tested stolen keys but it has no evidence linking them directly to these platform breaches.
Ship Check Kit · $29
Rather not run it by hand?
The Ship Check Kit has your coding agent find every key that ever left the server (SC-SEC-01 for git history, SC-SEC-02 for the browser bundle) and then confirm each old value is actually revoked, not just replaced (SC-SEC-03, which fails while an old value still gets anything but a 401 or 403). Three of 44 launch checks, each with the evidence and the fix.
