10 ms·
A Roblox cheat and one AI tool brought down Vercel's platform
- EdwardDiego 6mo agoA frigging Roblox cheat... And I thought it was bad when my son got compromised by a Roblox cheat, but they only they grabbed his Gamepass cookies and bought 4 Minecraft licenses, which MS quickly refunded...
- uyzstvqs 6mo agoThis essentially means that Vercel got hacked by a bunch of teenage script kiddies. Though on the positive side, we'll probably see some arrests soon.
- 8cvor6j844qw_d6 6mo agoYeah, I'm curious why a game cheat is allowed to run in the first place. Do these companies not have device controls, or do they just not care? Feels like the employee pulled a LastPass Plex move.
- loloquwowndueo 6mo agoRoblox doesn’t care about anything other than squeezing money out of addicted children. It’s not a competitive platform like say WoW or overwatch; nobody is really there to win and there are zero stakes if you do or don’t.
- jesse_dot_id 6mo ago> How many developers do you think knew that checkbox existed? How many assumed their database credentials and API keys were encrypted by default? If I don't see asterisks, I'm not hitting save on the field with a secret in it. Maybe they were setting them programmatically? They should definitely still be looking to pass some kind of a secret flag, though. This is a weird problem for a company like Vercel to have.
- apgwoz 6mo agoYou pretty much have to assume someone is going to put sensitive data in an input like this. Encryption by default is the only sensible choice.
- lemagedurage 6mo agoBut the encrypted API key doesn't work, it needs to be decrypted first. Let's give the server access to the private key so it can decrypt the API key. We can do this by putting the private key in an env var. But now the private key is unencrypted. Ah, it doesn't work.
- apgwoz 6mo agoYou’re thinking too much. When you run the app, the system decrypts the secrets and makes them available as env vars (or some other mechanism). In an admin ui, you list the names of secrets only, and provide a “reveal” or a “replace” on each one. They are never decrypted unless explicitly asked for. Is this perfect? Absolutely not. The key is controlled by the company, but it can be derived in a manner that doesn’t allow for the dump of everything if it’s leaked.
- kstrauser 6mo agoMy understanding is this is exactly how Vercel works. The users hadn’t checked the “don’t ever reveal, even to me” box next to the sensitive values. If they had, the attacker would only have been able to see the names of the variables and not their values.
- apgwoz 6mo agoAh. The article has since been updated to point out that it’s not plaintext, but encrypted at rest (which would be expected). OK.
- lemagedurage 6mo agoMy gripe is that, if some additional authentication is then not required for deployments or SSH access, that whoever has access to the admin UI will still be able to access the box and extract all secrets, just with extra steps. There's usually no real security boundary between "admin UI controls the box" and "box requires secrets in plain text". I still like the approach, but I'm afraid that it feels more secure than it is, and people should be aware of that.
- ethin 6mo agoThis looks really really AI-generated even if the author did try to hide it by making some grammar elements improper. Idk if that diminishes it's accuracy though.
- progbits 6mo agoI don't know why you are downvoted. The article is AI blogspam, it doesn't have any more factual information than eg https://www.darkreading.com/application-security/vercel-employees-ai-tool-access-data-breach https://www.darkreading.com/application-security/vercel-empl... and is full of empty LLMisms. It's depressing people are willing to read this.
- mchl-mumo 6mo agoI didn't notice till I saw this comment and now I'm also confident it's significantly AI written.
- progval 6mo agoBecause a comment that just says it's AI generated provides no value to the readers. They could at least provide an alternative link like you did.
- croemer 6mo agoIt does provide value in that I know I shouldn't read it. It's clearly LLM written after a few glances.
- parable 6mo ago[dead]
- paganel 6mo agoThat article you linked to didn't mention that Context.ai, from where this mess originated, is a YCombinator company. Most probably its founders are on this very web-forum.
- varun_ch 6mo agoContext.ai seems like it was the SPOF. By definition it has a lot of your data, and they didn’t secure it properly.
- trick-or-treat 6mo agoClearly, Vercel should not have been compromised by this. I don't know who Context.ai is but I do know Vercel and I expected better from them. I also think we can expect to see a lot more stories like this.
- R41 6mo agogood article, these AI products are crazy supply chain risks.
- mudkipdev 6mo agoI'm getting a "failed to verify your browser" error on this article
- NitpickLawyer 6mo agoAnd, ironically, it's hosted on vercel :D
- ChrisArchitect 6mo agoRelated: Vercel April 2026 security incident https://news.ycombinator.com/item?id=47824463 https://news.ycombinator.com/item?id=47824463
- ryanisnan 6mo agoConvenience is our Achilles heel, as a society. We'll keep dangerous devices like the SuperBox in our homes, if it helps us get access to free movies and tv. We'll use single-use plastics, even if we know they're bad for the environment, because they're just so damn easy. We'll let AI run that thing for us, because it's just too easy. A whole generation has grown up without knowing what it was like to infect your computer with AIDS trying to download an MP3, and it shows. That caution will come back, just at a terrible cost.
- trick-or-treat 6mo agoWhen life gives you AIDS, make lemonAIDS!
- notpachet 6mo ago> Convenience is our Achilles heel More generically, our species' Achilles heel is our inability to factor in the long-term cost of negative externalities when evaluating processes that yield short-term positive results.
- kauli 6mo agoThis. From simple personal choices to the marker economy and politics. With games we're introduced to cheat codes pretty early in our lives. Some people outgrow them, some don't. Too bad our systems encourage their use, whether it's a time-to-market thing, cutting costs, or the next election.
- happygoose 6mo agojust because there's a chance of something bad happening doesn't mean its worth it to abandon all convenience and workflow improvements, though. if no one ever used workflow tools that could access the contents of their emails because of the risk of a leak, its possible the productivity loss across society from that would be much worse than from the security incidents (like this one). there are pros and cons to things. it's not wrong to choose something just because it has a small risk associated with it.
- yoaviram 6mo agoI believe this is inaccurate. Vercel env vars are all encrypted at rest (on their side). The 'sensitive' checkbox means you can't retrieve the value once it's set, which would have saved your ass in this case. Also, annoying to read an article like this without a single link to source material.
- trick-or-treat 6mo agoI think it's clear that some customers env vars got exposed, so that can only mean unencrypted, right?
- TheDong 6mo agoThey said "encrypted at rest", which they almost certainly are. If you spin up an EC2 instance with an ftp server and check the "Encrypt my EBS volume" checkbox, all those files are 'encrypted at rest', but if your ftp password is 'admin/admin', your files will be exposed in plaintext quite quickly. Vercel's backend is of course able to decrypt them too (or else it couldn't run your app for you), and so the attacker was able to view them, and presumably some other control on the backend made it so the sensitive ones can end up in your app, but can't be seen in whatever employee-only interface the attacker was viewing.
- trick-or-treat 6mo agoHmm, that's confusing. So they're eventually encrypted but plain-text at some point? Doesn't sound good TBH.
- magackame 6mo agoIt seems only encrypt and throw away the key would be the acceptable strategy
- TheDong 6mo agoThey need to give your app the environment variables later so they cannot throw away the key. For non-sensitive environment variables, they also show you the value in the dashboard so you can check and edit them later. Things like 'NODE_ENV=production' vs 'NODE_ENV=development' is probably something the user wants to see, so that's another argument for letting the backend decrypt and display those values even ignoring the "running your app" part. You're welcome to add an input that goes straight to '/dev/null' if you want, but it's not exactly a useful feature.
- kstrauser 6mo agoI think this is wrong about what “sensitive” means here. AFAIK, all Vercel env cars are encrypted. The sensitive checkbox means that a develop looking at the env var can’t see what value is stored there. It’s a write-only value. Only the app can see it, via an env var (which obviously can’t be encrypted in such a way that the app can’t see it, otherwise it’d be worthless). If you don’t check that box, you can view the value in the project UI. That’s reasonable for most config values. Imagine “DEFAULT_TIME_ZONE” or such. There’s nothing gained from hiding it, and it’d be a pain in the ass come troubleshooting time. So sensitive doesn’t mean encrypted. It means the UI doesn’t show the dev what value’s stored there after they’ve updated it. Not sensitive means it’s still visible. And again, I presume this is only a UI thing, and both kinds are stored encrypted in the backend. I don’t work for Vercel, but I’ve use them a bit. I’m sure there are valid reasons to dislike them, but this specific bit looks like a strawman.
- deleted 6mo ago[deleted]
- dkersten 6mo agoThis is also how other cloud providers do it, eg DigitalOcean.
- nallerooth 6mo agoI don't know how it works on Vercel, but on other platforms it usually means that the value will be redacted in logs as well.
- rcxdude 6mo agoYou always get people screaming about 'it should have been encrypted!' when there's a leak without understanding what encryption can and can't do in principle and in practice (it most certainly isn't a synonym for 'secure' or 'safe').
- ErroneousBosh 6mo agoWhenever someone says "But it should have been encrypted!" about things like configs on a server, I ask them how they'd implement that in practice. PoC or GTFO. I think you'll find it's a bit harder to do than you expect.
- trick-or-treat 6mo agoAccording to the email I got from Vercel it was a limited subset of customers and I'm not one: Initially, we identified a limited subset of customers whose Vercel credentials were compromised. We reached out to that subset and recommended that they rotate their credentials immediately. At this time, we do not have reason to believe that your Vercel credentials or personal data have been compromised.
- sitkack 6mo agoThat parentset was just you.
- doctorpangloss 6mo agoThis article is LLM authored and full of hallucinations. "Let that sink in for a second."
- deleted 6mo ago[deleted]
- souravroyetl 6mo ago[flagged]
- agent-kay 6mo ago[flagged]
- voidUpdate 6mo agoSomething has gone screwy with the timestamps on this page... They're saying they were posted "in 8 hours", "in a day", then the last one is "an hour ago"
- rightbyte 6mo agoLast edit maybe? It is so annoying when sites don't publish the original timestamp.
- voidUpdate 6mo agoIt's still showing a time in the future, which only makes sense if there is some kind of error with the server time or some kind of weird timezone conversion gone wrong
- ashirviskas 6mo agoCan confirm, in 6 hours.
- tesders 6mo ago[dead]
- aroido-bigcat 6mo agoFeels like the bigger issue here is how much implicit trust we’re starting to place in these AI-integrated workflows. Tools that sit in the middle (like Context.ai) end up becoming a pretty large attack surface without feeling like one.
- darkwater 6mo agoI don't want to do the easy finger-pointing and scapegoating but honestly, what should happen to the Context.ai employee that thought it was a good idea to play games in their work machine and, on top of that, install cheats which are by definition of dubious provenance? I know defense in depth, security layers etc etc but there is also some personal responsibility at play here. We can chalk up the Vercel's employee mistake to a defense in depth failure that's on the whole company and management, but installing a cheat...
- ErroneousBosh 6mo agoRight? This isn't "A Roblox cheat and an AI tool", this is a failure of basic basic basic opsec across two organisations. One for which the Context.ai employee needs to have their arse booted up and down the car park for.
- sitkack 6mo agoWhat about the context.ai security team? You can blame individuals, but security is a property of the system.
- baxtr 6mo agoIt’s a very fine line. How do you check if people adhere to policies and at the same time don’t monitor them permanently?
- Topfi 6mo agoEndpoint Detection and Response? Heck, not giving the person Admin privileges would have sufficed to prevent this. Or better hiring preventing people who install Roblox cheats on work devices... There is no excuse and no fine line here. Even outside them boasting about SOC 2 Type II, this would be embarrassing for an SME not in the tech sector.
- baxtr 6mo ago
- alanmercer 6mo ago[dead]
- jFriedensreich 6mo agoI don't see storing non-sensitive environment variables unencrypted as the main issue here. Sure at vercels scale, encryption at rest for any data would add some better baseline, but i see this article as two major user interface fails more than anything else. Oauth dialogs are just pathetic, they are years behind what is required and what UX research knows how to do things, none of the companies invested any amount of resources into it after it just worked well enough not to make most users churn. The env var problem is also ridiculous, you can only update, not see and check values in the interface if they are encrypted for most providers i know, that leads to really annoying UX and is the reason they are not marked as sensitive by default and opt out. Even if you could unlock them to edit, no one will enter their password again as that is too much hassle, meaning we need a way to read and edit encrypted env vars in the interface where they are created but not have more in the way than a passkey dialog. Its doable but afaik no provider would go the extra mile to get to this UX. (Of course there are tons of other red flags not looked at in the article, eg. how does an employees machine get access to production systems and from there access to customers connected with oauth and how does the attacker get to env vars from a google workspace account)
- azalemeth 6mo agoVery ironically, they seem to have upped their game. Trying to read TFA on an older version of firefox gives me the lovely message: Failed to verify your browser Code 11 Vercel Security Checkpoint, arn1::1776759703-rtDgRAtRyXvjD4IoU4RbqvkGmvQQCP7H Gah.
- Topfi 6mo agoOdd, they used Delve [0] and a SOC2 compliant company like Context.ai [1] should have an AUP, EDR, etc. that prevents their employees from installing a Roblox cheat on their work computer. Heck, even outside SOC2, I have never worked at a company without endpoint restrictions to prevent unauthorised installs. It's almost like the denials were in fact false and Delve truly was just selling a sticker, not providing an actual service. If I were a VC that had funded Delve for a considerable amount of time, I'd be embarrassed that we did not catch that. I'd probably rework my processes, publicly analyse how this alleged fraud got past me and go far and beyond in disclosing my findings to rebuild trust. I'd most certainly not think just cutting funding is sufficient given the situation. Even more so if I'd encouraged other companies funded by me to use their "services". I'd maybe even reevaluate whether a circular approach wherein our funded companies are incentivised to rely on other also by us funded companies leads to the best options being chosen and whether that isn't antithetical to a forward thinking environment and competition. At the same time, I'd also think that maybe such a setup just hides unsuccessful companies and potentially even alleged fraud which once it gets to the broader market, may cause significant harm... [0] https://web.archive.org/web/20250918025724/https://trust.delve.co/context https://web.archive.org/web/20250918025724/https://trust.del... [1] https://web.archive.org/web/20260217220817/https://www.context.ai/privacy-policy https://web.archive.org/web/20260217220817/https://www.conte...
- deleted 6mo ago[deleted]
- pama 6mo agoFailed to verify my iphone browser…. But my claw could read it and text me the contents. The web is turning silly…
- cyanydeez 6mo agore-read your sentences, are you sure it's the web...
- nslsm 6mo agoI can see how this happened: the employee was home, his kid wanted to play some roblox, he installed roblox and gave the kid the laptop, the kid decided to install the cheat.
- deleted 6mo ago[deleted]
- Nebsol 6mo agohow the heck did a roblox cheat do this with an AI??
- misswaterfairy 6mo ago> February 2026. An employee at Context.ai, one of those AI productivity tools that promises to "supercharge your workflow," downloads a Roblox cheat. The cheat contains an infostealer. > March 2026. The attacker uses Context.ai's compromised infrastructure to pivot into a Vercel employee's Google Workspace account. This Vercel employee had signed up for Context.ai's "AI Office Suite" using their enterprise credentials and granted "Allow All" permissions. Let that sink in for a second. A Vercel engineer gave a third-party AI tool full access to their corporate Google account. I swear this AI 'boom' is melting people's brains and zombifying them like Toxoplasma gondii[1] does to rodents, making them do risky things that ultimately get them eaten (or hacked...). [1] https://en.wikipedia.org/wiki/Toxoplasma_gondii https://en.wikipedia.org/wiki/Toxoplasma_gondii
- mnmnmn 6mo ago[dead]
- deleted 6mo ago[deleted]
- _pdp_ 6mo agoIf I have to make a guess, it wasn't just any Google Workspace app but Gmail. The attacker gained broad access to the victim's inbox. They where then able to login into some internal systems using magic links or one-time codes. It begs the question why there is no 2FA? And why did they had such a broad access to being with? If this is not case, the only other option I can muster is perhaps API credentials but stored in google workspaces? It is possible but odd.
- throwatdem12311 6mo agoLmaoooo this is why I never install anything but work stuff on my work machines. Always have everything separate. Even on my personal machines, I have separate non-sudoer user accounts for gaming because I’m often downloading random mods. My son even asked me just the other day why I don’t have Roblox on the Mac….yeah stuff like this is why.
- morningsam 6mo agoWhat's the source of the claim that it was a Roblox cheat? Neither the report linked at the start of this article nor Context.ai's and Vercel's notices mention this.
- r1ch 6mo agoThe timeline is off too if the Trend Micro report is to be believed. It makes for a catchy headline, but a source is definitely warranted here.
- pier25 6mo agoOk but how do you go from a Google Workspace to production env vars? Doesn’t Vercel use 2FA? Did the hackers also get the env vars encryption keys?
- jimmypk 6mo ago[flagged]
- nippoo 6mo ago"I went through about a dozen AI tools I've personally authorized in the last year after reading this. Nine of them have Google Workspace OAuth permissions that include reading all emails and accessing all Drive files. Nine. I authorized every one of them without reading the permissions because the onboarding flow asked and I was in a hurry." Do other (tech-literate) people do this?! Giving anything access to my emails and Google Drive would keep me up at night and I try and be very granular with permissions and revoke them when I don't use an app any more. I would assume that anything confidential/NDA in my emails had been compromised and leaked well before this point!
- coldpie 6mo ago> Do other (tech-literate) people do this?! I'm sure it's very common, yes. Permissions & popup fatigue is very real. Today, every application and website throws 6 dozen popups at you that you have to get through to get to the stuff you came there for. Most of it is marketing; some of it is from braindead lawyers; some of it is important; none of it gets read by users. At some point you give up and just click "yes, goddamnit, I have work to do" and all the security stuff is out the window. Always remember: there is no such thing as computer security. If your data is on a networked computer, consider it to be semi-public. The first and only rule of computer security is don't store or do anything on a networked computer that would devastate you if it were leaked or compromised And, make sure not to think about how much of our modern infrastructure is built on top of computers connected to the Internet.
- inetknght 6mo ago> *Nine of them have Google Workspace OAuth permissions that include reading all emails and accessing all Drive files. Nine. I authorized every one of them without reading the permissions because the onboarding flow asked and I was in a hurry." No, you didn't authorize every one of them without reading the permissions because the onboarding flow asked and you were in a hurry. You authorized it because the onboarding flow asked, and you weren't given an opportunity to say no. What are you to do: say no, and then not use the app? This whole concept is just wrong. Instead of saying "no" and the app seeing that you didn't grant permission: you should be able to say "no", and the app shouldn't see any denial at all. It should just see empty data when requesting it. Problem fucking solved. You get to use whatever apps you want, apps get to ask for whatever permissions they want, and you get to deny that permission without the app fucking you over.
- deleted 6mo ago[deleted]
- sergiopreira 6mo ago[dead]
- shantanulume 6mo ago“The attack surface wasnt sophisticated. It was convenient. And convenience is the only product the entire AI tooling industry is actually selling.” Great read
- 11pyo 6mo ago[dead]
- cheesecompiler 6mo ago"join to see"??? gtfo
- jeremie_strand 6mo ago[dead]
- sandeepkd 6mo agoI read the original article, then the detailed statement and then this article to better understand what happened. I might consider myself as some one who has fairly good understanding of security flows. Here is my take: 1. The security flows are half baked and custom implemented, they do not present a coherent story 2. No one fully understands the ecosystem as a whole and so far no one has been able to track what actually happened, adding audit logs were not part of the product ask so no one ever added them in thoroughness If I have to put my money then its the second one. The possible down the road action, at the most this incident would trigger more security engineers to be hired which may give the impression of improving things but in reality its probably going to create more blindspots where product engineers would hand out the responsibility to security engineers and they do not have much of an idea about the product flows
- arcza 6mo agoI quit reading the moment your website said "sign in with google" took a single pixel of my real estate. It became spyware at that moment. Fuck all these dark patterns and trackers. I've had enough and hope your Google Webmasters console bounce rate shows it.