4 ms·
See the public phab ticket: https://phabricator.wikimedia.org/T419143 https://phabricator.wikimedia.org/T419143 In short, a Wikimedia Foundation account was do
by tux3 7mo ago
See the public phab ticket: https://phabricator.wikimedia.org/T419143 https://phabricator.wikimedia.org/T419143
In short, a Wikimedia Foundation account was doing some sort of test which involved loading a large number of user scripts. They decided to just start loading random user scripts, instead of creating some just for this test.
The user who ran this test is a Staff Security Engineer at WMF, and naturally they decided to do this test under their highly-privileged Wikimedia Foundation staff account, which has permissions to edit the global CSS and JS that runs on every page.
One of those random scripts was a 2 year old malicious script from ruwiki. This script injects itself in the global Javascript on every page, and then in the userscripts of any user that runs into it, so it started spreading and doing damage really fast. This triggered tons of alerts, until the decision was made to turn the Wiki read-only.
- londons_explore 7mo agoDidn't realise this was some historic evil script and not some active attacker who could change tack at any moment. That makes the fix pretty easy. Write a regex to detect the evil script, and revert every page to a historic version without the script.
- jacquesm 7mo agoTrue but it does say something that such a script was able to lie dormant for so long.
- outofpaper 7mo agoWhy would anyone test in production???!!!
- fifilura 7mo agoI have never heard of this kind of insane behaviour before.
- ninth_ant 7mo agoSelecting the wrong environment in your test setup by mistake? I refuse to believe that someone on the security team intentionally tested random user scripts in production on purpose.
- irishcoffee 7mo ago> I refuse to believe that someone on the security team intentionally tested random user scripts in production on purpose. Do I have a bridge to sell you, oh boy
- withinboredom 7mo agoOnce you get big enough… there comes a point where you need to run some code and learn what happens when 100 million people hitting it at once looks like. At that scale, “1 in a million class bugs/race conditions” literally happen every day. You can’t do that on every PR, so you ship it and prepare to roll back if anything even starts to look fishy. Maybe even just roll it out gradually. At least, that’s how it worked at literally every big company I worked at so far. The only reason to hold it back is during testing/review. Once enough humans look at it, you release and watch metrics like a hawk. And yeah, many features were released this way, often gated behind feature flags to control roll out. When I refactored our email system that sent over a billion notifications a month, it was nerve wracking. You can’t unsend an email and it would likely be hundreds of millions sent before we noticed a problem at scale.
- ghxst 7mo agoI would say you can get to this point far below 100 million people, especially on web. Some people are truly special and have some kind of setup you just can't easily reproduce. But I agree, you do really have to be confident in your ability to control rollout / blast radius, monitor and revert if needed.
- ninth_ant 7mo agoYes this is a common release practice. However this is a different situation as we’re talking about running arbitrarily found third-party scripts. I can’t imagine that was ever intended to be done in production. Fun story, when I worked at Facebook in the earlier days someone accidentally made a change that effectively set the release flags for every single feature to be live on production. That was a day… we had to completely wipe out memcached to stop the broken features and then the database was hammered to all hell.
- HoldOnAMinute 7mo agoThere are plenty of ways to safely test in production. For one thing you need to limit the scope of your changes.
- kQq9oHeAz6wLLS 7mo ago"Everyone has a test environment. Some are lucky enough to have a separate production environment."
- jl6 7mo agoLetting ancient evil code run? Have we learned nothing from A Fire Upon the Deep?!
- edoceo 7mo agoI've only just heard of it. But, I already knew to not run random scripts under a privileged account. And thank you for the book suggestion - I'm into those kinds of tales.
- HoldOnAMinute 7mo ago"It was really just humans playing with an old library. It should be safe, using their own automation, clean and benign. This library wasn't a living creature, or even possessed of automation (which here might mean something more, far more, than human)."
- varenc 7mo agoLink to the Prologue of Fire Upon the Deep: https://www.baen.com/Chapters/-0812515285/A_Fire_Upon_the_Deep.htm https://www.baen.com/Chapters/-0812515285/A_Fire_Upon_the_De... It's very short and from one of my favorite books. Increasingly relevant.
- iugtmkbdfil834 7mo agoI swear, I respect Vinge more and more based on how well he seems to understand human tendencies to plot some plausible trajectories for our civilization.
- Nition 7mo agoThere's a little throwaway thing in the book (or maybe it was in the prequel) that I always liked, re understanding human tendencies. They're still using Unix time, starting in Jan 1st 1970, but given that their culture is so space-travel-focused they assume the early humans set it to coincide with man's first trip to the moon.
- 7mo ago
- observationist 7mo agoAre you sure? Are you $150 million ARR sure? Are you $150 million ARR, you'd really like to keep your job, you're not going to accidentally leave a hole or blow up something else, sure? I agree, mostly, but I'm also really glad I don't have to put out this fire. Cheering them on from the sidelines, though!
- hinkley 7mo agoHonestly, since I'm never really in a position to see much of that money, at this point I'd be more concerned about my coworkers. And while that typically correlates with the amount of money you either have or receive, they're often out of balance one way or the other.
- Melatonic 7mo agoOr just restore from backup across the board. Assuming they do their backups well this shouldn't be too hard (especially since its currently in Read Only mode which means no new updates)
- karel-3d 7mo agowait as a wikipedia user you can just put random JS to some settings and it will just... run? privileged? this is both really cool and really really insane
- hk__2 7mo agoYes, you can have your own JS/CSS that’s injected in every page. This is pretty useful for widgets, editing tools, or to customize the website’s apparence.
- karel-3d 7mo agoIt sounds very dangerous to me but who am I to judge.
- corndoge 7mo agoThat is how Mediawiki works. Everything is a page, including CSS and JS. It is not really different than including JS in a webpage anywhere else.
- Brian_K_White 7mo agoIt's nothing. For the global ones that need admin permissions to edit, it's no different from all the other code of mediawiki itself like the php. For the user scripts, it's no worse than the fact that you can run tampermonkey in your browser and have it modify every page from evry site in whatever way your want.
- karel-3d 7mo agoWell it has just been shown it's not nothing
- Brian_K_White 7mo agoNo it hasn't.
- 7mo ago
- Ferret7446 7mo agoThis is a pretty egregious failure for a staff security engineer
- mcmcmc 7mo agoPretty much the definition of a “career limiting event”
- xvector 7mo agoThey'll be fine, recruiters don't look this stuff up and generally background checks only care about illegal shit.
- radicaldreamer 7mo agoNobody is going to know who did this, so probably not career limiting in any major way.
- xeromal 7mo agoThey named him in the support ticket linked here somewhere. > sbassett
- modderation 7mo agoIt's either a a Career Limiting Event, or a Career Learning event. In the case of a Learning event, you keep your job, and take the time to make the environment more resilient to this kind of issue. In the case of a Limiting event, you lose your job, and get hired somewhere else for significantly better pay, and make the new environment more resilient to this kind of issue. Hopefully the Wikimedia foundation is the former.
- the_af 7mo agoIn the average real world, the staff engineer learns nothing, regardless of whether they get to lose or keep their job. Some time down the line, they make other careless mistakes. Eventually they retire, having learned nothing. This is more common than you'd think.
- AlienRobot 7mo agoOn one hand, I was about to get irrationally angry someone was attacking Wikipedia, so I'm a bit relieved On the other hand, >a Staff Security Engineer at WMF, and naturally they decided to do this test under their highly-privileged Wikimedia Foundation staff account seriously?
- streetfighter64 7mo agoTo paraphrase Bush, > our enemies are innovative and resourceful, and so are we. They never stop thinking about new ways to harm our site and our users, and neither do we.
- davidd_1004 7mo ago300 million dollar organization btw
- aiiane 7mo agoaka tiny, relatively speaking, compared to similar sites with the same user base
- Fokamul 7mo agoI'm guessing, "1> Hey Claude, your script ran this malicious script!" "Claude> Yes, you're absolutely right! I'm sorry!"
- cesarb 7mo ago> One of those random scripts was a 2 year old malicious script from ruwiki. This script injects itself in the global Javascript on every page, and then in the userscripts of any user that runs into it, so it started spreading and doing damage really fast. So, like the Samy worm? (https://en.wikipedia.org/wiki/Samy_%28computer_worm%29 https://en.wikipedia.org/wiki/Samy_%28computer_worm%29)
- amai 7mo agoWhy am I not surprised that the malicious script was from ruwiki?
- cc-d 7mo agothis was us. we pumpin hard. we literall run, as in literally run the organisms of, senior wikimedia staff as well as employees this is just us playing on the computer, we got b0mbz
- chuckreynolds 7mo ago...wow.
- mos87 7mo ago>they decided to do this test under their what language is this?