15 ms·
Logging Everyone Out
- aerovistae 6y agoGeez, that's a new one.
- punnerud 6y agoThe same happened in Norway on a governmental page a couple of years ago. It then was a memcache problem that made other people see one persons tax report. It also happened a couple of days ago again. The cause is not known: https://translate.googleusercontent.com/translate_c?depth=1&nv=1&pto=aue&rurl=translate.google.com&sl=no&sp=nmt4&tl=en&u=https://www.digi.no/artikler/folk-kunne-komme-inn-som-andre-personer-i-id-porten/500328&usg=ALkJrhisEJgB_P_ZvVBKQuvMDZzKdxXr_g https://translate.googleusercontent.com/translate_c?depth=1&...
- rmetzler 6y agoI have seen the same issue with caching in varnish when cashing rules were naive or some proxy stripped relevant headers like Cookie.
- leoh 6y agoFun fact: income for every citizen is publicly available in Norway.
- punnerud 6y agoWas until 2017, only the highest earners is fully public now. You can search on everyone, but the person can see that you have checked them (cleared every year). There exist services where you can pay $5 per search to avoid this the history.
- gnud 6y agoIncome is sort-of publicly available (less so than it used to be), but the details of tax filings/returns are not.
- trynewideas 6y agoThey've done it a few times this year alone: https://lists.archive.carbon60.com/engine?list=wiki;do=search_results;search_type=AND;search_forum=forum_2;search_string=resetting%20sessions&sb=post_time https://lists.archive.carbon60.com/engine?list=wiki;do=searc... The other times just weren't noticed, posted, or upvoted on HN.
- s_dev 6y agoThis seems all very reasonable. Look forward to the post mortem. Given their size on both the web in terms of employees this unusual for Wikimedia. They typically fly under the radar. How many times has Wikipedia ever been down? I recall AWS, Google, Microsoft having more outages -- mind they probably are considerably bigger but still they're doing something right.
- jshen 6y agoit's not hard to have high uptime for content that is largely static. Has less to do with size than static versus dynamic, and accurate (think bank transactions) vs fuzzy (search results) edit: fixed a typo
- outworlder 6y ago> it's not hard to have high uptime for content that is largely static Largely static? There are edits happening all the time.
- marcan_42 6y agoBut everyone sees the same edited version when not logged in, which is the vast majority of users, so you can just throw a huge cache in front, which is what they do. And most edits only touch one page, so the churn is tiny relative to the cache size. This is a much easier service to reliably engineer than something like Twitter. For SRE purposes, Wikipedia is mostly static.
- logram 6y agoFrom [1]: > Wikipedia develops at a rate of over 1.9 edits per second, performed by editors from all over the world. Currently, the English Wikipedia includes 6,167,378 articles and it averages 598 new articles per day. Doesn't seem to be much, to be honest. [1]: https://en.wikipedia.org/wiki/Wikipedia:Statistics https://en.wikipedia.org/wiki/Wikipedia:Statistics
- bawolff 6y ago
- donatj 6y agoI believe the Steam Web Store had a similar issue of people being logged in as the wrong user a couple years back.
- gizmo385 6y agoThat was a caching issue if I remember correctly, as opposed to users be "logged in" as a different user.
- zapt02 6y agoIsn't that the same thing? Say that you forget to specify that "my_login_cookie" should vary the response from the backend. User A logs in. Now user B logs in, but because the page was cached from user As visit, user B sees the content for user A. B is not "logged in" as A, but can still see their content.
- cheeze 6y agoThis is a very common 'advanced failure scenario.' I've seen it on a handful of sites, session objects and caching are difficult and sometimes overlooked during migrations.
- punnerud 6y agoLove MediaWiki as a platform. I host several of them. Hope the interactive editor in PHP soon get out of beta.
- gerdesj 6y agoDo you mean the Visual Editor? That's been available for several years now. Getting the back end for it running can be a bit fraught. I wrote this: https://www.mediawiki.org/wiki/Intranet https://www.mediawiki.org/wiki/Intranet and keep it up to date every now and then. It's probably time for me to look at 1.36. Anyway, the current Parsoid based thing works fine.
- bawolff 6y agoI think the grandparent means the new version in 1.35 where the backend should just "work" (i think it is out of beta now)
- punnerud 6y agoBackend/Parsoid also using PHP I was thinking of.
- znpy 6y agoThe Visual editor is going to get bundled in 1.35. The beta go refers to in a version rewritten in PHP to drop the dependency on another service, parsoid, written in JavaScript/node.
- kevmo 6y agoI wish every service did this on a weekly basis.
- johnnyfaehell 6y agoYou can delete your cookies on a weekly basis and it'll be like they did.
- dylan604 6y agoOr just block cookies and never have this issue
- Xylakant 6y agoJudging from the error description, deleting cookies would likely not help in this case. This sounds as if sessions are mixed up on the server side - deleting your cookie will only remove your session token, not remove the session server-side. You’d need to actually send a logout request to the server.
- hinkley 6y agoClearing your cookies might mean you won't log into someone else's account, but somebody else might still log in as you.
- t0astbread 6y agoI wish every service that has something like user sessions provided the ability to revoke other sessions of your account. Then, if the site isn't too hard to automate, you could write a program for that.
- prionassembly 6y agoDo they mean employees? My Wikipedia login lasts for 30 days, I think. I basically only login to edit articles that require so. Otherwise I just edit as anonymous.
- dredmorbius 6y agoNo. Everyone.
- M2Ys4U 6y agoNo, users. I just had to log in, despite recently being active. I guess this is why.
- mmaunder 6y agoI'm reminded that MediaWiki runs on PHP, which isn't much loved here on HN, but does power some of the busiest sites in the world, and PHP 7 and 8 have done a great job of moving the language and performance forward.
- ibejoeb 6y agoIt's been quite a while since I've looked at mediawiki. Has any of the codebase been transition to modern PHP?
- edoceo 6y agoNot really, PHP is very backwards compatibility and I don't see lots of "modern" PHP (eg Traits, etc) in the code. I last looked at their release from a few months ago. I do lots of PHP migration work (4 to 5, 5 to 7) and for much of it it just works. Not much code change is necessary to the the upgraded features of the engine.
- bawolff 6y agoI feel like "traits" is just one particular feature, which by itself is not a good proxy for whether php is modern Disclaimer: am mediawiki dev. No opinion on if the codebase is "modern" because everyone defines that differently.
- edoceo 6y agoAgreed Traits is just a simple proxy. But nothing in there depends on new features of 7.3 right? MW has great backwards compatibility.
- bawolff 6y agoWikipedia is running php 7.2 https://en.wikipedia.org/wiki/Special:Version https://en.wikipedia.org/wiki/Special:Version so no. That said, new features do get used when they make sense, so we are definitely not maintaining back compat with ancient versions of php. (This includes a couple usages of traits, traits is just not something super common in mediawiki, but there are some cases where they are used)
- 8ytecoder 6y agoI have seen this behaviour on Facebook, ages ago. Reported it and yet nothing came out of it.
- jayar95 6y agoThat doesnt surprise me. They will let highly problematic security issues persist for weeks
- frakkingcylons 6y agoHave any examples?
- CGamesPlay 6y agoThe "view profile as another user" feature was infamous for being... really thorough. At one point you would receive chat messages directed to the person you were viewing the profile as (I believe this was before the integration of chat and messages).
- andrewla 6y agoI remember early in the history of E*Trade, I went to my account, and it showed someone else's name and account information. Didn't even bother reporting it (I was just a kid); just logged off and on again and withdrew everything from my account and never looked back.
- dionian 6y agoWhat is the (he/him) after the guy's name?
- dEnigma 6y agoHis preferred pronouns, I would assume.
- Raphmedia 6y agoPronouns. That way, if someone is named "Billy" or "Alex", you won't be spending years mistakenly assuming they were a man while they could have been a woman. This also has the beneficial side effect of covering trans individuals who use a birth name while having another gender identity. Even more useful for usernames such as yours, "dionian". I do not know if it is your first name, last name or made-up username. So I have no clue what your gender is and I'd default to "they". If I knew, I could call you by the right pronouns. It's just a slow shift away from assuming everyone on the internet is a man.
- fennecfoxen 6y agoIt also serves as a flag that allows people to identify you as having bought into the general set of practices which strive to change our culture to support trans and nonbinary gender ideology.
- Frondo 6y agoAnd if our culture doesn't already support trans and non-binary people (ideology? no, it's people we're talking about), then that seems like it'd be a good thing to change.
- fennecfoxen 6y agoSupport of people may be involved, and this is a poor forum to litigate deep problems -- but if I trusted the diversity-and-inclusion apparatus of Silicon Valley in general to support the entirety of Title VI of the Civil Rights Act, including the inconvenient parts about people of differing religions, then I might not been predisposed to choose that word. Even your reply highlights it. "It's about people." Notice the manner in which you _correct_ my rather anodyne words. The fault is that I did not frame the question as you would have me frame it, and looked at a part of the general phenomenon that is different from you think deserves to be centered on. You are welcome to your ideology, in any event, and it would be a poor world indeed where one could not act to change the world in accordance with one's conscience.
- usmannk 6y agoBugtracker: https://phabricator.wikimedia.org/T264370 https://phabricator.wikimedia.org/T264370
- foolfoolz 6y agoi worked at a company that spent 6 months and 2 devs to solve a huge infrastructure change without logging anyone out of their session
- dahart 6y agoIsn't that a tad different than discovering an active security hole?
- 2ion 6y agoLogging people out seems like a hella low price to pay to pay for a potential security issue. Of course, hats off if you hold your engineering organization to such a high standard for normal changes.
- kazinator 6y agoNot logging anyone out while upgrading the kernel of a time-sharing operating system: nice feat. Not logging anyone out while updating a suite of REST-ful applications: ho hum.
- foolfoolz 6y agothis was stupid. we were not a big enough org to spend 2 people like this. i never understood why forcing people to log in again was so controversial
- aeyes 6y agoDepends on the type of application, 99.9% of Wikimedia users are using the site without being logged in. If the site doesn't work without being logged in you could frustrate users and they might just use a different product instead of searching for their login after being logged in for a year or longer.
- kazinator 6y agoThat doesn't seem difficult; just don't change the format of the session tokens. Eg a web application: you can substantially rewrite a web application, without invalidating logged in sessions. The point here is that the logged in sessions were suspected of being unauthorized. The unauthorized sessions had to be turfed, and the clearest way of being sure that all unauthorized sessions are turfed is to delete all the sessions. Of course, some those with unauthorized access will also try to log in to resume that unauthorized access, but presumably there is some trap laid for that. Maybe for the accounts suspected of having been breached, there will be a mandatory password recovery procedure or whatever. Or they will monitor for suspicious logins from different IP addresses.
- tomc1985 6y agoAssuming session ID's are tied to individual users and looked up via a "SELECT * from sessions where id='?'" type query, how does this even happen?
- kace91 6y agoBadly designed caches for example. Though I have no idea what happened in this particular case.
- FilterSweep 6y agoThis (bad cache) used to happen with misconfigured shopify sites and had the same impact as described
- duderific 6y agoNot scoping a variable properly (causing a variable to leak out to global scope) could also cause this.
- bawolff 6y agoNot usually in php, where each request has no shared state with any other (outside of apc, memcached)
- TheRealPomax 6y agoRemember https://www.theregister.com/2015/12/30/steam_security_blip_explained/ https://www.theregister.com/2015/12/30/steam_security_blip_e... ?
- trynewideas 6y agoNone acknowledged publicly, but it's an open-source project with pretty open administration, so you can follow along: https://phabricator.wikimedia.org/T264370 https://phabricator.wikimedia.org/T264370
- deleted 6y ago[deleted]
- Kiro 6y agoThis reminds of a thing that happened on MSN back in the day when I suddenly started to receive random users' chat messages in one of my chats. It only lasted a few minutes and it was all different kind of languages. Never heard anything about it but in retrospect it felt very serious. Imagine the same thing happening on Facebook Messenger or Whatsapp.
- andrewmcwatters 6y agoI’ve always wondered if this class of cache issue resulted primarily from collisions of some sort. I’ve only seen it a handful of times over many years. Others here have mentioned it with services using Varnish, for instance.
- zapt02 6y agoVarnish cache works by creating a hash of the request consisting of method, url and for a site where you log in, that would also include the cookie you use for identification. Your proposed collision idea is very improbable but possible. There is actually a good article in the Varnish docs around hash collisions: https://varnish-cache.org/docs/4.1/phk/varnish_does_not_hash.html https://varnish-cache.org/docs/4.1/phk/varnish_does_not_hash... PS. Even though the article says "Varnish does not hash", I suggest everyone to keep reading until the end, when it turns out that they, in fact, do.
- harryf 6y agoI know the full analysis isn't online but I have a problem with this part... > This was done out of an abundance of caution, after we received one (1) user report of being logged in as someone else. This _seems_ like a knee-jerk reaction to one data point. There could be other causes for a user to report that, like a change to the cache key used for serving a users profile giving the _appearance_ that you're logged in as someone else, even though you're not really. Forcing everyone to re-login could potentially make the system worse, in that you're now overloading parts of the system that has to handle those logins, plus causing all kinds of cache expiry... I guess there's more to the story and someone who knows the system deeply knew this was the right choice but just reading the reports it seems knee-jerkish.
- jeremyjh 6y agoIt seems adequately explained in the next sentence. > Said report coincided with the deployment of a new MediaWiki release which caused other problems around User session objects;
- flemhans 6y agoI remember back in the days, I disconnected my dial-up modem and re-connected with another public IP. Upon refreshing Hotmail, I was presented with another user's mailbox. I was never able to reproduce the glitch.
- polygot 6y agoI had a similar issue happen to myself (not on wikimedia or anything related to wikipedia.) I clicked on "login" by accident without filling in my credentials and I was logged in as either an admin user or a user called "Adam".
- m-p-3 6y agoSo that's why my account was logged out from the mobile app. No big deal, as long as the security of the system remains intact.
- jacobwilliamroy 6y agoMy buddy told me that the U.S. marines in afghanistan once misplaced one of their fancy encrypted radios (or it might have been one of the keys to their fancy encrypted radio) so all radio communications were suspended until the next scheduled rotation.
- hexbinencoded 6y agoBattery ran out? Left one behind? Throw them all away and visit the quartermasters' Idiocracy/MIC version of Costco to pull another 100 pack off the shelf at a cost to the taxpayers of mere millions. Oh and F-35B's and Blackhawk turbines? just chuck those like Kleenex. My mom worked in civil service for the Navy and $20k parts in the '70's went "missing" and "lost" all of the time. Then there's a USAF Major I know who became unpopular for suggesting proper maintenance rather than ruining/replacing turbines might be a better approach (likely some sort of maintenance/HQ personnel-vendor kickbacks, MIC FTW). One of the few stories VICE didn't botch entirely: https://youtu.be/_CvWJVtEkUE https://youtu.be/_CvWJVtEkUE
- hexbinencoded 6y agoPHP developers gonna PHP. They're still at the level of fiddling with individual files in production and asking others if everything still works as their version of integration testing.
- tomcatfish 6y agoWow, this is the first time I've seen a company pick "better safe than sorry" when it had global effects (though this isn't a large hit, it's everywhere). I appreciate it a lot, thanks WikiMedia