23 ms·
Lots of speculation on Twitter about this—a failed attempt to re-open all closed subreddits and instate their own moderators. I can't imagine it'd be that, alth
by r0bbbo 3y ago
Lots of speculation on Twitter about this—a failed attempt to re-open all closed subreddits and instate their own moderators. I can't imagine it'd be that, although I do enjoy the conspiracy, and more likely they were using the window of reduced traffic to make some larger changes and they went awry
- paxys 3y agoPointless speculation. Reddit has constant outages of this exact nature almost weekly. There's nothing special about this one.
- ljm 3y ago80% of requests to Reddit via their web UI are basically 500s or CDN errors. Hyperbolic I know, but that’s what it feels like.
- jonathantf2 3y agoDoes it? I usually use Reddit on my lunch about this time every day and I can't remember the last time it was down.
- philote 3y agoI run into issues on occasion, usually with loading comments. I use the web interface. Interestingly, HN is being really slow for me right now and also gave an error when I first loaded it. Maybe something more global is going on, like network or cdn issues?
- gonzo41 3y agoI'm also seeing HN being sluggish. But I have crap Australian internet so YMMV.
- piva00 3y agoIt is sluggish and throwing some errors. I have a 1Gbps internet connection via fibre in Sweden, so definitely it's not an issue with your connection.
- pdimitar 3y agoI have a gigabit internet, am in Eastern Europe, most US services have 40ms or so latency -- pretty strong link. I had to reload this page 5 page until I was allowed to post a reply. Something is definitely going on.
- ktm5j 3y agoMe too on the HN issues. East Coast US here
- static_motion 3y agoSame thing here, Western Europe on a 500Mbps connection. Maybe higher traffic from people looking for Reddit alternatives?
- yreg 3y agoOr some of the people who would use reddit right now came here. I usually read hacker news when I try to reach reddit and it is down.
- sebzim4500 3y agoPresumably you are using old.reddit.com
- riffic 3y agooutages are incredibly common everywhere
- Capricorn2481 3y agoNo, it doesn't
- activiation 3y agoProbably DDoS more likely today
- b3orn 3y agoDid they DDoS themselves trying to fake numbers to show that the blackout has no effect on daily active user counts?
- activiation 3y agoThat would be funny
- SkyMarshal 3y agoReddit definitely doesn’t have outages like this almost weekly. Maybe 10+ years ago, but it’s rare now.
- atchoo 3y agoIt's often not working for me in the UK e.g. "You've broken reddit" and "Can't reach CDN". Maybe it is a regional thing?
- Merad 3y agoEastern US here. I will get those errors a few times a week but I'd say 80% of the time it works on a refresh and most of the rest of the time it's back within 30s.
- SkyMarshal 3y agoMight be, I’m US west coast fwiw.
- lamontcg 3y agoAlso very common for when you post a comment that it appears to not have posted it and returns an error, then you retry multiple times and get multiple comments posted. And even if you don't get hit by one of those outages, they tend to get saved in the comments from everyone else having hit them.
- unaindz 3y agoFor me in the EU it fails once a year or so and only on the few occasions I use reddit's own webpage instead of a third party app or web like libreddit.
- jeremy_k 3y agoNot true at all in regards to 10+ years ago. Once at week, if not more, Reddit fails to load for me. I refresh the page and it appears I've been logged out. Continue refreshing the page until my session is revived and things are back to normal. I'll often open a post and the header will load but Reddit will fail to load the comments with a click to retry loading comments.
- nprateem 3y agoI've been a long-time user of Reddit (I still remember when WSB had only 12m users) and I can't remember an outage.
- SkyMarshal 3y agoReddit definitely doesn’t have outages like this almost weekly. Maybe ten+ years ago, but it’s rare now.
- pyeri 3y agoBut this time it's different because most high traffic subs have already gone dark, so the traffic must be minimal as there will be very few posters relative to what the server can handle. And yet, I'm getting "You broke reddit" when I try to visit which is quite ironical.
- treis 3y agoProbably a massive shift of traffic to still open subs. That'd probably take it down due to caches suddenly having all the wrong data. No inside info, though. HN seems to be groaning under the load too and might go down with Reddit.
- rozap 3y agoMaybe it's expensive to generate the 403 for a private subreddit. A place I once worked had a 404 handler that was extremely expensive, but nobody noticed for a while because 404s are relatively rare. One time a vulnerability scanner took down the site because it was just hitting known vulnerability paths which all 404'd. The code that executed during a 404 was n^2.
- flutas 3y agoReddit 100% has stated that it was because of the sub blackout. > According to Reddit, the blackout is responsible for the problems. “A significant number of subreddits shifting to private caused some expected stability issues, and we’ve been working on resolving the anticipated issue,” spokesperson Tim Rathschmidt tells The Verge. https://www.theverge.com/2023/6/12/23758002/reddit-crashing-api-protest-subreddit-private-going-dark https://www.theverge.com/2023/6/12/23758002/reddit-crashing-...
- ilyt 3y agoThem "anticipating" "expected" issues with... people visiting reddit less sounds a bit weird...
- Dylan16807 3y agoMaking a ton of subs private does a lot more than change view counts.
- FireBeyond 3y agoI can't imagine it either, though it wouldn't be outside the realm of some of the stupidity that has gone on. But it would almost certainly be the death knell for the site - those subreddits who didn't "go dark" now would.
- cfors 3y agoCould also make a guess that the blackout has changed how their normal traffic patterns operate, causing some issues with autoscaling/hot partitions.
- mrzimmerman 3y agoYeah, my assumption was that something in some layer of their application isn’t well optimized when asked to return posts from a subreddit that has “gone dark” in whatever fashions the mods chose to do that. For example, maybe it causes reads from the database take a lot longer than they normally would, locking up the database or causing the process the crash (again, that’s just pure speculation).
- paulmd 3y agoone I've been wondering about is user overview pages. People use those a lot (it's actually my bookmark for getting onto reddit) and yesterday I noticed that a post I made wasn't in my overview, and it's because that sub had gone dark early. What happens when a user has 99% of their posting in subs that are now hidden, and the API is programmed to produce a fixed 30 comments of history on the overview page? The answer is extremely deep database pulls... you might pull a year of comment history to get 30 comments that aren't hidden. And depending on how they do that, it may actually pull the whole comment history for that timespan, since most of the time posts aren't hidden like this. I worked at a backend team at work with some very overburdened legacy tables in mongo, and this is the kind of thing we'd think about. Yeah you can use an index, but then you have to maintain the index for every record, and change it every time a sub goes private/public (and we literally were hitting practical limits on how many indexes we could keep, we finally instituted a 1-in-1-out rule). And how often does that happen? Even deleted comments are overall probably a minority such that indexes don't matter, but, this is relational data, you have to know which subreddits are closed before you can filter their results, and mongo sucks at joins. And the mongo instance can become a hotspot, so, just filter it in the application instead for those "rare" instances. Even if they are doing it in mongo, the index/collection they're joining may suddenly be 100x the size, which could blow stuff up anyway. edit: for me, one overview page is now taking me back one month in comment history. And I comment a lot on subs that are currently closed, so it could easily be throwing away 5-10 comments for every comment it displays.
- hgsgm 3y agoThey don't have people to be moderators of all those subs. That's why subs have volunteer mods.
- skhr0680 3y agoA meme pointing out that u/spez was a mod of r/jailbait in 2008 was the top post on r/all a few hours ago
- trompetenaccoun 3y agoDoesn't seem that unbelievable when you look into some of the other stuff he's done. For example he secretly used his admin powers to edit user comments from users he didn't like or who criticized him.
- mmastrac 3y agoI don't like what he's done one bit, nor him personally, but this is a really big stretch and not really the level of comment HN should strive for. EDIT: to clarify -- a big stretch to connect editing comments (which was well-known to have happened) with other actions.
- tipsysquid 3y agothe accusation of editing comments I believe to be confirmed as fact, everything else I am not sure of.
- mrgalaxy 3y agoParent commenter shouldn't have said secretly. That was something he openly admitted to doing. https://www.theverge.com/2016/11/23/13739026/reddit-ceo-steve-huffman-edit-comments https://www.theverge.com/2016/11/23/13739026/reddit-ceo-stev...
- AlecSchueler 3y agoI believe the issue the commenter above was taking was that just because someone commits, for lack of a better term, comment fraud, we shouldn't jump to suggesting he's also a paedophile.
- deleted 3y ago
- irthomasthomas 3y agoHN getting hugged, too.
- deleted 3y ago[deleted]
- nicce 3y agoAn alternative conspiracy theory: put Reddit down for two days to hide the existence of the protests from major users. E.g. many Google searches lead into private communities now. Word is spreading.
- hn_go_brrrrr 3y ago"You can't take us down! We'll take _ourselves_ down, so there."
- bombcar 3y agoIf people see the dead reddit dude they just assume it broke and come back later. If they see a "this sub is private because reddit is dumb and also evil" they may come back later, they may not.
- marcosdumay 3y agoHonestly, looks as reasonable as anything else about this thing.
- rangledangle 3y agoThis is my guess. Looks better on the record, since people are committing only 48 hours anyways (for whatever reason, should be permanent).
- BoxOfRain 3y agoMaybe the protest was so successful at driving traffic away from Reddit it looked like something was broken to their monitoring which tried to failover unsuccessfully? If that happened it might be the first case of a site being brought down by the polar opposite of a DDoS.
- ryanstorm 3y agoAttempting this to be more of a discussion prompt than speculation, but as an advertiser or other consumer analytics customer of reddit, would this sort of protest and the potential fallout be concerning? I'm not sure if all this is a drop in the bucket for those customers or if it's significant. If it was significant, would reddit be scrambling to save face for these advertising customers? An outage like this would certainly skew the difference in their analytics for today's relative traffic.
- Algent 3y agoI wouldn't be surprised if so many big subreddits being dark is causing issues around denied API calls. As for the forced reopening, beside the conspiracy this is something that could happen. It's a private company, moderator on strike are a loss of business, they would be 100% in their right to remove all the "traitors" (I'm not saying this would be a smart move, simply that if they really plan to go down this self destructive path it's the best time to do this and prove potential investor they still have control).
- abluecloud 3y agowhere the hell are they going to fine thousands of willing users to become mods of thousands of communities, overnight?
- deleted 3y ago[deleted]
- brightlancer 3y agoI think there are millions of folks on Reddit who would jump at the chance to be a petty dictator of a subreddit.
- Deestan 3y agoThat's easy. Less if you add "competent" to the qualifications list :)
- brightlancer 3y ago> It's a private company, moderator on strike are a loss of business, they would be 100% in their right to Legally, of course. Morally, it is completely unacceptable. This isn't "oh they're jerks"; this is "the system is broken". A meatspace analogy: You host a weekly gathering at a restaurant. You decide to temporarily boycott the restaurant to protest some behavior of theirs -- your actions are a loss of business, _so the restaurant decides to host your weekly gathering without you_. We'd never accept that in the real world, but for some reason we do online -- we fall back to the legal argument that It's A Private Business (which is true) and completely ignore that Reddit doesn't own the community, that the community doesn't _belong_ to Reddit. They own the platform (the restaurant); they don't own the community.
- jackson1442 3y agodefinitely not the case, reddit has stated that they would not be doing that, at least for the two-day protest. would be incredibly bad PR at this scale, especially after repeatedly and directly making this statement. however, they have stated that may do this if the protest extends beyond the 48h mark.
- jallen_dot_dev 3y agoWild guess: visiting a private sub requires an extra call to a service/db to check if the user can view it. Normally there are only a small number of these checks because private subs were usually small communities. Now, many large subs having switched private is causing some poor mircoservice somewhere to get hammered.
- bakuninsbart 3y agoI never worked at this scale, but could it also be that different subs are horizontally scaled and with so many people reverting to the subs that are still open the load is unevenly balanced?
- q7xvh97o2pDhNrh 3y agoGood question! And few people get to work at this scale, so it's not an unreasonable guess. I'll join you in speculating wildly about this, since, hey, it's kind of fun. IMHO sharding traffic by subreddit doesn't pass the smell-test, though. Different subreddits have very different traffic patterns, so the system would likely end up with hotspots continuously popping up, and it'd probably be toilsome to constantly babysit hot shards and rebalance. (Consider some of the more academic subreddits vs. some of the more meme-driven subreddits — and then consider what happens when e.g. a particular subreddit takes off, or goes cold.) Sharding on a dimension that has a more random/uniform distribution is usually the way to go. Off the top of my head (still speculating wildly and basically doing a 5-minute version of the system-design question just for fun), I'd be curious to shard by a hash of the post ID, or something like that. The trick is always to have a hashing algorithm that's stable when it's time to grow the number of shards (otherwise you're kicking off a whole re-balancing every time), and of course I'm too lazy to sort that out in this comment. I vaguely remember the Instagram team had a really cool sharding approach that they blogged about in this vein. (This would've been pre-acquisition, so ancient history by Silicon Valley standards.) As for subreddit metadata (public, private, whatever), I'd really expect all of that to be in a global cache at this point. It's read-often/write-rarely data, and close-to-realtime cache-invalidation when it does change is a straightforward and solved problem.
- tester457 3y ago> (I used to work as a backend developer at Reddit - I left 6 years ago but I doubt the way things work has changed much) > I think it's extremely unlikely that this is deliberate. The way that Reddit builds "mixed" subreddit listings (where you see posts from multiple subreddits, like users' front pages) is inefficient and strange, and relies heavily on multiple layers of caches. Having so many subreddits private with their posts inaccessible has never happened before, and is probably causing a bunch of issues with this process. https://tildes.net/~tech/163e/reddit_appears_to_be_down_during_blackout_day_1#comment-87v1 https://tildes.net/~tech/163e/reddit_appears_to_be_down_duri...
- deleted 3y ago[deleted]
- commandlinefan 3y ago> a failed attempt to re-open all closed subreddits and instate their own moderators My first thought this morning was "if I was reddit, I would re-open all closed subreddits and instate my own moderators"... I'm kind of surprised they didn't. I'd have to imagine that if they wanted to, the attempt wouldn't fail - this seems like it could be done almost trivially.
- Sebb767 3y agoThey'd just replace their unpaid workers by paid ones and annoy the userbase even more in the process. Doesn't sound like a great plan.
- humansubjects 3y agoI think you are underestimating the amount of unpaid labor that keeps reddit functioning
- ben7799 3y agoWild. Probably 50% of the subs I subscribed to are gone, and some of the others have deleted all the content and locked posting to the sub. Others look like they got the hostile takeover and a new mod.
- lsaferite 3y agoCould you point out some that appear to have had a hostile takeover? I'd love to check it out.
- nonstopdev 3y agoThat is an interesting point. Would assume your subreddit isn’t yours so at any point you could grow and generate content and users and Reddit could take it over and add in their own moderators?