17 ms·
Reddit crashed because of the growing subreddit blackout
- Capricorn2481 3y agoI have not read a reddit devblog or anything like that. My guess is they are doing some kind of caching of posts via elasticsearch/redis or something similar. It must depend on the privacy settings of a sub. So when 1000 subs go private, they potentially had to update hundreds of millions of comments. Anyone else have a theory?
- hospitalJail 3y agoThe subreddits that are most popular are on high performance servers while the subreddits that are less popular are on lower performance servers?
- philip1209 3y agoCache invalidation makes sense. Could also be some kind of fanout query that assumes the top subreddits are open and wastes CPU cycles.
- Scoundreller 3y agoI’ve noticed that my comment history in privatized subreddits are now unavailable, so I wonder if it’s related to that.
- kevincox 3y agoIt could even be that lots of queries to generate pages just fetch private posts and comments. The assumption was that users will not be subscribed to lots of subreddits that they have access to so skipping these posts is typically negligible and not worth using a more complex query for. If the fraction of skipped items becomes significant these queries will be doing lots more work. Its pretty believable that some very common query getting 25-50% more expensive will be more server capacity then Reddit has available. Another possible explanation is that operations on private subreddits are not as optimized as it is expected that they have a small number of subscribers and take up a small amount of total resources. These assumptions would be flipped.
- delecti 3y agoI'm skeptical. Most of the outages I've been involved in have been for way dumber reasons than that. Based on my dumb experience, my guess is that the handling for a user trying to access a private subreddit stressed some system that doesn't make sense to even exist.
- 2bitencryption 3y agothe cynic in me says, they're being DDOSd (I mean, they pissed off millions of internet nerds who have their own definition of justice), and instead of admitting it they're using the opportunity to blame the subreddit mods. then again, not sure why that messaging would be any better for them than admitting to a DDOS...
- manicdee 3y agoBlame the user makes sense because they don't want to show how fragile Reddit is while they're trying to attract investors for the IPO.
- ElevenLathe 3y agoThat makes sense to me. It would be one variant of a larger theory of "a subreddit going private is an expensive process that is poorly optimized because it's normally so rare". Another variant would be that clicking the "go private" button might do something like start a huge cascade of queries/calls to mutate the status of individual posts and comments (and possibly hence also trigger the massive cache invalidation you mentioned). Possibly also "going private" infra, whatever it is, is some bespoke snowflake that doesn't autoscale (since it's never needed to and there was always something better to be doing with engineering time, or people just forgot). One theory I have seen floated but which seems unlikely to me is that this was some kind of internal sabotage. I could maybe buy this if the protest was about a war or human rights issue or something, but I really don't think any IT pros would be willing to risk jail time for this one.
- 2OEH8eoCRo0 3y agoWould a better protest be rapidly switching subreddits from public/private?
- falcolas 3y agoToo easy to shut down, since there's a finite number of mod enabled accounts.
- joecot 3y agoA better protest would have been for the moderators to all stop moderating. Turn off their automod filters, allow all posts. Show reddit what happens when their volunteer moderators just stop.
- graypegg 3y agoI still think this is coming. After the 48hr strike, how many mods are going to turn their sub back to public, look at their mod inbox filled with confused and angry users, and then see some pending non-response from Reddit post-strike, and wonder why they do this?
- 3y ago
- putnambr 3y agoMy theory is that some middleware applications have policies to scale to zero or one based on traffic patterns, when scaled down due to lack of traffic things start breaking.
- jedberg 3y agoPosted elsewhere, but yeah that's my theory too (having actually helped built it in the first place). My guess is the caches got blown out because the entire system is built on the idea that most people are looking at the most popular content. When you close down all the most popular content, you have to dig deep into the long tail for fresh content. Also, my guess is that the code for building homepages is not optimized for having a lot of skips due to private Reddits, since most people have probably never been subscribed to a private reddit, or if they have it wasn't for very long, or even if for very long, never more than one or two at at time.
- biesnecker 3y agoMakes sense to me. I remember several lunch conversations at FB about whether or not we even could restart the site with totally cold caches. I’m sure it was possible but it wouldn’t have been clean or pretty.
- _dain_ 3y ago> I remember several lunch conversations at FB about whether or not we even could restart the site with totally cold caches so you're saying, we just have to knock it out once, and it'll be gone forever? that makes things easier ...
- jedberg 3y agoWe did restart reddit with cold caches a few times when I was there. It took hours to recover but eventually caught up if it happened near peak. If we did it during low traffic it wasn't too bad. But that was 13 years ago, no idea how it would do today. :)
- biesnecker 3y agoI’m sure Facebook could too (their disaster recovery planning and drills were always top notch) but I’m glad I won’t be oncall if it ever happens :-D
- EMCymatics 3y agoWell that is how wallstreetbets freezes the subreddit to log all of the potentially actionable comments.
- amyjess 3y agoMy theory is that the algorithm that produces the front page relies on the largest subreddits all being public. But if it can't draw on posts from the largest subs, it'll have to dip into a bunch of smaller subs, and the part of the algorithm where it delves deeper and deeper into less popular subreddits is terribly optimized and puts a huge strain on the servers.
- micromacrofoot 3y agoTwitter was like this at one point, flipping over from public to private was one of the most resource intensive things an individual user could do... especially with a long tweet history, because it involves changing every tweet.
- tester457 3y agoHere's a guess from a former reddit dev. > (I used to work as a backend developer at Reddit - I left 6 years ago but I doubt the way things work has changed much) > I think it's extremely unlikely that this is deliberate. The way that Reddit builds "mixed" subreddit listings (where you see posts from multiple subreddits, like users' front pages) is inefficient and strange, and relies heavily on multiple layers of caches. Having so many subreddits private with their posts inaccessible has never happened before, and is probably causing a bunch of issues with this process https://tildes.net/~tech/163e/reddit_appears_to_be_down_during_blackout_day_1#comment-87v1 https://tildes.net/~tech/163e/reddit_appears_to_be_down_duri...
- synicalx 3y agoThat was my thinking as well, and assuming they're still cloud-heavy (it's been years since I've checked) this could potentially be an expensive event for them. That opens up a very interesting for of protest/attack that reddit really won't like; "privating" then "un-privating" a subreddit several times a day.
- Solvency 3y agoI thought it was a cloudflare outage.
- deleted 3y ago[deleted]
- almost_usual 3y agoPretty sure Reddit uses Fastly as a CDN.
- stillbourne 3y ago[flagged]
- swarnie 3y agoGreat comment, Reddit can't resume fast enough.
- thewataccount 3y agoCan you elaborate? For us cloudflare has been incredibly reliable and incredibly cheap, although we'd consider moving if there was a better option. I'm not aware of this perception, have you had issues with cloudflare? Does fastly have a better offering?
- stillbourne 3y agoIt was a joke.
- stillbourne 3y agoHow long before they admins swoop in and change ownership of the blacked out subreddits?
- falcolas 3y agoI personally believe they will wait to see which ones remain shuttered. 48 hours is not very long to wait to maintain your free civilian labor. After that, they'll definitely reclaim the shuttered subreddits with new moderators.
- phpisthebest 3y agoReddit is odd.... while most communities hate the mods, they hate admins far more. Admins taken over a subreddit would kill it in the eyes of the users.
- falcolas 3y agoI think the admins will care more about the appearance of "business as usual" in the face of an IPO than the opinions of the users who would notice a moderator replacement. In their eyes, this is all nothing more than a tempest in a teapot; a small pothole on the road to public investor dollars.
- phpisthebest 3y agothe API thing was that, it is moved beyond that now, and admin action on that level will make the API issue an after thought... If they do that, forget IPO, there will be no more reddit at all. right now they already missed the IPO point years ago, today for a company to have a successful IPO they need more than good MAU numbers, and this API change was never going to get to the profitability numbers they need for a good IPO Their IPO dreams were fucked long before today
- falcolas 3y agoThe internet forgets too quickly. And so long as Reddit can present a "everything's awesome" front, their IPO will likely take off. Having /r/Pics down permanently because of a protest would be a problem, but a poorly moderated /r/Pics would not. The quality would just go down, and Reddit would get a good justification for locking it down. "No, /r/Pics wasn't shut down because of the protest, it was shut down because there were too many law-breaking posts. And you know how tough we have to be on those kinds of posts." At least, that's my take on how most investors would look at it. Investors love companies who focus on profitability, regardless of their long term perspective.
- cdme 3y agoWhat happens if the mods were to coordinate cycling subs from private to public?
- ch4s3 3y agoProbably rolling outages
- TheAdamist 3y agoCfaa charges.
- cryptonector 3y agoWhat's the theory?
- chx 3y ago18 U.S.C. §§ 1030(a)(5)(B) > intentionally accesses a protected computer without authorization, and as a result of such conduct, recklessly causes damage; I would think it's reasonably easy to establish mens rea when someone switches their sub from public to private and back every few hours, even if manually done because what else would they want to do but take down Reddit? Not to mention the punishes much harshly "conspiring" to commit such a crime. Careful there. IANAL, of course.
- JohnFen 3y agoIANAL either, but my reading of that is the "and" in there is important. That is, you'd have to be "intentionally accessing a protected computer without authorization" and recklessly cause damage. Reddit mods are not accessing a protected computer without authorization.
- cryptonector 3y agoThey're mods. They're authorized. Taking their subs private -temporarily, intermittently, permanently- is within the scope of the authority delegated to them by Reddit. A prosecution on the basis of CFAA would be ridiculous -- though, of course, that wouldn't mean it couldn't happen.
- freespirit77 3y agoCould be due to home feeds not having enough cached content as a lot of the first page posts went private and had to be skipped? Would have led to have to go deeper in lists of posts to fill the first page and causing more cache misses and db reads and performance issues.
- derefr 3y agoVery likely; before the crash, I noticed that the home page failed to load, but individual subreddits continued to load.
- klodolph 3y agoThe way Reddit displays content is basically all caches, as far as I can tell. Every query you do is supposed to directly hit a cache of 1000 objects (like comments or posts). You can’t go past 1000, either through the web interface or through the API. Kinda frustrating design, IMO, speaking as an end-user.
- kenhwang 3y agoIt was always obvious when you scrolled past the cached content because errors or timeouts would start happening pretty frequently.
- sofixa 3y ago> A significant number of subreddits shifting to private caused some expected stability issues, and we’ve been working on resolving the anticipated issue,” Calling bullshit on that one. If they had expected stability issues, and knew exactly which subreddits will go private and when, how is it possible they had so much "anticipated issues"?
- schaefer 3y agoThe world has finite resources... such as man-hours.
- sofixa 3y agoSuch a critical moment for the future of the company (mass boycott after controversial changes, presumably on the path to IPO; they really don't need any more bad publicity) should be an all hands on deck situation. Again, they've had advanced notice, and knew the full list of subreddits. They should have prepared better.
- pohl 3y ago> should be an all hands on deck situation [The Mythical Man Month has entered the chat]: Nine mothers, all hands on deck, should be able to gestate a baby in 1 month.
- tiffanyg 3y agoAt the risk of adding further low quality content, right here and now, this proposed "Mythical Mom Month" book is an intriguing pitch. Do you have a more detailed pitch sheet? :)
- TillE 3y agoYou're not going to significantly rewrite a caching system in a matter of days, not if you want to fix more problems than you cause.
- falcolas 3y ago
- deleted 3y ago[deleted]
- subarctic 3y agoIt'd be funny if the engineers at reddit decided now was a good time to upgrade postgres or do some other kind of scheduled maintenance, and just not worry about downtime because the blackout is on anyway.
- jklinger410 3y agoUnless there is some serious damage control happening from corporate which makes them unable, I think a team of engineers would talk themselves into doing this 99% of the time.
- joeross 3y agoI don't see any damage control... they're still committed to the "Apollo sucks" line: > Mr. Rathschmidt added that some apps are more efficient and require significantly fewer A.P.I. calls and that “Apollo is notably less efficient than other third-party apps.” Source: https://archive.ph/y92P7#selection-681.0-681.173 https://archive.ph/y92P7#selection-681.0-681.173
- AH4oFVbPT4f8 3y agoDeniability is a powerful tool, and considering their track record this year, this decision wouldn't rank among their worst.
- Tao3300 3y agoOr rolling out the big blackout override switch
- poyu 3y agoEasy: ``` UPDATE subreddits SET is_private = FALSE WHERE is_private = TRUE AND updated_at BETWEEN 2023-06-11 AND 2023-06-12; ``` Bonus: Run it every hour
- EA-3167 3y agoI don't think anyone doubts that Reddit could pretty easily unlock those subs, but without mods there to run them it would be a legal nightmare. How long before illegal material was flooding the site beyond the capacity of the admins alone to manage? I'd also worry that if Reddit moved in that direction, the mods would feel like they didn't have control over their own subs and bail. I don't believe that Reddit can afford that, especially given how many subs and what %age of the user base we're talking about here.
- throwaway4837 3y agoMy unpopular theory is that the Reddit blackout increased Reddit usage, having the opposite of the intended effect.
- falcolas 3y agoIt would certainly change usage patterns. And there are enough popular subreddits who didn't go in on the outage, so there's content to be had. The adage "there's no such thing as bad press" could definitely apply, given how broadly covered this is (I mean, hell, I saw it show up on NPR).
- pkju62dqmx 3y agoCurrently over 90% of subs are private or restricted. https://reddark.untone.uk https://reddark.untone.uk Re-hosts of that site have the figure even higher.
- cpncrunch 3y agoThat is in no way accurate. It says 7806 subreddits, but there are actually over 3 million subreddits, so the figure is actually only 0.24% rather than 90%. It's wonderful what you can do by just faking statistics! I also don't see some subreddits listed there that are part of the strike.
- Tao3300 3y agoLike reading a list of war dead. Always remember where you were and what you were doing on that fateful day when r/amateurcumsluts went dark, and the heroic sacrifices made on that hill.
- deleted 3y ago[deleted]
- Tao3300 3y ago> there are enough popular subreddits who didn't go in on the outage Yeah, this morning I noticed only one sub that I follow was actually private. I was going to mention it here, but I forgot which one it was - and now I can't find the one that was blacked out.
- Anonasty 3y agoIt seems that Reddit will go down like Digg did.
- PUSH_AX 3y agoMore likely that people will simply forget in 3-4 weeks, like literally every other issue that can sort of blow over.
- willio58 3y agoUnfortunately, I've been hearing this for like a decade and it doesn't seem to ever happen.
- wmeredith 3y agoDigg.com’s implosion was so dramatic because Reddit was there to immediately start serving the same users just about the same thing. There are obviously plenty of other things Reddit users can do with their time, but there isn’t a monolith solution waiting in the wings to very closely replace Reddit with almost the exact same thing. Its destruction will be much more prolonged.
- sebazzz 3y agoExactly the same comments have been made about Twitter, and Twitter is still here.
- mvdtnz 3y agoI see Reddit themselves confirmed that the blackout caused the outage, but just to be clear Reddit is a very unstable website and outages are commonplace. It's clear their engineering team is not exactly top notch (or have been poorly resourced).
- mkl95 3y agoThey should disclose what happens in their system when a subreddit goes private, and how it scales with subreddit size. Otherwise, I call BS.
- seydor 3y agoI think reddit will never be the same after this, not because of the API thing (I dont agree with the blackouts), but because it shows how much they neglected their website. Over the years they have made it impossible to compete with the established subreddits. They have allowed a single uniform mob to take over the site, establish themselves as permanent tyrants and grow their ban lists to enormous scale. Even today with the blackouts, you can't find alternative subs to the closed ones, because new subreddits are hidden in search results and popular lists. Without this competition, how do they expect their subs to improve? It's as if their mismanagement has come back to bite them. Mods have been allowed to run the show and ruin communities for years and years. Very few people are using 3rd party apps but now they are causing inconvenience to millions for their own capricious reasons. I knew that the site was ran by a small powertripping group of mods but now that they make it so obvious it has kind of chepened the whole medium in my mind
- btmiller 3y agoThe subreddit experience is bifurcated into (at least) two camps: big/popular/default or small/curated/niche. The tyrannical behaviors are most often found in the former camp where being a moderator is synonymous with “internet power”. The latter is populated by enthusiasts tending to their communities. I agree that Reddit won’t be the same after this, but there’s just not a ton of remaining value in the big defaults. They’re already plagued with low value, low effort content, low cardinality content. The true damage that’s being done right now is to the small communities which represent(ed) the best of Reddit. Out of the spotlight but deep in quality content for those that care about the topic. The inherent risk of stability is more certain now with Reddit’s new policies. And I suspect now with many subreddits weighing indefinite shutdowns that many users (perhaps more importantly: the best content creators) will scatter to the wind for greener pastures. Those that are left are given crappy apps, lots of ads, and hollow communities eagerly looking for something better.
- phil21 3y ago> The tyrannical behaviors are most often found in the former camp where being a moderator is synonymous with “internet power”. What's old is new again, I suppose! Similar happened on IRC back in the day. All the big channel wars and drama were over the big popular channels for internet points. I certainly participated in my fair share. Then you had the small channels of folks who were just pretty chill chatting about whatever random topic of interest. Might be nostalgia, but I still think the best balance of this was the phpbb/vbulletin era. The amount of "reference material" lost behind the walled gardens is crazy.
- WXLCKNO 3y agoI'd love to see an API only version of Reddit where people build the clients on top of it but are forced to serve ads from the global provider, from which they get a generous cut.