7 ms·
Reddit is experiencing a partial outage
- MuffinFlavored 4y ago"just roll it back" /s Anybody want to guess root cause? Do we have a "root cause" bingo card? DNS Database What else are super likely?
- Trung0246 4y agoJust for fun I've bet on hardware failure.
- technothrasher 4y agoCat walking on keyboard?
- cududa 4y agoWhat an odd comment. As a software engineer, my professional guess is the computers aren’t working the way they should.
- MuffinFlavored 4y agoI used to work for a very high level director who was promoted many, many times (probably VP level+, $300k/yr easily total comp, probably 80 indirect reports overall in the org, probably 10-20 years experience) whose entire incident playbook handling philosophy was "how quickly can we roll it back/why hasn't it been rolled back yet/have you tried rolling it back yet"
- lordnacho 4y ago"It works on my career"
- cududa 4y ago“And that’s how docker was born”
- NobleLie 4y ago"Let me run the reverse reverse migration script again"
- Spivak 4y agoWoe be upon thine fools that change code and database schema simultaneously.
- ketralnis 4y agoIt's weird for it to be their _entire_ playbook, but most outages that I've exacerbated were because I panicedly tried to fix things instead of just rolling it back and then taking stock. I often have to work hard to convince people of all experience levels that it's the best way forward. - "It's just a little bug I can just fix it [and definitely won't make it worse with code that I haven't tested as rigorously right?]" - "My KPI/bonus/project plan relies on this going out today" - "My code is fine it's the infrastructure [that I didn't warn] that can't handle it. They need to fix their side now." I don't know about your VP but "how fast can we get back to before it was broken?" is reasonably the first thing you should be asking
- bombcar 4y agoIncident response should always be: (1) get people enacting the final disaster recovery plan and rollback whilst we (2) see if we can recover from where we stand. Doing #1 puts some serious boundaries on how bad it can get
- spydum 4y agoi find its usually the same persons or teams responsible for both. hard to do them in parallel
- cortesoft 4y agoThis probably really depends on the type of business you have. I work for a CDN, our outages are usually caused by one of our network peers/providers borking things. There is nothing to rollback.
- ketralnis 4y agoFor sure, and you're not going to be able to roll back a failed power supply. I'm just saying it's a totally reasonable first and maybe even second question
- packetslave 4y agoIf you're a director with 10-20 years experience and you're making $300k/year, you're REALLY doing it wrong.
- jandrese 4y agoInvestor left their bottle of Tequilla sitting on the delete key?
- MuffinFlavored 4y agothat sounds Twitter-Musk-esque, amirite?
- martin8412 4y agoIt's from the series Silicon Valley
- deleted 4y ago[deleted]
- monitron 4y agoPre-IPO jitters?
- SgtBastard 4y agoSeems to be limited to loading comments in-app. Posts are loading fine and Reddit.com is showing comment threads. Busted API deployment?
- bguebert 4y agoYou should add "BGP config problems" and "cryptolocker"
- sen 4y agoIt’s always DNS.
- mattbrewsbytes 4y agoYou missed the other two common ones: permissions change and a disk filled up somewhere. Before finding out the dead simple failure mode and fix, engineers need to spend countless hours diving into the most technically complex scenarios that might be happening but are irrelevant. Then they can reset permissions or add disk space or add back a DNS entry.
- bombcar 4y agoReminds me of the old sysadmin who always made a file 10% the size of the disk named .root-emergency or similar. Disk filled up? Delete the file, get some breathing time, fix the problem, recreate the file.
- buggeryorkshire 4y agoIsn't that what ext filesystems on Linux do? IIRC the reserved portion is 5% which can be dropped if you need some headroom.
- yamtaddle 4y agoYeah, the root-reserved blocks are tunable. Won't save you if someone's running-as-root reporting job goes rogue and fills up the disk, though, while the file might... I mean, obviously one ought not have done that in the first place, but the real world is a whole thing.
- bombcar 4y agoYep. Often a system crash is caused by logging, which often logs ... as root.
- throwaway81523 4y agoTry SCE to AUX.
- JKCalhoun 4y agoA Tesla fire in Newark, CA took out an internet backbone?
- HideousKojima 4y agoOr a North American Fiber-Seeking Backhoe.
- wozer 4y agoFor me it looks like the "old" reddit design is especially affected by the outage. "new.reddit.com" works fine for me.
- sondr3 4y agoCan't wait for this to be one of the reasons the hard deprecated old reddit :(
- duckmysick 4y agoMaybe the cause and effect is reversed. We're seeing errors because the old reddit is deprecated.
- ericmay 4y agoIt will suck when they do that, but at least I’ll get whatever time I spend on Reddit back to funnel somewhere else.
- nickstinemates 4y agoAs long as the API exists, so can your reddit consumption.
- unshavedyak 4y agoIs there a way to use this on desktop? I’m not sure I care enough about Reddit to worry too much. Tbh I look forward to the day I can cut that time sink out. I just can’t peel myself away entirely.
- throwaway81523 4y agoI just use some home cooked python scripts on desktop, but I think there is a gnus.el client somewhere too. On my phone, Red Reader is great. It's on F-droid.org.
- 4y ago
- deleted 4y ago[deleted]
- henriquez 4y agoNow how will I look at porn while I’m supposed to be working?
- unsupp0rted 4y agoI noticed after last week’s “don’t be cute” discussion the error page stopped saying “you broke Reddit” and switched to a generic 503-sounding message
- ketralnis 4y agoWe didn't change anything AFAIK, different parts of the stack have different error messages
- dcdc123 4y agoLet's be honest here...Reddit is _always_ experiencing a partial outage...
- marcosdumay 4y agoWell, the outage was in a larger part than usual.
- JohnBooty 4y agoOne way they appear to be managing load is randomly banning users. I've never had a report against me, I never post anything remotely spicy, I never even argue or flame. I moderated multiple subs. And yet my 9-year, paid Reddit Gold account was randomly nuked. No explanation. Appeals denied. Support emails never answered. I guess if your servers can't keep up, one way to cope is to just ban the folks who contribute the most to your site?
- nso 4y agoYou really believe they ban people as a way of mitigating high server loads? I'm sorry but like, come on...
- JohnBooty 4y agoI do not literally believe that, no. However I do often wonder what in the world is happening at Reddit HQ. Mitigating server load by banning users isn't too much sillier than some of their actual decisions.
- corbulo 4y agoOnly partial?
- assimpleaspossi 4y agoWhat is reddit and why would anyone go there?
- TheMaskedCoder 4y agoI think it's like teddit, but worse
- nothrowaways 4y agoIt has a rich niche community for everything, you will find brutally honest experts in everything.
- whatshisface 4y agoReddit is a website where anyone can make their own hackernews.
- randomdata 4y agoWith the added bonus of a CEO who will helpfully edit your comments if your attempt wasn’t funny enough.
- filoleg 4y agoLet me be clear, I think it was an absolutely moronic move by the reddit CEO. However, you make it sound like it happens all the time. In reality, it happened literally just once (multiple comments, but all within one hour), the edit itself was made obvious on purpose, and it has never happened again (it's been almost 7 years since then). For additional context to what happened: the CEO replaced his own username (u/spez) in a few comments on r/the_donald (that were just variations of "fuck u/spez") with usernames of r/the_donald mods. Again, I have a very low opinion of Reddit leadership team and their direction in general, but "CEO editing user comments" is not a concern I have about the website at all. Mostly because it was an isolated incident that happened over half a decade ago.
- blibble 4y ago