21 ms·

Reddit was down for over two hours

232 points by bluedino 7y ago

quickthrower2 7y ago
No problems from Australia in the last 2 hours (including just now.) For example https://www.reddit.com/r/menwritingwomen/new/ https://www.reddit.com/r/menwritingwomen/new/ shows a post from just now (verified the timestamp).
- LilBytes 7y ago
  It's 'up' for me but, I'm regularly having problems opening comments and posts.
  - tossAfterUsing 7y ago
    i miss the fail whale. IMO, their javascript UX is awful... impossible to see all the comments in a thread! what sort of nonsense is that? old.reddit.com ftw
    - quickthrower2 7y ago
      https://i.reddit.com/ https://i.reddit.com/ ft-mobile-w
      - spraak 7y ago
        Whoa, that is orders of magnitude faster than the default mobile site. Thank you for sharing, I didn't know about it
      - krferriter 7y ago
        Wow it's so fast an the image links are links to actual images, not the post page with an image embedded on it.
      - LilBytes 7y ago
        Wow, so quick. Thank you.
- drusepth 7y ago
  It's just a white page for me in Japan.
- domnomnom 7y ago
  Odd example
- nineteen999 7y ago
  Also in Australia, but it has been both up and down for me in the past 2 hours.
deleted 7y ago
[deleted]
notori0us 7y ago
I noticed this outage coincided with elevated timeouts in one of our EC2 us-east-1 availability zones. We had recovery by removing that zone from our Auto Scaling Groups, but I haven't been able to find many others affected online (other than Reddit). Did anyone else experience something like this? edit: Our errors started at 13:35 Pacific Time.
aldoushuxley001 7y ago
They've been having lots of crazy errors all week (like e.g. all comments posting 3-4 times, other downtimes, etc.). Something's broken on their end for sure.
kryogen1c 7y ago
Webroots website started acting up a couple his ago
deleted 7y ago
[deleted]
dymk 7y ago
Reddit is the only major social media/aggregator I can think of that actually has to go offline for scheduled maintenance still. I can’t imagine the kind of shitshow it must be internally. Imagine if Facebook or Twitter had to go down regularly for “maintenance”. Heads would roll.
- cdiddy2 7y ago
  they don't make much money so don't have the same redundancy luxuries as other sites
  - dymk 7y ago
    They’re Alexa rank #16 globally - that they’re having trouble monetizing their userbase just means they have absolutely no idea what they’re doing.
    - zbentley 7y ago
      That doesn't follow at all. There are all sorts of insanely popular websites which have significant monetization issues (think tumblr, wikipedia). Just because lots of people use something doesn't make them willing to pay for it, and it doesn't make a platform inherently monetizable.
      - lazyjones 7y ago
        Indeed, also not everybody is trying to squeeze money out of their user base regardless of costs (to popularity, principles...).
        dymk 7y ago
        Reddit gold, the new Reddit platinum and bronze, they already show ads, redesign shows ads much more prominently. Literally dozens of ad trackers when you load their site. Reddit is just as much of a profit driven company as Facebook or Twitter, I’m not sure why you’d think otherwise.
paganel 7y ago
At first I thought it was of me refreshing /r/soccer so many times this evening because of CL matches.
bitwize 7y ago
Y'know, the toll that all this activity takes on Reddit's servers gave me an idea for a sort of distributed Reddit. News links and stories assorted into topic-based groups, with the option for users to comment or respond to others' comments in a threaded discussion. It would be completely decentralized and belong to the users, not some central media company. We could call it Users' Network, or USENET for short. We could implement it with blockchain.
- Sendotsh 7y ago
  Sign me up. I’d happily contribute to that as much as I can, whenever my parents aren’t using the phone line.
  - thrownblown 7y ago
    Don't forget to turn off call waiting. I always forgot to turn off call waiting....
    - _asummers 7y ago
      I'm showing my youth here, but this is a legitimate question. Isn't call waiting just a thing that exists on the line? Did phones used to have a switch to toggle that? And what was the harm in keeping it on? When I had dial up, I don't recall having to consider call waiting.
      - uptown 7y ago
        The incoming call tone would disrupt modem traffic so you had to dial a code prior to the number you wanted to dial to disable this temporarily for the duration of your outgoing call.
      - xen2xen1 7y ago
        Dialing *67, I think, in the dial string for dialup turned off call waiting for the duration of the call. Otherwise a second call would try to ring through and disrupt your Super Z-modem download.
banana_giraffe 7y ago
Number of new posts every 15 minutes in the past 24 hours: https://imgur.com/a/HqUsYGn https://imgur.com/a/HqUsYGn It's interesting to see how it fails. Clearly some users can still post. I'm sure there are things that still work (maybe the phone apps .. got me), but also very clearly, there's a sharp dip in new posts.
ropiwqefjnpoa 7y ago
Stuff like this is great, you can always point to it when management demands an explanation why the company email server was unavailable for 5 minutes.
a-wu 7y ago
Marked as resolved now, mods should probably update the title.
StreamBright 7y ago
The productivity of IT teams skyrocketed world wide.
- rapfaria 7y ago
  I remember reading that when a judge ordered Whatsapp to be taken down in Brazil for a few hours, the GDP through that small period went up by 2%.
loraa 7y ago
To be fair, Reddit sucks.
opportune 7y ago
I wonder what Reddit devops is like, I find it so strange that such a trafficked website can’t handle rollouts without what appears (on the outside) to be a full reboot. And also strange that the rollout takes so long. Makes me think they are doing something naive like running on naked vms and having one guy running a script to take them down, update os, redeploy new binary, etc. all at once. Then probably encountering issues and live debugging them until it works again
- gbrayut 7y ago
  some details at https://www.youtube.com/watch?v=z7TIzCAEo0M https://www.youtube.com/watch?v=z7TIzCAEo0M
Deimorz 7y ago
They were doing another test run of their new live-streaming service "RPAN" today, I wouldn't be surprised if that's connected to the downtime. If you can get the page to load: https://old.reddit.com/r/pan/comments/d5paue/a_midweek_treat_rpan_will_be_onair_tomorrow/ https://old.reddit.com/r/pan/comments/d5paue/a_midweek_treat...
JohnBerea 7y ago
And nothing of value was lost.
OrgNet 7y ago
reddit is going back to the reliability that they had a few years ago
bonerman69 7y ago
Does anyone know why they tweeted this??? https://web.archive.org/web/20190918200317/https://twitter.com/reddit/status/821409753 https://web.archive.org/web/20190918200317/https://twitter.c...
- hobabaObama 7y ago
  I believe it was on r/worldnews . They just tweeted it Its still showing up on reddit. https://old.reddit.com/r/worldnews/comments/6l17g/though_blacks_are_only_24_percent_of_new_york/?st=k0q71tmt&sh=dbb3f82e https://old.reddit.com/r/worldnews/comments/6l17g/though_bla...
- srbby 7y ago
  What do you mean? Is it false or something?
Animats 7y ago
And US productivity increased 15% during the outage. Maybe social media should be turned off on Tuesdays and Thursdays, as suggested in Ready Player One.
- 0xdeadb00f 7y ago
  I'd like to see how that would go.
frequentnapper 7y ago
That explains the strange uptick in my productivity chart during that time.
aceon48 7y ago
I dont know how, but reddit's tech actually feels WORSE now. The old site was simple, but it was fast, clean, and it worked for the one thing it needed to do: upvote downvote content. Now the newer version is always trying to kick me into either Chrome or the App. I don't want the f'ing App. And the mobile experience is supppper slow. Its literally just a text webpage it should load near instantly, if they didnt have to redesign the whole thing in some new web framework.

232 points by bluedino 7y ago

No problems from Australia in the last 2 hours (including just now.) For example https://www.reddit.com/r/menwritingwomen/new/ https://www.reddit.com/r/menwritingwomen/new/ shows a post from just now (verified the timestamp).

LilBytes 7y ago
It's 'up' for me but, I'm regularly having problems opening comments and posts.
- tossAfterUsing 7y ago
  i miss the fail whale. IMO, their javascript UX is awful... impossible to see all the comments in a thread! what sort of nonsense is that? old.reddit.com ftw
  - quickthrower2 7y ago
    https://i.reddit.com/ https://i.reddit.com/ ft-mobile-w
    - spraak 7y ago
      Whoa, that is orders of magnitude faster than the default mobile site. Thank you for sharing, I didn't know about it
    - krferriter 7y ago
      Wow it's so fast an the image links are links to actual images, not the post page with an image embedded on it.
    - LilBytes 7y ago
      Wow, so quick. Thank you.
drusepth 7y ago
It's just a white page for me in Japan.
domnomnom 7y ago
Odd example
nineteen999 7y ago
Also in Australia, but it has been both up and down for me in the past 2 hours.

deleted 7y ago

[deleted]

notori0us 7y ago

I noticed this outage coincided with elevated timeouts in one of our EC2 us-east-1 availability zones. We had recovery by removing that zone from our Auto Scaling Groups, but I haven't been able to find many others affected online (other than Reddit). Did anyone else experience something like this? edit: Our errors started at 13:35 Pacific Time.

aldoushuxley001 7y ago

They've been having lots of crazy errors all week (like e.g. all comments posting 3-4 times, other downtimes, etc.). Something's broken on their end for sure.

kryogen1c 7y ago

Webroots website started acting up a couple his ago

deleted 7y ago

[deleted]

dymk 7y ago

Reddit is the only major social media/aggregator I can think of that actually has to go offline for scheduled maintenance still. I can’t imagine the kind of shitshow it must be internally. Imagine if Facebook or Twitter had to go down regularly for “maintenance”. Heads would roll.

cdiddy2 7y ago
they don't make much money so don't have the same redundancy luxuries as other sites
- dymk 7y ago
  They’re Alexa rank #16 globally - that they’re having trouble monetizing their userbase just means they have absolutely no idea what they’re doing.
  - zbentley 7y ago
    That doesn't follow at all. There are all sorts of insanely popular websites which have significant monetization issues (think tumblr, wikipedia). Just because lots of people use something doesn't make them willing to pay for it, and it doesn't make a platform inherently monetizable.
    - lazyjones 7y ago
      Indeed, also not everybody is trying to squeeze money out of their user base regardless of costs (to popularity, principles...).
      - dymk 7y ago
        Reddit gold, the new Reddit platinum and bronze, they already show ads, redesign shows ads much more prominently. Literally dozens of ad trackers when you load their site. Reddit is just as much of a profit driven company as Facebook or Twitter, I’m not sure why you’d think otherwise.

paganel 7y ago

At first I thought it was of me refreshing /r/soccer so many times this evening because of CL matches.

bitwize 7y ago

Y'know, the toll that all this activity takes on Reddit's servers gave me an idea for a sort of distributed Reddit. News links and stories assorted into topic-based groups, with the option for users to comment or respond to others' comments in a threaded discussion. It would be completely decentralized and belong to the users, not some central media company. We could call it Users' Network, or USENET for short. We could implement it with blockchain.

Sendotsh 7y ago
Sign me up. I’d happily contribute to that as much as I can, whenever my parents aren’t using the phone line.
- thrownblown 7y ago
  Don't forget to turn off call waiting. I always forgot to turn off call waiting....
  - _asummers 7y ago
    I'm showing my youth here, but this is a legitimate question. Isn't call waiting just a thing that exists on the line? Did phones used to have a switch to toggle that? And what was the harm in keeping it on? When I had dial up, I don't recall having to consider call waiting.
    - uptown 7y ago
      The incoming call tone would disrupt modem traffic so you had to dial a code prior to the number you wanted to dial to disable this temporarily for the duration of your outgoing call.
    - xen2xen1 7y ago
      Dialing *67, I think, in the dial string for dialup turned off call waiting for the duration of the call. Otherwise a second call would try to ring through and disrupt your Super Z-modem download.

banana_giraffe 7y ago

Number of new posts every 15 minutes in the past 24 hours: https://imgur.com/a/HqUsYGn https://imgur.com/a/HqUsYGn It's interesting to see how it fails. Clearly some users can still post. I'm sure there are things that still work (maybe the phone apps .. got me), but also very clearly, there's a sharp dip in new posts.

ropiwqefjnpoa 7y ago

Stuff like this is great, you can always point to it when management demands an explanation why the company email server was unavailable for 5 minutes.

a-wu 7y ago

Marked as resolved now, mods should probably update the title.

StreamBright 7y ago

The productivity of IT teams skyrocketed world wide.

rapfaria 7y ago
I remember reading that when a judge ordered Whatsapp to be taken down in Brazil for a few hours, the GDP through that small period went up by 2%.

loraa 7y ago

To be fair, Reddit sucks.

opportune 7y ago

I wonder what Reddit devops is like, I find it so strange that such a trafficked website can’t handle rollouts without what appears (on the outside) to be a full reboot. And also strange that the rollout takes so long. Makes me think they are doing something naive like running on naked vms and having one guy running a script to take them down, update os, redeploy new binary, etc. all at once. Then probably encountering issues and live debugging them until it works again

gbrayut 7y ago
some details at https://www.youtube.com/watch?v=z7TIzCAEo0M https://www.youtube.com/watch?v=z7TIzCAEo0M

Deimorz 7y ago

They were doing another test run of their new live-streaming service "RPAN" today, I wouldn't be surprised if that's connected to the downtime. If you can get the page to load: https://old.reddit.com/r/pan/comments/d5paue/a_midweek_treat_rpan_will_be_onair_tomorrow/ https://old.reddit.com/r/pan/comments/d5paue/a_midweek_treat...

JohnBerea 7y ago

And nothing of value was lost.

OrgNet 7y ago

reddit is going back to the reliability that they had a few years ago

bonerman69 7y ago

Does anyone know why they tweeted this??? https://web.archive.org/web/20190918200317/https://twitter.com/reddit/status/821409753 https://web.archive.org/web/20190918200317/https://twitter.c...

hobabaObama 7y ago
I believe it was on r/worldnews . They just tweeted it Its still showing up on reddit. https://old.reddit.com/r/worldnews/comments/6l17g/though_blacks_are_only_24_percent_of_new_york/?st=k0q71tmt&sh=dbb3f82e https://old.reddit.com/r/worldnews/comments/6l17g/though_bla...
srbby 7y ago
What do you mean? Is it false or something?

Animats 7y ago

And US productivity increased 15% during the outage. Maybe social media should be turned off on Tuesdays and Thursdays, as suggested in Ready Player One.

0xdeadb00f 7y ago
I'd like to see how that would go.

frequentnapper 7y ago

That explains the strange uptick in my productivity chart during that time.

aceon48 7y ago

I dont know how, but reddit's tech actually feels WORSE now. The old site was simple, but it was fast, clean, and it worked for the one thing it needed to do: upvote downvote content. Now the newer version is always trying to kick me into either Chrome or the App. I don't want the f'ing App. And the mobile experience is supppper slow. Its literally just a text webpage it should load near instantly, if they didnt have to redesign the whole thing in some new web framework.