5 ms·

It looks like the Obama AMA has really caused some serious fires in reddit's backend infrastructure. The site's been down for the last 10 minutes. I would've t

by ashray 14y ago

It looks like the Obama AMA has really caused some serious fires in reddit's backend infrastructure. The site's been down for the last 10 minutes. I would've thought that they'd have brought in some additional computing power for such an event, should've been easy for them considering they have a cloud deployment. Maybe this gives them greater reason to hire more engineers. I found it impressive that they served billions of impressions with just 2 engineers a short while ago.. Also, it says a lot about the "Come Cloud with us, we'll help you scale" marketing bandwagon. We've seen time and again issues with EC2s infrastructure and if EC2 doesn't have issues right now (http://status.aws.amazon.com/ http://status.aws.amazon.com/) then it's just sad that they can't order a gazillion instances for this event and have it scale easily. Definitely makes me think that we still have a long way to go to compute in a truly 'elastic' way. reddit definitely does have some crazy infrastructure in place but this would've been one of the most important moments in reddit history (so far..) and I'm sad to see that their engineers are probably going to get blamed for this.. EDIT: Okay, they're back in read only mode.. I wonder how they'll hack in some write access for the AMA while keeping everything else read only. Time for some app server redeployments! Funsies! EDIT2: And they're gone again sigh

untog 14y ago
I'd be more interested in seeing how a site using a service like Google App Engine responds. In theory their Cloud Storage should just keep on scaling and scaling, rather than forcing the admin to spin up more instances, set up database replication, etc, etc.
- ljd 14y ago
  AWS Elastic Beanstalk will do the same thing.
  - untog 14y ago
    AFAIK Reddit aren't using it. If anyone can clarify I'd love to hear more.
  - deno 14y ago
    Beanstalk has nothing to do with the database, which is the point of contention. Spinning up more app servers only helps with serving signed in users.
    - ljd 14y ago
      You can configure beanstalk to scale your database out as well.
      - deno 14y ago
        Scaling out database is nothing at all like spinning up stateless app servers.
        ljd 14y ago
        Well there is no magical database solution where you can do that and I have to make decisions on what I can buy today not on what may be magically developed in the future. In this respect, application partitioning with a deployable schema is not a bad solution for what I do.
- ashray 14y ago
  Well, to be fair maybe they didn't have enough notice to spin up additional instances and get replication completed beforehand but I highly doubt that considering how high profile this AMA is. The US President doesn't probably drop in on people / or websites :P Yeah GAE might work better but they're a bit out of reach mostly due to the technologies that you are required to use. Any migration at that scale is a seriously massive matter. I guess the funny thing is that you can't plan to grow that big. Usually it just happens suddenly (facebook/reddit style) and then you've got to make some tough architectural calls. It's funny how folks are like "That's a problem I'd like to have..". I'm not sure they're very happy with this problem at this very moment.. sigh
  - mattmaroon 14y ago
    I don't know about websites, but the US President drops in on lots of places (restaurants for instance) as a matter of security. From what I understand with interviews, it's often not uncommon for the interviewer to get very little notice either at times.
lobster45 14y ago
I agree that this is more of a problem with elastic cloud services than reddit engineers. I am sure that they anticipated the load and did what they could, but the traffic is surely immense
lobster45 14y ago
Looks like part of their infrastructure is still up. website is down, but I can connect on my mobile using baconreader app
pippy 14y ago
Stupid question: is the because reddits backend is in python? I often hear that PHP + varnish is only limited by the size of the pipe. Can this scale on cloud platforms?
- ashray 14y ago
  No, it's probably due to the massive number of reads/writes pumping through their system. I'm pretty sure that their database isn't able to handle what's going on at the moment. Also, if you were to use Varnish as a cache, that wouldn't help you much with write access because every time there is a write (a new post..) you'd have to invalidate the cache. In a high volume scenario such as this (tons of posts coming in..), the page is as good as dynamic, even if you DO cache it for a bit.
  - natrius 14y ago
    Varnish can only really help Reddit with users who aren't logged in since every page is customized. It'd still help in high volume scenarios since Varnish solves the thundering herd problem: only one request (at most) will hit the database for each invalidation.
    - emidln 14y ago
      Vanish can cache snippets of content. They refer to this as Edge Side Includes.
      - natrius 14y ago
        Reddit's customization is far more than snippets. I've used ESIs with Varnish extensively, and I don't think it would help much for authenticated Reddit users.
  - pippy 14y ago
    Thank you, this is the answer that I was looking for
- ubernostrum 14y ago
philwelch 14y ago
Cloud computing is cruise control for scaling, but you still have to steer.
morsegeek 14y ago
I'm still able to access the AMA through Alien Blue on iPhone, so that part of their infrastructure must still be up.
tzs 14y ago
How much advance warning did Reddit have that Obama was doing an AMA?
Achshar 14y ago
I believe it was a surprise AMA, so no heads up to Reddit team.
bolasanibk 14y ago
AskReddit thread about the stats: http://www.reddit.com/r/AskReddit/comments/z2fo7/admins_of_reddit_today_you_probably_all_almost/ http://www.reddit.com/r/AskReddit/comments/z2fo7/admins_of_r... Page views/uniques http://i.imgur.com/vPnEK.png http://i.imgur.com/vPnEK.png

by ashray 14y ago

untog 14y ago

I'd be more interested in seeing how a site using a service like Google App Engine responds. In theory their Cloud Storage should just keep on scaling and scaling, rather than forcing the admin to spin up more instances, set up database replication, etc, etc.

ljd 14y ago
AWS Elastic Beanstalk will do the same thing.
- untog 14y ago
  AFAIK Reddit aren't using it. If anyone can clarify I'd love to hear more.
- deno 14y ago
  Beanstalk has nothing to do with the database, which is the point of contention. Spinning up more app servers only helps with serving signed in users.
  - ljd 14y ago
    You can configure beanstalk to scale your database out as well.
    - deno 14y ago
      Scaling out database is nothing at all like spinning up stateless app servers.
      - ljd 14y ago
        Well there is no magical database solution where you can do that and I have to make decisions on what I can buy today not on what may be magically developed in the future. In this respect, application partitioning with a deployable schema is not a bad solution for what I do.
ashray 14y ago
Well, to be fair maybe they didn't have enough notice to spin up additional instances and get replication completed beforehand but I highly doubt that considering how high profile this AMA is. The US President doesn't probably drop in on people / or websites :P Yeah GAE might work better but they're a bit out of reach mostly due to the technologies that you are required to use. Any migration at that scale is a seriously massive matter. I guess the funny thing is that you can't plan to grow that big. Usually it just happens suddenly (facebook/reddit style) and then you've got to make some tough architectural calls. It's funny how folks are like "That's a problem I'd like to have..". I'm not sure they're very happy with this problem at this very moment.. sigh
- mattmaroon 14y ago
  I don't know about websites, but the US President drops in on lots of places (restaurants for instance) as a matter of security. From what I understand with interviews, it's often not uncommon for the interviewer to get very little notice either at times.

lobster45 14y ago

I agree that this is more of a problem with elastic cloud services than reddit engineers. I am sure that they anticipated the load and did what they could, but the traffic is surely immense

lobster45 14y ago

Looks like part of their infrastructure is still up. website is down, but I can connect on my mobile using baconreader app

pippy 14y ago

Stupid question: is the because reddits backend is in python? I often hear that PHP + varnish is only limited by the size of the pipe. Can this scale on cloud platforms?

ashray 14y ago
No, it's probably due to the massive number of reads/writes pumping through their system. I'm pretty sure that their database isn't able to handle what's going on at the moment. Also, if you were to use Varnish as a cache, that wouldn't help you much with write access because every time there is a write (a new post..) you'd have to invalidate the cache. In a high volume scenario such as this (tons of posts coming in..), the page is as good as dynamic, even if you DO cache it for a bit.
- natrius 14y ago
  Varnish can only really help Reddit with users who aren't logged in since every page is customized. It'd still help in high volume scenarios since Varnish solves the thundering herd problem: only one request (at most) will hit the database for each invalidation.
  - emidln 14y ago
    Vanish can cache snippets of content. They refer to this as Edge Side Includes.
    - natrius 14y ago
      Reddit's customization is far more than snippets. I've used ESIs with Varnish extensively, and I don't think it would help much for authenticated Reddit users.
- pippy 14y ago
  Thank you, this is the answer that I was looking for
ubernostrum 14y ago

philwelch 14y ago

Cloud computing is cruise control for scaling, but you still have to steer.

morsegeek 14y ago

I'm still able to access the AMA through Alien Blue on iPhone, so that part of their infrastructure must still be up.

tzs 14y ago

How much advance warning did Reddit have that Obama was doing an AMA?

Achshar 14y ago

I believe it was a surprise AMA, so no heads up to Reddit team.

bolasanibk 14y ago

AskReddit thread about the stats: http://www.reddit.com/r/AskReddit/comments/z2fo7/admins_of_reddit_today_you_probably_all_almost/ http://www.reddit.com/r/AskReddit/comments/z2fo7/admins_of_r... Page views/uniques http://i.imgur.com/vPnEK.png http://i.imgur.com/vPnEK.png