10 ms·

Hacker News is hosted at M5 and they are having a network outage: http://status.m5hosting.com/pages/incident/5407b8e2b002442510000272/604fa9a35aef05052e955afe

Hacker News is hosted at M5 and they are having a network outage: http://status.m5hosting.com/pages/incident/5407b8e2b002442510000272/604fa9a35aef05052e955afe http://status.m5hosting.com/pages/incident/5407b8e2b00244251... edit: Unrelated to the Azure outage.

fotta 6y ago
I'm surprised that a site as big as HN is only hosted in one place.
- cm2187 6y ago
  You should look at stackoverflow's hosting!
  - aspectmin 6y ago
    Is this described somewhere? :)
    - cm2187 6y ago
      The most recent resource I found. I think they basically use a rack in one datacentre. https://meta.stackexchange.com/questions/10369/which-tools-and-technologies-are-used-to-build-the-stack-exchange-network https://meta.stackexchange.com/questions/10369/which-tools-a...
    - giantrobot 6y ago
      This is the newest version of their architecture I've seen [0]. Compare to an overview from 2009 [1]. tl;dr StackOverflow's architecture is fairly simple and has done mostly vertical scaling (more powerful machines) and bare metal servers rather than virtual servers. They also realize their use patterns are read-heavy so there's a lot of caching and they take advantage of CDNs for static content which completely offloads that traffic off their main servers. [0] https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... [1] http://highscalability.com/stack-overflow-architecture http://highscalability.com/stack-overflow-architecture
- Sahbak 6y ago
  Sometimes, it pays off being extremely simple. In HN, it definitely does
  - mromanuk 6y ago
    After this event, they should switch to two servers in different DC.
    - centimeter 6y ago
      Having two servers is a lot more than 2x as complicated and expensive as having 1 server.
      - jpwgarrison 6y ago
        A wise colleague recently explained to me that if you build HA things HA from the start, it's only a little more than 2x. If you try to make an _existing_ system HA, it's 3x at best. HN is not a paid service, they can be down for a few hours per year, no problem. We're not all going to walk away in disgust.
        joshmanders 6y ago
        Not to mention the HNStatus[0] twitter account has so few tweets[1] I don't think it's even worth it. [0]: https://twitter.com/HNStatus https://twitter.com/HNStatus [1]: Last tweet since before today's incident was 2 years ago, 4 years ago since the one before that.
    - johannes1234321 6y ago
      When going to two you need to handle split brain some way probably, otherwise you end up with an database state hard to merge, thus you better get three, so two can find consensus, or at least an external arbitration node, deciding on who is up. At that point you have lots of complexity ... while for HN being down for a bit isn't much of a (business) loss. For other sites that maths probably is different. (I assume they keep off-site backups and could recover from there fairly quickly)
- jsty 6y ago
  Until 2018 at least it was ... wait for it ... a single server! https://news.ycombinator.com/item?id=18496344 https://news.ycombinator.com/item?id=18496344 (Anyone know if that's still the case?)
  - dang 6y ago
    One production server and one failover (in the same data center, obviously).
    - betamaxthetape 6y ago
      I assume there are off-site backups, though? Asking as someone who was impacted by the OVH fire last week, and I didn't have recent backups and therefore lost data.
      - ghotli 6y ago
        I've been waiting to see a comment like this somewhere. Just a hugops from the internet and a reminder to all who see this to get your backups fire-proof and off-site.
      - dang 6y ago
        Yes, we've got a good backup system thanks to the greatly-missed sctb. Sorry to hear that, that sucks.
- Aperocky 6y ago
  HN is probably very small. Curious as to the minimum size of the backend that will hold up the website. There may need to be read replicas, but maybe not even that is needed.
  - _joel 6y ago
    They only have one server, iirc.
    - voxadam 6y ago
      And, if I'm not mistaken, the site is single threaded.
  - aspectmin 6y ago
    Would love to see the HN architecture.
    - krapp 6y ago
      arclanguage.org hosts the current version of Arc Lisp, including an old version of the forum, but HN has made a lot of changes locally that they won't disclose for business reasons. There's an open source fork at https://github.com/arclanguage/anarki https://github.com/arclanguage/anarki, but it doesn't have any direct relationship with HN.
    - mike_d 6y ago
      Single threaded LISP application running on a single machine. Ta-da.
      - dang 6y ago
        The application is multi-threaded. But it runs over a green-thread language runtime, which maps everything to one OS thread. That's a significant distinction because if you swap the underlying implementation then the same application should magically become multithreaded, which is exactly the plan.
    - deleted 6y ago
- mwcampbell 6y ago
  Running on a single server is cheaper, and nobody loses money if HN is down (as far as I know), so it makes sense.
nodesocket 6y ago
Not sure why HN would still be a hosted at a 3rd tier provider. A few EC2 instances (multi-zone) behind a application load balancer should do the trick.
- colinmhayes 6y ago
  Because it doesn't make any money
  - heavyset_go 6y ago
    YC's companies get free advertising and job listings on the front page.
    - mike_d 6y ago
      You just described not making money.
  - nodesocket 6y ago
    Who cares it’s a few hundred a month to host on AWS
    - mike_d 6y ago
      Then where will people go to learn about AWS outages?
- dang 6y ago
  They're first-tier to us.
1vuio0pswjnm7 6y ago
HN is also available through Cloudflare but that seems to depend on M5. Don't take my word for it. Test it for yourself: printf 'GET / HTTP/1.1\r\nHost: news.ycombinator.com\r\nConnection: close\r\n\r\n' \ |openssl s_client -connect cloudflare.com:443 -ign_eof -servername news.ycombinator.com
- slig 6y ago
  Cloudflare only proxies dynamic websites.
  - robocat 6y ago
    CloudFlare will proxy whatever site you configure, even if it is static. Static websites will get the best speed boost from locally served assets (much reduced latency from the local POP) because the page itself can be cached (presuming headers on origin site are correctly set). Especially for page requests from international users.
    - slig 6y ago
      Sorry, I wasn't clear. What I meant was that since HN is dynamic its dynamic content is not usually cached. I mentioned specifically "dynamic" sites on my parent comment because as of this month Cloudflare can host static pages.
- dang 6y ago
  We stopped using Cloudflare a few years ago. https://news.ycombinator.com/item?id=18188832 https://news.ycombinator.com/item?id=18188832 https://news.ycombinator.com/item?id=21799045 https://news.ycombinator.com/item?id=21799045
  - ksec 6y ago
    >The desire to have fewer moving parts This actually got me thinking. Do we really need CDN? This is one of those thing we take and use without actually thinking whether we could do without it. Interesting thought experiment.
tootie 6y ago
We had some AWS network issues today too...

fotta 6y ago

I'm surprised that a site as big as HN is only hosted in one place.

cm2187 6y ago
You should look at stackoverflow's hosting!
- aspectmin 6y ago
  Is this described somewhere? :)
  - cm2187 6y ago
    The most recent resource I found. I think they basically use a rack in one datacentre. https://meta.stackexchange.com/questions/10369/which-tools-and-technologies-are-used-to-build-the-stack-exchange-network https://meta.stackexchange.com/questions/10369/which-tools-a...
  - giantrobot 6y ago
    This is the newest version of their architecture I've seen [0]. Compare to an overview from 2009 [1]. tl;dr StackOverflow's architecture is fairly simple and has done mostly vertical scaling (more powerful machines) and bare metal servers rather than virtual servers. They also realize their use patterns are read-heavy so there's a lot of caching and they take advantage of CDNs for static content which completely offloads that traffic off their main servers. [0] https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... [1] http://highscalability.com/stack-overflow-architecture http://highscalability.com/stack-overflow-architecture
Sahbak 6y ago
Sometimes, it pays off being extremely simple. In HN, it definitely does
- mromanuk 6y ago
  After this event, they should switch to two servers in different DC.
  - centimeter 6y ago
    Having two servers is a lot more than 2x as complicated and expensive as having 1 server.
    - jpwgarrison 6y ago
      A wise colleague recently explained to me that if you build HA things HA from the start, it's only a little more than 2x. If you try to make an _existing_ system HA, it's 3x at best. HN is not a paid service, they can be down for a few hours per year, no problem. We're not all going to walk away in disgust.
      - joshmanders 6y ago
        Not to mention the HNStatus[0] twitter account has so few tweets[1] I don't think it's even worth it. [0]: https://twitter.com/HNStatus https://twitter.com/HNStatus [1]: Last tweet since before today's incident was 2 years ago, 4 years ago since the one before that.
  - johannes1234321 6y ago
    When going to two you need to handle split brain some way probably, otherwise you end up with an database state hard to merge, thus you better get three, so two can find consensus, or at least an external arbitration node, deciding on who is up. At that point you have lots of complexity ... while for HN being down for a bit isn't much of a (business) loss. For other sites that maths probably is different. (I assume they keep off-site backups and could recover from there fairly quickly)
jsty 6y ago
Until 2018 at least it was ... wait for it ... a single server! https://news.ycombinator.com/item?id=18496344 https://news.ycombinator.com/item?id=18496344 (Anyone know if that's still the case?)
- dang 6y ago
  One production server and one failover (in the same data center, obviously).
  - betamaxthetape 6y ago
    I assume there are off-site backups, though? Asking as someone who was impacted by the OVH fire last week, and I didn't have recent backups and therefore lost data.
    - ghotli 6y ago
      I've been waiting to see a comment like this somewhere. Just a hugops from the internet and a reminder to all who see this to get your backups fire-proof and off-site.
    - dang 6y ago
      Yes, we've got a good backup system thanks to the greatly-missed sctb. Sorry to hear that, that sucks.
Aperocky 6y ago
HN is probably very small. Curious as to the minimum size of the backend that will hold up the website. There may need to be read replicas, but maybe not even that is needed.
- _joel 6y ago
  They only have one server, iirc.
  - voxadam 6y ago
    And, if I'm not mistaken, the site is single threaded.
- aspectmin 6y ago
  Would love to see the HN architecture.
  - krapp 6y ago
    arclanguage.org hosts the current version of Arc Lisp, including an old version of the forum, but HN has made a lot of changes locally that they won't disclose for business reasons. There's an open source fork at https://github.com/arclanguage/anarki https://github.com/arclanguage/anarki, but it doesn't have any direct relationship with HN.
  - mike_d 6y ago
    Single threaded LISP application running on a single machine. Ta-da.
    - dang 6y ago
      The application is multi-threaded. But it runs over a green-thread language runtime, which maps everything to one OS thread. That's a significant distinction because if you swap the underlying implementation then the same application should magically become multithreaded, which is exactly the plan.
  - deleted 6y ago
mwcampbell 6y ago
Running on a single server is cheaper, and nobody loses money if HN is down (as far as I know), so it makes sense.

nodesocket 6y ago

Not sure why HN would still be a hosted at a 3rd tier provider. A few EC2 instances (multi-zone) behind a application load balancer should do the trick.

colinmhayes 6y ago
Because it doesn't make any money
- heavyset_go 6y ago
  YC's companies get free advertising and job listings on the front page.
  - mike_d 6y ago
    You just described not making money.
- nodesocket 6y ago
  Who cares it’s a few hundred a month to host on AWS
  - mike_d 6y ago
    Then where will people go to learn about AWS outages?
dang 6y ago
They're first-tier to us.

1vuio0pswjnm7 6y ago

HN is also available through Cloudflare but that seems to depend on M5. Don't take my word for it. Test it for yourself: printf 'GET / HTTP/1.1\r\nHost: news.ycombinator.com\r\nConnection: close\r\n\r\n' \ |openssl s_client -connect cloudflare.com:443 -ign_eof -servername news.ycombinator.com

slig 6y ago
Cloudflare only proxies dynamic websites.
- robocat 6y ago
  CloudFlare will proxy whatever site you configure, even if it is static. Static websites will get the best speed boost from locally served assets (much reduced latency from the local POP) because the page itself can be cached (presuming headers on origin site are correctly set). Especially for page requests from international users.
  - slig 6y ago
    Sorry, I wasn't clear. What I meant was that since HN is dynamic its dynamic content is not usually cached. I mentioned specifically "dynamic" sites on my parent comment because as of this month Cloudflare can host static pages.
dang 6y ago
We stopped using Cloudflare a few years ago. https://news.ycombinator.com/item?id=18188832 https://news.ycombinator.com/item?id=18188832 https://news.ycombinator.com/item?id=21799045 https://news.ycombinator.com/item?id=21799045
- ksec 6y ago
  >The desire to have fewer moving parts This actually got me thinking. Do we really need CDN? This is one of those thing we take and use without actually thinking whether we could do without it. Interesting thought experiment.

tootie 6y ago

We had some AWS network issues today too...