5 ms·

Why the move? Hopefully it’s simply M5 didn’t have a server ready and they’ll migrate back. Vultr has a great assortment of bare metal servers. https://www.v

Why the move? Hopefully it’s simply M5 didn’t have a server ready and they’ll migrate back. Vultr has a great assortment of bare metal servers. https://www.vultr.com/products/bare-metal/#pricing https://www.vultr.com/products/bare-metal/#pricing

jacooper 4y ago
There is also Hetzner, their dedicated server pricing is very good.
- ta988 4y ago
  But they don't have servers in US do they?
  - Amfy 4y ago
    they do, in Ashburn. I would not host production with them.
    - whitepoplar 4y ago
      Why?
      - ev1 4y ago
        I host a lot of production there, but always some layers underneath - like a high CPU or backing storage layer or database, worker nodes that pick up tasks. Never the front user facing. Do not let any user generated content be accessible from any hetzner IP or you are one email away from shutdown pretty much. Don't forget germany's laws on speech too; they are nothing remotely similar to US. I would host, for example, a corporate site just fine, but last thing ever would be a forum or image hosting site or w/e
        Amfy 4y ago
        this. matches my experience and the ones in my circle
    - closeparen 4y ago
      That’s a VPS offering only, they don’t do dedicated servers here.
- Amfy 4y ago
  they are not very good for hosting user generated content, they suspend very fast for any sort of Abuse complaint. User generated content, will result in some Abuse complaints.
  - jacooper 4y ago
    I host a forum in it and there is nor problems. Of course a site as big as HN would've different rules and treatment.
dang 4y ago
> Why the move? Our primary server died around 11pm last night (PST), so we switched to our secondary server, but then our secondary server died around 6am, and we didn't have a third. The plan was always "in the unlikely event that both servers die at the same time, be able to spin HN up on AWS." We knew it would take us several hours to do that, but it seemed an ok tradeoff given how unlikely the both-servers-die-at-the-same-time scenario seemed at the time. (It doesn't seem so unlikely now. In fact it seems to have a probability of 1.) Given what we knew when we made that plan, I'm pretty pleased with how things have turned out so far (fingers crossed—no jinx—definitely not gloating). We had done dry runs of this and made good-enough notes. It sucks to have been down for 8 hours, but it could have been worse, and without good backups (thank you sctb!) it would have been catastrophic. Having someone as good as mthurman do most of the work is also a really good idea.
- omegalulw 4y ago
  Do you have a postmortem on why both servers died so fast?
  - dang 4y ago
    It was an SSD that failed in each case, and in a similar way (e.g. both were in RAID arrays but neither could be rebuilt from the array - but I am over my skis in reporting this, as I barely know what that means). The disks were in two physically separate servers that were not connected to each other. I believe, however, that they were of similar make and model. So the leading hypothesis seems to be that perhaps the SSDs were from the same manufacturing batch and shared some defect. In other words, our servers were inbred! Which makes me want to link to the song 'Second Cousin' by Flamin' Groovies. The HN hindsight consensus, to judge by the replies to https://news.ycombinator.com/item?id=32026606 https://news.ycombinator.com/item?id=32026606, is that this happens all the time, is not surprising at all, and is actually quite to be expected. Live and learn!
    - whitepoplar 4y ago
      Do you happen to know the make and model of SSD?

jacooper 4y ago

There is also Hetzner, their dedicated server pricing is very good.

ta988 4y ago
But they don't have servers in US do they?
- Amfy 4y ago
  they do, in Ashburn. I would not host production with them.
  - whitepoplar 4y ago
    Why?
    - ev1 4y ago
      I host a lot of production there, but always some layers underneath - like a high CPU or backing storage layer or database, worker nodes that pick up tasks. Never the front user facing. Do not let any user generated content be accessible from any hetzner IP or you are one email away from shutdown pretty much. Don't forget germany's laws on speech too; they are nothing remotely similar to US. I would host, for example, a corporate site just fine, but last thing ever would be a forum or image hosting site or w/e
      - Amfy 4y ago
        this. matches my experience and the ones in my circle
  - closeparen 4y ago
    That’s a VPS offering only, they don’t do dedicated servers here.
Amfy 4y ago
they are not very good for hosting user generated content, they suspend very fast for any sort of Abuse complaint. User generated content, will result in some Abuse complaints.
- jacooper 4y ago
  I host a forum in it and there is nor problems. Of course a site as big as HN would've different rules and treatment.

dang 4y ago

> Why the move? Our primary server died around 11pm last night (PST), so we switched to our secondary server, but then our secondary server died around 6am, and we didn't have a third. The plan was always "in the unlikely event that both servers die at the same time, be able to spin HN up on AWS." We knew it would take us several hours to do that, but it seemed an ok tradeoff given how unlikely the both-servers-die-at-the-same-time scenario seemed at the time. (It doesn't seem so unlikely now. In fact it seems to have a probability of 1.) Given what we knew when we made that plan, I'm pretty pleased with how things have turned out so far (fingers crossed—no jinx—definitely not gloating). We had done dry runs of this and made good-enough notes. It sucks to have been down for 8 hours, but it could have been worse, and without good backups (thank you sctb!) it would have been catastrophic. Having someone as good as mthurman do most of the work is also a really good idea.

omegalulw 4y ago
Do you have a postmortem on why both servers died so fast?
- dang 4y ago
  It was an SSD that failed in each case, and in a similar way (e.g. both were in RAID arrays but neither could be rebuilt from the array - but I am over my skis in reporting this, as I barely know what that means). The disks were in two physically separate servers that were not connected to each other. I believe, however, that they were of similar make and model. So the leading hypothesis seems to be that perhaps the SSDs were from the same manufacturing batch and shared some defect. In other words, our servers were inbred! Which makes me want to link to the song 'Second Cousin' by Flamin' Groovies. The HN hindsight consensus, to judge by the replies to https://news.ycombinator.com/item?id=32026606 https://news.ycombinator.com/item?id=32026606, is that this happens all the time, is not surprising at all, and is actually quite to be expected. Live and learn!
  - whitepoplar 4y ago
    Do you happen to know the make and model of SSD?