4 ms·

The customer.io story seems like a great example of why NOT to use budget providers like OVH and Hetzner for mission-critical applications. You get what you pa

by ch4ch4 13y ago

The customer.io story seems like a great example of why NOT to use budget providers like OVH and Hetzner for mission-critical applications. You get what you pay for.

jamescun 13y ago
Not so much an example of why not to use budget providers, more an example of why you should build highly available infrastructure. I don't believe there is any provider, "budget" or not, that guarantees a servers reliability 100% of the time.
- ch4ch4 13y ago
  I was alluding more to the customer support aspect of it. If a technician spends one hour troubleshooting your network problems, then they've already lost their profit for the month.
  - vacri 13y ago
    This is one thing where I find AWS shines. I'm on the lowest level of paid support, and I've had nothing but excellent service from good technicians who will try to actually replicate your problem, then contact other teams if they fail or there's follow up. Out of a dozen or so tickets, I've only had one where the response wasn't genuinely useful, and that was for an issue that may have been due to internet weather anyway. Support is one of things that you can get along without... until you need it. Then you really, really wish you had it.
bluedino 13y ago
I've spent nearly 2 hours with sites down at Rackspace, which is certainly not a budget provider, because our hardware firewall crapped out and they "couldn't find" our hardware in the datacenter. Entirely different problem from a provider that loses one internet connection and their other links can't keep up with traffic, but you can still have major problems even if you're spending thousands of dollars a month compared to hundreds.
sybhn 13y ago
Echoing some of the other comments... the place i worked at was one of the 5 biggest Rackspace customer, and that didn't stop them from regularly cutting traffic or bringing down servers for hours in some case. You might get better uptime overall from reputable providers, but ultimately it's all about distributed application/service architecture.
- jrallison 13y ago
  Absolutely. Removing SPOFs and moving to a service oriented architecture has been a major focus for us over the last few months.
chiph 13y ago
The high-dollar ones aren't necessarily better. At a previous job we hosted with {HAL} out of Atlanta. A NOC operator there saw/heard/smelled something that indicated to him that he should hit the Big Red Switch. So he did. This removed power to every machine in that part of the DC. After management confirmed that there was no life-threatening emergency, they started bringing everything back up. Only to have machines start going down again 20 minutes later, as their local UPSes ran out of juice. Someone had to walk around to every cage and recycle them all manually.
AznHisoka 13y ago
Other than the bridge problem with OVH a month back, OVH has been fairly stable network-wise for me for the past year or so. And I rely on the network a ton since I run distributed crawlers.
cstrat 13y ago
Yeah they had a terrible experience! The idea of using DC hosting providers is because the uptime, environment and usually strong network connectivity. Given the poor run they had I think the developer would have had a better uptime if he had hosted this at home on a server.

by ch4ch4 13y ago

The customer.io story seems like a great example of why NOT to use budget providers like OVH and Hetzner for mission-critical applications. You get what you pay for.

jamescun 13y ago

Not so much an example of why not to use budget providers, more an example of why you should build highly available infrastructure. I don't believe there is any provider, "budget" or not, that guarantees a servers reliability 100% of the time.

ch4ch4 13y ago
I was alluding more to the customer support aspect of it. If a technician spends one hour troubleshooting your network problems, then they've already lost their profit for the month.
- vacri 13y ago
  This is one thing where I find AWS shines. I'm on the lowest level of paid support, and I've had nothing but excellent service from good technicians who will try to actually replicate your problem, then contact other teams if they fail or there's follow up. Out of a dozen or so tickets, I've only had one where the response wasn't genuinely useful, and that was for an issue that may have been due to internet weather anyway. Support is one of things that you can get along without... until you need it. Then you really, really wish you had it.

bluedino 13y ago

I've spent nearly 2 hours with sites down at Rackspace, which is certainly not a budget provider, because our hardware firewall crapped out and they "couldn't find" our hardware in the datacenter. Entirely different problem from a provider that loses one internet connection and their other links can't keep up with traffic, but you can still have major problems even if you're spending thousands of dollars a month compared to hundreds.

sybhn 13y ago

Echoing some of the other comments... the place i worked at was one of the 5 biggest Rackspace customer, and that didn't stop them from regularly cutting traffic or bringing down servers for hours in some case. You might get better uptime overall from reputable providers, but ultimately it's all about distributed application/service architecture.

jrallison 13y ago
Absolutely. Removing SPOFs and moving to a service oriented architecture has been a major focus for us over the last few months.

chiph 13y ago

The high-dollar ones aren't necessarily better. At a previous job we hosted with {HAL} out of Atlanta. A NOC operator there saw/heard/smelled something that indicated to him that he should hit the Big Red Switch. So he did. This removed power to every machine in that part of the DC. After management confirmed that there was no life-threatening emergency, they started bringing everything back up. Only to have machines start going down again 20 minutes later, as their local UPSes ran out of juice. Someone had to walk around to every cage and recycle them all manually.

AznHisoka 13y ago

Other than the bridge problem with OVH a month back, OVH has been fairly stable network-wise for me for the past year or so. And I rely on the network a ton since I run distributed crawlers.

cstrat 13y ago

Yeah they had a terrible experience! The idea of using DC hosting providers is because the uptime, environment and usually strong network connectivity. Given the poor run they had I think the developer would have had a better uptime if he had hosted this at home on a server.