5 ms·

It's amusing that Techcrunch considers themselves and a handful of other startups "the internet". This is why we architect provider independent solutions. Keep

It's amusing that Techcrunch considers themselves and a handful of other startups "the internet". This is why we architect provider independent solutions. Keep your TTLs low. Replicate your site to another provider, and be prepared to hit the switch when your primary provider inevitably fails. All datacenters lose power, all networks get borked, and all storage systems fail. It's a fact of life.

iigs 17y ago
Yeah the snotty attitude advanced in the article is pretty offensive. If you run something on the internet and rely on it for your dinner, you don't really have an excuse for single points of failure. Do your job. It's exceptionally easy to engineer around these issues if you don't just outsource all of your thinking to a provider.
- antirez 17y ago
  without to mention how trivial is to replicate a blog.
- drusenko 17y ago
  I don't know, I sort of side with TechCrunch on this one. Yes, you can replicate a site (some easier than others, and a blog should be relatively easy), but that's still no excuse for datacenter downtime, and Rackspace has had a prolific amount of downtime. Any datacenter downtime is serious business -- at a good datacenter, it's a small-ish incident every few years. Rackspace, on the other hand, seems to have a large incident every few months, meaning it's probably best to take your servers elsewhere.
  - ghshephard 17y ago
    I used to think that Data Center Downtime was serious business, but I've come around to believing that it shouldn't be. If your data center going down is a problem, then you don't really have a very robust failover plan. If, instead, you _plan_ for the Data Center to go down, and treat a data center failure as a trivial issue, then your DR plans start to become significantly more robust. In particular, the companies that I admire are ones that routinely swap, on a routine basis, their DR and production facilities - and when a production data center goes down they don't even bother to wait - they just light up the DR center and are back in business. Most of the SaaS Financial Hosting companies (Oracle Financials) that I've talked with will provide you with that feature.
- aaronbrethorst 17y ago
  MG Siegler, the author, isn't exactly the most technically astute person around, which isn't inherently bad except that he doesn't seem to recognize his own limitations around such matters. MG has a bad tendency of breathlessly reporting and misreporting on issues that are far outside of his area of expertise. One of my favorite examples happened a month ago when MG misdiagnosed CSS files not loading on Twitter as 'BRAND NEW FEATURES THAT ARE UGLY, OMG!' (http://www.techcrunch.com/2009/11/17/twitter-just-ui-puked-on-my-timeline/ http://www.techcrunch.com/2009/11/17/twitter-just-ui-puked-o...).
- nikcub 17y ago
  We have a backup. Guess where it was. Now guess where it will be tonight.
  - deleted 17y ago
    [deleted]
- Confusion 17y ago
  It's exceptionally easy to engineer around these issues if you don't just outsource all of your thinking to a provider. Can we stop blaming internet companies for outsourcing certain responsibilities? A decent hosting provider should ensure there isn't a single point of failure in their service. You don't ask a random company to set up their own accountancy department either: there are specialized companies to which you can outsource that. They bear the responsibility for faulty reports, as Rackspace bears the responsibility for not having a decent failover setup. As a specialist of some sort, you have better things to do than to think about what is needed for a proper failover setup. Especially because you will still miss certain subtle details, because you are not an expert in handling a datacentre failure. You should hire experts to set that up for you and a hosting provider has those experts. Honest to God, what's with this attitude of 'if something on the internet fails, it's your own fault, because we are all smart enough to take care of all those contingencies'? 1: we aren't; failover handling is prone to subtle bugs. What tptacek keeps saying about security also holds for contingency handling. 2: it's not smart to spend your time bothering with intricate, but ultimately mundane, issues that can be outsourced.
endtime 17y ago
Pardon my ignorance, but how do you hit the switch? Or, rather, what is the switch? DNS is slow to propagate, and if you have some kind of server for your "switch" then isn't that still a single point of failure?
- iigs 17y ago
  Great question. In the common case, the solution is to set low TTLs (time to live attributes: they tell the DNS server how long to cache the name) on the DNS records, so they expire quickly. You can also run a pair of "front end" servers at all times, and use magic in the back-end to replicate the DB between sites. Depending on the type of site in question this can be really easy or really hard. Another (generally commercial hardware) solution is to receive the query to one DNS server. This DNS server waits to respond for a few milliseconds while it sends a copy of the query to the other DNS server. They have different answers for the question (dns-east will return the IP for www-east, likewise for dns-west and www-west). They return the answers at the same time, and the closer server "wins" by getting to the recipient faster. This is called GSLB, Global Server Load Balancing.
  - pbhjpbhj 17y ago
    But your GSLB requires your DNS server to be up, which it won't be if the datacenter blacks-out. Ditto if you're running a "pair of 'front end' servers". Shouldn't DNS records have a reserve address too then the client could decide if failure had occurred to use the alternate address?
michaelneale 17y ago
Are low TTLs effective in practice? Sounds like a good idea, but what % of users are using DNSs that just ignore it and cache things well beyond what they should?
- andrewvc 17y ago
  With a 60 second TTL I see most traffic drop off within 5 minutes, and almost all of it gone within 30 minutes. There's a few stragglers who seem to still be there even an hour after the switch, but by and large DNS TTL does work.

iigs 17y ago

Yeah the snotty attitude advanced in the article is pretty offensive. If you run something on the internet and rely on it for your dinner, you don't really have an excuse for single points of failure. Do your job. It's exceptionally easy to engineer around these issues if you don't just outsource all of your thinking to a provider.

antirez 17y ago
without to mention how trivial is to replicate a blog.
drusenko 17y ago
I don't know, I sort of side with TechCrunch on this one. Yes, you can replicate a site (some easier than others, and a blog should be relatively easy), but that's still no excuse for datacenter downtime, and Rackspace has had a prolific amount of downtime. Any datacenter downtime is serious business -- at a good datacenter, it's a small-ish incident every few years. Rackspace, on the other hand, seems to have a large incident every few months, meaning it's probably best to take your servers elsewhere.
- ghshephard 17y ago
  I used to think that Data Center Downtime was serious business, but I've come around to believing that it shouldn't be. If your data center going down is a problem, then you don't really have a very robust failover plan. If, instead, you _plan_ for the Data Center to go down, and treat a data center failure as a trivial issue, then your DR plans start to become significantly more robust. In particular, the companies that I admire are ones that routinely swap, on a routine basis, their DR and production facilities - and when a production data center goes down they don't even bother to wait - they just light up the DR center and are back in business. Most of the SaaS Financial Hosting companies (Oracle Financials) that I've talked with will provide you with that feature.
aaronbrethorst 17y ago
MG Siegler, the author, isn't exactly the most technically astute person around, which isn't inherently bad except that he doesn't seem to recognize his own limitations around such matters. MG has a bad tendency of breathlessly reporting and misreporting on issues that are far outside of his area of expertise. One of my favorite examples happened a month ago when MG misdiagnosed CSS files not loading on Twitter as 'BRAND NEW FEATURES THAT ARE UGLY, OMG!' (http://www.techcrunch.com/2009/11/17/twitter-just-ui-puked-on-my-timeline/ http://www.techcrunch.com/2009/11/17/twitter-just-ui-puked-o...).
nikcub 17y ago
We have a backup. Guess where it was. Now guess where it will be tonight.
- deleted 17y ago
  [deleted]
Confusion 17y ago
It's exceptionally easy to engineer around these issues if you don't just outsource all of your thinking to a provider. Can we stop blaming internet companies for outsourcing certain responsibilities? A decent hosting provider should ensure there isn't a single point of failure in their service. You don't ask a random company to set up their own accountancy department either: there are specialized companies to which you can outsource that. They bear the responsibility for faulty reports, as Rackspace bears the responsibility for not having a decent failover setup. As a specialist of some sort, you have better things to do than to think about what is needed for a proper failover setup. Especially because you will still miss certain subtle details, because you are not an expert in handling a datacentre failure. You should hire experts to set that up for you and a hosting provider has those experts. Honest to God, what's with this attitude of 'if something on the internet fails, it's your own fault, because we are all smart enough to take care of all those contingencies'? 1: we aren't; failover handling is prone to subtle bugs. What tptacek keeps saying about security also holds for contingency handling. 2: it's not smart to spend your time bothering with intricate, but ultimately mundane, issues that can be outsourced.

endtime 17y ago

Pardon my ignorance, but how do you hit the switch? Or, rather, what is the switch? DNS is slow to propagate, and if you have some kind of server for your "switch" then isn't that still a single point of failure?

iigs 17y ago
Great question. In the common case, the solution is to set low TTLs (time to live attributes: they tell the DNS server how long to cache the name) on the DNS records, so they expire quickly. You can also run a pair of "front end" servers at all times, and use magic in the back-end to replicate the DB between sites. Depending on the type of site in question this can be really easy or really hard. Another (generally commercial hardware) solution is to receive the query to one DNS server. This DNS server waits to respond for a few milliseconds while it sends a copy of the query to the other DNS server. They have different answers for the question (dns-east will return the IP for www-east, likewise for dns-west and www-west). They return the answers at the same time, and the closer server "wins" by getting to the recipient faster. This is called GSLB, Global Server Load Balancing.
- pbhjpbhj 17y ago
  But your GSLB requires your DNS server to be up, which it won't be if the datacenter blacks-out. Ditto if you're running a "pair of 'front end' servers". Shouldn't DNS records have a reserve address too then the client could decide if failure had occurred to use the alternate address?

michaelneale 17y ago

Are low TTLs effective in practice? Sounds like a good idea, but what % of users are using DNSs that just ignore it and cache things well beyond what they should?

andrewvc 17y ago
With a 60 second TTL I see most traffic drop off within 5 minutes, and almost all of it gone within 30 minutes. There's a few stragglers who seem to still be there even an hour after the switch, but by and large DNS TTL does work.