4 ms·

Wow, I'm surprised TechCrunch is included in this, you would think they have a bit more of a redundant setup.

pavs 17y ago
Why would any website which gets few million hits a month not have a redundant system is beyond me. If you consider the amount of money they make the cost of a redundant setup is really nothing.
- CoachRufus87 17y ago
  When you consider the amount of money their losing right now...
- icey 17y ago
  I was under the impression that people used Wordpress VIP hosting so that they wouldn't have to worry about using a redundant system (meaning that Wordpress VIP should take care of it).
  - andrewpbrett 17y ago
    We were under that impression as well.
  - benofsky 17y ago
    yeah, that's what I thought too.
  - brk 17y ago
    You can't outsource your liability. At a certain point (a point which TC passed quite a long time ago) you simply have to own your own infrastructure. That does not make it bullet proof, but it puts it under your own control and you can respond accordingly and better control the outcome of any outage. I know that in Web2.0 everybody wants to think that they can outsource all the messy bits of running a website to the lowest bidder, but that approach has never worked. When you get REALLY big (like facebook and google big) you move from a self-managed server in a carrier-neutral datacenter to building and maintaining your own datacenters. For some reason people think it's perfectly normal that facebook builds its own datacenter, and yet also perfectly normal that a large blog (Techcrunch) take no ownership of the very equipment that makes their site go.
    - mrkurt 17y ago
      Really? You'd be hard pressed to own all of your own infrastructure if you're Techcrunch, especially for anything approaching the cost of the Wordpress VIP thing (or even Rackspace Cloud for that matter). You know that lovely self managed server in the carrier neutral datacenter? It's still going to go down when power in one of the datacenter's grids fail and their backup generators don't come on properly. The network's still going to fail when someone screws up a config and routes your traffic improperly. Once you own it all, you're still banking on the fact that you can do a better job than some company that specializes in what you're trying to do. You'd be quite a bit better off sticking with some "cloud" provider and putting up a static failover cache on EC2 or something. Even then you're at the mercy of your DNS provider (unless you're going to run your own anycasted DNS) and any misbehaving DNS caches.
    - icey 17y ago
      Why would TC want to spend the money on a datacenter, staffing, bandwidth, etc for their blog when there are services out there that will do all of that stuff for them at a fraction of the cost? Private datacenters have outages as well, they wouldn't have been immune from outages if they were hosted internally. If anything, they are somewhat shielded from liability because everyone knew Wordpress was down. It makes sense for Facebook and Google to build and own their own data centers because they deal with volume on an hourly basis that is magnitudes larger than TC probably sees in a week (or possibly even a month). Techcrunch is a blog. That's it. That doesn't require owning your own datacenter. It requires a hosting company that can keep the lights on. Wordpress is about as good as you're going to get in that situation.
      - brk 17y ago
        Why would TC want to spend the money on a datacenter, staffing, bandwidth I don't think they should/would do that. They are nowhere near large enough in traffic volume to justify that. But they SHOULD have more direct control over the servers that run their blog so that WHEN something happens (again, I never claimed that maintaining your own server made you immune) they can directly control the outcome.
    - 17y ago
dangrossman 17y ago
It's not really critical to anyone for some blog to have perfect uptime. TechCrunch's advertisers don't pay on a CPM basis except possibly a few of the traditional banner slots.
Sidnicious 17y ago
Isn't that why they're giving the WordPress folks all that money?
- Aleran 17y ago
  As far as I can tell Wordpress VIP is free. You just have to be accepted.
jbyers 17y ago
Wordpress serves traffic live out of three datacenters. That is the redundant system. And I can say from experience that it's really, really hard to do that. Doing so well is probably the reason they've had such a good record for the last four years. An unexpected condition on the network that keeps the datacenters from talking to each other (I'm speculating based on their report) might have cascaded into a wide-scale outage. It's also worth considering how hard it is to cold-start a large site, if the outage had any impact on their MySQL servers or caches. We can't cold-start Wikispaces anymore (nights and weekends included) and we're only doing ~10M unique visitors a month -- WP has something like 300+ million.
- moe 17y ago
  it's really, really hard to do that ...if you're doing it for the first time. It's also worth considering how hard it is to cold-start a large site ...if your architecture is flawed.

pavs 17y ago

Why would any website which gets few million hits a month not have a redundant system is beyond me. If you consider the amount of money they make the cost of a redundant setup is really nothing.

CoachRufus87 17y ago
When you consider the amount of money their losing right now...
icey 17y ago
I was under the impression that people used Wordpress VIP hosting so that they wouldn't have to worry about using a redundant system (meaning that Wordpress VIP should take care of it).
- andrewpbrett 17y ago
  We were under that impression as well.
- benofsky 17y ago
  yeah, that's what I thought too.
- brk 17y ago
  You can't outsource your liability. At a certain point (a point which TC passed quite a long time ago) you simply have to own your own infrastructure. That does not make it bullet proof, but it puts it under your own control and you can respond accordingly and better control the outcome of any outage. I know that in Web2.0 everybody wants to think that they can outsource all the messy bits of running a website to the lowest bidder, but that approach has never worked. When you get REALLY big (like facebook and google big) you move from a self-managed server in a carrier-neutral datacenter to building and maintaining your own datacenters. For some reason people think it's perfectly normal that facebook builds its own datacenter, and yet also perfectly normal that a large blog (Techcrunch) take no ownership of the very equipment that makes their site go.
  - mrkurt 17y ago
    Really? You'd be hard pressed to own all of your own infrastructure if you're Techcrunch, especially for anything approaching the cost of the Wordpress VIP thing (or even Rackspace Cloud for that matter). You know that lovely self managed server in the carrier neutral datacenter? It's still going to go down when power in one of the datacenter's grids fail and their backup generators don't come on properly. The network's still going to fail when someone screws up a config and routes your traffic improperly. Once you own it all, you're still banking on the fact that you can do a better job than some company that specializes in what you're trying to do. You'd be quite a bit better off sticking with some "cloud" provider and putting up a static failover cache on EC2 or something. Even then you're at the mercy of your DNS provider (unless you're going to run your own anycasted DNS) and any misbehaving DNS caches.
  - icey 17y ago
    Why would TC want to spend the money on a datacenter, staffing, bandwidth, etc for their blog when there are services out there that will do all of that stuff for them at a fraction of the cost? Private datacenters have outages as well, they wouldn't have been immune from outages if they were hosted internally. If anything, they are somewhat shielded from liability because everyone knew Wordpress was down. It makes sense for Facebook and Google to build and own their own data centers because they deal with volume on an hourly basis that is magnitudes larger than TC probably sees in a week (or possibly even a month). Techcrunch is a blog. That's it. That doesn't require owning your own datacenter. It requires a hosting company that can keep the lights on. Wordpress is about as good as you're going to get in that situation.
    - brk 17y ago
      Why would TC want to spend the money on a datacenter, staffing, bandwidth I don't think they should/would do that. They are nowhere near large enough in traffic volume to justify that. But they SHOULD have more direct control over the servers that run their blog so that WHEN something happens (again, I never claimed that maintaining your own server made you immune) they can directly control the outcome.
  - 17y ago

dangrossman 17y ago

It's not really critical to anyone for some blog to have perfect uptime. TechCrunch's advertisers don't pay on a CPM basis except possibly a few of the traditional banner slots.

Sidnicious 17y ago

Isn't that why they're giving the WordPress folks all that money?

Aleran 17y ago
As far as I can tell Wordpress VIP is free. You just have to be accepted.

jbyers 17y ago

Wordpress serves traffic live out of three datacenters. That is the redundant system. And I can say from experience that it's really, really hard to do that. Doing so well is probably the reason they've had such a good record for the last four years. An unexpected condition on the network that keeps the datacenters from talking to each other (I'm speculating based on their report) might have cascaded into a wide-scale outage. It's also worth considering how hard it is to cold-start a large site, if the outage had any impact on their MySQL servers or caches. We can't cold-start Wikispaces anymore (nights and weekends included) and we're only doing ~10M unique visitors a month -- WP has something like 300+ million.

moe 17y ago
it's really, really hard to do that ...if you're doing it for the first time. It's also worth considering how hard it is to cold-start a large site ...if your architecture is flawed.