5 ms·

True. You'd also save a ton of less operations and engineering staff. Running anything on a single server, however, is really a non starter for anything remote

True. You'd also save a ton of less operations and engineering staff. Running anything on a single server, however, is really a non starter for anything remotely serious. What do you do if you need to do an OS update? I suppose you could just never do those, like a former employer (1000+ day uptimes...)

kijin 4y ago
Compare the cost of operating multiple servers, on one hand, with the lost revenue from having weekly or monthly maintenance windows during which you just put up a Fail Whale page. Most people overestimate the latter by a huge margin.
- IshKebab 4y ago
  That's fine if your service is really local - you can do it at night. Not really an option for a global site. Imagine if Twitter went down for a few hours every month. People are addicted to Twitter. It might be at a critical time for an entire country (e.g. the Queen dies). Even worse you can't guarantee how long the upgrade will take. You'd definitely need at least two servers. But I think you could surely just have simple master/slave replication and switch between them.
  - kijin 4y ago
    Yeah, I was just responding to the "how about OS updates" part of the parent comment, for which scheduled downtime is a reasonable option. To protect your service from unscheduled downtime though, like a failing RAID array, you would need at least two servers. Personally I wouldn't run a critical service on only one server, but two servers? Definitely doable. I actually have a service running on two servers in different DCs 700 miles apart. Zero downtime in 9 years. :)
glass3 4y ago
Twitter is not serious enough that there cannot be a timeout. With a second OS partition, the server can alternate between the working copy and a copy that is updated in a VM. For a free service, customers cannot complain even if there is a reboot every day and the service is down for a couple of minutes. Realistically, there would be a mirroring server to be prepared for hardware failures. One server can be restarted while the other is the main server.
- Sebb767 4y ago
  > With a second OS partition, the server can alternate between the working copy and a copy that is updated in a VM. For a free service, customers cannot complain even if there is a reboot every day and the service is down for a couple of minutes. Are you seriously suggesting that a service (the size of Twitter, no less) has an acceptable downtime of a few minutes a day? > Realistically, there would be a mirroring server to be prepared for hardware failures. One server can be restarted while the other is the main server. But for that mirroring, you need to replicate disk writes, databases, backups etc.. This additional load would easily bring the server to a point where a single server would no longer suffice, even an insanely spec'ed one.
  - vidarh 4y ago
    > has an acceptable downtime of a few minutes a day? We kinda know the answer to this: Twitter was struggling with harm to its reputation for a long time because of regular Fail Whales. It absolutely was a huge problem for them at the time.
    - glass3 4y ago
      There is a difference between open-ended, unscheduled downtime and a known 5 minute reboot window. If I can choose between an ad-free service that reboots at noon and a downtime-free service with ads, I would choose the ad-free service. Who needs Twitter to be a service without any downtime?
femiagbabiaka 4y ago
Twitter actually used to have downtime during certain updates IIRC. The state department asked them to skip one such maintenance in service of some Iranian protests in 2009. I highly doubt that the era of US ops in twitter has ended, so downtime is probably a nonstarter. Source: https://foreignpolicy.com/2009/06/16/state-department-intervened-with-twitter-over-iran/ https://foreignpolicy.com/2009/06/16/state-department-interv...
Sebb767 4y ago
> You'd also save a ton of less operations and engineering staff. You absolutely would not. The cost of having developers put out extremely optimized code (due to the scaling limits) and cuddle that single server to never, ever fail easily eclipses the cost of a having a multiple servers by a few orders of magnitude. EDIT: To the downvoters, I'd really love to see the calculation on how engineering time would be cheaper than buying a second server in any reasonable timeframe.
- icedchai 4y ago
  I was comparing the costs to building/operating on a single server to the real Twitter. Twitter is a massive, distributed, hugely complex system, which requires a large team to build, maintain, and operate. That costs $$$. Both for people and servers. (You are right, servers are cheap compared to employee costs though.)

kijin 4y ago

Compare the cost of operating multiple servers, on one hand, with the lost revenue from having weekly or monthly maintenance windows during which you just put up a Fail Whale page. Most people overestimate the latter by a huge margin.

IshKebab 4y ago
That's fine if your service is really local - you can do it at night. Not really an option for a global site. Imagine if Twitter went down for a few hours every month. People are addicted to Twitter. It might be at a critical time for an entire country (e.g. the Queen dies). Even worse you can't guarantee how long the upgrade will take. You'd definitely need at least two servers. But I think you could surely just have simple master/slave replication and switch between them.
- kijin 4y ago
  Yeah, I was just responding to the "how about OS updates" part of the parent comment, for which scheduled downtime is a reasonable option. To protect your service from unscheduled downtime though, like a failing RAID array, you would need at least two servers. Personally I wouldn't run a critical service on only one server, but two servers? Definitely doable. I actually have a service running on two servers in different DCs 700 miles apart. Zero downtime in 9 years. :)

glass3 4y ago

Twitter is not serious enough that there cannot be a timeout. With a second OS partition, the server can alternate between the working copy and a copy that is updated in a VM. For a free service, customers cannot complain even if there is a reboot every day and the service is down for a couple of minutes. Realistically, there would be a mirroring server to be prepared for hardware failures. One server can be restarted while the other is the main server.

Sebb767 4y ago
> With a second OS partition, the server can alternate between the working copy and a copy that is updated in a VM. For a free service, customers cannot complain even if there is a reboot every day and the service is down for a couple of minutes. Are you seriously suggesting that a service (the size of Twitter, no less) has an acceptable downtime of a few minutes a day? > Realistically, there would be a mirroring server to be prepared for hardware failures. One server can be restarted while the other is the main server. But for that mirroring, you need to replicate disk writes, databases, backups etc.. This additional load would easily bring the server to a point where a single server would no longer suffice, even an insanely spec'ed one.
- vidarh 4y ago
  > has an acceptable downtime of a few minutes a day? We kinda know the answer to this: Twitter was struggling with harm to its reputation for a long time because of regular Fail Whales. It absolutely was a huge problem for them at the time.
  - glass3 4y ago
    There is a difference between open-ended, unscheduled downtime and a known 5 minute reboot window. If I can choose between an ad-free service that reboots at noon and a downtime-free service with ads, I would choose the ad-free service. Who needs Twitter to be a service without any downtime?

femiagbabiaka 4y ago

Twitter actually used to have downtime during certain updates IIRC. The state department asked them to skip one such maintenance in service of some Iranian protests in 2009. I highly doubt that the era of US ops in twitter has ended, so downtime is probably a nonstarter. Source: https://foreignpolicy.com/2009/06/16/state-department-intervened-with-twitter-over-iran/ https://foreignpolicy.com/2009/06/16/state-department-interv...

Sebb767 4y ago

> You'd also save a ton of less operations and engineering staff. You absolutely would not. The cost of having developers put out extremely optimized code (due to the scaling limits) and cuddle that single server to never, ever fail easily eclipses the cost of a having a multiple servers by a few orders of magnitude. EDIT: To the downvoters, I'd really love to see the calculation on how engineering time would be cheaper than buying a second server in any reasonable timeframe.

icedchai 4y ago
I was comparing the costs to building/operating on a single server to the real Twitter. Twitter is a massive, distributed, hugely complex system, which requires a large team to build, maintain, and operate. That costs $$$. Both for people and servers. (You are right, servers are cheap compared to employee costs though.)