3 ms·

Servers can and do scale effectively linearly depending on the workload. The broader point stands though. The more servers you have, the more money a 1% cpu op

Servers can and do scale effectively linearly depending on the workload. The broader point stands though. The more servers you have, the more money a 1% cpu optimization saves you in raw dollar amounts. At a certain scale it always makes sense to optimize your code.

hinkley 6y ago
The moment the servers need any consensus at all, you've opened the door to Amdahl's law. That could be sharing caches, running Raft, or talking to a transactional store. Even if it's just to keep my UI in sync between the desktop and mobile site, you have costs. But since so many sites have social aspects now, you need to be able to communicate between accounts, which means interprocess communication. The Amdahl's law fraction is at least logarithmic to the number of users, but more likely to be proportional to the 'surface area' - sqrt(n). Every new server gets you a little bit less oomph than the one before.
- hinkley 6y ago
  And remember, even if we aren't sending more and more data between our users, we're collecting more and more telemetry data from our servers all the time. Coalescing logs and gathering stats requires funneling data from everywhere down to one machine (the one that built the graph)
- missblit 6y ago
  I actually wasn't thinking of social sites when I said "certain workloads". It's true that every server extra has a bit more overhead because of stuff like caching / load balancing / whatnot, but the amount extra can be tiny even at very large scale. For certain workloads that are parellelizable. (It'd be fun to talk in details from personal experience instead of in general terms, but you know those pesky NDAs...)
- CyberDildonics 6y ago
  There is basically no overlap with Amdahl's law here. Amdahl's law just says that there are diminishing returns on the whole when increasing parallelization in a program with significant serial parts. It is a fairly obvious observation, but people seem to distort it and apply it to general scalability. First, everything needs to be synchronized but that doesn't mean synchronization is expensive or blocks computation from happening. Lots of synchronization isn't even dealt with explicitly by users, it happens in queues in the kernel, in networking hardware, or even in the memory controller when talking about reading from memory. The closest thing is the database becoming a bottleneck, but that is not because of Amdahl's law or even synchronization, it just about reads, and those are extremely parallel, so it likely comes down to total CPU power when it is a problem. That's why query caching can be so effective - it isn't about synchronization and definitely not about serial parts of a program, it is just about usage. For connected social media sites, not everything has to be done on a page request. Anything done ahead of time is also done in parallel.
fefe23 6y ago
Unfortunately the probability of failure also scales up. Let's say you bought good hardware and the probability it will fail is x. Let's say you buy a hundred of those. The probability that one of them fails is now 100x. You just increased the probability of failure a hundredfold. You will need a load balancer who can detect failure and not forward to that machine anymore. The load balancer can fail, too. Now it's 101 fold. You also need a separate database if you share sate. 102 fold. Make it redundant. 103 fold. Add router / switches. 105 fold. Make those redundant. 107 fold. I have never used more than one crappy PC for any project of mine, and some of those have had/have pretty substantial load on them. When I say one server, I mean one PC but with two power supplies and RAID storage. I'm not trying to tempt fate here. But I have no devops team, no admins juggling failing machines, I don't have racks. I have one slot in a rack somewhere. The machines have been running for years. Every few years, as I approach MTBF of the hardware, I move to a new server. That entails a small downtime. Result: People use my servers in their "is the internet up" scripts. Because my failure rate is a fraction of what distributed systems have. In fact, practically all of my downtimes have been software bugs in my code, or me fat-fingering input somewhere. I can't even remember the last time the hardware failed. Not even sure I ever had an actual hardware failure that caused downtime. I once had a raid hdd throwing errors and it had to be replaced. I think that's it.

hinkley 6y ago

The moment the servers need any consensus at all, you've opened the door to Amdahl's law. That could be sharing caches, running Raft, or talking to a transactional store. Even if it's just to keep my UI in sync between the desktop and mobile site, you have costs. But since so many sites have social aspects now, you need to be able to communicate between accounts, which means interprocess communication. The Amdahl's law fraction is at least logarithmic to the number of users, but more likely to be proportional to the 'surface area' - sqrt(n). Every new server gets you a little bit less oomph than the one before.

hinkley 6y ago
And remember, even if we aren't sending more and more data between our users, we're collecting more and more telemetry data from our servers all the time. Coalescing logs and gathering stats requires funneling data from everywhere down to one machine (the one that built the graph)
missblit 6y ago
I actually wasn't thinking of social sites when I said "certain workloads". It's true that every server extra has a bit more overhead because of stuff like caching / load balancing / whatnot, but the amount extra can be tiny even at very large scale. For certain workloads that are parellelizable. (It'd be fun to talk in details from personal experience instead of in general terms, but you know those pesky NDAs...)
CyberDildonics 6y ago
There is basically no overlap with Amdahl's law here. Amdahl's law just says that there are diminishing returns on the whole when increasing parallelization in a program with significant serial parts. It is a fairly obvious observation, but people seem to distort it and apply it to general scalability. First, everything needs to be synchronized but that doesn't mean synchronization is expensive or blocks computation from happening. Lots of synchronization isn't even dealt with explicitly by users, it happens in queues in the kernel, in networking hardware, or even in the memory controller when talking about reading from memory. The closest thing is the database becoming a bottleneck, but that is not because of Amdahl's law or even synchronization, it just about reads, and those are extremely parallel, so it likely comes down to total CPU power when it is a problem. That's why query caching can be so effective - it isn't about synchronization and definitely not about serial parts of a program, it is just about usage. For connected social media sites, not everything has to be done on a page request. Anything done ahead of time is also done in parallel.

fefe23 6y ago

Unfortunately the probability of failure also scales up. Let's say you bought good hardware and the probability it will fail is x. Let's say you buy a hundred of those. The probability that one of them fails is now 100x. You just increased the probability of failure a hundredfold. You will need a load balancer who can detect failure and not forward to that machine anymore. The load balancer can fail, too. Now it's 101 fold. You also need a separate database if you share sate. 102 fold. Make it redundant. 103 fold. Add router / switches. 105 fold. Make those redundant. 107 fold. I have never used more than one crappy PC for any project of mine, and some of those have had/have pretty substantial load on them. When I say one server, I mean one PC but with two power supplies and RAID storage. I'm not trying to tempt fate here. But I have no devops team, no admins juggling failing machines, I don't have racks. I have one slot in a rack somewhere. The machines have been running for years. Every few years, as I approach MTBF of the hardware, I move to a new server. That entails a small downtime. Result: People use my servers in their "is the internet up" scripts. Because my failure rate is a fraction of what distributed systems have. In fact, practically all of my downtimes have been software bugs in my code, or me fat-fingering input somewhere. I can't even remember the last time the hardware failed. Not even sure I ever had an actual hardware failure that caused downtime. I once had a raid hdd throwing errors and it had to be replaced. I think that's it.