3 ms·

> So shard more, or get more disks, or get flash Your intuition is excellent. These are all valid techniques to alleviate disk seek limitations. They are likel

by xirium 18y ago

> So shard more, or get more disks, or get flash Your intuition is excellent. These are all valid techniques to alleviate disk seek limitations. They are likely to be successful. Unfortunately, they're expensive options. You can partially overcome disk seek limitations by performing database batch inserts. So, rather than inserting messages indiviually and updating indexes for each message, you insert 100 messages and perform less disk seeks updating the indexes for each batch. Unfortunately, it can be hard to reliably merge data into suitable batches.

wmf 18y ago
Hank's proposed solution is expensive; we should just admit that up front. But is it more expensive than being down and fighting fires all the time?
- xirium 18y ago
  The economics of downtime is a very important consideration. I read that three minutes of downtime reduces the value of a website by 1%. However, that may be a figure bandied by vendors of high-availability equipment. It is also generally acknowledged that high-availability is achieved through strict processes and sheer redundancy of hardware. Therefore, it gets exponentially more expensive to minimise downtime. So, you've got two non-linear curves, cost and benefit. If you've got an exceptional team, a great architecture and good finances then getting 99.999% uptime should be easy. Of course, this assumes that you don't have unforeseen circumstances, glaring omissions in redundancy - or exponential growth and utilisation without a revenue stream. Even with generous financing, exponential growth can be deadly. A small but sustained increment in load can be enough to cause a backlog in requests that never recovers. If you're growing fast then it can occur anywhere in your system at any time. I've been a DBA for a renderfarm with thousands of cores and this situation is stressing. We've also had this situation while working on search. Thankfully, it occurred on a smaller scale and most thankfully it occurred before launch. We had two database servers, two app servers, no redundancy, on unreliable hardware. We didn't make much headway with efficiency improvements because long tests were likely to encounter hardware failure. It was demoralising. This slowed the development and testing of a database failover feature in the database wrapper. Now this has been implemented, progress is much better. A test that previously took three hours now takes five minutes and we've had much more time to migrate to reliable hardware and introduce redundancy. We've also got a setup which can be used to test resilience to real modes of failure.
  - wmf 18y ago
    My guess is that Twitter's downtime is due to overload, not hardware failures. Thus a scalable system (even one with a high slope) reduces the overload/downtime problem to "buy more servers -- now!" instead of the "analyze bottleneck, rewrite code" loop that it sounds like Twitter is in now.

by xirium 18y ago

wmf 18y ago

Hank's proposed solution is expensive; we should just admit that up front. But is it more expensive than being down and fighting fires all the time?

xirium 18y ago
The economics of downtime is a very important consideration. I read that three minutes of downtime reduces the value of a website by 1%. However, that may be a figure bandied by vendors of high-availability equipment. It is also generally acknowledged that high-availability is achieved through strict processes and sheer redundancy of hardware. Therefore, it gets exponentially more expensive to minimise downtime. So, you've got two non-linear curves, cost and benefit. If you've got an exceptional team, a great architecture and good finances then getting 99.999% uptime should be easy. Of course, this assumes that you don't have unforeseen circumstances, glaring omissions in redundancy - or exponential growth and utilisation without a revenue stream. Even with generous financing, exponential growth can be deadly. A small but sustained increment in load can be enough to cause a backlog in requests that never recovers. If you're growing fast then it can occur anywhere in your system at any time. I've been a DBA for a renderfarm with thousands of cores and this situation is stressing. We've also had this situation while working on search. Thankfully, it occurred on a smaller scale and most thankfully it occurred before launch. We had two database servers, two app servers, no redundancy, on unreliable hardware. We didn't make much headway with efficiency improvements because long tests were likely to encounter hardware failure. It was demoralising. This slowed the development and testing of a database failover feature in the database wrapper. Now this has been implemented, progress is much better. A test that previously took three hours now takes five minutes and we've had much more time to migrate to reliable hardware and introduce redundancy. We've also got a setup which can be used to test resilience to real modes of failure.
- wmf 18y ago
  My guess is that Twitter's downtime is due to overload, not hardware failures. Thus a scalable system (even one with a high slope) reduces the overload/downtime problem to "buy more servers -- now!" instead of the "analyze bottleneck, rewrite code" loop that it sounds like Twitter is in now.