4 ms·

It seems truly bizarre to me that in 2012, there is a major operation that is still trying to vertically scale such a trivially shardable DB. Scaling a static

It seems truly bizarre to me that in 2012, there is a major operation that is still trying to vertically scale such a trivially shardable DB. Scaling a static Q/A site with a few widgets that require user info/counters should be table stakes.

jacques_chester 15y ago
No matter how you slice it, sharding introduces substantial complication. You need to modify your application to shard or to use an auto-sharding store, and then maintain that overhead forever. The thing is, they don't need to do that. Moore's Law, here represented as SSDs and RAM, is keeping ahead of their traffic growth.
latch 15y ago
They've proudly blogged about it a few times http://blog.stackoverflow.com/category/server/ http://blog.stackoverflow.com/category/server/ I think they've successfully shown how far you can take it. Nevertheless, I agree with you that scaling this horizontally makes a lot of sense.
- wavetossed 15y ago
  They ARE scaling horizontally. Remember where they mentioned that they are putting StackOverflow, one of several sites that they run, on its own box? That is horizontal scaling. They now have two shards, one with SO and one with all the other sites.
staunch 15y ago
The situation is the opposite of what you seem to think. It's far easier and cheaper to scale up than out than it has ever been. Just 5-6 years ago it would have been nearly unthinkable to have a half a TB of memory, 16 CPU cores, and a TB of SSD for a few thousand dollars. Most people's traffic/data hasn't grown nearly as fast as Moore's law. Scaling up makes more sense than ever. It may not be hip but it's the right choice for 99% of cases.
- marshray 15y ago
  Just be ready for the day when you max out your IO bandwidth, or some other hard-to-forsee bottleneck.
  - jacques_chester 15y ago
    There is always a bottleneck. Moving it from "disk I/O" to "cache coherency network traffic" doesn't magically make that go away. Bottlenecks are not magically confined to only affect scale-up designs. And if you look at the design of scale-out systems, they usually surrender some desirable quality such as easy joins, isolation, atomicity etc.
  - InclinedPlane 15y ago
    It depends on how fast you're growing, since technology keeps chugging along too. Case in point, here's a 16TB SSD with 560 MB/s peformance: http://www.engadget.com/2012/01/09/ocz-goes-ssd-crazy-at-ces-leaves-no-port-unplugged/ http://www.engadget.com/2012/01/09/ocz-goes-ssd-crazy-at-ces...
    - learc83 15y ago
      You might be able to keep cramming in more RAM until memristor storage makes RAM obsolete.
      - jacques_chester 15y ago
        I know it's a cliche, but memristors are going to cause an earthquake in the algo-economics landscape.
        InclinedPlane 15y ago
        Potentially memristors could lead to smartphones and tablets with the power of a server rack of today crammed into them, but also with improved battery life. And that's one of the least revolutionary implications of memristor technology.

jacques_chester 15y ago

No matter how you slice it, sharding introduces substantial complication. You need to modify your application to shard or to use an auto-sharding store, and then maintain that overhead forever. The thing is, they don't need to do that. Moore's Law, here represented as SSDs and RAM, is keeping ahead of their traffic growth.

latch 15y ago

They've proudly blogged about it a few times http://blog.stackoverflow.com/category/server/ http://blog.stackoverflow.com/category/server/ I think they've successfully shown how far you can take it. Nevertheless, I agree with you that scaling this horizontally makes a lot of sense.

wavetossed 15y ago
They ARE scaling horizontally. Remember where they mentioned that they are putting StackOverflow, one of several sites that they run, on its own box? That is horizontal scaling. They now have two shards, one with SO and one with all the other sites.

staunch 15y ago

The situation is the opposite of what you seem to think. It's far easier and cheaper to scale up than out than it has ever been. Just 5-6 years ago it would have been nearly unthinkable to have a half a TB of memory, 16 CPU cores, and a TB of SSD for a few thousand dollars. Most people's traffic/data hasn't grown nearly as fast as Moore's law. Scaling up makes more sense than ever. It may not be hip but it's the right choice for 99% of cases.

marshray 15y ago
Just be ready for the day when you max out your IO bandwidth, or some other hard-to-forsee bottleneck.
- jacques_chester 15y ago
  There is always a bottleneck. Moving it from "disk I/O" to "cache coherency network traffic" doesn't magically make that go away. Bottlenecks are not magically confined to only affect scale-up designs. And if you look at the design of scale-out systems, they usually surrender some desirable quality such as easy joins, isolation, atomicity etc.
- InclinedPlane 15y ago
  It depends on how fast you're growing, since technology keeps chugging along too. Case in point, here's a 16TB SSD with 560 MB/s peformance: http://www.engadget.com/2012/01/09/ocz-goes-ssd-crazy-at-ces-leaves-no-port-unplugged/ http://www.engadget.com/2012/01/09/ocz-goes-ssd-crazy-at-ces...
  - learc83 15y ago
    You might be able to keep cramming in more RAM until memristor storage makes RAM obsolete.
    - jacques_chester 15y ago
      I know it's a cliche, but memristors are going to cause an earthquake in the algo-economics landscape.
      - InclinedPlane 15y ago
        Potentially memristors could lead to smartphones and tablets with the power of a server rack of today crammed into them, but also with improved battery life. And that's one of the least revolutionary implications of memristor technology.