7 ms·

This is precisely the point made by McSherry, Isard and Murray in their lovely paper, "Scalability! But at what COST?" (Usenix HotOS '15). They demonstrate how

This is precisely the point made by McSherry, Isard and Murray in their lovely paper, "Scalability! But at what COST?" (Usenix HotOS '15). They demonstrate how much performance headroom there is in modern CPU and memory, and show how simple cache-sensitive batch algorithms running on a single core can outperform hundreds of cores running distributed map-reduce style jobs. https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf https://www.usenix.org/system/files/conference/hotos15/hotos...

wiradikusuma 7y ago
But nobody in BigCo(tm) would do that, because everyone (including the executives) involved want to have Hadoop/BigData(tm) in their resume. /sarcasm
- tlb 7y ago
  No doubt there are people who do it for cynical reasons. But at least some people do it sincerely thinking it’s the right choice. It’d be more interesting to talk about them, and how they came to make the wrong decision for what they thought were the right reasona.
  - asdfasgasdgasdg 7y ago
    I think it's a bit presumptuous to say that the decision is "wrong". That paper demonstrated that a single server can outperform a small server farm on a toy problem. Nobody, not even google, solves pagerank in production as a batch job. Real problems are often more complex.
    - tluyben2 7y ago
      For many workloads, companies have tech leads and CTOs absolutely overarchitecting their stack. If you can run your entire system off of 4 loadbalanced $200/mo servers, why am I seeing hadoop/kafka/kubernetes/etc running on $10k+/mo premoney, so paid with investor money? Sure there are a lot of cases where this is fine, but I would say (from real life) that there are far more cases where this was a pretty poor choice. Usually proven to be even poorer when the tech lead leaves and no-one seemingly having a clue how it all works together, despite the whole docker/kubernetes/ci stories peddled to management. That is usually when I get asked to take a look. In my experience these problems are definitely not always toy problems while some of them can, easily, be ran on 1 server for the expected lifespan of the company because the company will never get that much users/clients/data even though immensely profitable. Not everyone (almost no one) is a FAANG and I find it offensive to use the company money (which can be investor money) to realise the wet dreams of the cto while the case for that architecture and the cost of it, makes 0 sense to the business and bottomline. Obviously in life, things are nuanced and differ case by case but nah, real world problems are, generally, not more complex. They are more complex in some (rare) cases, like the one you named. But most companies are not doing anything like that, but they do have their cto’s gearing up for that incredibly unlikely future.
  - FpUser 7y ago
    Talked to many people lit you mentioned. Large percentage repeats these points: 1) We always wait for database, so performance of our code does not matter. This comes from places where ironically the whole database can fit into RAM. 10+ gbps connectivity - well it is almost commodity for business so the latency is not much of a bottleneck. Fast IO to store data - well imagine array of Optane drives. Not very cheap but really peanuts for normal business. All this means that purposely built data server which is a heart of many type of business residing on decent computer, can be blindingly fast and most of the businesses will never ever outgrow it. I've written quite a few of such servers serving loads of businesses in NA so hopefully my experience is not irrelevant. 2) Scripting languages are so convenient to use and save so much time and this is what matters as the salary is a main expense. - Personally I use those mostly for management/deployment/etc.etc type of scripts and maybe to quickly test some small ideas. I clearly see their benefits there. Anything that resembles a product that would actually run business. Sorry but experienced developer can implement those just as fast as in any scripting language and it will save a ton on maintenance. 3) And finally do not do premature optimization mantra . - In my opinion switching from scripting language to something like C++ has nothing to do with premature optimization. Developer can be very productive with compiled languages as they also have megatons of libraries for any imaginable task.
    - kortilla 7y ago
      > Sorry but experienced developer can implement those just as fast as in any scripting language and it will save a ton on maintenance. This is just a no true Scotsman argument. For 10 years I’ve watched python/ruby shops drastically outpace projects in C++/Java shops. What you’re failing to realize is how trivial most apps are and how fast fully functional back ends can be created with frameworks in those languages (django/rails). Your bit about maintenance is also bullshit. I’ve spent entire days peeling apart complex C++ code bases to make a small change to some core abstraction. Ability to maintain code is entirely up to how well organized and documented it is. It has nothing to do with “scripting”.
      - AmericanChopper 7y ago
        > What you’re failing to realize is how trivial most apps are and how fast fully functional back ends can be created with frameworks in those languages A vast majority of web dev is incredibly simple. People tend to easily convince themselves that their app is very special and requires a lot of complex solutions to make it work. There’s a lot of incentives that lead people to that conclusion, but most of the time following the golden path laid out for you by an existing framework is going to produce a better product with much less effort.
  - 0x445442 7y ago
    > It’d be more interesting to talk about them, and how they came to make the wrong decision for what they thought were the right reason My guess, essentially holding their finger to the wind. Most orgs can't afford to do real engineering.
- Sevii 7y ago
  Its more like 'Here at BigCo, we do x because it scales. We have learned it from various incidents that we won't tell you about, but trust us we have to do it this way."
  - buzzkillington 7y ago
    I've worked at BigCo. It's resume padding with the fear of looking like an idiot for not knowing about the new tech. We were going to go all in on Snowflake with everyone on the team being for it. I sat down, read the original whitepaper, wrote a simulation of what the costs would look like with the current read/write statistics and tested a small batch of data on it to double check. Turns out we would have paid between x500 to x10,000 what we were currently paying for using Postgres. I moved on three months ago, and last I heard they were trying to use Snowflake again.
    - pnako 7y ago
      I'm starting to wonder if there is market to do "technology laundering", use things like PostgreSQL, SQLite, standard Unix tools, put it under some cloud marketing and charge a x10 premium. Or perhaps not only there is market, but that's more or less what everyone is already doing.
      - IanCal 7y ago
        If you can set it up so it solves my problems, yes. I could manage my own server and fine tune everything, or I can throw it on snowflake. Snowflake means I've spent almost no time managing anything and it was costing less than an aws box running postgres but absolutely blew it out of the water performance wise. Depends on your workload but it's been perfect for one of my use cases. If they were just using postgres under the hood and I got the same en experience - fine. Big things are time sharing resources and management/updates/etc. Can you charge me 10x the underlying cost but let me pay for two minutes on a wildly powerful machine? Great, that's a net win for me.
        FpUser 7y ago
jpz 7y ago
Big data is about I/O not CPU I’m a C++ veteran btw and understand the point but big data is about how to process petabytes of I/O not how to consume CPU.
- sriram_malhar 7y ago
  Yes, you are right. That said, the COST paper's presumption is that not every "big data" problem is big (petabyte size), that most probably fit in a single system's disk & memory.
- emsy 7y ago
  Correct me if I’m wrong, but isn’t the point of writing cache coherent code the fact that memory I/O is the bottleneck these days?
- growse 7y ago
  > Big data is about I/O not CPU > I’m a C++ veteran btw and understand the point but big data is about how to process petabytes of I/O not how to consume CPU. I'm not sure this is cut-and-dry. Back when I was working on Spark workloads, there was some interesting research being done on where the bottlenecks were for jobs. I think it turned out for a lot of jobs, infinite disk / network io didn't give as much of an improvement as you'd expect. https://databricks.com/session/making-sense-of-spark-performance https://databricks.com/session/making-sense-of-spark-perform...
- philipov 7y ago
  Not for everyone. In finance, a query might only use gigabytes or terabytes of data, but need to do a ton of simulation and calculation on top of that. Optimization of e.g. trading algorithms is entirely CPU-bound.
  - jpz 7y ago
    Yes, I'm aware. I've written and maintained several of these systems in exotic derivatives space. Scale-out of CPU was required for these. These are not big data problems, however, by definition.
- 7y ago
lonelappde 7y ago
This was part of the excitement/promise of Go lang, to eliminate the need for small-scale mapreduce jobs, with library code that launched a year or two before the cost paper was published.

wiradikusuma 7y ago

But nobody in BigCo(tm) would do that, because everyone (including the executives) involved want to have Hadoop/BigData(tm) in their resume. /sarcasm

tlb 7y ago
No doubt there are people who do it for cynical reasons. But at least some people do it sincerely thinking it’s the right choice. It’d be more interesting to talk about them, and how they came to make the wrong decision for what they thought were the right reasona.
- asdfasgasdgasdg 7y ago
  I think it's a bit presumptuous to say that the decision is "wrong". That paper demonstrated that a single server can outperform a small server farm on a toy problem. Nobody, not even google, solves pagerank in production as a batch job. Real problems are often more complex.
  - tluyben2 7y ago
    For many workloads, companies have tech leads and CTOs absolutely overarchitecting their stack. If you can run your entire system off of 4 loadbalanced $200/mo servers, why am I seeing hadoop/kafka/kubernetes/etc running on $10k+/mo premoney, so paid with investor money? Sure there are a lot of cases where this is fine, but I would say (from real life) that there are far more cases where this was a pretty poor choice. Usually proven to be even poorer when the tech lead leaves and no-one seemingly having a clue how it all works together, despite the whole docker/kubernetes/ci stories peddled to management. That is usually when I get asked to take a look. In my experience these problems are definitely not always toy problems while some of them can, easily, be ran on 1 server for the expected lifespan of the company because the company will never get that much users/clients/data even though immensely profitable. Not everyone (almost no one) is a FAANG and I find it offensive to use the company money (which can be investor money) to realise the wet dreams of the cto while the case for that architecture and the cost of it, makes 0 sense to the business and bottomline. Obviously in life, things are nuanced and differ case by case but nah, real world problems are, generally, not more complex. They are more complex in some (rare) cases, like the one you named. But most companies are not doing anything like that, but they do have their cto’s gearing up for that incredibly unlikely future.
- FpUser 7y ago
  Talked to many people lit you mentioned. Large percentage repeats these points: 1) We always wait for database, so performance of our code does not matter. This comes from places where ironically the whole database can fit into RAM. 10+ gbps connectivity - well it is almost commodity for business so the latency is not much of a bottleneck. Fast IO to store data - well imagine array of Optane drives. Not very cheap but really peanuts for normal business. All this means that purposely built data server which is a heart of many type of business residing on decent computer, can be blindingly fast and most of the businesses will never ever outgrow it. I've written quite a few of such servers serving loads of businesses in NA so hopefully my experience is not irrelevant. 2) Scripting languages are so convenient to use and save so much time and this is what matters as the salary is a main expense. - Personally I use those mostly for management/deployment/etc.etc type of scripts and maybe to quickly test some small ideas. I clearly see their benefits there. Anything that resembles a product that would actually run business. Sorry but experienced developer can implement those just as fast as in any scripting language and it will save a ton on maintenance. 3) And finally do not do premature optimization mantra . - In my opinion switching from scripting language to something like C++ has nothing to do with premature optimization. Developer can be very productive with compiled languages as they also have megatons of libraries for any imaginable task.
  - kortilla 7y ago
    > Sorry but experienced developer can implement those just as fast as in any scripting language and it will save a ton on maintenance. This is just a no true Scotsman argument. For 10 years I’ve watched python/ruby shops drastically outpace projects in C++/Java shops. What you’re failing to realize is how trivial most apps are and how fast fully functional back ends can be created with frameworks in those languages (django/rails). Your bit about maintenance is also bullshit. I’ve spent entire days peeling apart complex C++ code bases to make a small change to some core abstraction. Ability to maintain code is entirely up to how well organized and documented it is. It has nothing to do with “scripting”.
    - AmericanChopper 7y ago
      > What you’re failing to realize is how trivial most apps are and how fast fully functional back ends can be created with frameworks in those languages A vast majority of web dev is incredibly simple. People tend to easily convince themselves that their app is very special and requires a lot of complex solutions to make it work. There’s a lot of incentives that lead people to that conclusion, but most of the time following the golden path laid out for you by an existing framework is going to produce a better product with much less effort.
- 0x445442 7y ago
  > It’d be more interesting to talk about them, and how they came to make the wrong decision for what they thought were the right reason My guess, essentially holding their finger to the wind. Most orgs can't afford to do real engineering.
Sevii 7y ago
Its more like 'Here at BigCo, we do x because it scales. We have learned it from various incidents that we won't tell you about, but trust us we have to do it this way."
- buzzkillington 7y ago
  I've worked at BigCo. It's resume padding with the fear of looking like an idiot for not knowing about the new tech. We were going to go all in on Snowflake with everyone on the team being for it. I sat down, read the original whitepaper, wrote a simulation of what the costs would look like with the current read/write statistics and tested a small batch of data on it to double check. Turns out we would have paid between x500 to x10,000 what we were currently paying for using Postgres. I moved on three months ago, and last I heard they were trying to use Snowflake again.
  - pnako 7y ago
    I'm starting to wonder if there is market to do "technology laundering", use things like PostgreSQL, SQLite, standard Unix tools, put it under some cloud marketing and charge a x10 premium. Or perhaps not only there is market, but that's more or less what everyone is already doing.
    - IanCal 7y ago
      If you can set it up so it solves my problems, yes. I could manage my own server and fine tune everything, or I can throw it on snowflake. Snowflake means I've spent almost no time managing anything and it was costing less than an aws box running postgres but absolutely blew it out of the water performance wise. Depends on your workload but it's been perfect for one of my use cases. If they were just using postgres under the hood and I got the same en experience - fine. Big things are time sharing resources and management/updates/etc. Can you charge me 10x the underlying cost but let me pay for two minutes on a wildly powerful machine? Great, that's a net win for me.
      - FpUser 7y ago

jpz 7y ago

Big data is about I/O not CPU I’m a C++ veteran btw and understand the point but big data is about how to process petabytes of I/O not how to consume CPU.

sriram_malhar 7y ago
Yes, you are right. That said, the COST paper's presumption is that not every "big data" problem is big (petabyte size), that most probably fit in a single system's disk & memory.
emsy 7y ago
Correct me if I’m wrong, but isn’t the point of writing cache coherent code the fact that memory I/O is the bottleneck these days?
growse 7y ago
> Big data is about I/O not CPU > I’m a C++ veteran btw and understand the point but big data is about how to process petabytes of I/O not how to consume CPU. I'm not sure this is cut-and-dry. Back when I was working on Spark workloads, there was some interesting research being done on where the bottlenecks were for jobs. I think it turned out for a lot of jobs, infinite disk / network io didn't give as much of an improvement as you'd expect. https://databricks.com/session/making-sense-of-spark-performance https://databricks.com/session/making-sense-of-spark-perform...
philipov 7y ago
Not for everyone. In finance, a query might only use gigabytes or terabytes of data, but need to do a ton of simulation and calculation on top of that. Optimization of e.g. trading algorithms is entirely CPU-bound.
- jpz 7y ago
  Yes, I'm aware. I've written and maintained several of these systems in exotic derivatives space. Scale-out of CPU was required for these. These are not big data problems, however, by definition.
7y ago

lonelappde 7y ago

This was part of the excitement/promise of Go lang, to eliminate the need for small-scale mapreduce jobs, with library code that launched a year or two before the cost paper was published.