5 ms·

There's a reason it's a classic. The best summation I heard of it was, "If you can't get the solution to a problem to fit on one machine, you probably don't und

There's a reason it's a classic. The best summation I heard of it was, "If you can't get the solution to a problem to fit on one machine, you probably don't understand the problem well enough." This is probably one of my favorite quotes in software. I'm probably slightly misquoting it here, but I don't care.

nickpsecurity 10y ago
It's been formalized past that into the COST methodology of assessing distribution solutions: Configuration that Outperforms a Single Thread. If that configuration doesn't exist, just ditch the solution in favor of a single thread. :) http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/... It just gets more hilarious the more you read the measurements of all these tech vs a single thread.
- qwertyuiop924 10y ago
  A single thread frequently has less code: less code means less code being executed. This is a contributing factor to speed. The less instructions you have to execute, the faster it runs, give or take. Not that instructions executed do not necessarily correlate to code size (a loop is only written once, but executed many times). By the way, bane responded to my original post: it's worth reading.
  - deleted 10y ago
    [deleted]
  - nickpsecurity 10y ago
    His response just loaded on my end for some reason. Yeah, that's the points I was getting from his longer write-up. If you see him redo it & I'm not there, definitely ping me so I can see the future copy. We were actually both in the best example I've seen of this effect on HN: https://news.ycombinator.com/item?id=12472905 https://news.ycombinator.com/item?id=12472905 It's the kind of title that, if proven in resulting article, should make many people pause to think about how effectively they're solving business problems.
    - qwertyuiop924 10y ago
      Ah, I remember that. To paraphrase myself at the time, "never send a 747 to do a racecar's job." Yeah, as it turns out, it's not a good idea to use a piece of software optimized for workloads to large to even be handled by a single machine on large quantities of data, and maybe you should use tools designed for small workloads (Gigabytes, or even Terabytes, is pretty small, nowadays) for small workloads. This should be obvious. Besides, things like Manta exist, so you can scale up your small workload scripts to larger workloads in the future. Mind, I don't know if I trust Manta yet (I don't know the perf, and I haven't tried it personally), so maybe exercise a little bit of caution about going that route.

nickpsecurity 10y ago

It's been formalized past that into the COST methodology of assessing distribution solutions: Configuration that Outperforms a Single Thread. If that configuration doesn't exist, just ditch the solution in favor of a single thread. :) http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/... It just gets more hilarious the more you read the measurements of all these tech vs a single thread.

qwertyuiop924 10y ago
A single thread frequently has less code: less code means less code being executed. This is a contributing factor to speed. The less instructions you have to execute, the faster it runs, give or take. Not that instructions executed do not necessarily correlate to code size (a loop is only written once, but executed many times). By the way, bane responded to my original post: it's worth reading.
- deleted 10y ago
  [deleted]
- nickpsecurity 10y ago
  His response just loaded on my end for some reason. Yeah, that's the points I was getting from his longer write-up. If you see him redo it & I'm not there, definitely ping me so I can see the future copy. We were actually both in the best example I've seen of this effect on HN: https://news.ycombinator.com/item?id=12472905 https://news.ycombinator.com/item?id=12472905 It's the kind of title that, if proven in resulting article, should make many people pause to think about how effectively they're solving business problems.
  - qwertyuiop924 10y ago
    Ah, I remember that. To paraphrase myself at the time, "never send a 747 to do a racecar's job." Yeah, as it turns out, it's not a good idea to use a piece of software optimized for workloads to large to even be handled by a single machine on large quantities of data, and maybe you should use tools designed for small workloads (Gigabytes, or even Terabytes, is pretty small, nowadays) for small workloads. This should be obvious. Besides, things like Manta exist, so you can scale up your small workload scripts to larger workloads in the future. Mind, I don't know if I trust Manta yet (I don't know the perf, and I haven't tried it personally), so maybe exercise a little bit of caution about going that route.