9 ms·

Past: How to Analyze Billions of Records per Second on a Supercomputer Present: How to Analyze Billions of Records per Second on a Single Desktop PC Future:

Past: How to Analyze Billions of Records per Second on a Supercomputer Present: How to Analyze Billions of Records per Second on a Single Desktop PC Future: How to Analyze 5 Records per Second on Thousands of Machines Distributed Across the Globe Over the Blockchain.

TrevorJ 8y ago
That's axiomatic of the direction consumer software has gone over the last 20 years as well. How a word processor can feel sluggish on a modern PC is beyond me.
- zeusk 8y ago
  Because that word processor can do a lot more things and is not the sole thing running on a modern machine.
  - _emacsomancer_ 8y ago
    and, one would imagine, a lot of accumulated cruft.
    - zeusk 8y ago
      what you're calling cruft is some other person's usecase.
      - frankling_ 8y ago
        I'd still prioritize the basic use case of comfortably typing words, which to my admittedly sensitive mind is not supported that well anymore by some common pieces of software. Between the apparent round-trip per keypress in Outlook Web, the noticeable general display delay of Windows 10, and the frequent wait for characters to appear in a reasonably small Powerpoint presentation, I feel like I might be holding it wrong.
      - TeMPOraL 8y ago
        Quite often it would not be usecases, but externalities. That is, trading the quality and efficiency of your product for some benefits in selling or developing. Software, for better or worse, is a social enterprise, so you can get away with dumping a lot on your users before they pack up and look for alternatives.
      - _emacsomancer_ 8y ago
        Ok, and a lot of non-user-facing cruft too.
- mmt 8y ago
  It's not just consumer software and not just the last 20 years. There is an axiom that software gets bigger, for no particularly good reason, for at least 40 years. This can be witnessed in how the size of /bin/true on Unix went from 0 bytes to whatever size it is today [1]. I can't find it now, but there was an article with a bar chart showing a gradual progression through time. Of course, on-disk size doesn't translate directly to performance, but the trivial example demonstrates mechanisms by which bloat/cruft can accumulate so gradually that no outcry is ever caused. It also refutes the sibling/downthread comment the "cruft" is (necessarily) someone else's use case. [1] e.g. 13k MacOS native, 52k GNU coreutils from MacPorts
  - deleted 8y ago
    [deleted]
  - majewsky 8y ago
    > There is an axiom that software gets bigger, for no particularly good reason, for at least 40 years. Sounds like a computer variant of Parkinson's law: "Work expands to fill the time available for its completion."
    - mmt 8y ago
      Close, but not quite, since what's available (space in the specific example) isn't being filled. Alternatively, with the complaint that desktop software feels more sluggish on a modern computer than contemporaneous software did on much older hardware suggests that something is even being overfilled. My intuition says there's a different phenomenon of psychology at work here.
- quickthrower2 8y ago
  Not experienced a sluggish word processor recently (using 6 yr old lenovo laptop, both Word and OpenOffice are fine)
  - majewsky 8y ago
    Word 2016 takes 2 seconds to start on my pretty beefy notebook (2017 Lenovo X1 Yoga). Pretty sure that when you measure it, Word 2003 or so would take a similar amount of time to start on a 2003 PC, even though all of CPU, RAM and disk storage are vastly faster nowadays. Why does the Word window not appear within milliseconds of me clicking on the application icon?
taneq 8y ago
I can't find it now but there was a great blog post about parallel processing of large data sets just using Linux streams from the command line. They ended up comparing a single laptop (iirc) against a Hadoop cluster and coming out ahead.
- wcrichton 8y ago
  https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
  - taneq 8y ago
    That's the one, thanks!
    - adamdrake 8y ago
      Author here. If I can answer any questions, feel free to ask away!
      - itronitron 8y ago
        I'm interested to know if you have any pointers on learning resources for awk.
        pedro84 8y ago
        I learned from The AWK Programming Language: https://news.ycombinator.com/item?id=13451454 https://news.ycombinator.com/item?id=13451454
        adamdrake 8y ago
        I use it from time to time, but I certainly wouldn't consider myself an expert. When I need to do something beyond my current knowledge I search around a bit until I find the awk syntax I need to get the job done. Sorry I can't be more helpful!
adamdrake 8y ago
Or, as is my preference, how to analyze 366,000 Records per Second on my laptop: https://adamdrake.com/big-data-small-machine.html https://adamdrake.com/big-data-small-machine.html And that's with non-optimized code!
- mmt 8y ago
  You may want to re-submit that, since it didn't get any traction 38 days ago but might today, if this article and this comment get enough eyeballs. I now hear actual data scientists complain that startups trying to hire them think they have "big data" problems when all they have are "data" problems.
  - itronitron 8y ago
    That sounds like something AI can help solve. /s
deltateam 8y ago
thats good tongue in cheek comedy as long as you know why it isn't accurate. in this case the nodes in a blockchain can execute arbitrary tasks, its like access to a ton of Amazon Lambda machines. You have to pay them gas - or whatever currency/fuel/commodity powers that ecosystem - and you can execute commands to your heart's content, as long as the gas is tolerable. Ethereum's nodes would probably be limited in what they could execute, more in line with your joke, but there are several distributed processing applications built on top of Ethereum such as iExecRLC. You can use their token to run your billions of records analysis. Its more in line with your "past" observation. Just distributed computing with the economic incentives more broadly aligned instead of praying for altruism.

TrevorJ 8y ago

That's axiomatic of the direction consumer software has gone over the last 20 years as well. How a word processor can feel sluggish on a modern PC is beyond me.

zeusk 8y ago
Because that word processor can do a lot more things and is not the sole thing running on a modern machine.
- _emacsomancer_ 8y ago
  and, one would imagine, a lot of accumulated cruft.
  - zeusk 8y ago
    what you're calling cruft is some other person's usecase.
    - frankling_ 8y ago
      I'd still prioritize the basic use case of comfortably typing words, which to my admittedly sensitive mind is not supported that well anymore by some common pieces of software. Between the apparent round-trip per keypress in Outlook Web, the noticeable general display delay of Windows 10, and the frequent wait for characters to appear in a reasonably small Powerpoint presentation, I feel like I might be holding it wrong.
    - TeMPOraL 8y ago
      Quite often it would not be usecases, but externalities. That is, trading the quality and efficiency of your product for some benefits in selling or developing. Software, for better or worse, is a social enterprise, so you can get away with dumping a lot on your users before they pack up and look for alternatives.
    - _emacsomancer_ 8y ago
      Ok, and a lot of non-user-facing cruft too.
mmt 8y ago
It's not just consumer software and not just the last 20 years. There is an axiom that software gets bigger, for no particularly good reason, for at least 40 years. This can be witnessed in how the size of /bin/true on Unix went from 0 bytes to whatever size it is today [1]. I can't find it now, but there was an article with a bar chart showing a gradual progression through time. Of course, on-disk size doesn't translate directly to performance, but the trivial example demonstrates mechanisms by which bloat/cruft can accumulate so gradually that no outcry is ever caused. It also refutes the sibling/downthread comment the "cruft" is (necessarily) someone else's use case. [1] e.g. 13k MacOS native, 52k GNU coreutils from MacPorts
- deleted 8y ago
  [deleted]
- majewsky 8y ago
  > There is an axiom that software gets bigger, for no particularly good reason, for at least 40 years. Sounds like a computer variant of Parkinson's law: "Work expands to fill the time available for its completion."
  - mmt 8y ago
    Close, but not quite, since what's available (space in the specific example) isn't being filled. Alternatively, with the complaint that desktop software feels more sluggish on a modern computer than contemporaneous software did on much older hardware suggests that something is even being overfilled. My intuition says there's a different phenomenon of psychology at work here.
quickthrower2 8y ago
Not experienced a sluggish word processor recently (using 6 yr old lenovo laptop, both Word and OpenOffice are fine)
- majewsky 8y ago
  Word 2016 takes 2 seconds to start on my pretty beefy notebook (2017 Lenovo X1 Yoga). Pretty sure that when you measure it, Word 2003 or so would take a similar amount of time to start on a 2003 PC, even though all of CPU, RAM and disk storage are vastly faster nowadays. Why does the Word window not appear within milliseconds of me clicking on the application icon?

taneq 8y ago

I can't find it now but there was a great blog post about parallel processing of large data sets just using Linux streams from the command line. They ended up comparing a single laptop (iirc) against a Hadoop cluster and coming out ahead.

wcrichton 8y ago
https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
- taneq 8y ago
  That's the one, thanks!
  - adamdrake 8y ago
    Author here. If I can answer any questions, feel free to ask away!
    - itronitron 8y ago
      I'm interested to know if you have any pointers on learning resources for awk.
      - pedro84 8y ago
        I learned from The AWK Programming Language: https://news.ycombinator.com/item?id=13451454 https://news.ycombinator.com/item?id=13451454
      - adamdrake 8y ago
        I use it from time to time, but I certainly wouldn't consider myself an expert. When I need to do something beyond my current knowledge I search around a bit until I find the awk syntax I need to get the job done. Sorry I can't be more helpful!

adamdrake 8y ago

Or, as is my preference, how to analyze 366,000 Records per Second on my laptop: https://adamdrake.com/big-data-small-machine.html https://adamdrake.com/big-data-small-machine.html And that's with non-optimized code!

mmt 8y ago
You may want to re-submit that, since it didn't get any traction 38 days ago but might today, if this article and this comment get enough eyeballs. I now hear actual data scientists complain that startups trying to hire them think they have "big data" problems when all they have are "data" problems.
- itronitron 8y ago
  That sounds like something AI can help solve. /s

deltateam 8y ago

thats good tongue in cheek comedy as long as you know why it isn't accurate. in this case the nodes in a blockchain can execute arbitrary tasks, its like access to a ton of Amazon Lambda machines. You have to pay them gas - or whatever currency/fuel/commodity powers that ecosystem - and you can execute commands to your heart's content, as long as the gas is tolerable. Ethereum's nodes would probably be limited in what they could execute, more in line with your joke, but there are several distributed processing applications built on top of Ethereum such as iExecRLC. You can use their token to run your billions of records analysis. Its more in line with your "past" observation. Just distributed computing with the economic incentives more broadly aligned instead of praying for altruism.