5 ms·

very simple processing, not memory bound, tiny data-set - of course its going to be faster locally when the command itself takes less time than the networking,

very simple processing, not memory bound, tiny data-set - of course its going to be faster locally when the command itself takes less time than the networking, distribution, coordination and collation overhead of using any distributed tool...

numlock86 7y ago
Yes, the article even states that in the first paragraph. Processing a few GB (the example in the post is not even in the two digit range) is easily done with basically any tool. Heck, someone could probably pull off something like that in Excel in a couple of minutes. I think the point if the article is exactly that: Unless you have a couple of TB to process don't even bother. I remember a similar post in the same spirit that had a couple of GB in a Postgres database and ended like "Just do a simple query!" instead of a Hadoop cluster for simple map-reduce.
- throwawaybbb 7y ago
  You can easily run a workload with 20tb/day on a single thread ripper. My last contract we wrote a c tool that did dsp on 10tb/day data with a latency of 50 microseconds. We are using less than 10% of the CPU on that.
  - fxtentacle 7y ago
    Yeah, the ThreadRippers are amazing. I was truly astonished when I noticed that one hour of V-Ray on 3970X outperformed $1000 in ChaosCloud rendering. At a price of $2500 once for the TR versus $24,000 per day for cloud, it's a no-brainer to buy them. Sadly, availability in my area is pretty spotty and even Amazon currently predicts 2+ weeks of delivery time.
    - threeseed 7y ago
      Not sure where you are getting your costs from. Threadripper 3970x = 32 cores @ $1999 AWS c5.9xlarge = 36 vCPUs @ $8.34/day (spot) You could buy a year of AWS for the same price as a Threadripper based PC.
      - fxtentacle 7y ago
        I was comparing with ChaosCloud: https://www.chaosgroup.com/cloud https://www.chaosgroup.com/cloud For EC2, one would need to rent licenses for the V-Ray renderer plus V-Ray tends to be heavily CPU-cache- and memory-bound, so exactly the parts that get slow if you enable the Intel security fixes. Due to c5.9xlarge being Xeons, they are like 80% slower than a comparable AMD CPU. BTW, calculator.s3.amazonaws.com quotes me $1119 per month for a single c5.9xlarge, so I'm not sure how you would reach $8/day when I see $37/day.
        threeseed 7y ago
        Then switch to the AMD based c5a.9xlarge which is even cheaper. And you can reach $8/day using Spot or Reserved Instances. Not sure where you get $1119.
  - muzakthings 7y ago
    what kind of DSP were you doing? can I come work for you haha
lostcolony 7y ago
Hence the use of the word "can". The fact that "deal with it locally" was obvious to the author, but not to the people who inspired the author to deal with it that way (he mentions an EMR job), means there are those it may be worth pointing it out to. Certainly, being reminded that you probably should solve your problem without distribution, unless you actually demonstrate that you CAN'T solve your problem without distribution, is nothing but a positive.
llarsson 7y ago
You know that, I know that, and we can be happy that we have the experience to know what the right tool for this job would be by sizing up and describing the characteristics of the problem like you just did. But those with less experience may not be able to do that unless shown stuff like this in practice. Some may think this problem requires MapReduce. The quote from the original implementation blog post certainly seems to indicate so.
- threeseed 7y ago
  MapReduce as a paradigm and technology was popular about a decade ago and then died shortly after in favour of Hive, Spark etc. Pretty confident that not a single developer anywhere in this world would be first thinking of MapReduce. Just like they wouldn't jump straight to Cobol.
  - wbl 7y ago
    Back in high school I did a map reduce job using split, sort, and a handwritten transformation and accumulation tool. It worked!
  - jbergens 7y ago
    Some places still seem to go for large Kafka clusters just to calc some stats and forward some messages. I am sure some of their solutions are MapReduce below.
    - threeseed 7y ago
      Very curious to understand more about these mythical developers who are recommending a technology/paradigm that stopped being used a decade ago. You've seen people writing pages and pages of Java code to do ETL ?
      - doteka 7y ago
        I worked for a large multinational last year that had multiple teams rolling out terrible Hadoop solutions. The rest of the world does not particularly care about SV fashion trends.

numlock86 7y ago

Yes, the article even states that in the first paragraph. Processing a few GB (the example in the post is not even in the two digit range) is easily done with basically any tool. Heck, someone could probably pull off something like that in Excel in a couple of minutes. I think the point if the article is exactly that: Unless you have a couple of TB to process don't even bother. I remember a similar post in the same spirit that had a couple of GB in a Postgres database and ended like "Just do a simple query!" instead of a Hadoop cluster for simple map-reduce.

throwawaybbb 7y ago
You can easily run a workload with 20tb/day on a single thread ripper. My last contract we wrote a c tool that did dsp on 10tb/day data with a latency of 50 microseconds. We are using less than 10% of the CPU on that.
- fxtentacle 7y ago
  Yeah, the ThreadRippers are amazing. I was truly astonished when I noticed that one hour of V-Ray on 3970X outperformed $1000 in ChaosCloud rendering. At a price of $2500 once for the TR versus $24,000 per day for cloud, it's a no-brainer to buy them. Sadly, availability in my area is pretty spotty and even Amazon currently predicts 2+ weeks of delivery time.
  - threeseed 7y ago
    Not sure where you are getting your costs from. Threadripper 3970x = 32 cores @ $1999 AWS c5.9xlarge = 36 vCPUs @ $8.34/day (spot) You could buy a year of AWS for the same price as a Threadripper based PC.
    - fxtentacle 7y ago
      I was comparing with ChaosCloud: https://www.chaosgroup.com/cloud https://www.chaosgroup.com/cloud For EC2, one would need to rent licenses for the V-Ray renderer plus V-Ray tends to be heavily CPU-cache- and memory-bound, so exactly the parts that get slow if you enable the Intel security fixes. Due to c5.9xlarge being Xeons, they are like 80% slower than a comparable AMD CPU. BTW, calculator.s3.amazonaws.com quotes me $1119 per month for a single c5.9xlarge, so I'm not sure how you would reach $8/day when I see $37/day.
      - threeseed 7y ago
        Then switch to the AMD based c5a.9xlarge which is even cheaper. And you can reach $8/day using Spot or Reserved Instances. Not sure where you get $1119.
- muzakthings 7y ago
  what kind of DSP were you doing? can I come work for you haha

lostcolony 7y ago

Hence the use of the word "can". The fact that "deal with it locally" was obvious to the author, but not to the people who inspired the author to deal with it that way (he mentions an EMR job), means there are those it may be worth pointing it out to. Certainly, being reminded that you probably should solve your problem without distribution, unless you actually demonstrate that you CAN'T solve your problem without distribution, is nothing but a positive.

llarsson 7y ago

You know that, I know that, and we can be happy that we have the experience to know what the right tool for this job would be by sizing up and describing the characteristics of the problem like you just did. But those with less experience may not be able to do that unless shown stuff like this in practice. Some may think this problem requires MapReduce. The quote from the original implementation blog post certainly seems to indicate so.

threeseed 7y ago
MapReduce as a paradigm and technology was popular about a decade ago and then died shortly after in favour of Hive, Spark etc. Pretty confident that not a single developer anywhere in this world would be first thinking of MapReduce. Just like they wouldn't jump straight to Cobol.
- wbl 7y ago
  Back in high school I did a map reduce job using split, sort, and a handwritten transformation and accumulation tool. It worked!
- jbergens 7y ago
  Some places still seem to go for large Kafka clusters just to calc some stats and forward some messages. I am sure some of their solutions are MapReduce below.
  - threeseed 7y ago
    Very curious to understand more about these mythical developers who are recommending a technology/paradigm that stopped being used a decade ago. You've seen people writing pages and pages of Java code to do ETL ?
    - doteka 7y ago
      I worked for a large multinational last year that had multiple teams rolling out terrible Hadoop solutions. The rest of the world does not particularly care about SV fashion trends.