3 ms·

Yes, the article even states that in the first paragraph. Processing a few GB (the example in the post is not even in the two digit range) is easily done with b

Yes, the article even states that in the first paragraph. Processing a few GB (the example in the post is not even in the two digit range) is easily done with basically any tool. Heck, someone could probably pull off something like that in Excel in a couple of minutes. I think the point if the article is exactly that: Unless you have a couple of TB to process don't even bother. I remember a similar post in the same spirit that had a couple of GB in a Postgres database and ended like "Just do a simple query!" instead of a Hadoop cluster for simple map-reduce.

throwawaybbb 7y ago
You can easily run a workload with 20tb/day on a single thread ripper. My last contract we wrote a c tool that did dsp on 10tb/day data with a latency of 50 microseconds. We are using less than 10% of the CPU on that.
- fxtentacle 7y ago
  Yeah, the ThreadRippers are amazing. I was truly astonished when I noticed that one hour of V-Ray on 3970X outperformed $1000 in ChaosCloud rendering. At a price of $2500 once for the TR versus $24,000 per day for cloud, it's a no-brainer to buy them. Sadly, availability in my area is pretty spotty and even Amazon currently predicts 2+ weeks of delivery time.
  - threeseed 7y ago
    Not sure where you are getting your costs from. Threadripper 3970x = 32 cores @ $1999 AWS c5.9xlarge = 36 vCPUs @ $8.34/day (spot) You could buy a year of AWS for the same price as a Threadripper based PC.
    - fxtentacle 7y ago
      I was comparing with ChaosCloud: https://www.chaosgroup.com/cloud https://www.chaosgroup.com/cloud For EC2, one would need to rent licenses for the V-Ray renderer plus V-Ray tends to be heavily CPU-cache- and memory-bound, so exactly the parts that get slow if you enable the Intel security fixes. Due to c5.9xlarge being Xeons, they are like 80% slower than a comparable AMD CPU. BTW, calculator.s3.amazonaws.com quotes me $1119 per month for a single c5.9xlarge, so I'm not sure how you would reach $8/day when I see $37/day.
      - threeseed 7y ago
        Then switch to the AMD based c5a.9xlarge which is even cheaper. And you can reach $8/day using Spot or Reserved Instances. Not sure where you get $1119.
- muzakthings 7y ago
  what kind of DSP were you doing? can I come work for you haha

throwawaybbb 7y ago

You can easily run a workload with 20tb/day on a single thread ripper. My last contract we wrote a c tool that did dsp on 10tb/day data with a latency of 50 microseconds. We are using less than 10% of the CPU on that.

fxtentacle 7y ago
Yeah, the ThreadRippers are amazing. I was truly astonished when I noticed that one hour of V-Ray on 3970X outperformed $1000 in ChaosCloud rendering. At a price of $2500 once for the TR versus $24,000 per day for cloud, it's a no-brainer to buy them. Sadly, availability in my area is pretty spotty and even Amazon currently predicts 2+ weeks of delivery time.
- threeseed 7y ago
  Not sure where you are getting your costs from. Threadripper 3970x = 32 cores @ $1999 AWS c5.9xlarge = 36 vCPUs @ $8.34/day (spot) You could buy a year of AWS for the same price as a Threadripper based PC.
  - fxtentacle 7y ago
    I was comparing with ChaosCloud: https://www.chaosgroup.com/cloud https://www.chaosgroup.com/cloud For EC2, one would need to rent licenses for the V-Ray renderer plus V-Ray tends to be heavily CPU-cache- and memory-bound, so exactly the parts that get slow if you enable the Intel security fixes. Due to c5.9xlarge being Xeons, they are like 80% slower than a comparable AMD CPU. BTW, calculator.s3.amazonaws.com quotes me $1119 per month for a single c5.9xlarge, so I'm not sure how you would reach $8/day when I see $37/day.
    - threeseed 7y ago
      Then switch to the AMD based c5a.9xlarge which is even cheaper. And you can reach $8/day using Spot or Reserved Instances. Not sure where you get $1119.
muzakthings 7y ago
what kind of DSP were you doing? can I come work for you haha