2 ms·
I really don't understand comments like this. Yes your company's data may fit in RAM. But does every intermediate data set also fit in RAM ? Because I've also
by threeseed 7y ago
I really don't understand comments like this.
Yes your company's data may fit in RAM. But does every intermediate data set also fit in RAM ? Because I've also worked at a bank and we had thousands of complex ETLs often needing tens to hundreds of intermediate sets along the way. There is no AWS server that can keep all of that inflight at one time.
And what about your Data Analysts/Scientists. Can all of their random data sets reside in RAM on the same server too ?
- lmilcin 7y agoNever said EVERY data set fits in RAM, but that doesn't mean MOST of them don't. There is a trend, when the application is inefficient, to spend huge amount of resources on scaling it instead of making the application more efficient.
- andyjpb 7y agoBuy them a machine each. $100K has always been "cheap" for a "business computer" and today you can get more computer for that money than ever. $100K of hardware (per year or so) is small-fry compared to almost every other R&D industry out there. Just compare with the cost of debuggers, oscilloscopes and EMC labs for electronic engineers.
- threeseed 7y agoMy company has over 400 Data Scientists and 1000s of Data Analysts. Buy them a machine each at a cost of $40-60 billion ? Or would it make more sense to buy one Spark cluster and then share the resources at a fraction of the cost.
- Aeolun 7y agoI don’t get your numbers, getting one for 400 people is 40M, for thousands it may be 100-999M. Still expensive, but much less than 40 billion.
- threeseed 7y agoHe said $100k for each user but it's a dumb idea anyway. We have a Spark cluster which supports all of those users for $10-$20k a month.