3 ms·
That is one of the selling points of Hadoop, you can write garbage code and scale your way out of any problem, turning the $$$ knob up to more nodes.
by nerpderp82 4y ago
That is one of the selling points of Hadoop, you can write garbage code and scale your way out of any problem, turning the $$$ knob up to more nodes.
- angry_moose 4y agoYeah, that's why I got involved (I was infrastructure at the time) - how can we throw more hardware at it as the kubernetes setup they had wasn't cutting it. One of the "data scientists" point blank said in a meeting "My time is too valuable to be spent optimizing the code, I should be solving problems. We can always just buy more hardware". Admittedly the last little bit of analysis was pretty cool, but >>99% of that runtime was massaging all of the data into a format that allowed the last step to happen.
- mejakethomas 4y agoSnowflake too. Inefficient sql? Crank the virtual warehouse.