3 ms·

This is a pretty snarky outside view and just not actually true (I spent the first part of my career trying to reduce compute spend as a data engineer). It was

This is a pretty snarky outside view and just not actually true (I spent the first part of my career trying to reduce compute spend as a data engineer). It was extremely difficult to get > 64gb on a machine for a very long time, and implementation complexity gets hard FAST when you have a hard cap. And it's EXTREMELY disruptive to have a process that fails every 1/50 times, when data is slightly too large, because your team will be juggling dozens of these routine crons, and if each of them breaks regularly, you do nothing but dumb oncall trying to trim bits off of each strong. No, Hadoop and MapReduce were not hyperefficient, but it was OK if you write it correctly, and having something that ran reliably is WAY more valuable than boutique bit-optimized C++ crap that nobody trusts or can maintain and fails every thursday with insane segfaults. (nowdays, just use Snowflake. but it was a reasonable tool for the time).

stakhanov 2y ago
Why do you assume it's an "outside" view? I was freelancing for many years on projects with "Data Scientist" / "Data Engineer" in my job title, having been hired by managers who heard about "Big Data" at an event by KPMG or Accenture or whatever. And I don't understand why you're reading "boutique bit-optimized C++ crap" into "most basic and obvious optimizations". One of those most basic and obvious optimizations is to avoid reading a dataframe into memory in its entirety, when the math that you want to do on top of it can actually be done as a running accumulator while reading the data from a stream. This is possible in 90% of realistic use cases, but the fraction of software written back then that took advantage of this was shockingly small. Solving the problem by buying more machines, chopping up the dataframe into smaller pieces, and farming out the payload through Hadoop had management buy-in. Yet, for some reason, doing the sane thing, namely rewriting poorly-written software, didn't.

stakhanov 2y ago

Why do you assume it's an "outside" view? I was freelancing for many years on projects with "Data Scientist" / "Data Engineer" in my job title, having been hired by managers who heard about "Big Data" at an event by KPMG or Accenture or whatever. And I don't understand why you're reading "boutique bit-optimized C++ crap" into "most basic and obvious optimizations". One of those most basic and obvious optimizations is to avoid reading a dataframe into memory in its entirety, when the math that you want to do on top of it can actually be done as a running accumulator while reading the data from a stream. This is possible in 90% of realistic use cases, but the fraction of software written back then that took advantage of this was shockingly small. Solving the problem by buying more machines, chopping up the dataframe into smaller pieces, and farming out the payload through Hadoop had management buy-in. Yet, for some reason, doing the sane thing, namely rewriting poorly-written software, didn't.