3 ms·

The funny thing about "big data" was that it came with a perverse incentive to avoid even the most basic and obvious optimizations on the software level, becaus

The funny thing about "big data" was that it came with a perverse incentive to avoid even the most basic and obvious optimizations on the software level, because the hardware requirement was how you proved how badass you were. Like: "Look, boss, I can compute all those averages for that report on just my laptop, by ingesting a SAMPLE of the data, rather than making those computations across the WHOLE dataset". Boss: "What do you mean 'sample'? I just don't know what you're trying to imply with your mathmo engineeringy gobbledigook! Me having spent those millions on nothing can clearly not be it, right?"

Spooky23 2y ago
It came with a few cohorts of Xooglers cashing their options out. The amount of salesman hype and chatter about big data, followed by the dick measuring contests about whose data was big enough to be worthy was intense for awhile.
bpodgursky 2y ago
This is a pretty snarky outside view and just not actually true (I spent the first part of my career trying to reduce compute spend as a data engineer). It was extremely difficult to get > 64gb on a machine for a very long time, and implementation complexity gets hard FAST when you have a hard cap. And it's EXTREMELY disruptive to have a process that fails every 1/50 times, when data is slightly too large, because your team will be juggling dozens of these routine crons, and if each of them breaks regularly, you do nothing but dumb oncall trying to trim bits off of each strong. No, Hadoop and MapReduce were not hyperefficient, but it was OK if you write it correctly, and having something that ran reliably is WAY more valuable than boutique bit-optimized C++ crap that nobody trusts or can maintain and fails every thursday with insane segfaults. (nowdays, just use Snowflake. but it was a reasonable tool for the time).
- stakhanov 2y ago
  Why do you assume it's an "outside" view? I was freelancing for many years on projects with "Data Scientist" / "Data Engineer" in my job title, having been hired by managers who heard about "Big Data" at an event by KPMG or Accenture or whatever. And I don't understand why you're reading "boutique bit-optimized C++ crap" into "most basic and obvious optimizations". One of those most basic and obvious optimizations is to avoid reading a dataframe into memory in its entirety, when the math that you want to do on top of it can actually be done as a running accumulator while reading the data from a stream. This is possible in 90% of realistic use cases, but the fraction of software written back then that took advantage of this was shockingly small. Solving the problem by buying more machines, chopping up the dataframe into smaller pieces, and farming out the payload through Hadoop had management buy-in. Yet, for some reason, doing the sane thing, namely rewriting poorly-written software, didn't.

Spooky23 2y ago

It came with a few cohorts of Xooglers cashing their options out. The amount of salesman hype and chatter about big data, followed by the dick measuring contests about whose data was big enough to be worthy was intense for awhile.

bpodgursky 2y ago

This is a pretty snarky outside view and just not actually true (I spent the first part of my career trying to reduce compute spend as a data engineer). It was extremely difficult to get > 64gb on a machine for a very long time, and implementation complexity gets hard FAST when you have a hard cap. And it's EXTREMELY disruptive to have a process that fails every 1/50 times, when data is slightly too large, because your team will be juggling dozens of these routine crons, and if each of them breaks regularly, you do nothing but dumb oncall trying to trim bits off of each strong. No, Hadoop and MapReduce were not hyperefficient, but it was OK if you write it correctly, and having something that ran reliably is WAY more valuable than boutique bit-optimized C++ crap that nobody trusts or can maintain and fails every thursday with insane segfaults. (nowdays, just use Snowflake. but it was a reasonable tool for the time).

stakhanov 2y ago
Why do you assume it's an "outside" view? I was freelancing for many years on projects with "Data Scientist" / "Data Engineer" in my job title, having been hired by managers who heard about "Big Data" at an event by KPMG or Accenture or whatever. And I don't understand why you're reading "boutique bit-optimized C++ crap" into "most basic and obvious optimizations". One of those most basic and obvious optimizations is to avoid reading a dataframe into memory in its entirety, when the math that you want to do on top of it can actually be done as a running accumulator while reading the data from a stream. This is possible in 90% of realistic use cases, but the fraction of software written back then that took advantage of this was shockingly small. Solving the problem by buying more machines, chopping up the dataframe into smaller pieces, and farming out the payload through Hadoop had management buy-in. Yet, for some reason, doing the sane thing, namely rewriting poorly-written software, didn't.