4 ms·

I once had a job interview where they gave me a problem and expected some answer of "use a spark cluster, da da" and I said -- well given the problem specs and

I once had a job interview where they gave me a problem and expected some answer of "use a spark cluster, da da" and I said -- well given the problem specs and upper bounds you've given, I'd throw a $200 stick of RAM at it and be done. They wanted some answer involving using spark engineers you'd effectively pay for at $200/hr for many weeks. I didn't get the job. OK, but seriously speaking, if the upper bound is beyond 786gb RAM or whatever the current max is, I might want to use dask distributed. https://distributed.dask.org/en/latest/ https://distributed.dask.org/en/latest/ edit: wrote mb instead of gb

nicoburns 7y ago
> 786mb RAM Did you mean gb? In any case you can actually do at least 24TB these days: https://aws.amazon.com/blogs/aws/ec2-high-memory-update-new-18-tb-and-24-tb-instances/ https://aws.amazon.com/blogs/aws/ec2-high-memory-update-new-...
- foota 7y ago
  That's ludicrous.
ClumsyPilot 7y ago
Bet their job description was like:"flexible", "open minded people" and "best tool for the job". But then they dont usually like when you demonstrate anything but the answer they want to a highly contrived problem.
- mark-r 7y ago
  Yes, technical interviewing is hard.
mumblemumble 7y ago
It's slightly mystifying. The only company I've worked at that did "big data" _really_ well just plugged a few TB of RAM into some sharded databases and got on with life. Usually when I tell that story, I get a lot of objections about how that solution won't scale and they must not have really had big data from people who are, truth be told, used to working with data on a fraction of the scale that this company did. That said, it's not a turnkey solution. This company also was more meticulous about data engineering than others, and that certainly had its own cost.
- hinkley 7y ago
  People do not want to hear about how they’re working way, way too hard to feel special.
- RivieraKid 7y ago
  I've always disliked the term "big data" because all of the attempts at a definition seemed either stupid or vague. After a while, I came up with this definition: it's a set of technnologies used for processing data that are too large to be processed in a single machine.
  - colechristensen 7y ago
    And more often than not, it's a set of technologies used for processing data on many machines which could be processed faster on one.
  - mumblemumble 7y ago
    The thing that gets me about that definition is that "too large to be processd in a single machine" leaves out a lot of variables. How's the machine specced? How's it being analyzed? Using what kinds of software? If the only single-machine option you consider is Pandas, which doesn't do streaming well and is built on a platform that makes ad-hoc multiprocessing a chore, you'll hit the ceiling a lot faster than if you had done it in Java, which might in turn be hard to push as far as something like C# (largely comparable to Java, but some platform features make it easier to be frugal with memory and mind your cache lines) or, dare I say it, something native like ocaml or C++. Alternatively, if you start right off with Spark, you won't be able to push even one node as far as if you hadn't, because Spark is designed from the ground up for running on a cluster, and therefore has a tendency to manage memory the same way a 22-year-old professional basketball player handles money. It makes scale-out something of a self-fulfilling prophecy. Also, as someone who was doing distributed data processing pipelines well before Hadoop and friends came along, I'm not sure I can swallow "big data" being one and the same as "handling data that is too big to run on one computer." Big data sort of implies a certain culture of handling data at that scale, too. Because of that, I tend to think of "big data" as describing a culture as much as it describes anything practical. It's a set (not the only set) of technologies for procesing data on multiple machines. Whether you actually need multiple machines to do the job seems to be less relevant than the marketing team at IBM (to pick an easy punching back) would have us believe.
pletnes 7y ago
Unlike most big data solutions, dask is just so convenient. I sometimes use it just to concatenate a bunch of files into a dataframe. And yet, it scales up to really large datasets, too.
omgtehlion 7y ago
768G is not current max. Recently bought a machine (for less than $3k) for a side project, it has 768G inside and is 4 years old. Today most commodity servers (which you can buy out of pocket without lengthy preorder) can accomodate from 1.5 to 3 TiB of memory.

nicoburns 7y ago

> 786mb RAM Did you mean gb? In any case you can actually do at least 24TB these days: https://aws.amazon.com/blogs/aws/ec2-high-memory-update-new-18-tb-and-24-tb-instances/ https://aws.amazon.com/blogs/aws/ec2-high-memory-update-new-...

foota 7y ago
That's ludicrous.

ClumsyPilot 7y ago

Bet their job description was like:"flexible", "open minded people" and "best tool for the job". But then they dont usually like when you demonstrate anything but the answer they want to a highly contrived problem.

mark-r 7y ago
Yes, technical interviewing is hard.

mumblemumble 7y ago

It's slightly mystifying. The only company I've worked at that did "big data" _really_ well just plugged a few TB of RAM into some sharded databases and got on with life. Usually when I tell that story, I get a lot of objections about how that solution won't scale and they must not have really had big data from people who are, truth be told, used to working with data on a fraction of the scale that this company did. That said, it's not a turnkey solution. This company also was more meticulous about data engineering than others, and that certainly had its own cost.

hinkley 7y ago
People do not want to hear about how they’re working way, way too hard to feel special.
RivieraKid 7y ago
I've always disliked the term "big data" because all of the attempts at a definition seemed either stupid or vague. After a while, I came up with this definition: it's a set of technnologies used for processing data that are too large to be processed in a single machine.
- colechristensen 7y ago
  And more often than not, it's a set of technologies used for processing data on many machines which could be processed faster on one.
- mumblemumble 7y ago
  The thing that gets me about that definition is that "too large to be processd in a single machine" leaves out a lot of variables. How's the machine specced? How's it being analyzed? Using what kinds of software? If the only single-machine option you consider is Pandas, which doesn't do streaming well and is built on a platform that makes ad-hoc multiprocessing a chore, you'll hit the ceiling a lot faster than if you had done it in Java, which might in turn be hard to push as far as something like C# (largely comparable to Java, but some platform features make it easier to be frugal with memory and mind your cache lines) or, dare I say it, something native like ocaml or C++. Alternatively, if you start right off with Spark, you won't be able to push even one node as far as if you hadn't, because Spark is designed from the ground up for running on a cluster, and therefore has a tendency to manage memory the same way a 22-year-old professional basketball player handles money. It makes scale-out something of a self-fulfilling prophecy. Also, as someone who was doing distributed data processing pipelines well before Hadoop and friends came along, I'm not sure I can swallow "big data" being one and the same as "handling data that is too big to run on one computer." Big data sort of implies a certain culture of handling data at that scale, too. Because of that, I tend to think of "big data" as describing a culture as much as it describes anything practical. It's a set (not the only set) of technologies for procesing data on multiple machines. Whether you actually need multiple machines to do the job seems to be less relevant than the marketing team at IBM (to pick an easy punching back) would have us believe.

pletnes 7y ago

Unlike most big data solutions, dask is just so convenient. I sometimes use it just to concatenate a bunch of files into a dataframe. And yet, it scales up to really large datasets, too.

omgtehlion 7y ago

768G is not current max. Recently bought a machine (for less than $3k) for a side project, it has 768G inside and is 4 years old. Today most commodity servers (which you can buy out of pocket without lengthy preorder) can accomodate from 1.5 to 3 TiB of memory.