4 ms·

Straight up, what's people's views on all this big data stuff? I see it in so many job ads and I really can't believe it's necessary. Sure, at the far end of on

Straight up, what's people's views on all this big data stuff? I see it in so many job ads and I really can't believe it's necessary. Sure, at the far end of one side of the bell curve are companies like uber but otherwise, are people using it to process a few terabytes that could be done better on one multicore server? How many companies have enough data to justify it? Personal opinions welcome.

royjacobs 6y ago
I've found that a lot of big data solutions in use by companies are only in use because business requirements were "we want all the data and we'll figure out what to do with it later", but then it turns out only a fraction of the data is used or only in a heavily aggregated form. That makes sense, of course, nobody is going to manually go through a few terabytes worth of records manually.
trollied 6y ago
Traditional SQL-based data warehouses still rule the world, and probably will for the foreseeable. HN reports on a small number of businesses that use bleeding-edge technologies (often just for the sake of it). The vast majority of businesses stick to the "traditional" way of doing things. At the end of the day, most places just want something to point PowerBI at etc.
cyberdrunk 6y ago
What I've seen is that thousands of jobs for dozens of team are run on company's Hadoop cluster. Sure, each team could provision their own custom infra and run the job there, but having a centralized way to do it, with all the extra niceties (scalable capacity, good monitoring, logging, HA), can provide some company-wide efficiencies. Plus, some of the jobs can in fact be huge and you may need dozens of nodes to process them (we have such jobs in a bank, where we don't really have big data) - doing it without a cluster would be problematic.
- throwaway_pdp09 6y ago
  Sounds like you're one who might actually need it.
cmrdporcupine 6y ago
"Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it. " https://www.quora.com/Big-data-is-like-teenage-sex-everyone-talks-about-it-nobody-really-knows-how-to-do-it-everyone-thinks-everyone-else-is-doing-it-so-everyone-claims-they-are-doing-it-What-is-your-opinion-about-this-statement https://www.quora.com/Big-data-is-like-teenage-sex-everyone-...
- devonkim 6y ago
  It’s the same with almost all hyped up technology or movements trying to push stuff into the C suite for conversations.
zten 6y ago
Answer the question about how to process a few TB on a multicore server, and you might still find yourself using Spark, or something like it. If you start from the assumption that you've been ingesting data and storing it in a compressed columnar format like Parquet or ORC, then you're already locked into a solution that exists in the Hadoop ecosystem. This turns out to be an effective way to deal with terabytes of data because depending on the query you want to execute, the file format (and a smart partitioning structure) helps turn your problem of reading terabytes into one of reading gigabytes. And, everything in Hadoop land is generally a query, so you're going to need a query engine like Hive, Impala, Spark, etc. AND, you probably need something like Hive metadata so that you're not crawling some directory structure for the schema and input files every time you start up a new process to run a query. You _could_ write something on your own that just forks out a bunch of threads in a single process to rip through the data, but why? Think about what you're effectively implementing - a bespoke query engine that runs one query plan. Spark has already written an API and a query engine (multi-process distributed, unlike whatever you're likely to hand-roll) and lots of input/output code. You can spark-submit --master local[*] and use up all of the cores and ram everything into one monster JVM if you really wanted to. Finally, you could stuff all of this in memory, but at what cost, and what are you going to use to do it? (Worse, what happens if the VM goes down - those terabytes are going to take their sweet time reloading over a network link.)
- throwaway_pdp09 6y ago
  How; plenty of disks to give you the IO. It's usually about the IO. If you want memory, go to a server site and configure to max out a server with lots of DRAM slots. You might be surprised. But really getting enough mem is less important than IO, so stick with disks, SSDs I suppose. You can get single chip with dozens of cores for not too much. I guess that's how I'd do it. > This turns out to be an effective way to deal with terabytes of data A few terabytes don't need cluster, typically. > You _could_ write something on your own that just forks out a bunch of threads in a single process to rip through the data, but why? Because it's simple and easy. I wrote one in a few days. Not much code. > You can spark-submit --master local[] and use up all of the cores and ram everything into one monster JVM if you really wanted to. Point is, do you need to? > those terabytes are going to take their sweet time reloading over a network link You do not run big IO over a network like that if you can avoid it. With a single server with plenty of SSDs, you trivially don't. My take anyway.
StreamBright 6y ago
Exactly. Btw. the other end of companies do not use Hadoop either. S3 + Presto is super popular nowadays in case of cloud.

royjacobs 6y ago

I've found that a lot of big data solutions in use by companies are only in use because business requirements were "we want all the data and we'll figure out what to do with it later", but then it turns out only a fraction of the data is used or only in a heavily aggregated form. That makes sense, of course, nobody is going to manually go through a few terabytes worth of records manually.

trollied 6y ago

Traditional SQL-based data warehouses still rule the world, and probably will for the foreseeable. HN reports on a small number of businesses that use bleeding-edge technologies (often just for the sake of it). The vast majority of businesses stick to the "traditional" way of doing things. At the end of the day, most places just want something to point PowerBI at etc.

cyberdrunk 6y ago

What I've seen is that thousands of jobs for dozens of team are run on company's Hadoop cluster. Sure, each team could provision their own custom infra and run the job there, but having a centralized way to do it, with all the extra niceties (scalable capacity, good monitoring, logging, HA), can provide some company-wide efficiencies. Plus, some of the jobs can in fact be huge and you may need dozens of nodes to process them (we have such jobs in a bank, where we don't really have big data) - doing it without a cluster would be problematic.

throwaway_pdp09 6y ago
Sounds like you're one who might actually need it.

cmrdporcupine 6y ago

"Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it. " https://www.quora.com/Big-data-is-like-teenage-sex-everyone-talks-about-it-nobody-really-knows-how-to-do-it-everyone-thinks-everyone-else-is-doing-it-so-everyone-claims-they-are-doing-it-What-is-your-opinion-about-this-statement https://www.quora.com/Big-data-is-like-teenage-sex-everyone-...

devonkim 6y ago
It’s the same with almost all hyped up technology or movements trying to push stuff into the C suite for conversations.

zten 6y ago

Answer the question about how to process a few TB on a multicore server, and you might still find yourself using Spark, or something like it. If you start from the assumption that you've been ingesting data and storing it in a compressed columnar format like Parquet or ORC, then you're already locked into a solution that exists in the Hadoop ecosystem. This turns out to be an effective way to deal with terabytes of data because depending on the query you want to execute, the file format (and a smart partitioning structure) helps turn your problem of reading terabytes into one of reading gigabytes. And, everything in Hadoop land is generally a query, so you're going to need a query engine like Hive, Impala, Spark, etc. AND, you probably need something like Hive metadata so that you're not crawling some directory structure for the schema and input files every time you start up a new process to run a query. You _could_ write something on your own that just forks out a bunch of threads in a single process to rip through the data, but why? Think about what you're effectively implementing - a bespoke query engine that runs one query plan. Spark has already written an API and a query engine (multi-process distributed, unlike whatever you're likely to hand-roll) and lots of input/output code. You can spark-submit --master local[*] and use up all of the cores and ram everything into one monster JVM if you really wanted to. Finally, you could stuff all of this in memory, but at what cost, and what are you going to use to do it? (Worse, what happens if the VM goes down - those terabytes are going to take their sweet time reloading over a network link.)

throwaway_pdp09 6y ago
How; plenty of disks to give you the IO. It's usually about the IO. If you want memory, go to a server site and configure to max out a server with lots of DRAM slots. You might be surprised. But really getting enough mem is less important than IO, so stick with disks, SSDs I suppose. You can get single chip with dozens of cores for not too much. I guess that's how I'd do it. > This turns out to be an effective way to deal with terabytes of data A few terabytes don't need cluster, typically. > You _could_ write something on your own that just forks out a bunch of threads in a single process to rip through the data, but why? Because it's simple and easy. I wrote one in a few days. Not much code. > You can spark-submit --master local[] and use up all of the cores and ram everything into one monster JVM if you really wanted to. Point is, do you need to? > those terabytes are going to take their sweet time reloading over a network link You do not run big IO over a network like that if you can avoid it. With a single server with plenty of SSDs, you trivially don't. My take anyway.

StreamBright 6y ago

Exactly. Btw. the other end of companies do not use Hadoop either. S3 + Presto is super popular nowadays in case of cloud.