4 ms·

There is no point running something like Hadoop or Spark or Flink if your problem is not bottlenecked by the resources of of a single machine. E.g. how would t

There is no point running something like Hadoop or Spark or Flink if your problem is not bottlenecked by the resources of of a single machine. E.g. how would this solution work if the data is so large or the problem requires more context (looking at more than one line at the time) and therefore more memory? Answer: it wouldn't .

Cthulhu_ 5y ago
That's it, for a long time, what a lot of companies that jumped on the big data bandwagon didn't realize is that they didn't actually have big data. I'm sure there's some arbitrary lines that can be drawn, but anything under 1 TB is not big data anymore and can be processed on a single machine. Other things to consider is data volume though. A popular use case of hadoop was to take e.g. server access logs - high volume data that traditionally you wouldn't hold onto for long - and get something meaningful out of that.
- rizkeyz 5y ago
  At this point in time (2022) I consider everything below say 40TB not big (textual) data at all. It can be compressed 40TB -> 10TB (or less) and that fits fine on a single 16T drive. For many questions, you won't need all the raw data, so you end up with some form of projection of the data that is maybe 1/10 in size, so 10TB -> 1TB. Heck, if you tune GNU sort a bit, it will blast through that TB quite quickly.
  - trashtester 5y ago
    If you just want cold storage, you can put 10TB of compressed textual data on a spinning hard drive. If want to run some processing of that data within a workday, you need multiple drives in parallel (still possible on a single server). However, if you want to process it in less than 30 mins you need a cluster.
  - anonymousDan 5y ago
    The problem isn't necessarily (just) capacity. It's also the he I/O bandwidth needed to read the data.
- NateEag 5y ago
  My favorite litmus test for "Big Data" is https://yourdatafitsinram.net https://yourdatafitsinram.net. If you can fit it into RAM on a single machine, you probably shouldn't be using complex distributed systems for working with it. The developer time spent setting it up and fixing arcane bugs due to the distributed system is likely to cost you more than a single monster server will. If your processes generate large amounts of data rapidly, you can get a decent idea when you should start working on a move to a "big data" solution by looking at your rate of growth. ...though in my experience, stuff like logs can be tarballed and tossed on S3 as insurance just in case you really do need to get an overview of some pattern over the past three years. Mostly only about the past six months of logs are really worth keeping on hand actively, IMO.

Cthulhu_ 5y ago

That's it, for a long time, what a lot of companies that jumped on the big data bandwagon didn't realize is that they didn't actually have big data. I'm sure there's some arbitrary lines that can be drawn, but anything under 1 TB is not big data anymore and can be processed on a single machine. Other things to consider is data volume though. A popular use case of hadoop was to take e.g. server access logs - high volume data that traditionally you wouldn't hold onto for long - and get something meaningful out of that.

rizkeyz 5y ago
At this point in time (2022) I consider everything below say 40TB not big (textual) data at all. It can be compressed 40TB -> 10TB (or less) and that fits fine on a single 16T drive. For many questions, you won't need all the raw data, so you end up with some form of projection of the data that is maybe 1/10 in size, so 10TB -> 1TB. Heck, if you tune GNU sort a bit, it will blast through that TB quite quickly.
- trashtester 5y ago
  If you just want cold storage, you can put 10TB of compressed textual data on a spinning hard drive. If want to run some processing of that data within a workday, you need multiple drives in parallel (still possible on a single server). However, if you want to process it in less than 30 mins you need a cluster.
- anonymousDan 5y ago
  The problem isn't necessarily (just) capacity. It's also the he I/O bandwidth needed to read the data.
NateEag 5y ago
My favorite litmus test for "Big Data" is https://yourdatafitsinram.net https://yourdatafitsinram.net. If you can fit it into RAM on a single machine, you probably shouldn't be using complex distributed systems for working with it. The developer time spent setting it up and fixing arcane bugs due to the distributed system is likely to cost you more than a single monster server will. If your processes generate large amounts of data rapidly, you can get a decent idea when you should start working on a move to a "big data" solution by looking at your rate of growth. ...though in my experience, stuff like logs can be tarballed and tossed on S3 as insurance just in case you really do need to get an overview of some pattern over the past three years. Mostly only about the past six months of logs are really worth keeping on hand actively, IMO.