4 ms·

At an absolute minimum, I'd say "big data" begins when you can't buy hardware with that much memory. Apparently, in 2017, that was somewhere around 16 terabyte

by Groxx 8y ago

At an absolute minimum, I'd say "big data" begins when you can't buy hardware with that much memory. Apparently, in 2017, that was somewhere around 16 terabytes. https://www.theregister.co.uk/2017/05/16/aws_ram_cram/ https://www.theregister.co.uk/2017/05/16/aws_ram_cram/ Heck, you can trivially get a 4TB instance from Amazon nowadays: https://aws.amazon.com/ec2/instance-types/ https://aws.amazon.com/ec2/instance-types/ The biggest DBs I've worked on have been a few tens of billions of rows, and several hundreds of gigabytes. That's like... nothing. A laughable start. You can make trivially inefficient mistakes and for the most part e.g. MySQL will still work fine. Absolutely nowhere near "big data". And odds are pretty good you could just toss existing more-than-4TB-data through grep and then into your DB and still ignore "big data" problems.

gaius 8y ago
10 years ago I was working on a 50Tb datawarehouse. Now I see people who think 50Gb is “big data” because Pandas on a laptop chokes on it.
- Groxx 8y ago
  Yeah, or R. When I need to do stuff bigger than RAM on a laptop I just reach for sqlite (assuming it can even fit on disk).
  - gaius 8y ago
    The funny thing is we didn’t even consider our 50Tb to be particularly big, we knew guys running 200Tb... BTW in R data.table is better for larger datasets than the default dataframe... or use the Revolution Analytics stuff on a server.
    - Groxx 8y ago
      Oh yeah, the default dataframe is practically only for toy problems / very-already-filtered data (which accounts for a LOT of useful scenarios!). It's more that I've run into "big data" people who (╯°□°)╯︵ ┻━┻ when the default settings + dataframe choke, which is around 200MB of heap.
- Swinx43 8y ago
  This, a million times this!!! I see people doing consulting promoting the most over engineered solutions I have ever seen. Doing it for data that is a few hundred GB or maybe at best a TB. It makes me want to cry, knowing we handled that with a single server and a relational database 10 years ago. Lets also not forget that everyone today forgets that the majority of data actually has some sort of structure. There is no point in pretending that every piece of data is a BLOB or a JSON document. I have given up on our industry ever becoming sane, I now fully expect each hype cycle to be pushed to the absolute maximum. Only to be replaced by the next buzzword cycle when the current starts failing to deliver on promises.
  - gaius 8y ago
    I have given up on our industry ever becoming sane, I now fully expect each hype cycle to be pushed to the absolute maximum. Only to be replaced by the next buzzword cycle when the current starts failing to deliver on promises I sometimes wish I was unscrupulous enough to cash in on these trends, I’d be a millionaire now. Instead I’m just a sucker who tries to make solid engineering decisions on behalf of my employers and clients. It’s depressing to think what professionalism has cost me in cash terms. But you’ve got to be able to look at yourself in the mirror.
  - therealdrag0 8y ago
    Yep. I left consulting after working on a project that was a Ferrari when the customer only needed a Honda. Our architect kept being like "it needs to be faster", and I'm like "our current rate would process the entire possible dataset (everyone on earth) in a night, do we really need a faster Apache Storm cluster?" :S "Resume driven development" is a great term.
  - attaboyjon 8y ago
    It's important to remember what drives this - employers often like to think their problems are 'big data' and by god, they need the over-engineered solution. Your peers who interview you will toss your resume in the trash if you are not buzzword compliant. Hate the game not the player.
- sgt101 8y ago
  There's a big difference between a 50Tb data warehouse and a data warehouse which has a number of 10Tb tables. Our data warehouse used to have 20k tables and 18Tb data. Our big data instance has 4k tables and 800Tb data
- geezerjay 8y ago
  > Now I see people who think 50Gb is “big data” because Pandas on a laptop chokes on it. IIRC the term "big data" was coined to refer to data volumes that are too large for exiting applications to process without having to rethink how the data and/or the application was deployed and organized. Thus, although the laptop reference is on point, large enough data volumes that make Pandas choke in a desktop environment does fit the definition of big data.
  - gaius 8y ago
    large enough data volumes that make Pandas choke in a desktop environment does fit the definition of big data What about too large for Excel, is that “big data” too? (Neither are)
  - samuel_ms 8y ago
    That definition might be a bit loose. If you have an existing application designed to run on a Raspberry Pi, it'll hit that limit pretty quickly. This is obviously an over-exaggeration, but I don't think a dataset that breaks Pandas in a desktop environment even comes close to big data.
phs318u 8y ago
15 years ago I was working at a utility and was re-architecting our meter-data solution as the state government was about to mandate a change from manually read meters to remotely read interval meters. Had to rescale the database from processing less than 2 million reads a year, to over 50 million reads _a day_ (for our roughly 1M meters). Needed to keep 16 months online, and several years worth in near-line storage. We went from a bog-standard Oracle database to Oracle Enterprise with partitions by month. This was being fed by an event-driven microservices[0] architecture of individually deployable Pro*C/C++ unix executables. The messaging was Oracle A/Q. At the time, I thought "wow - this is big!", and it was for me. [0] we didn't call it microservices back then though that's clearly what it was. The term I used was "event-driven, loosely coupled, transactional programs processing units of work comprised of a day's worth of meter readings per meter". Doesn't roll off the tongue quite so easily.
- opinionator1 8y ago
  You can say "composed of", or you can say "comprises" -- "comprised of" is always wrong.
sgt101 8y ago
Sorta, depends on your workload and if you are a solo hero dev or if you are an enterprise requiring shared data resources. For 16TB you'll definitely get benefits if you are doing something embarrassingly parallel and processor bound over a cluster of a handful of machines. It's just parallelism, and it's a good thing. I totally agree about the hundreds of Gigs - that is unless you are in a setting where many teams need to access that database and join it with others, in which case a proper data lake implemented on something beefy is a good idea. Hadoop has the benefit of distribution and replication, but another data warehouse might work better - say Oracle or Teradata if you are a small shop.
wilsonnb 8y ago
I'm pretty sure you can buy IBM's newest mainframe, the z14, with 32TB of memory. I don't think that mainframes are that popular in trendy "big data" though.
flal_ 8y ago
I'd say more than the boundary is not on a 1 dimensional axis but more on a size x time x [access - pattern ] space : 100Gb of data may not be that much to you, but it's another story if you have to process all of it every minute. Also everything is fine if you have nicely structured ( and thus indexed ... ) data, but I do have a few 100s Gb of images as well...

by Groxx 8y ago

gaius 8y ago

10 years ago I was working on a 50Tb datawarehouse. Now I see people who think 50Gb is “big data” because Pandas on a laptop chokes on it.

Groxx 8y ago
Yeah, or R. When I need to do stuff bigger than RAM on a laptop I just reach for sqlite (assuming it can even fit on disk).
- gaius 8y ago
  The funny thing is we didn’t even consider our 50Tb to be particularly big, we knew guys running 200Tb... BTW in R data.table is better for larger datasets than the default dataframe... or use the Revolution Analytics stuff on a server.
  - Groxx 8y ago
    Oh yeah, the default dataframe is practically only for toy problems / very-already-filtered data (which accounts for a LOT of useful scenarios!). It's more that I've run into "big data" people who (╯°□°)╯︵ ┻━┻ when the default settings + dataframe choke, which is around 200MB of heap.
Swinx43 8y ago
This, a million times this!!! I see people doing consulting promoting the most over engineered solutions I have ever seen. Doing it for data that is a few hundred GB or maybe at best a TB. It makes me want to cry, knowing we handled that with a single server and a relational database 10 years ago. Lets also not forget that everyone today forgets that the majority of data actually has some sort of structure. There is no point in pretending that every piece of data is a BLOB or a JSON document. I have given up on our industry ever becoming sane, I now fully expect each hype cycle to be pushed to the absolute maximum. Only to be replaced by the next buzzword cycle when the current starts failing to deliver on promises.
- gaius 8y ago
  I have given up on our industry ever becoming sane, I now fully expect each hype cycle to be pushed to the absolute maximum. Only to be replaced by the next buzzword cycle when the current starts failing to deliver on promises I sometimes wish I was unscrupulous enough to cash in on these trends, I’d be a millionaire now. Instead I’m just a sucker who tries to make solid engineering decisions on behalf of my employers and clients. It’s depressing to think what professionalism has cost me in cash terms. But you’ve got to be able to look at yourself in the mirror.
- therealdrag0 8y ago
  Yep. I left consulting after working on a project that was a Ferrari when the customer only needed a Honda. Our architect kept being like "it needs to be faster", and I'm like "our current rate would process the entire possible dataset (everyone on earth) in a night, do we really need a faster Apache Storm cluster?" :S "Resume driven development" is a great term.
- attaboyjon 8y ago
  It's important to remember what drives this - employers often like to think their problems are 'big data' and by god, they need the over-engineered solution. Your peers who interview you will toss your resume in the trash if you are not buzzword compliant. Hate the game not the player.
sgt101 8y ago
There's a big difference between a 50Tb data warehouse and a data warehouse which has a number of 10Tb tables. Our data warehouse used to have 20k tables and 18Tb data. Our big data instance has 4k tables and 800Tb data
geezerjay 8y ago
> Now I see people who think 50Gb is “big data” because Pandas on a laptop chokes on it. IIRC the term "big data" was coined to refer to data volumes that are too large for exiting applications to process without having to rethink how the data and/or the application was deployed and organized. Thus, although the laptop reference is on point, large enough data volumes that make Pandas choke in a desktop environment does fit the definition of big data.
- gaius 8y ago
  large enough data volumes that make Pandas choke in a desktop environment does fit the definition of big data What about too large for Excel, is that “big data” too? (Neither are)
- samuel_ms 8y ago
  That definition might be a bit loose. If you have an existing application designed to run on a Raspberry Pi, it'll hit that limit pretty quickly. This is obviously an over-exaggeration, but I don't think a dataset that breaks Pandas in a desktop environment even comes close to big data.

phs318u 8y ago

15 years ago I was working at a utility and was re-architecting our meter-data solution as the state government was about to mandate a change from manually read meters to remotely read interval meters. Had to rescale the database from processing less than 2 million reads a year, to over 50 million reads _a day_ (for our roughly 1M meters). Needed to keep 16 months online, and several years worth in near-line storage. We went from a bog-standard Oracle database to Oracle Enterprise with partitions by month. This was being fed by an event-driven microservices[0] architecture of individually deployable Pro*C/C++ unix executables. The messaging was Oracle A/Q. At the time, I thought "wow - this is big!", and it was for me. [0] we didn't call it microservices back then though that's clearly what it was. The term I used was "event-driven, loosely coupled, transactional programs processing units of work comprised of a day's worth of meter readings per meter". Doesn't roll off the tongue quite so easily.

opinionator1 8y ago
You can say "composed of", or you can say "comprises" -- "comprised of" is always wrong.

sgt101 8y ago

Sorta, depends on your workload and if you are a solo hero dev or if you are an enterprise requiring shared data resources. For 16TB you'll definitely get benefits if you are doing something embarrassingly parallel and processor bound over a cluster of a handful of machines. It's just parallelism, and it's a good thing. I totally agree about the hundreds of Gigs - that is unless you are in a setting where many teams need to access that database and join it with others, in which case a proper data lake implemented on something beefy is a good idea. Hadoop has the benefit of distribution and replication, but another data warehouse might work better - say Oracle or Teradata if you are a small shop.

wilsonnb 8y ago

I'm pretty sure you can buy IBM's newest mainframe, the z14, with 32TB of memory. I don't think that mainframes are that popular in trendy "big data" though.

flal_ 8y ago

I'd say more than the boundary is not on a 1 dimensional axis but more on a size x time x [access - pattern ] space : 100Gb of data may not be that much to you, but it's another story if you have to process all of it every minute. Also everything is fine if you have nicely structured ( and thus indexed ... ) data, but I do have a few 100s Gb of images as well...