5 ms·

I'm beginning to think that "big data" is the most unfortunate possible term. By any reasonable definition of 'big' even a gigabyte or two counts as big. And

I'm beginning to think that "big data" is the most unfortunate possible term. By any reasonable definition of 'big' even a gigabyte or two counts as big. And since 32bit isn't quite one for the history books just yet, many of us still have 2-4GB fresh in our heads as a threshold where you might have to start thinking about the data needing special treatment.

mfisher87 12y ago
There have been conversations in my job where I refer to a 40GB file as "big" and someone else will go "that's nothing." Well, obviously there are bigger files somewhere, but that doesn't mean 40GB is nothing. "Big" is always a relative term. Anything is big in the right perspective, so to say that something "isn't big" is almost always wrong. We should really try to find a term with a more absolute meaning.
- andybak 12y ago
  I think a rough definition might be something like "datasets big enough to cause problems with standard tools on standard hardware" - i.e. where you need to change your tools to get things done. Still pretty vague but it captures what I think many people are getting at.
  - blumkvist 12y ago
    Big data does not refer only to the size of the dataset. It's variance and velocity, in addition to volume.
    - jghn 12y ago
      I was going to say something similar except for me it's a function of size and multidimensionality. We're sitting on several PB but it's generally in two dimensions - standard analysis techniques work, we just have problem moving the data around efficiently and processing it. I like your inclusion of velocity, I'm including that in my definition.
vonmoltke 12y ago
Not only that, most users of "big data" make assumptions about what the underlying dataset is. Not all large datasets are unstructured text you are trying to run information extraction on.
- bunderbunder 12y ago
  Indeed. It's amazing how impressive you can look rescuing someone from a "big data" quagmire by just loading that CSV file into a halfway decent RDBMS and letting a simple SELECT statement spin away from a couple tens of μs. Not that I've got anything against container ships, mind you, it's just that I find they're a rather unwieldy vehicle to use for getting groceries home from the market.
gaius 12y ago
5 years ago at least, I was dealing with 50Tb databases on hundreds of Tb of physical storage. We didn't call it big data, it was just data, we didn't call ourselves data scientists, we were just database guys, DBAs and devs. Nowadays a million rows is "big data" and only a "data scientist" can handle it. These guys are a joke.

mfisher87 12y ago

There have been conversations in my job where I refer to a 40GB file as "big" and someone else will go "that's nothing." Well, obviously there are bigger files somewhere, but that doesn't mean 40GB is nothing. "Big" is always a relative term. Anything is big in the right perspective, so to say that something "isn't big" is almost always wrong. We should really try to find a term with a more absolute meaning.

andybak 12y ago
I think a rough definition might be something like "datasets big enough to cause problems with standard tools on standard hardware" - i.e. where you need to change your tools to get things done. Still pretty vague but it captures what I think many people are getting at.
- blumkvist 12y ago
  Big data does not refer only to the size of the dataset. It's variance and velocity, in addition to volume.
  - jghn 12y ago
    I was going to say something similar except for me it's a function of size and multidimensionality. We're sitting on several PB but it's generally in two dimensions - standard analysis techniques work, we just have problem moving the data around efficiently and processing it. I like your inclusion of velocity, I'm including that in my definition.

vonmoltke 12y ago

Not only that, most users of "big data" make assumptions about what the underlying dataset is. Not all large datasets are unstructured text you are trying to run information extraction on.

bunderbunder 12y ago
Indeed. It's amazing how impressive you can look rescuing someone from a "big data" quagmire by just loading that CSV file into a halfway decent RDBMS and letting a simple SELECT statement spin away from a couple tens of μs. Not that I've got anything against container ships, mind you, it's just that I find they're a rather unwieldy vehicle to use for getting groceries home from the market.

gaius 12y ago

5 years ago at least, I was dealing with 50Tb databases on hundreds of Tb of physical storage. We didn't call it big data, it was just data, we didn't call ourselves data scientists, we were just database guys, DBAs and devs. Nowadays a million rows is "big data" and only a "data scientist" can handle it. These guys are a joke.