4 ms·

if your data set can be disposed by an awk script, it should not be called “big data”. Why not? I don't see how awk is limited to a certain amount of data.

gsich 8y ago
If your data fits on a single harddrive it's not big data. So I would set the current limit to at least 14 TB.
- rzzzt 8y ago
  Medium data. Substantial data. Just-enough-data.
  - anon1253 8y ago
    My favorite term is "annoying sized" data, not enough to warrant clusters and HPC, but enough to make a decent laptop crawl to a halt. It's that uncomfortable in-between that makes up the bulk of the data I usually encounter.
    - icelancer 8y ago
      This is pretty much where we are at, especially with a lot of transforms and processing on high-speed video captures.
    - zokier 8y ago
      The one nice thing about AWS and its ilk is the ability to spin up big chunky VM for few hours/days for ad-hoc data processing for pretty small amount of money. That can in many cases shift the boundary of annoying sized enough, with the added bonus that your own workstation is not fully occupied by the processing. Of course such approach is not applicable to all scenarios, but it is a useful trick to keep in your sleeve.
      - marmaduke 8y ago
        A while back on RDS we did our migrations by standing up a super fat instance, fail over, migrate, fail back and shutdown the fat instance. This was a great asset, and AWS made this easy.
- gizmo686 8y ago
  I thought the boundary point was RAM. It is relativly simple to work with data across multiple drives. When you pass the boundary of being able to work in a single systems RAM, you genneally need a more significant rework
  - Aeolun 8y ago
    Most stream processing doesn’t rely too much on RAM, unless you literally need all the data in memory at the same time.
    - groestl 8y ago
      If it _needs_ to be in RAM, then either you got a big enough machine (then by definition it's not Big Data) or it's impossible. If you manage to come by with RAM using smart algorithms, although the full dataset would never fit in RAM, then it's Big Data. So I'd argue, stream processing is Big Data, exactly because it doesn't rely too much on RAM.
      - Aeolun 8y ago
        True, but I can stream process 20GB of data on my tiny 2GB RAM home server as well. That’s not really ‘big data’ in my opinion.
  - gsich 8y ago
    That is harder to define. Server mainboards can hold more RAM then consumer mainboards. So with 32GB per slot and 4 slots I would set a limit to 128GB? Also this would make so much more tasks "big data". Games with 50+GB are not big data, neither is e.g video conversion.
    - groestl 8y ago
      I think the limit is more in the double digits TB range right now.
      - 8y ago
- Aeolun 8y ago
  You can still connect a fairly big NAS to a beefy server and do the processing, unless thoughput rate becomes an issue. Saturating a 10Gbit link means you can probably process up to 100TB a day.
- geocar 8y ago
  One time I met a company who insisted they were sending tens of TB of data per day and would need multi-PB per year storage compressed. Took one look at the data: All json, all GUIDS and bools. If we just pre-parse it, the entire dataset for a year fits in a few 100s GB uncompressed -- literally could fit on a macbook air for most of the year. The funny thing about "big data" in my experience, is just how small it actually becomes when you start using the right tools. And yet so much energy goes into just getting the wrong tools to do more...
  - dredmorbius 8y ago
    All of the source text for all of Google+ Communities posts is a few hundred GB. This for 8.1 million communities and ~10 million active users. Add in images and the rest of the Web payload (800 KiB per page), and that swells to Petabyte range. But the actual scale of the user-entered text is stunningly small.
  - tetha 8y ago
    > The funny thing about "big data" in my experience, is just how small it actually becomes when you start using the right tools. Rings way too true for me atm. My current workplace is currently struggling, because one of our application stores something like a combined 300G of analytics data in the database with the application data. Modifying the table causes hours of downtime because everyone claims that backwards compatible db changes are too hard. And everyone is scared because with more users there's "so much more analytics data" incoming. Yes, with 300Gb across 3-4 years. And I'm just wondering why it's not an option to just move all of that into one decently sized mysql/postgres instance. Give it SSDs, 30 - 60Gb of ram for the hot dataset (1-2 month) and it'll just solve our problems. But apparently, "that's too hard to do and takes too much time" without further reason.

gsich 8y ago

If your data fits on a single harddrive it's not big data. So I would set the current limit to at least 14 TB.

rzzzt 8y ago
Medium data. Substantial data. Just-enough-data.
- anon1253 8y ago
  My favorite term is "annoying sized" data, not enough to warrant clusters and HPC, but enough to make a decent laptop crawl to a halt. It's that uncomfortable in-between that makes up the bulk of the data I usually encounter.
  - icelancer 8y ago
    This is pretty much where we are at, especially with a lot of transforms and processing on high-speed video captures.
  - zokier 8y ago
    The one nice thing about AWS and its ilk is the ability to spin up big chunky VM for few hours/days for ad-hoc data processing for pretty small amount of money. That can in many cases shift the boundary of annoying sized enough, with the added bonus that your own workstation is not fully occupied by the processing. Of course such approach is not applicable to all scenarios, but it is a useful trick to keep in your sleeve.
    - marmaduke 8y ago
      A while back on RDS we did our migrations by standing up a super fat instance, fail over, migrate, fail back and shutdown the fat instance. This was a great asset, and AWS made this easy.
gizmo686 8y ago
I thought the boundary point was RAM. It is relativly simple to work with data across multiple drives. When you pass the boundary of being able to work in a single systems RAM, you genneally need a more significant rework
- Aeolun 8y ago
  Most stream processing doesn’t rely too much on RAM, unless you literally need all the data in memory at the same time.
  - groestl 8y ago
    If it _needs_ to be in RAM, then either you got a big enough machine (then by definition it's not Big Data) or it's impossible. If you manage to come by with RAM using smart algorithms, although the full dataset would never fit in RAM, then it's Big Data. So I'd argue, stream processing is Big Data, exactly because it doesn't rely too much on RAM.
    - Aeolun 8y ago
      True, but I can stream process 20GB of data on my tiny 2GB RAM home server as well. That’s not really ‘big data’ in my opinion.
- gsich 8y ago
  That is harder to define. Server mainboards can hold more RAM then consumer mainboards. So with 32GB per slot and 4 slots I would set a limit to 128GB? Also this would make so much more tasks "big data". Games with 50+GB are not big data, neither is e.g video conversion.
  - groestl 8y ago
    I think the limit is more in the double digits TB range right now.
    - 8y ago
Aeolun 8y ago
You can still connect a fairly big NAS to a beefy server and do the processing, unless thoughput rate becomes an issue. Saturating a 10Gbit link means you can probably process up to 100TB a day.
geocar 8y ago
One time I met a company who insisted they were sending tens of TB of data per day and would need multi-PB per year storage compressed. Took one look at the data: All json, all GUIDS and bools. If we just pre-parse it, the entire dataset for a year fits in a few 100s GB uncompressed -- literally could fit on a macbook air for most of the year. The funny thing about "big data" in my experience, is just how small it actually becomes when you start using the right tools. And yet so much energy goes into just getting the wrong tools to do more...
- dredmorbius 8y ago
  All of the source text for all of Google+ Communities posts is a few hundred GB. This for 8.1 million communities and ~10 million active users. Add in images and the rest of the Web payload (800 KiB per page), and that swells to Petabyte range. But the actual scale of the user-entered text is stunningly small.
- tetha 8y ago
  > The funny thing about "big data" in my experience, is just how small it actually becomes when you start using the right tools. Rings way too true for me atm. My current workplace is currently struggling, because one of our application stores something like a combined 300G of analytics data in the database with the application data. Modifying the table causes hours of downtime because everyone claims that backwards compatible db changes are too hard. And everyone is scared because with more users there's "so much more analytics data" incoming. Yes, with 300Gb across 3-4 years. And I'm just wondering why it's not an option to just move all of that into one decently sized mysql/postgres instance. Give it SSDs, 30 - 60Gb of ram for the hot dataset (1-2 month) and it'll just solve our problems. But apparently, "that's too hard to do and takes too much time" without further reason.