4 ms·

I thought the boundary point was RAM. It is relativly simple to work with data across multiple drives. When you pass the boundary of being able to work in a sin

I thought the boundary point was RAM. It is relativly simple to work with data across multiple drives. When you pass the boundary of being able to work in a single systems RAM, you genneally need a more significant rework

Aeolun 8y ago
Most stream processing doesn’t rely too much on RAM, unless you literally need all the data in memory at the same time.
- groestl 8y ago
  If it _needs_ to be in RAM, then either you got a big enough machine (then by definition it's not Big Data) or it's impossible. If you manage to come by with RAM using smart algorithms, although the full dataset would never fit in RAM, then it's Big Data. So I'd argue, stream processing is Big Data, exactly because it doesn't rely too much on RAM.
  - Aeolun 8y ago
    True, but I can stream process 20GB of data on my tiny 2GB RAM home server as well. That’s not really ‘big data’ in my opinion.
gsich 8y ago
That is harder to define. Server mainboards can hold more RAM then consumer mainboards. So with 32GB per slot and 4 slots I would set a limit to 128GB? Also this would make so much more tasks "big data". Games with 50+GB are not big data, neither is e.g video conversion.
- groestl 8y ago
  I think the limit is more in the double digits TB range right now.
  - gsich 8y ago
    See? As I said this not easily definable. A single HDD is though. What you could do is a single RAM module.
    - groestl 8y ago
      Idk, it's fuzzy, obviously, and a moving goalpost, but in the end it comes down to this: do I buy/rent a bigger server? Or do I put more engineers to the task of replacing the naive algorithm. The latter is Big Data and usually the case when you run out of RAM, not out of disk. And the first one is an underrated option.
twic 8y ago
"Big Data is any thing which is crash Excel." https://twitter.com/devops_borat/status/288698056470315008?lang=en https://twitter.com/devops_borat/status/288698056470315008?l...
mpweiher 8y ago
"EC2 High Memory instances offer 6 TB, 9 TB, and 12 TB of memory in an instance. " https://aws.amazon.com/ec2/instance-types/high-memory/ https://aws.amazon.com/ec2/instance-types/high-memory/

Aeolun 8y ago

Most stream processing doesn’t rely too much on RAM, unless you literally need all the data in memory at the same time.

groestl 8y ago
If it _needs_ to be in RAM, then either you got a big enough machine (then by definition it's not Big Data) or it's impossible. If you manage to come by with RAM using smart algorithms, although the full dataset would never fit in RAM, then it's Big Data. So I'd argue, stream processing is Big Data, exactly because it doesn't rely too much on RAM.
- Aeolun 8y ago
  True, but I can stream process 20GB of data on my tiny 2GB RAM home server as well. That’s not really ‘big data’ in my opinion.

gsich 8y ago

That is harder to define. Server mainboards can hold more RAM then consumer mainboards. So with 32GB per slot and 4 slots I would set a limit to 128GB? Also this would make so much more tasks "big data". Games with 50+GB are not big data, neither is e.g video conversion.

groestl 8y ago
I think the limit is more in the double digits TB range right now.
- gsich 8y ago
  See? As I said this not easily definable. A single HDD is though. What you could do is a single RAM module.
  - groestl 8y ago
    Idk, it's fuzzy, obviously, and a moving goalpost, but in the end it comes down to this: do I buy/rent a bigger server? Or do I put more engineers to the task of replacing the naive algorithm. The latter is Big Data and usually the case when you run out of RAM, not out of disk. And the first one is an underrated option.

twic 8y ago

"Big Data is any thing which is crash Excel." https://twitter.com/devops_borat/status/288698056470315008?lang=en https://twitter.com/devops_borat/status/288698056470315008?l...

mpweiher 8y ago

"EC2 High Memory instances offer 6 TB, 9 TB, and 12 TB of memory in an instance. " https://aws.amazon.com/ec2/instance-types/high-memory/ https://aws.amazon.com/ec2/instance-types/high-memory/