3 ms·

More succinct version of the same from Gary Bernhardt of WAT fame (from 2015, same era) https://twitter.com/garybernhardt/status/600783770925420546 https://twit

by chx 3y ago

More succinct version of the same from Gary Bernhardt of WAT fame (from 2015, same era) https://twitter.com/garybernhardt/status/600783770925420546 https://twitter.com/garybernhardt/status/600783770925420546 > Consulting service: you bring your big data problems to me, I say "your data set fits in RAM", you pay me $10,000 for saving you $500,000.

CodesInChaos 3y ago
Reminds me of one of my favourite twitter posts: > Small Data is when is fit in RAM. Big Data is when is crash because is not fit in RAM. https://twitter.com/DEVOPS_BORAT/status/299176203691098112 https://twitter.com/DEVOPS_BORAT/status/299176203691098112
- tetha 3y ago
  DEVOPS_BORAT contains a lot of truth if you think about it, hah. Our sarcastic team-motto is very much this: https://twitter.com/DEVOPS_BORAT/status/41587168870797312 https://twitter.com/DEVOPS_BORAT/status/41587168870797312
h4kor 3y ago
https://yourdatafitsinram.net/ https://yourdatafitsinram.net/
- thelastgallon 3y ago
  I see servers with more RAM, 32 TB[1], 48TB RAM[2], probably more at: https://buy.hpe.com/us/en/compute/mission-critical-x86-servers/c/1010550750 https://buy.hpe.com/us/en/compute/mission-critical-x86-serve... [1]https://buy.hpe.com/us/en/compute/mission-critical-x86-servers/compute-scale-up-servers/compute-scale-up-servers/hpe-compute-scale-up-server-3200/p/1014774076 https://buy.hpe.com/us/en/compute/mission-critical-x86-serve... [2]https://buy.hpe.com/us/en/compute/mission-critical-x86-servers/integrity-mc990-x-servers/integrity-mc900-x-servers/hpe-integrity-mc990-x-server/p/1008798952 https://buy.hpe.com/us/en/compute/mission-critical-x86-serve...
  - chx 3y ago
    64TB max: IBM Power System E980 https://www.ibm.com/downloads/cas/VX0AM0EP https://www.ibm.com/downloads/cas/VX0AM0EP (notably the E880 did 32TB in 2014) SGI UV 300 https://www.uvhpc.com/sgi-uv-300 https://www.uvhpc.com/sgi-uv-300 SGI UV 3000 https://www.uvhpc.com/sgi-uv-3000 https://www.uvhpc.com/sgi-uv-3000
deleted 3y ago
[deleted]
attractivechaos 3y ago
Actually the awk solution in the blog post doesn’t load the entire dataset into memory. It is not limited by RAM. Even if you make the input 100x larger, mawk will still be hundreds of times faster than Hadoop. An important lesson here is streaming. In our field, we often process >100GB data in <1GB memory this way.
- tosh 3y ago
  This. For many analytical use cases the whole dataset doesn't have to fit into memory. Still: of course worthwhile to point out how oversized a compute cluster approach is when the whole dataset would actually fit into memory of a single machine.
deleted 3y ago
[deleted]

by chx 3y ago

CodesInChaos 3y ago

Reminds me of one of my favourite twitter posts: > Small Data is when is fit in RAM. Big Data is when is crash because is not fit in RAM. https://twitter.com/DEVOPS_BORAT/status/299176203691098112 https://twitter.com/DEVOPS_BORAT/status/299176203691098112

tetha 3y ago
DEVOPS_BORAT contains a lot of truth if you think about it, hah. Our sarcastic team-motto is very much this: https://twitter.com/DEVOPS_BORAT/status/41587168870797312 https://twitter.com/DEVOPS_BORAT/status/41587168870797312

h4kor 3y ago

https://yourdatafitsinram.net/ https://yourdatafitsinram.net/

thelastgallon 3y ago
I see servers with more RAM, 32 TB[1], 48TB RAM[2], probably more at: https://buy.hpe.com/us/en/compute/mission-critical-x86-servers/c/1010550750 https://buy.hpe.com/us/en/compute/mission-critical-x86-serve... [1]https://buy.hpe.com/us/en/compute/mission-critical-x86-servers/compute-scale-up-servers/compute-scale-up-servers/hpe-compute-scale-up-server-3200/p/1014774076 https://buy.hpe.com/us/en/compute/mission-critical-x86-serve... [2]https://buy.hpe.com/us/en/compute/mission-critical-x86-servers/integrity-mc990-x-servers/integrity-mc900-x-servers/hpe-integrity-mc990-x-server/p/1008798952 https://buy.hpe.com/us/en/compute/mission-critical-x86-serve...
- chx 3y ago
  64TB max: IBM Power System E980 https://www.ibm.com/downloads/cas/VX0AM0EP https://www.ibm.com/downloads/cas/VX0AM0EP (notably the E880 did 32TB in 2014) SGI UV 300 https://www.uvhpc.com/sgi-uv-300 https://www.uvhpc.com/sgi-uv-300 SGI UV 3000 https://www.uvhpc.com/sgi-uv-3000 https://www.uvhpc.com/sgi-uv-3000

deleted 3y ago

[deleted]

attractivechaos 3y ago

Actually the awk solution in the blog post doesn’t load the entire dataset into memory. It is not limited by RAM. Even if you make the input 100x larger, mawk will still be hundreds of times faster than Hadoop. An important lesson here is streaming. In our field, we often process >100GB data in <1GB memory this way.

tosh 3y ago
This. For many analytical use cases the whole dataset doesn't have to fit into memory. Still: of course worthwhile to point out how oversized a compute cluster approach is when the whole dataset would actually fit into memory of a single machine.

deleted 3y ago

[deleted]