4 ms·
If that “data lake” is going to be less than 100 GB, all these people are idiots.
by nynx 3y ago
If that “data lake” is going to be less than 100 GB, all these people are idiots.
- replygirl 3y agoThis sounds arbitrary. I wouldn't set up a lake just to generate reports, but why 100 GB?
- manquer 3y agoyou could load that size in memory these days in a consumer machine let alone a server . AWS these days have 24TB RAM servers available on demand . 100 GB data in even dozen different formats require just regular database . A data lake just for generating reports on 100 GB of data which are going to be used by very few and built by even fewer seems overkill .
- deleted 3y ago[deleted]
- yamtaddle 3y agoOK I'm glad I wasn't the only one reading this going, "wait, this data lake thing does seem like wild overkill"
- mcphage 3y agoGiven the "data lake" ended up being only 4 weeks of work, that might be a slight overstatement of what was being designed.
- manquer 3y ago100GB ? You could store that in RAM easily these days . I would say no for anything less then 50-100TB today. We do have to keep in mind she was fired in 2018 and this was probably few years before that . Keeping 2015 as the reference frame a bare of minimum of few TBs to consider any lakes or puddles