4 ms·
At some point one has to ask, what are you guys doing ingesting 5pb of data per day?! Unless this is google, that doesn’t make any sense. That’s an average of
by d--b 5y ago
At some point one has to ask, what are you guys doing ingesting 5pb of data per day?!
Unless this is google, that doesn’t make any sense. That’s an average of 7.5mb per human on the planet, every day.
- fn1 5y agoWhy the concept "data lake" emerged in the first place: Thinking about what to store (and what to log) is not trivial and takes careful consideration. Plus, there's always the argument: "But what if we need something that we forgot to store or log?". Answering that takes time and risk-acceptance that most developers in most projects don't get. "Datalake" was just a pseudo-solution to gain peace-of-mind: We throw everything into a big bucket and figure it out later.
- ricardobeat 5y agoWorked in a similar environment. Events and logs. When a single page view does 200+ database queries, triggers another hundred requests, ML services, analytics and other tracking, and a transaction kicks off a chain of a hundred events in your stream, it's pretty easy to reach those insane numbers. That one page view can easily add 1MB of data. Just like performance optimization, it's cheaper to buy more hardware than to pay for humans to think about it and coordinate.
- kevinsundar 5y agoI did say FAANG :) Believe it or not this is just for security data. But that fact combined with SOA leads to lots of logs. (5Pbs is the uncompressed amount, but we decompress incoming data, then ETL).
- echion 5y agoWould love to know some of the answers to the questions you posed