3 ms·
Where are they getting "petabytes" of daily data from? Even if you took every single transaction anywhere near the industry, in all its details, and represented
by jfengel 6y ago
Where are they getting "petabytes" of daily data from? Even if you took every single transaction anywhere near the industry, in all its details, and represented it in a verbose format like JSON or XML, that's still only kilobytes of data times billions of transactions = terabytes.
If they were gathering tons of video, maybe. Or vast numbers of high-precision sensors generating data continuously. But I didn't see anything like that.
It's just hard for any human activity to generate that much data. Is "petabytes" just some kind of hype, or is there something I'm missing?
- YoungWeb 6y agoTo me it seems to be more of a volume data point rather than storage. So, it might only be terabytes of data, but for instance in ML algorithms, you could be running that data through the algorithm many times every two minutes. So by the end of the day, you've processed "petabytes" of data. This is just my interpretation of it. I could be wrong to.
- TrainedMonkey 6y agoMaybe data streams collected from phones of people ordering groceries and buyers going to stores could add up to petabytes.
- sn_master 6y agoA lot of big data datasets are de-normalized to make queries faster. Joining is very expensive. Source: Worked in big datasets at Bing for a couple years.
- hnpobdo 6y ago1 user searches for a product, and then clicks on 1: although you estimate this transaction generates 1kb of data, in fact this is more on the order of 1kb x the number of candidates, so a few MB.