2 ms·
Ballpark numbers - an employee typing at 100wpm for 40 hours a week generates not quite 70MB worth of keystrokes in a year - and I'd guess that estimate is prob
by bigiain 13y ago
Ballpark numbers - an employee typing at 100wpm for 40 hours a week generates not quite 70MB worth of keystrokes in a year - and I'd guess that estimate is probably something like 2 orders of magnitude too high - no-one actually types 100wpm nonstop all day every single day at work. Even ignoring that, I've got enough disk space sitting under my TV to store (uncompressed) every keystroke typed in a year by something like 100,000 such mythical "100wpm typists working at 100% utilisation" employees. 1PB would, to a first approximation store every single keypress it'd be possible for the rumored 4 million "top secret or above clearance" people in the US typing flat out for something like 15 years.
So I don't think your "do you have the infrastructure" argument holds too much water here.
The "do you have internal Hadoop clusters" question falls the same way. Sure, I don't have that connected to my 8TB of external USB drives plugged into my media server - but if I worked at Google or FaceBook or Twitter, I'd fully expect to be able to provision and spin up adequately sized clusters of VMs and storage to effectively consume and run reports on data that size and bigger. And it's surely not just the NSA and Google/Facebook/Twitter routinely dealing with collecting and processing data at that scale - any decent sized telco, any non-trivial web analytics service, any large financial institution, every HFT business, most bio-med businesses, probably every physics and astronomy department at any university – there must be tens of thousands of businesses routinely dealing with that sort of sized data sets.
It's not simple - certainly not simple enough for me to do it on a Mac Mini and a bunch of external hard drives – but I also don't think it's anything like "uncharted waters" territory. (I'm pretty sure I could find the expertise required in my 1st level LinkedIn connections, and have absolutely no doubt I'd be able to manage designing, developing and deploying exactly such a system if someone came to me with a high six or low seven figure budget.)
- mirkules 13y agoI think your approximations are focusing solely on keystrokes, whereas the parent specified just "data", which makes me believe that network traffic, application logs, etc are included in this. I can believe the 1PB/day number, it's not that far fetched when you consider the above. I also agree with the parent. There are so many possible scenarios and things to log that eventually you're playing a "logging" version of whack-a-mole. Even just managing these files (as the parent talks about) is really no trivial task. Honestly, I wouldn't even know how to begin managing a petabyte worth of daily data.