8 ms·
Wow, 4.3 petabytes a day. Operating at Google's scale these days is just nuts.
by hn_throwaway_99 6y ago
Wow, 4.3 petabytes a day. Operating at Google's scale these days is just nuts.
- est31 6y agoOne of the advantages of being on an exponential growth curve is that this figure has been way lower in the past. Just go back 5 years and it's only a fraction of that. 5 more years and it's even lower. The gmail/drive/etc. data that come from Google's first decade of existence probably fit into those 4.3 petabytes.
- vageli 6y agoHow is this an advantage?
- est31 6y agoAdvantage in the sense that they have to store less. 4.3 petabytes times the age of Google in days is way way more than they actually have to store, as the time period when they were adding 4.3 petabytes daily is short in comparison.
- jstanley 6y agoSo the advantage of being on an exponential growth curve is that today's problems are tiny in comparison to tomorrow's problems? That seems like a disadvantage!
- sagarm 6y agoThe other implication of exponential data growth is that shortening retention periods is only a short term solution to storage costs.
- dvfjsdhgfv 6y agoI'd say it's the main implication. Google will have to use more drastic means soon. It will be interesting to see how this plays out.
- asdfasgasdgasdg 6y agoI used to work on a certain Google product and I remember thinking a petabyte was big when our raw content store crossed that milestone. Now Linus Tech Tips can put that much storage in a single box.
- znpy 6y agoThanks for posting that, most people don't realise how much compute/storage you can actually put on a single rack cabinet nowadays.
- gogopuppygogo 6y agoI remember about ten years ago when 40TB of usable storage was a mid sized five five figure investment. That’s nas territory now.
- bryanrasmussen 6y agoI remember a time when 3mb of black market data was enough to get you out of the sprawl. Now you can barely get a Uber downtown for that.
- dreamcompiler 6y agoWhoa.
- nucleardog 6y agoYep. Thats the flip side the the growth is that storage capacity keeps growing relative to price as well. I spent $600 over a decade ago and put 7.5TB in my server. I spent $600 this year to add another 28TB. I’m at nearly 40TB in my home server for maybe $1200 invested over the last decade.
- asdfasgasdgasdg 6y agoOne problem that we run into though is that spindle count /random access capability is not increasing correspondingly. It's an interesting challenge.
- knowaveragejoe 6y agoThat's mainly media. What's the deal with deleting emails of all things, as if they couldn't support that aspect indefinitely?
- hanniabu 6y agoPeople use email as a sneaky method of storage
- perryizgr8 6y agoSometimes it's the most practical way to share photos and documents, especially when iphones are involved.
- rblatz 6y agoWe just use shared albums or text them to each other.
- fps_doug 6y agoHow prevalent is that? Some nerds doing this is a drop in the ocean. Rather my bet would be on Youtube as the hungry beast that can't get enough storage. There's a loooooooot of videos that have single digit views yet need to be accessible at any time, instantly. Video is large. Doesn't youtube even keep the original file nowadays?
- blackboxlogic 6y agoSounds like we need a way to encode arbitrary data into video that can survive lossy compression. Youtube filesystem?
- basscomm 6y agoIn my experience, this is super common. Lots of average computer users send video clips, uncompressed 48 megapixel photos from their cell phones, PowerPoint presentations with embedded videos and 48 megapixel images, pdfs, and on and on. At the company where I work now several people have a 1MB gif of the company logo that they put in their signature. Combine that with Gmail telling users for years that they had plenty of space, so there was no need to ever delete anything, and people believed it. Now you have a decade of emails with giant attachments or embeds that may or may not ever get looked at, and that keeps growing.
- pishpash 6y agoI think the main issue is storage Moore's law has really hit a wall and it is no longer the case that the data Google stores grows slower than storage per dollar like in the early days. Google's freebie business model is going to have to drastically change if the two exponents have crossed paths and they'll have to do it fast because, exponents.
- fatnoah 6y agoI interviewed at G and eventually matched with teams writing software for core networking functions. In that process, the discussion came out that the software was responsible for roughly 11% of traffic...for the entire Internet.
- yudlejoza 6y agoI wonder what kind of HDD setup they (and AWS, and Azure, etc) are using. A reasonable price point is that of a 10 TB HDD, you would need 430 new disks of that size everyday. Would love to learn about what kind of contracts they enter in with hardware manufacturers.
- yencabulator 6y agoThe storage company most willing to explain what they're doing is Backblaze. Their blog is great. https://www.backblaze.com/blog/category/cloud-storage/storage-pod/ https://www.backblaze.com/blog/category/cloud-storage/storag...
- bg24 6y agoIs there any tutorial type doc on how businesses use the years old data? I understand the use or hot data, need for reports, queries, etc. Looking to beef up my knowledge.