5 ms·

I wonder how many of these hard drives are needed to completely scan the internet every month and create a "time capsule" drive that you can revisit and browse

by LAMike 6y ago

I wonder how many of these hard drives are needed to completely scan the internet every month and create a "time capsule" drive that you can revisit and browse ten years later like a yearbook

soulofmischief 6y ago
The proposed project entails constructing a global-scale shared supercomputer capable of hosting the entire Internet on one platform as an application https://en.wikipedia.org/wiki/IBM_Kittyhawk https://en.wikipedia.org/wiki/IBM_Kittyhawk
- eloff 6y ago
  So, basically where AWS is headed. I joke, but there's a grain truth there.
  - soulofmischief 6y ago
    It's true, though. IBM misunderstood the implications of cloud computing at the time. They clearly realized their mistake: In 2013 researchers used the Kittyhawk project to demonstrate a novel high-performance cloud computing platform by merging a cloud computing environment with a supercomputer.
- bserge 6y ago
  Wow... is IBM the Kodak of computing/hosting?
oliverobscure 6y ago
I suppose one could use the current size of the Internet Archive as a yardstick for how much public-facing stuff there is. Of course in general terms this also depends on your definition of 'the internet' - consider databases and other such stores. Huge numbers.
deleted 6y ago
[deleted]
cbhl 6y ago
If you actually do this, just make sure your crawler doesn't get caught on some programmatic infinite webpage generator. Like one of those calendar widgets.
manquer 6y ago
With video and other rich media ? You will need hundreds of thousands or even millions of these disks. YT alone adds 500hrs of video per min . That is at minimum 9000 TB / day (500 18TB disks/day - 180,000 disks a year) [1] If YT is no more than say 1% of the public internet (Facebook is probably storing lot more , most of the data is not public though, Instagram is ) Then you are looking probably at 18 million disks a year without any backups and copies considered . [1] it is assuming SD quality 220 mb per hour single format and one copy.
- exikyut 6y ago
  > YT alone adds 500hrs of video per min . That is at minimum 9000 TB / day (500 18TB disks/day ...) ... assuming SD quality 220 mb per hour single format and one copy. Nitpick: Each bigger resolution stacks on top of the previous one, so 8K60fps means storing 8K60 (webm+av1), alongside 4K60 HDR (webm), 4K60 (webm+av1), 4K (webm), 1440p60 HDR (webm), 1440p60 (webm+av1), 1440p (webm), 1080p60 HDR (webm), 1080p60 (webm+h264+av1), 1080p (webm+h264), 720p60 HDR (webm), 720p60 (webm+h264+av1), 720p (webm+h264), 480p60 HDR (webm), 480p (webm+h264+av1), 360p60 HDR (webm), 360p (webm+h264+av1), 240p60 HDR (webm), 240p (webm+h264+av1), 144p60 HDR (webm), and 144p (webm+h264+av1). And as noted, a given resolution is often stored in multiple video formats, so there's that too. The data needed to store a single video thus adds up quickly: $ youtube-dl -F 1La4QzGeaaQ | grep -o '[0-9\.]\+.iB' | sed 's/MiB/*(1024^2)/;s/GiB/*(1024^3)/;s/.*/(&)/' \ | tr '\n' + | sed 's/$.*$+$/scale=3;print(\1)\/(1024^3),"G\n"\n/' | bc -ql 8.423G So for https://youtu.be/1La4QzGeaaQ https://youtu.be/1La4QzGeaaQ (a random 8K video I found), storing 5 minutes 37 seconds requires about 8.5GB due to all the transcoding. It's quite possible YouTube also keeps the original uploads around for a bit as well. --- It incidentally proved so complicated to correctly type out the format list above manually it was actually easier to figure out the following regex: $ youtube-dl -F 1La4QzGeaaQ | sed -n '/video/{s/^.\{35\}//;s/ \+[0-9]\+k \+, \+/ (/;s/webm.*\|vp9.*/webm/;s/avc.*/h264/;s/av01.*/av1/;s/4320p/8K/;s/2160p/4K/;p}' \ | sort -hr | sed ':1;H;N;${:2;s/$[^\(]\+$($[^\n]\+$\n$\1$(/\1(\2+/;t2;s/$\n\|$$/), /g;s/, $//};b1' (The above took probably twice as long to type out as the list it produced, but it was a lot more fun, and importantly verified its own correctness, which I was having a hard time doing manually. Plus, it's cool to make sed do new things - like token deduplication and newline coalescence in this case (remove the 2nd sed command to see what I mean).)
elorant 6y ago
A fuckload. There are 160M active domains. Say 2% of them are news sites with tens of thousands of active articles, another 2% is e-commerce sites and forums with also tens of thousands of pages, and the rest have like 50 pages on average. So you have something like 154M sites x 50 pages each x let's say 200KB per page that makes it 1.540 TB of data. And for the rest 4% let's assume 20.000 pages on average by 2MB each, that gives another 250.000 TB. And then you add sites like YouTube which could easily triple the above numbers. You'll end-up with a million TBs at least. That's like 100.000 HDDs.
- catsdanxe 6y ago
  The website that lists all possible Bitcoin private keys would take up all of those drives anyway
recuter 6y ago
Stackoverflow is about 10GB: https://archive.org/details/stackexchange https://archive.org/details/stackexchange English Wikipedia is about 15GB: https://en.wikipedia.org/wiki/Wikipedia:Database_download#English-language_Wikipedia https://en.wikipedia.org/wiki/Wikipedia:Database_download#En... Reddit is about 1TB: https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment/ https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_eve... Common Crawl (May/June 2020): "It contains 2.75 billion web pages or 255 TiB of uncompressed content". About 50TB compressed. https://commoncrawl.org/2020/06/may-june-2020-crawl-archive-now-available/ https://commoncrawl.org/2020/06/may-june-2020-crawl-archive-... I think 100TB would be plenty. We're not quite there yet but I wouldn't be surprised if you could get "The Internet" on one consumer drive in a few years. This would be a cool product.
- rasz 6y ago
  Something tells me that 50TB doesnt include video media. There are dozens of chinese pirate streaming sites with more movies and TV than whole of netflix and all its competitors. Then there is YT.
  - blaser-waffle 6y ago
    Or pictures. If you look at that reddit link it's about "every single comment" -- and that's about 1 TB. I'd bet that the pictures from one fairly active if niche sub would exceed that, e.g. r/battlecars or r/breaddit

by LAMike 6y ago

I wonder how many of these hard drives are needed to completely scan the internet every month and create a "time capsule" drive that you can revisit and browse ten years later like a yearbook

soulofmischief 6y ago

The proposed project entails constructing a global-scale shared supercomputer capable of hosting the entire Internet on one platform as an application https://en.wikipedia.org/wiki/IBM_Kittyhawk https://en.wikipedia.org/wiki/IBM_Kittyhawk

eloff 6y ago
So, basically where AWS is headed. I joke, but there's a grain truth there.
- soulofmischief 6y ago
  It's true, though. IBM misunderstood the implications of cloud computing at the time. They clearly realized their mistake: In 2013 researchers used the Kittyhawk project to demonstrate a novel high-performance cloud computing platform by merging a cloud computing environment with a supercomputer.
bserge 6y ago
Wow... is IBM the Kodak of computing/hosting?

oliverobscure 6y ago

I suppose one could use the current size of the Internet Archive as a yardstick for how much public-facing stuff there is. Of course in general terms this also depends on your definition of 'the internet' - consider databases and other such stores. Huge numbers.

deleted 6y ago

[deleted]

cbhl 6y ago

If you actually do this, just make sure your crawler doesn't get caught on some programmatic infinite webpage generator. Like one of those calendar widgets.

manquer 6y ago

With video and other rich media ? You will need hundreds of thousands or even millions of these disks. YT alone adds 500hrs of video per min . That is at minimum 9000 TB / day (500 18TB disks/day - 180,000 disks a year) [1] If YT is no more than say 1% of the public internet (Facebook is probably storing lot more , most of the data is not public though, Instagram is ) Then you are looking probably at 18 million disks a year without any backups and copies considered . [1] it is assuming SD quality 220 mb per hour single format and one copy.

exikyut 6y ago
> YT alone adds 500hrs of video per min . That is at minimum 9000 TB / day (500 18TB disks/day ...) ... assuming SD quality 220 mb per hour single format and one copy. Nitpick: Each bigger resolution stacks on top of the previous one, so 8K60fps means storing 8K60 (webm+av1), alongside 4K60 HDR (webm), 4K60 (webm+av1), 4K (webm), 1440p60 HDR (webm), 1440p60 (webm+av1), 1440p (webm), 1080p60 HDR (webm), 1080p60 (webm+h264+av1), 1080p (webm+h264), 720p60 HDR (webm), 720p60 (webm+h264+av1), 720p (webm+h264), 480p60 HDR (webm), 480p (webm+h264+av1), 360p60 HDR (webm), 360p (webm+h264+av1), 240p60 HDR (webm), 240p (webm+h264+av1), 144p60 HDR (webm), and 144p (webm+h264+av1). And as noted, a given resolution is often stored in multiple video formats, so there's that too. The data needed to store a single video thus adds up quickly: $ youtube-dl -F 1La4QzGeaaQ | grep -o '[0-9\.]\+.iB' | sed 's/MiB/*(1024^2)/;s/GiB/*(1024^3)/;s/.*/(&)/' \ | tr '\n' + | sed 's/$.*$+$/scale=3;print(\1)\/(1024^3),"G\n"\n/' | bc -ql 8.423G So for https://youtu.be/1La4QzGeaaQ https://youtu.be/1La4QzGeaaQ (a random 8K video I found), storing 5 minutes 37 seconds requires about 8.5GB due to all the transcoding. It's quite possible YouTube also keeps the original uploads around for a bit as well. --- It incidentally proved so complicated to correctly type out the format list above manually it was actually easier to figure out the following regex: $ youtube-dl -F 1La4QzGeaaQ | sed -n '/video/{s/^.\{35\}//;s/ \+[0-9]\+k \+, \+/ (/;s/webm.*\|vp9.*/webm/;s/avc.*/h264/;s/av01.*/av1/;s/4320p/8K/;s/2160p/4K/;p}' \ | sort -hr | sed ':1;H;N;${:2;s/$[^\(]\+$($[^\n]\+$\n$\1$(/\1(\2+/;t2;s/$\n\|$$/), /g;s/, $//};b1' (The above took probably twice as long to type out as the list it produced, but it was a lot more fun, and importantly verified its own correctness, which I was having a hard time doing manually. Plus, it's cool to make sed do new things - like token deduplication and newline coalescence in this case (remove the 2nd sed command to see what I mean).)

elorant 6y ago

A fuckload. There are 160M active domains. Say 2% of them are news sites with tens of thousands of active articles, another 2% is e-commerce sites and forums with also tens of thousands of pages, and the rest have like 50 pages on average. So you have something like 154M sites x 50 pages each x let's say 200KB per page that makes it 1.540 TB of data. And for the rest 4% let's assume 20.000 pages on average by 2MB each, that gives another 250.000 TB. And then you add sites like YouTube which could easily triple the above numbers. You'll end-up with a million TBs at least. That's like 100.000 HDDs.

catsdanxe 6y ago
The website that lists all possible Bitcoin private keys would take up all of those drives anyway

recuter 6y ago

Stackoverflow is about 10GB: https://archive.org/details/stackexchange https://archive.org/details/stackexchange English Wikipedia is about 15GB: https://en.wikipedia.org/wiki/Wikipedia:Database_download#English-language_Wikipedia https://en.wikipedia.org/wiki/Wikipedia:Database_download#En... Reddit is about 1TB: https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment/ https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_eve... Common Crawl (May/June 2020): "It contains 2.75 billion web pages or 255 TiB of uncompressed content". About 50TB compressed. https://commoncrawl.org/2020/06/may-june-2020-crawl-archive-now-available/ https://commoncrawl.org/2020/06/may-june-2020-crawl-archive-... I think 100TB would be plenty. We're not quite there yet but I wouldn't be surprised if you could get "The Internet" on one consumer drive in a few years. This would be a cool product.

rasz 6y ago
Something tells me that 50TB doesnt include video media. There are dozens of chinese pirate streaming sites with more movies and TV than whole of netflix and all its competitors. Then there is YT.
- blaser-waffle 6y ago
  Or pictures. If you look at that reddit link it's about "every single comment" -- and that's about 1 TB. I'd bet that the pictures from one fairly active if niche sub would exceed that, e.g. r/battlecars or r/breaddit