5 ms·

One the one hand it's sad that it'll be impossible, practically and perhaps even physically, to archive all of Youtube should Google ever decide to axe it, on t

One the one hand it's sad that it'll be impossible, practically and perhaps even physically, to archive all of Youtube should Google ever decide to axe it, on the other hand even though I'm fanatically a datahoarder, the thought of archiving youtube just seems... exhausting? Pointless? Haha I'm not sure. Then again just imagine how fantastically useful 40 years worth of video a day could be to a historian 200 years from now. I think the most important technology going forward will be analysis and indexing software, to make searching videos like this possible. Ted Chiang wrote a short story about that, where you could search like "that time I got in a fight with my wife about the toilet seat being up," and you'd get a video from the incident.

Fiveplus 6y ago
Given we are living in 2020, there have to have been efforts or thought-experiments dedicated to archiving YouTube (would love to read about them if they exist). It seems 'almost' impossible like you said, even if we disregard, say channels with less than 100 subscribers (just as an example).
- 3pt14159 6y ago
  It's not impossible. It's merely a matter of resources and will. Google not only has to store all that video they also need to transmit it over and over again. An archivist would only have to write to store the video. If humanity makes it to the year 3000 we're going to be pretty disappointed that all that video was lost due to a lack of foresight.
  - jeffbee 6y ago
    I bet the number of times the typical youtube upload is watched is zero. I think YouTube itself already fills the archival, non-transmitting role here.
    - formerly_proven 6y ago
      Average views per video is around 10k. I don't know what the mode (most frequent number of views) is, but I'm almost certain it's not zero, since the uploader checking if the video uploaded correctly already counts as a view. Anyway, for archival purposes you don't need to save the plethora of formats Youtube keeps at hand for each video, you just need one size in one codec, and it probably doesn't have to be 4K or 8K, which probably reduces the storage requirements by one order of magnitude or so.
      - komali2 6y ago
        Is the average as relevant as some other statistical measure that I'm not knowledgeable enough to know about? For example if the vast majority of views are on 10% of the videos, the other 90% could be stored on less expensive hardware that has less bandwidth maybe? edit: for example the average net worth of an American is 76,000$ but toss a couple billionaires in the mix, whose net worth is larger by such an enormous factor, that the "average" is misleading
        deleted 6y ago
        [deleted]
        doersino 6y ago
        You’re thinking of the median (the value that ends up in the middle after sorting), which is indeed more useful in such cases. (Of course, boiling a wide distribution of values down to a single value, no matter by which process, is fraught with problems.)
      - jeffbee 6y ago
        Well, to show the video to the person who uploaded it you don't need to reencode it or push it to the edge or any of that. And I see no reason to believe that most videos are even watched by their creators.
      - 3pt14159 6y ago
        I'm the highly downvoted parent that said this is possible. It's only 45 years of video per day. That is very, very possible to save. That's roughly 600 terabytes of storage space a day. Which means that per month you're generating a long term storage bill that is about $18k more per month. So after a decade of operation your monthly bill is about $2.1m which is far, far, far cheaper than what we spend on plenty of things. Like I said above: This is not impossible. In fact, I bet the NSA is probably already doing it. At least for public videos. This truly is peanuts.
  - op03 6y ago
    We get time travel by 2034 so no issues - https://en.wikipedia.org/wiki/John_Titor https://en.wikipedia.org/wiki/John_Titor
- gumby 6y ago
  I was at a conference at IBM around 1993 at which a very smart person pointed out that we had reached peak “knowledge”, in the sense that the new explosion of data on the Internet, including the the new “www” meant indexing would soon be impossible. “You’d need to keep a copy of every host”. We all nodded because this seemed pretty obviously true. It also wasn’t an urgent problem as experiments like http were unlikely to reach broad usage due to critical failures like the one way links. I mention this not to mock your “‘almost’ impossible” statement but to point out that you’re in good company
  - jschwartzi 6y ago
    From an indexing/search perspective it's already impossible to find some of the more obscure videos I saw in 2013 because of the relentless crap flood of "things sort of like what you searched for that are newer."
    - metiscus 6y ago
      Well that and the growing penchant for revising history by destroying dissent, blacking out unpleasant thoughts and facts, and the onslaught of ridiculous copyright strikes against fair use content.
    - gumby 6y ago
      Doesn’t that suggest a market need for another search engine for this purpose?
      - Avicebron 6y ago
        Meta-search? Searching the search engine..I like it.
        boogies 6y ago
        Metasearch engines are a thing: https://en.wikipedia.org/wiki/Metasearch_engine https://en.wikipedia.org/wiki/Metasearch_engine Here's a nice one with many public instances you can try: https://github.com/asciimoo/searx https://github.com/asciimoo/searx
marta_morena_25 6y ago
There is nothing impossible about it. We barely scratched the surface of data storage mechanisms. All it takes is another few breakthroughs and we likely can store all of Youtube of all times in a block the size of an ice cube.
- boogies 6y ago
  What makes you so confident? SSD tech is experiencing scaling challenges¹. IIRC DNA’s supposed to be incredibly dense and last long, but too expensive for much more than wedding photos². Do you have a more promising emerging tech in mind? 1: https://news.ycombinator.com/item?id=23393293 https://news.ycombinator.com/item?id=23393293 2: I think I heard this use case predicted on some NPR thing I can't find, but this one https://www.npr.org/2013/01/24/170082404/shall-i-encode-thee-in-dna-sonnets-stored-on-double-helix https://www.npr.org/2013/01/24/170082404/shall-i-encode-thee... says “Agilent waived the cost of DNA synthesis for this project, but the researchers estimate it would normally cost about $12,400 per megabyte. "It's an unthinkably large amount of money ... at the moment," Goldman says.”
  - perl4ever 6y ago
    >$12,400 per megabyte I believe the first hard drive cost about $100K/MB in today's money.
- K2h 6y ago
  The Sandisk microSD card SDSQXCZ-1T00 rated at 1TB for MSRP of $400 is listed at sdcard.org as physical dimensions of 11 x 15 x 1.0 mm, Approx 0.5g. That is 165 mm^2. If a medium ice cube is 25 x 25 x 32 mm for a volume of 20,000 mm^2 - then today an 'ice cube' could hold 20,000 / 165 = 121 TB with normal off the shelf accessible technology for $48K USD.
  - ranma42 6y ago
    Given that ~1 hour of video is uploaded to youtube per second and assuming 5mbit/s for the video data, then you can store under a days worth of youtube videos on that current-day ice cube (it will be full in about ~15 hours worth of uploads (if I didn't miscalculate)).
pythonaut_16 6y ago
It's crazy to think that 40 years worth of video can be uploaded daily, and yet that only captures a minuscule amount of everything that goes on in the world in a given day.
- AnHonestComment 6y ago
  There’s ~21,000,000 years of human life lived a day. At 40 years per day uploaded, that’s 0.000002 of the human experience for that day. It’s interesting to think that YouTube content is only a “two in a million” reflection of human life. Averaged out, that’s about 1 minute per person per year.
  - pythonaut_16 6y ago
    Apply a similar comparison to the past and it's humbling to realize just how little we know about history, and how quickly we collectively forget the farther back you go.
- TheSpiceIsLife 6y ago
  On any given day approximately 23.37 million years of human experience occurs.
  - komali2 6y ago
    But there's a lot of crossover surely - families that live together we could consider "one" day's worth experience, minus the time they spend apart pooping or whatever.
    - TheSpiceIsLife 6y ago
      Yeah that’s an interesting way of slicing it. When people say “everyone’s different” I’m quite fond of saying things like “well, there’s 7.8 billion of us, surely we’re not all that different?”
- kortilla 6y ago

Fiveplus 6y ago

Given we are living in 2020, there have to have been efforts or thought-experiments dedicated to archiving YouTube (would love to read about them if they exist). It seems 'almost' impossible like you said, even if we disregard, say channels with less than 100 subscribers (just as an example).

3pt14159 6y ago
It's not impossible. It's merely a matter of resources and will. Google not only has to store all that video they also need to transmit it over and over again. An archivist would only have to write to store the video. If humanity makes it to the year 3000 we're going to be pretty disappointed that all that video was lost due to a lack of foresight.
- jeffbee 6y ago
  I bet the number of times the typical youtube upload is watched is zero. I think YouTube itself already fills the archival, non-transmitting role here.
  - formerly_proven 6y ago
    Average views per video is around 10k. I don't know what the mode (most frequent number of views) is, but I'm almost certain it's not zero, since the uploader checking if the video uploaded correctly already counts as a view. Anyway, for archival purposes you don't need to save the plethora of formats Youtube keeps at hand for each video, you just need one size in one codec, and it probably doesn't have to be 4K or 8K, which probably reduces the storage requirements by one order of magnitude or so.
    - komali2 6y ago
      Is the average as relevant as some other statistical measure that I'm not knowledgeable enough to know about? For example if the vast majority of views are on 10% of the videos, the other 90% could be stored on less expensive hardware that has less bandwidth maybe? edit: for example the average net worth of an American is 76,000$ but toss a couple billionaires in the mix, whose net worth is larger by such an enormous factor, that the "average" is misleading
      - deleted 6y ago
        [deleted]
      - doersino 6y ago
        You’re thinking of the median (the value that ends up in the middle after sorting), which is indeed more useful in such cases. (Of course, boiling a wide distribution of values down to a single value, no matter by which process, is fraught with problems.)
    - jeffbee 6y ago
      Well, to show the video to the person who uploaded it you don't need to reencode it or push it to the edge or any of that. And I see no reason to believe that most videos are even watched by their creators.
    - 3pt14159 6y ago
      I'm the highly downvoted parent that said this is possible. It's only 45 years of video per day. That is very, very possible to save. That's roughly 600 terabytes of storage space a day. Which means that per month you're generating a long term storage bill that is about $18k more per month. So after a decade of operation your monthly bill is about $2.1m which is far, far, far cheaper than what we spend on plenty of things. Like I said above: This is not impossible. In fact, I bet the NSA is probably already doing it. At least for public videos. This truly is peanuts.
- op03 6y ago
  We get time travel by 2034 so no issues - https://en.wikipedia.org/wiki/John_Titor https://en.wikipedia.org/wiki/John_Titor
gumby 6y ago
I was at a conference at IBM around 1993 at which a very smart person pointed out that we had reached peak “knowledge”, in the sense that the new explosion of data on the Internet, including the the new “www” meant indexing would soon be impossible. “You’d need to keep a copy of every host”. We all nodded because this seemed pretty obviously true. It also wasn’t an urgent problem as experiments like http were unlikely to reach broad usage due to critical failures like the one way links. I mention this not to mock your “‘almost’ impossible” statement but to point out that you’re in good company
- jschwartzi 6y ago
  From an indexing/search perspective it's already impossible to find some of the more obscure videos I saw in 2013 because of the relentless crap flood of "things sort of like what you searched for that are newer."
  - metiscus 6y ago
    Well that and the growing penchant for revising history by destroying dissent, blacking out unpleasant thoughts and facts, and the onslaught of ridiculous copyright strikes against fair use content.
  - gumby 6y ago
    Doesn’t that suggest a market need for another search engine for this purpose?
    - Avicebron 6y ago
      Meta-search? Searching the search engine..I like it.
      - boogies 6y ago
        Metasearch engines are a thing: https://en.wikipedia.org/wiki/Metasearch_engine https://en.wikipedia.org/wiki/Metasearch_engine Here's a nice one with many public instances you can try: https://github.com/asciimoo/searx https://github.com/asciimoo/searx

marta_morena_25 6y ago

There is nothing impossible about it. We barely scratched the surface of data storage mechanisms. All it takes is another few breakthroughs and we likely can store all of Youtube of all times in a block the size of an ice cube.

boogies 6y ago
What makes you so confident? SSD tech is experiencing scaling challenges¹. IIRC DNA’s supposed to be incredibly dense and last long, but too expensive for much more than wedding photos². Do you have a more promising emerging tech in mind? 1: https://news.ycombinator.com/item?id=23393293 https://news.ycombinator.com/item?id=23393293 2: I think I heard this use case predicted on some NPR thing I can't find, but this one https://www.npr.org/2013/01/24/170082404/shall-i-encode-thee-in-dna-sonnets-stored-on-double-helix https://www.npr.org/2013/01/24/170082404/shall-i-encode-thee... says “Agilent waived the cost of DNA synthesis for this project, but the researchers estimate it would normally cost about $12,400 per megabyte. "It's an unthinkably large amount of money ... at the moment," Goldman says.”
- perl4ever 6y ago
  >$12,400 per megabyte I believe the first hard drive cost about $100K/MB in today's money.
K2h 6y ago
The Sandisk microSD card SDSQXCZ-1T00 rated at 1TB for MSRP of $400 is listed at sdcard.org as physical dimensions of 11 x 15 x 1.0 mm, Approx 0.5g. That is 165 mm^2. If a medium ice cube is 25 x 25 x 32 mm for a volume of 20,000 mm^2 - then today an 'ice cube' could hold 20,000 / 165 = 121 TB with normal off the shelf accessible technology for $48K USD.
- ranma42 6y ago
  Given that ~1 hour of video is uploaded to youtube per second and assuming 5mbit/s for the video data, then you can store under a days worth of youtube videos on that current-day ice cube (it will be full in about ~15 hours worth of uploads (if I didn't miscalculate)).

pythonaut_16 6y ago

It's crazy to think that 40 years worth of video can be uploaded daily, and yet that only captures a minuscule amount of everything that goes on in the world in a given day.

AnHonestComment 6y ago
There’s ~21,000,000 years of human life lived a day. At 40 years per day uploaded, that’s 0.000002 of the human experience for that day. It’s interesting to think that YouTube content is only a “two in a million” reflection of human life. Averaged out, that’s about 1 minute per person per year.
- pythonaut_16 6y ago
  Apply a similar comparison to the past and it's humbling to realize just how little we know about history, and how quickly we collectively forget the farther back you go.
TheSpiceIsLife 6y ago
On any given day approximately 23.37 million years of human experience occurs.
- komali2 6y ago
  But there's a lot of crossover surely - families that live together we could consider "one" day's worth experience, minus the time they spend apart pooping or whatever.
  - TheSpiceIsLife 6y ago
    Yeah that’s an interesting way of slicing it. When people say “everyone’s different” I’m quite fond of saying things like “well, there’s 7.8 billion of us, surely we’re not all that different?”
kortilla 6y ago