4 ms·

Given we are living in 2020, there have to have been efforts or thought-experiments dedicated to archiving YouTube (would love to read about them if they exist)

Given we are living in 2020, there have to have been efforts or thought-experiments dedicated to archiving YouTube (would love to read about them if they exist). It seems 'almost' impossible like you said, even if we disregard, say channels with less than 100 subscribers (just as an example).

3pt14159 6y ago
It's not impossible. It's merely a matter of resources and will. Google not only has to store all that video they also need to transmit it over and over again. An archivist would only have to write to store the video. If humanity makes it to the year 3000 we're going to be pretty disappointed that all that video was lost due to a lack of foresight.
- jeffbee 6y ago
  I bet the number of times the typical youtube upload is watched is zero. I think YouTube itself already fills the archival, non-transmitting role here.
  - formerly_proven 6y ago
    Average views per video is around 10k. I don't know what the mode (most frequent number of views) is, but I'm almost certain it's not zero, since the uploader checking if the video uploaded correctly already counts as a view. Anyway, for archival purposes you don't need to save the plethora of formats Youtube keeps at hand for each video, you just need one size in one codec, and it probably doesn't have to be 4K or 8K, which probably reduces the storage requirements by one order of magnitude or so.
    - komali2 6y ago
      Is the average as relevant as some other statistical measure that I'm not knowledgeable enough to know about? For example if the vast majority of views are on 10% of the videos, the other 90% could be stored on less expensive hardware that has less bandwidth maybe? edit: for example the average net worth of an American is 76,000$ but toss a couple billionaires in the mix, whose net worth is larger by such an enormous factor, that the "average" is misleading
      - deleted 6y ago
        [deleted]
      - doersino 6y ago
        You’re thinking of the median (the value that ends up in the middle after sorting), which is indeed more useful in such cases. (Of course, boiling a wide distribution of values down to a single value, no matter by which process, is fraught with problems.)
    - jeffbee 6y ago
      Well, to show the video to the person who uploaded it you don't need to reencode it or push it to the edge or any of that. And I see no reason to believe that most videos are even watched by their creators.
    - 3pt14159 6y ago
      I'm the highly downvoted parent that said this is possible. It's only 45 years of video per day. That is very, very possible to save. That's roughly 600 terabytes of storage space a day. Which means that per month you're generating a long term storage bill that is about $18k more per month. So after a decade of operation your monthly bill is about $2.1m which is far, far, far cheaper than what we spend on plenty of things. Like I said above: This is not impossible. In fact, I bet the NSA is probably already doing it. At least for public videos. This truly is peanuts.
- op03 6y ago
  We get time travel by 2034 so no issues - https://en.wikipedia.org/wiki/John_Titor https://en.wikipedia.org/wiki/John_Titor
gumby 6y ago
I was at a conference at IBM around 1993 at which a very smart person pointed out that we had reached peak “knowledge”, in the sense that the new explosion of data on the Internet, including the the new “www” meant indexing would soon be impossible. “You’d need to keep a copy of every host”. We all nodded because this seemed pretty obviously true. It also wasn’t an urgent problem as experiments like http were unlikely to reach broad usage due to critical failures like the one way links. I mention this not to mock your “‘almost’ impossible” statement but to point out that you’re in good company
- jschwartzi 6y ago
  From an indexing/search perspective it's already impossible to find some of the more obscure videos I saw in 2013 because of the relentless crap flood of "things sort of like what you searched for that are newer."
  - metiscus 6y ago
    Well that and the growing penchant for revising history by destroying dissent, blacking out unpleasant thoughts and facts, and the onslaught of ridiculous copyright strikes against fair use content.
  - gumby 6y ago
    Doesn’t that suggest a market need for another search engine for this purpose?
    - Avicebron 6y ago
      Meta-search? Searching the search engine..I like it.
      - boogies 6y ago
        Metasearch engines are a thing: https://en.wikipedia.org/wiki/Metasearch_engine https://en.wikipedia.org/wiki/Metasearch_engine Here's a nice one with many public instances you can try: https://github.com/asciimoo/searx https://github.com/asciimoo/searx

3pt14159 6y ago

It's not impossible. It's merely a matter of resources and will. Google not only has to store all that video they also need to transmit it over and over again. An archivist would only have to write to store the video. If humanity makes it to the year 3000 we're going to be pretty disappointed that all that video was lost due to a lack of foresight.

jeffbee 6y ago
I bet the number of times the typical youtube upload is watched is zero. I think YouTube itself already fills the archival, non-transmitting role here.
- formerly_proven 6y ago
  Average views per video is around 10k. I don't know what the mode (most frequent number of views) is, but I'm almost certain it's not zero, since the uploader checking if the video uploaded correctly already counts as a view. Anyway, for archival purposes you don't need to save the plethora of formats Youtube keeps at hand for each video, you just need one size in one codec, and it probably doesn't have to be 4K or 8K, which probably reduces the storage requirements by one order of magnitude or so.
  - komali2 6y ago
    Is the average as relevant as some other statistical measure that I'm not knowledgeable enough to know about? For example if the vast majority of views are on 10% of the videos, the other 90% could be stored on less expensive hardware that has less bandwidth maybe? edit: for example the average net worth of an American is 76,000$ but toss a couple billionaires in the mix, whose net worth is larger by such an enormous factor, that the "average" is misleading
    - deleted 6y ago
      [deleted]
    - doersino 6y ago
      You’re thinking of the median (the value that ends up in the middle after sorting), which is indeed more useful in such cases. (Of course, boiling a wide distribution of values down to a single value, no matter by which process, is fraught with problems.)
  - jeffbee 6y ago
    Well, to show the video to the person who uploaded it you don't need to reencode it or push it to the edge or any of that. And I see no reason to believe that most videos are even watched by their creators.
  - 3pt14159 6y ago
    I'm the highly downvoted parent that said this is possible. It's only 45 years of video per day. That is very, very possible to save. That's roughly 600 terabytes of storage space a day. Which means that per month you're generating a long term storage bill that is about $18k more per month. So after a decade of operation your monthly bill is about $2.1m which is far, far, far cheaper than what we spend on plenty of things. Like I said above: This is not impossible. In fact, I bet the NSA is probably already doing it. At least for public videos. This truly is peanuts.
op03 6y ago
We get time travel by 2034 so no issues - https://en.wikipedia.org/wiki/John_Titor https://en.wikipedia.org/wiki/John_Titor

gumby 6y ago

I was at a conference at IBM around 1993 at which a very smart person pointed out that we had reached peak “knowledge”, in the sense that the new explosion of data on the Internet, including the the new “www” meant indexing would soon be impossible. “You’d need to keep a copy of every host”. We all nodded because this seemed pretty obviously true. It also wasn’t an urgent problem as experiments like http were unlikely to reach broad usage due to critical failures like the one way links. I mention this not to mock your “‘almost’ impossible” statement but to point out that you’re in good company

jschwartzi 6y ago
From an indexing/search perspective it's already impossible to find some of the more obscure videos I saw in 2013 because of the relentless crap flood of "things sort of like what you searched for that are newer."
- metiscus 6y ago
  Well that and the growing penchant for revising history by destroying dissent, blacking out unpleasant thoughts and facts, and the onslaught of ridiculous copyright strikes against fair use content.
- gumby 6y ago
  Doesn’t that suggest a market need for another search engine for this purpose?
  - Avicebron 6y ago
    Meta-search? Searching the search engine..I like it.
    - boogies 6y ago
      Metasearch engines are a thing: https://en.wikipedia.org/wiki/Metasearch_engine https://en.wikipedia.org/wiki/Metasearch_engine Here's a nice one with many public instances you can try: https://github.com/asciimoo/searx https://github.com/asciimoo/searx