11 ms·
I know this is just filtering for now, but I've been backing up entire channels for fear they'll one day disappear because global policy or legacy media conglom
by ravenstine 8y ago
I know this is just filtering for now, but I've been backing up entire channels for fear they'll one day disappear because global policy or legacy media conglomerates won't allow for it. I imagine the "you" in YouTube will be mostly gone in the next decade. The writing's on the wall.
- amatecha 8y agoNice, do you have any summary of what process you take to back up channels? I've considered the same in the past, but found that there are a lot of crappy/sketchy plugins, and not too many legit/decent ones that actually get the highest quality copy of the content.
- icebraining 8y agoyoutube-dl is great for this, you can run it periodically against a channel and only download new videos. See the --download-archive option. Using "-f best" will fetch the highest quality format, but you can be more specific with filters like "-f best[filesize<50M]".
- walterbell 8y agoRecommended storage device and disk drives?
- toomuchtodo 8y agohttps://www.backblaze.com/blog/open-source-data-storage-server/ https://www.backblaze.com/blog/open-source-data-storage-serv... ; it's what I use to keep anything in cold storage that can't immediately go into the Internet Archive or that they're unable to dark after I upload.
- craftyguy 8y agoWait so you bought a $10k 60 disk storage platform? Or built one?
- toomuchtodo 8y agoBuilt (several). My background in tech is ops/infrastructure/networking/devops.
- craftyguy 8y agoThat's awesome. I would like to build something similar, though maybe scaled down to 1/2 size/capacity. Did you follow their parts list exactly or deviate (e.g. what did you use for chassis since theirs is custom)?
- reitanqild 8y ago> what did you use for chassis since theirs is custom FWIW, in addition to their awesome disk stats Backblaze also provides blueprints for their chassis as well as a BOM for the parts inside
- craftyguy 8y ago> Backblaze also provides blueprints for their chassis Wow, I had no idea they did that. I definitely have a new respect for them.
- icebraining 8y agoWhat does "dark" mean in this context?
- toomuchtodo 8y agoPublishing of content that would cause legal or other harm to the Internet Archive. It’s still stored on disk, but not accessible.
- toomuchtodo 8y agoTo clarify, to “dark” an object is to make it unavailable but continue to store it.
- amatecha 8y agoReally! I had no clue about this one (it has been a while since I searched around for such a thing).. that's great! Thanks for the tip, I'll check it out.
- ravenstine 8y agoYeah, I'm basically doing what you describe: `youtube-dl -f best -citw -v <url of channel or playlist>` If I recall, that's going to ignore errors. At an airport right now so I can't remember exactly what those flags are.
- jwilk 8y agoyoutube-dl has this in their FAQ: > Do I always have to pass -citw? > By default, youtube-dl intends to have the best options (incidentally, if you have a convincing case that these should be different, please file an issue where you explain that ( https://yt-dl.org/bug https://yt-dl.org/bug )). Therefore, it is unnecessary and sometimes harmful to copy long option strings from webpages. In particular, the only option out of -citw that is regularly useful is -i. But OK, here's explantion of the options you used: • -f best: Select the best quality format represented by a single file with video and audio. (By default, yt-dl will merge best video with best audio if that's what's available.) • -c: Force resume of partially downloaded files. (By default, yt-dl will resume downloads if possible.) • -i: Continue on download errors, for example to skip unavailable videos in a playlist. • -t: Use title in file name. (Deprecated. This is the default.) • -w: Do not overwrite files. • -v: Print various debugging information.
- ascom 8y agoI actually use -f bestvideo+bestaudio, since it downloads and muxes the best combination of DASH video and audio. ArchiveTeam also has a recommended list of flags for archival: https://www.archiveteam.org/index.php?title=YouTube https://www.archiveteam.org/index.php?title=YouTube
- ravenstine 8y agoThanks! I'll be doing this from now on. Had no idea about that.
- Mashimo 8y ago>I actually use -f bestvideo+bestaudio, It does that by default. (If you have ffmpeg)
- earenndil 8y agoI thought it already fetches the highest quality by default?
- d0lph 8y agoIt has for a while.
- morganvachon 8y agoI'll throw in my vote for youtube-dl as well. I love that it's so cross-platform; there's even a port for OpenBSD which makes it super easy to enjoy YT videos on that OS without having to run a resource-heavy browser, and it can be scripted.
- yizzy 8y agoYou can also use Yizzy[0]. It’s just a GUI on top of youtube-dL but it’s very easy if you don’t like command line. It’s only MacOS compatible by default but should work anywhere if you change the python path. Like youtube-dl it also allows you to download whole playlists by just pasting the link. 0 - https://github.com/biko-the-bird/Yizzy https://github.com/biko-the-bird/Yizzy
- chii 8y agoalso https://github.com/MrS0m30n3/youtube-dl-gui https://github.com/MrS0m30n3/youtube-dl-gui and there are plenty of others too.
- deleted 8y ago[deleted]
- michaelsjoeberg 8y agoi have been doing the same, in particular unofficial music remixes, which likely will disappear at some point and not allowed on spotify etc.
- reificator 8y agoIf we lose Mouth Moods[0] it will be a horrific cultural loss for the internet itself. Bustin' actually came up next on my "everything" playlist while I was typing this comment. [0]: http://www.neilcic.com/mouthmoods/ http://www.neilcic.com/mouthmoods/
- oliveshell 8y agoThankfully— at least in this case— Neil offering the album as a free lossless download on his website means countless archival-quality copies are saved on PCs the world over!
- reificator 8y agoYep, though I must admit I downloaded the lossless and uploaded it straight to Google Play Music where it gets converted to lossy.
- ravenstine 8y agoThere's a lot of fan made instrumentals out there(of astounding quality) that I've been archiving for fear of losing forever. I learned my lesson after Nintendo went berzerk and requested takedowns for gameplay videos and walkthroughs; smashrockgroin's series for Goldeneye 64 was both funny and informative when playing that game, but now those videos are gone forever. I've vowed never to buy a Nintendo product again because of that alone.
- userbinator 8y agoI learned my lesson after Nintendo went berzerk and requested takedowns for gameplay videos and walkthroughs Going after pirates and such I can sort of understand, but people enjoying your product (and obviously helping to promote it)? WTF. It reminds me of the complete opposite of this old meme: http://lurkmore.so/images/9/94/360_Kid.jpg http://lurkmore.so/images/9/94/360_Kid.jpg
- ballenf 8y agoThe really frustrating aspect is that one would need to implement advanced search functionality on those videos otherwise the more you archive the less chance you have of finding anything. The "hosting" aspect of YouTube is really a small part of why the service dominates. The connections between channels and links from one video to another channel -- network effects and discovery -- are what make disrupting YT seem impossible. Kind of crazy given how YT originated in juvenile joke vids, but YT may have the potential to become Google's biggest source of revenue despite any number of overbearing content policies. The new ads below videos for products of potential interest based on the video content are annoying but probably very profitable.
- chii 8y ago> network effects and discovery -- are what make disrupting YT seem impossible. i highly doubt that. The youtube monopoly is due to both audience, and the monitization model available (which, atm is getting shoddier and shoddier every day with random demonitizations). Hosting is a big issue, but it's a technical issue, and other platforms (such as vemeo, or some other platform) has "solved". But those platforms don't dominate (or even show up as a blip!) because content creators can't use them to make their living.
- JTbane 8y ago>I've been backing up entire channels for fear they'll one day disappear because global policy or legacy media conglomerates won't allow for it. Not a bad idea, as entire channels have gotten the boot for much less, from Youtube itself.
- nkrisc 8y agoOut of curiosity, what sorts of channels/content are you backing up?
- ravenstine 8y agoPeople have a tendency to misundertand everything I do, so I'd rather not get into specifics, but in summary: - Intellectual discussions - Fan made instrumentals and remixes (especially for Rammstein) - Found footage, old shows, and other nostalgic material from VHS tapes. (wouldn't surprise me if they get flagged for having Folgers commercials from 1993) - Random funny videos with swearing or vulgar language Not a whole lot of things that are likely to be relevant to most people, but I've a culture of my own and I'd like to preserve that. To forever lose something I enjoy means losing a bit of myself and not being able to share that with others.
- nkrisc 8y agoThat definitely seems like the kind of content that would end up getting removed, but ironically is the kind of the content that services like YouTube grew up on.
- jlykins 8y agoI've been thinking a lot lately about how various small internet subcultures and communities that I've been a part of have been lost to the void. It's one of those things that makes me a special kind of sad and nostalgic. Would you be at all interested in software that helps to construct a permanent showcase of the stuff that matters to you? It would handle things like archiving to enable a focus on curation and presentation. I've been thinking of making something like this because services like the internet archive don't seem quite right. There's no context. I just worry that nobody would be interested in doing the work of providing that context.
- ravenstine 8y agoYeah, I'd definitely be interested in seeing something like that.
- dilap 8y agoit's quite a shame. google are not being good chaperones of what has become an amazing service and resource. hopefully a completely decentralized replacement will arise. (OT: wow, HN strips emoji!)
- emodendroket 8y agoWhat reason is there to think it would? There used to be a bunch of services YouTube competed with; now few of them are even worth mentioning, let alone serious competitors.
- dilap 8y agoi think there's a couple of reasons for optimism: 1. high-speed net access is becoming more and more common; higher speed help makes the possible overhead of distributed solutions less of a barrier 2. there is more awareness now of the downsides of having a centralized provider. many youtube creators and users have been burned by google; i think that'll generate real demand for a decentralized solution. 3. existence/arguably success of decentralized currencies provides a lot of psychic energy/motivation for making decentralized things.
- Silhouette 8y agohopefully a completely decentralized replacement will arise. Ironically, we used to have a nice, decentralized Internet, and services like YouTube have been probably the biggest cause of damage to it. Not so many years ago, everyone got a bit of web space and their own URL along with the connection from their ISP. You could put your own web site there, for all to see. Many ISPs would allow running some basic software. Visitors would find these sites through search engines or [gasps] manually created links from other sites with relevant material. An analogous service today would probably involve providing a useful amount of space and bandwidth, and probably some ready-made blogging or video hosting software so non-geeks could just start creating something instead of having to install anything first. But in reality, almost no ISPs (at least here in the UK) routinely offer that sort of service any more, because the likes of YouTube and Medium have killed it.
- asdgionion 8y agoMaybe we should formalize this process. If everyone just rips the videos they want we'd wind up with a bunch of silos. There would be important videos overlooked, a huge amount of duplicated effort and, most importantly, only you could view your archived videos. I think the technical problems would be solvable. youtube-dl and existing peer-to-peer filesharing systems have done most of the work for us. Making a system to browse videos would be harder. We'd need working channels, playlists, and links. We'd need search functionality. Those don't cleanly fit into normal filesharing. The problem would be the social side. It would take a lot of volunteers to archive a substantial chunk of Youtube and would most definitely be against the law.
- stordoff 8y agoI've been doing the same to back up fan translations of various things. Almost certain they will disappear to some point, as they are definitely infringing, but it's the only way I can enjoy those media (most of which I otherwise own).
- kakarot 8y agoSame here. Backing up a lot of channels I like but also lots of music videos, interviews, live shows, informational videos, etc. Hopefully the next great online video service will have support for displaying the original encoding for uploaded videos, saving them all another generation of quality degradation due to transcoding.
- nomel 8y agoAnother reason to backup HD Youtube videos is that they'll "demote" them to SD if they don't get enough views.
- lucb1e 8y agoI've never heard or seen that. Seems weird as well. Source?
- thecatspaw 8y agothey are probably talking about their caching. Videos with low views are not cached at all servers, especially not higher definitions. So you want to watch the video, the server realizes "oh hang on, I threw this one out, let me grab it again from another server". while the server fetches the fullHD video, you get it in 360p
- lucb1e 8y agoI recently had a video with very low views that just wouldn't load for minutes on end. Was wondering if they were down, or if it had to be grabbed from some tape archive because a disk crashed. After about 10-15 minutes, it slowly loaded. This was at all qualities, browsers, and even when trying from another continent.
- Aaronn 8y agoIs that true? Do you have a source on that?
- nomel 8y agoMaybe I'm an isolated case, but many of my videos from 5 years ago with few views. They were filmed on a Nikon DSLR in 1080p. A few weeks after uploading they were fine, 1080p. Several months later, I noticed a few were 720p, and now many are 480p, but a extremely poor quality 480p that looks more like < 240. For a few that are still 720p, and some that are 480p, the framerate has been reduced to maybe 4 fps. Maybe it was the codec Nikon was using, but regardless, they're basically gone, and I no longer consider Youtube a "safe" place for videos. I would send links, but I don't want to dox myself.