6 ms·
youtube-dl is great for this, you can run it periodically against a channel and only download new videos. See the --download-archive option. Using "-f best" wil
by icebraining 8y ago
youtube-dl is great for this, you can run it periodically against a channel and only download new videos. See the --download-archive option. Using "-f best" will fetch the highest quality format, but you can be more specific with filters like "-f best[filesize<50M]".
- walterbell 8y agoRecommended storage device and disk drives?
- toomuchtodo 8y agohttps://www.backblaze.com/blog/open-source-data-storage-server/ https://www.backblaze.com/blog/open-source-data-storage-serv... ; it's what I use to keep anything in cold storage that can't immediately go into the Internet Archive or that they're unable to dark after I upload.
- craftyguy 8y agoWait so you bought a $10k 60 disk storage platform? Or built one?
- toomuchtodo 8y agoBuilt (several). My background in tech is ops/infrastructure/networking/devops.
- craftyguy 8y agoThat's awesome. I would like to build something similar, though maybe scaled down to 1/2 size/capacity. Did you follow their parts list exactly or deviate (e.g. what did you use for chassis since theirs is custom)?
- reitanqild 8y ago> what did you use for chassis since theirs is custom FWIW, in addition to their awesome disk stats Backblaze also provides blueprints for their chassis as well as a BOM for the parts inside
- craftyguy 8y ago> Backblaze also provides blueprints for their chassis Wow, I had no idea they did that. I definitely have a new respect for them.
- icebraining 8y agoWhat does "dark" mean in this context?
- toomuchtodo 8y agoPublishing of content that would cause legal or other harm to the Internet Archive. It’s still stored on disk, but not accessible.
- toomuchtodo 8y agoTo clarify, to “dark” an object is to make it unavailable but continue to store it.
- amatecha 8y agoReally! I had no clue about this one (it has been a while since I searched around for such a thing).. that's great! Thanks for the tip, I'll check it out.
- ravenstine 8y agoYeah, I'm basically doing what you describe: `youtube-dl -f best -citw -v <url of channel or playlist>` If I recall, that's going to ignore errors. At an airport right now so I can't remember exactly what those flags are.
- jwilk 8y agoyoutube-dl has this in their FAQ: > Do I always have to pass -citw? > By default, youtube-dl intends to have the best options (incidentally, if you have a convincing case that these should be different, please file an issue where you explain that ( https://yt-dl.org/bug https://yt-dl.org/bug )). Therefore, it is unnecessary and sometimes harmful to copy long option strings from webpages. In particular, the only option out of -citw that is regularly useful is -i. But OK, here's explantion of the options you used: • -f best: Select the best quality format represented by a single file with video and audio. (By default, yt-dl will merge best video with best audio if that's what's available.) • -c: Force resume of partially downloaded files. (By default, yt-dl will resume downloads if possible.) • -i: Continue on download errors, for example to skip unavailable videos in a playlist. • -t: Use title in file name. (Deprecated. This is the default.) • -w: Do not overwrite files. • -v: Print various debugging information.
- ascom 8y agoI actually use -f bestvideo+bestaudio, since it downloads and muxes the best combination of DASH video and audio. ArchiveTeam also has a recommended list of flags for archival: https://www.archiveteam.org/index.php?title=YouTube https://www.archiveteam.org/index.php?title=YouTube
- ravenstine 8y agoThanks! I'll be doing this from now on. Had no idea about that.
- Mashimo 8y ago>I actually use -f bestvideo+bestaudio, It does that by default. (If you have ffmpeg)
- earenndil 8y agoI thought it already fetches the highest quality by default?
- d0lph 8y agoIt has for a while.
- morganvachon 8y agoI'll throw in my vote for youtube-dl as well. I love that it's so cross-platform; there's even a port for OpenBSD which makes it super easy to enjoy YT videos on that OS without having to run a resource-heavy browser, and it can be scripted.