7 ms·
Browser extension and local backend that automatically archives YouTube videos
- jz10 1y agoI gave Claude access to supadata YT transcription and obsidian MCP to convert them to "permanent note" format and it's helped tone down my YT addiction a lot
- fcpguru 1y agooh wow they have free plan? https://supadata.ai/pricing https://supadata.ai/pricing nice!
- fcpguru 1y ago~/os/starchive (main)[56daf7] $ ls -lh data total 3207312 -rw-r--r-- 1 aa staff 525M Aug 2 09:11 2PMzaym-StM.mov -rw-r--r-- 1 aa staff 362M Aug 2 09:10 CHbawkGc_os.mov -rw-r--r-- 1 aa staff 658M Aug 2 09:11 lqR7VV8ftys.mov ~/os/starchive (main)[56daf7] $ ./starachive Server starting on port 3009... JSON received: map[videoId:CHbawkGc_os] Added video CHbawkGc_os to queue. Queue length: 1 Processing video CHbawkGc_os. Remaining in queue: 0
- polarbear67 1y agoInteresting. I was looking into creating an extension that manually manipulates and intercepts the vnd.yt-ump [1] requests, then use webcodecs to process everything in the browser. [1]: https://github.com/gsuberland/UMP_Format/blob/main/UMP_Format.md https://github.com/gsuberland/UMP_Format/blob/main/UMP_Forma...
- fcpguru 1y agooh wow, yeah https://github.com/yt-dlp/yt-dlp https://github.com/yt-dlp/yt-dlp sounds like the easier path.
- mikae1 1y ago> Videos are saved to the ./data/ directory and converted to MOV format using ffmpeg with hardware acceleration Transcoded (ouch) or just remuxed to a mov container? Have to investigate.
- atahanacar 1y agoRelevant lines: https://github.com/andrewarrow/starchive/blob/136030c6ef11a5c92f53739700310ac8cb215e09/youtube.go#L27 https://github.com/andrewarrow/starchive/blob/136030c6ef11a5...
- fcpguru 1y agothe video has to be re-encoded because apple quicktime doesn't like the youtube video format. But the audio can just be copied. My mac's fan never spins with the hardware acceleration so it runs in the background and I just forget about it.
- ahoef 1y agoI detest QuickTime more than any other piece of software
- 1718627440 1y agoWhy does Apple take the effort to maintain and ship different encoding libraries? I would've expected to both the Safari engine and Quicktime to simply depend on libappleavsmth.dylib?
- fcpguru 1y agowow i went down an AI rabbit whole learning the answer to this: https://chatgpt.com/share/688e818d-67ac-8010-913d-618f3534f1a0 https://chatgpt.com/share/688e818d-67ac-8010-913d-618f3534f1...
- 1718627440 1y agoFirst of all how do you know it's true? That being said, that makes zero sense. Just linking to a library, doesn't precluded using a protocol over a socket to talk to a graphic/audio server. Access control like remote code isolation (webAPIs), CORS and DRM also don't change anything about decoding and mixing video streams.
- WithinReason 1y agoNow add DHT so clients can download videos from each other as a torrent and you solved global video distribution.
- rwmj 1y agoThat's basically PeerTube?
- WithinReason 1y agoPeerTube doesn't have all of youtube's videos on it
- rwmj 1y agoIsn't that PeerTube that's going to be DMCA'd out of existence by Google?
- WithinReason 1y agoWhy? How is it different from yt-dlp from a DMCA standpoint? You're downloading a video in both cases, with this you're not using YouTube’s server bandwidth so it doesn't cost them money.
- rwmj 1y agoBecause users would also be uploading, as PeerTube works somewhat like Bittorrent.
- WithinReason 1y agoI never mentioned PeerTube
- rwmj 1y agoWhether or not it's actual PeerTube, you're describing something that would have to work like PeerTube, since you can't do peer to peer video sharing without some people uploading, and so inevitably it's going to be DMCA'd out of existence as soon as Google find it. (Note to self: Never wrestle with a pig ...)
- erinnh 1y agoIve been using Tubearchivist with the extension for this. https://github.com/tubearchivist/browser-extension https://github.com/tubearchivist/browser-extension I really like the WebUI of Tubearchivist itself.
- fcpguru 1y agothe main feature I want is to just browse youtube like normal in firefox like I always do. And completely forget starchive is running. Then later in the day I'm pleasntly suprised that any video I want to clip is already downloaded and ready. I never know which one I'll want to download and I don't want to have to click any button.
- nemomarx 1y agoWhat are you clipping them for?
- fcpguru 1y agoUsually thoughout the day I'll be watching many different videos and then one will stick with me. Someone will have made a really good point at like time code 3 mins and 17 seconds or something. If I have to right then and there pause the video and start a download it takes me out of the moment. I like it so much better to just at the end of the day go back and find good moments and place them in a their own videos. Examples: https://www.youtube.com/watch?v=ksHaSnEs4WM https://www.youtube.com/watch?v=ksHaSnEs4WM https://www.youtube.com/watch?v=KRfsAufKrzk https://www.youtube.com/watch?v=KRfsAufKrzk https://www.youtube.com/watch?v=6EoH-Qy_xw8 https://www.youtube.com/watch?v=6EoH-Qy_xw8
- deleted 1y ago[deleted]
- socalgal2 1y agoCool but ... this also sounds like hording behavior. The number of things I've saved over the years only to throw them away years later and realize that saving them in the first place was a waste of time. In the 90s my friend's mom would video tape AMC movies. She had 300+ tapes. Maybe she had a few rare ones but now all those movies are available on demand either legally or illegally and in much better quality. Another friend kept all of his 1980s computer magazines (Byte, etc...) and moved these extremely heavy boxes through 30+ years of moves. I doubt he ever opened a single magazine since the moment he saved them. Then they all appeared on The Archive and he finally got rid of them. To be clear, I have a few youtube videos saved on my local storage. I'm just thinking that saving every video I watch reminds me of the things I've personally over-saved. Actually that reminds me. I met up with the magazine saving friend recently which is when I verified that he finally got rid of his stash. It made me think about things I'm still saving that if I reflect on I know I will never actually look at. For example I have box of about eight 3.5 inch floppy disks from my Amiga days. The odds that I'm going to get an Amiga or download an Amiga emu and get a drive to read those are close enough to zero that I should throw them away. Similarly I have a book of CD-ROMs of backed up data from the 90s. There's a close to 0% chance that I'm never going to bother look at their contents.
- fcpguru 1y agooh that's not why I want them local. I want to open them in final cut pro and edit them and use parts in other videos. I delete the data folder at the end of each day.
- nemomarx 1y agoGetting them on a public shared archive is probably a good outcome though. There was that lady who taped hundreds of hours of daytime TV and archiving that has some interesting historical uses? But a personal copy I'm not sure has much point yeah.
- toomuchtodo 1y ago> Marion Marguerite Stokes (née Butler; November 25, 1929 – December 14, 2012) was an American access television producer, businesswoman, investor, civil rights demonstrator, activist, librarian, and archivist, especially known for hoarding and archiving hundreds of thousands of hours of television news footage spanning 35 years [70,000 VHS tapes], from 1977 until her death in 2012, at which time she had been operating nine properties and three storage units. According to the Los Angeles Review of Books review of the 2019 documentary film Recorder, Stokes's massive project of recording the 24-hour news cycle "makes a compelling case for the significance of guerrilla archiving." https://en.wikipedia.org/wiki/Marion_Stokes https://en.wikipedia.org/wiki/Marion_Stokes https://archive.org/details/marionstokesvideo https://archive.org/details/marionstokesvideo https://recorderfilm.com/ https://recorderfilm.com/
- ivanjermakov 1y agoI'm achieving this with a single yt-dlp script reading url from a clipboard.
- fcpguru 1y agooh but there's still the thought of having to press copy. My favorite thing about this is I just forget I even have it running and browse youtube like normal. Then later anything I've watched that day is already downloaded.
- ivanjermakov 1y agoSounds like a waste of network and computer resources to me - copying url means the video is worth the effort.
- Szpadel 1y agoI creates something similar in concept but with different goal. I wanted to be able to watch videos with sponsor block on iPad ideally using Plex. I found self hosted solution like this but I was very dissatisfied with how that worked on other hand I wanted to check out loco.rs framework, so I decided to implement my own solution. basically you are able to add channels/playlists on many many platforms that yt-dlp supports, you can select what should be cut out using sponsor block and you choice how many days you want it (videos older that that are automatically deleted) if you are interested, you can check it out: https://github.com/Szpadel/LocalTube https://github.com/Szpadel/LocalTube
- amelius 1y agoIt would be nice if the extension wrote them to some shared repository. That way, the videos could be preserved for humanity without Google having a say in it. Added benefit: every video would have to be archived only once.
- Alive-in-2025 1y agoBut then companies could sue to wipe out the centralized repo. So to be safe, you'd copy things to the central repo and also have a local copy. ;-) Next, you try to centralize all the private copies so only one person has to keep theirs. Solution is end copyright for things over x years in age. Instead in the us we keep pushing back the date.
- amelius 1y agoDepends where the central server is. Nobody is wiping annas archive, for example.
- syntaxing 1y agoWhoa! I asked about something like this 2 years ago but never got to making anything [1]! Super exciting something like this exists! [1] https://news.ycombinator.com/item?id=37885584 https://news.ycombinator.com/item?id=37885584
- myself248 1y agoOh, this is huge and important. The number of things I watch that're just gone when I go back to look again! Youtube is an archive like a grocery store is a food archive. [1] If it was worth watching in the first place, it's worth saving. Reducing the friction of doing so is going to help a lot of people. (1: I'm getting this quote wrong, what's the actual and attribution??)
- fcpguru 1y agoha, I'm not sure who said Youtube is a video archive like a grocery store is a food archive but that's excellent.
- john01dav 1y agoThere's a specific removed video that I want back (the whole channel seems to have been bought by someone who wanted pre edition subscribers or something, then everything on it was nuked and replaced with content that doesn't interest me). I tried emailing the channel's new email address to ask for it, and got no response. Do you know of any practical way to try to get it back?
- myself248 1y agoTry https://findyoutubevideo.thetechrobo.ca/ https://findyoutubevideo.thetechrobo.ca/
- untech 1y agoSee also ArchiveBox, which supports YT saving as well, but can save other content too https://github.com/ArchiveBox/ArchiveBox https://github.com/ArchiveBox/ArchiveBox
- busymom0 1y agoMy archiving app called HEAP can be configured using a simple apple script and yt-dlp to do this too. And since it's a native macOS app instead of a browser extension, it works via all browsers: https://apps.apple.com/ca/app/heap-website-full-page-image/id6747727830?mt=12 https://apps.apple.com/ca/app/heap-website-full-page-image/i...
- globular-toast 1y agoI've had this idea myself so cool to see it implemented. What I'd really like is a kind of universal web caching backend. So everything I access goes through a cache and I have the option of viewing from cache if something goes offline or changes. I could also mark things as "favourite" so they don't ever expire from the cache. Does such a thing exist?
- fcpguru 1y agotrying to just grab from the actual browser cache is very hard for video. If you look at the complexity of yt-dlp you'll see why that's so much easier than trying to grab various formats from cache.
- ProofHouse 1y agoI don’t really get the purpose of this broadly, because doesn’t YouTube keep videos online unless the creator took them down which is probably not the case 95% of the time? That said for a niche or a high likelihood of a video being removed, or if you really want to be 100% certain it makes sense, but would I be accurate in that statement or am I missing something?
- fcpguru 1y agoI'm not trying to save them forever. I just want them local so I can take clips from them for other videos. I use them as source input to final cut pro.
- add-sub-mul-div 1y agoMy version of this downloads the files to my Plex filesystem so I can watch them on my TV without going through a Youtube app. Also the sponsorblock segments are cut out of my local version after download. I go even further and schedule TV "channels" that rotate through the local videos using ErsatzTV.
- globular-toast 1y agoWould you let someone else have the power to remove books from your library? Oh wait, you probably are What I'll say is you're either the kind of person who gets fucked once then does something about it, or you just keep getting screwed and complain pathetically when it keeps happening.
- ProofHouse 1y agoI’ve never gone to find a video and it’s gone. Nor have I ever felt the need to take a photo of every building I walk past and find interest in to ensure I have it for posterity in case it is later bulldozed. To each their own but don’t have to be rude about it. Thanks to the other responses. If someone wants to record all the tv they’ve ever watch that is their prerogative, as it is if they do not.
- imtringued 1y agoIf you ever created a long hundred+ video playlist on YouTube you'd realize that videos go missing very frequently and you'll never know which ones are missing.
- frou_dh 1y agoFor YouTube videos I feel are worth archiving, I just add them to playlists on my channel, then periodically download my entire channel using a single yt-dlp command (it can keep track of what's already been downloaded).
- xdfgh1112 1y agoSounds good. I've made playlists and it's ridiculous how many of the videos become unavailable even after a year.
- hkt 1y agoI've been toying with writing something like this for ages and now I don't have to. Brilliant work, thank you!
- lez 1y agoIt started a line of thoughts in me. What if the backend keeps the videos around for a longer period of time, and: * regularly checks youtube, and whenever one archived video gets deleted from youtube, it advertises the video ID on a specific set of Nostr relays * have a different browser extension for yt viewers that activates when the user hits a deleted video. * the backend can stream the video for continuous Bitcoin lightning payment until the stream is kept alive. So you can make some money on free disk space.
- quectophoton 1y agoRelated, but this is something that I would like if browsers gave more importance to. Literally a few moments ago I was playing a video from a Nitter instance (Twitter proxy), and wanted to save it with Firefox but nope. Firefox already has the video there, I can see in the developer tools that it's playing from a blob. Even if the blob is then modified on-the-fly, why can't I right-click it and save that blob that you already have? If it's a stream, why can't I dump it? You are already playing it, you already have the bytes, you can "cache" the whole playback into a local file and then let me Right Click -> Save that "cache" file. Had to use `yt-dlp`. Similar with bookmarks. When I bookmark something, it would be nice if the current state of the page were saved as a fallback (or at least have a checkbox for this behavior). You already have the rendered page right there. I can literally do `Ctrl+S` and save this page as it is right now, even with this half-typed comment. The code for saving the current state of a page is already there. And you can already link binary data to a bookmark (favicon), and text data (description, tags, etc). I wouldn't even mind if I have to give up the description field in exchange for a base64-encoded version of the bookmarked page, I'd be willing to accept that additional storage requirement.[1] Sites like YouTube that actively work to prevent easy downloads and archiving might still need extensions like the one linked in the submission, but browsers don't make it easy to save some data that they already have, and it's difficult enough to contribute to (or patch for yourself) those massive codebases unless you are already familiar with them. I'm willing to patch things for myself and repackage them for myself, but unfortunately for these things it's easier and more reliable long-term to just do workarounds. (Disclaimer: I'm using Firefox.) [1] EDIT: Having history enabled doesn't seem to do save an offline version, either.
- fcpguru 1y agoFirefox (Mozilla) could be sued by youtube if they allow this. But yt-dlp is open-source code, maintained by a loose group of contributors around the world, hosted on GitHub. No central office, no paid service, no ad revenue. If GitHub takes it down (due to a DMCA request), someone will re-upload it in hours. This decentralization makes lawsuits costly and potentially ineffective.
- codethief 1y agoSpeaking of archiving data, is there a search engine that also returns search results from older (no-longer-existent but archived) web pages? The issue I've had with things like the Wayback Machine is that I need to recall the URL or find some other web page still linking to the (now broken) URL.
- nottorp 1y agoWhy a backend? What's wrong with just a directory hierarchy?
- fcpguru 1y agoYou can’t run yt-dlp from a browser. So the backend listens for a message from the browser for which video to download.
- deleted 1y ago[deleted]
- nottorp 1y agoOh. I guessed it's doing its own downloading but if it isn't, it makes sense.
- fulafel 1y agoDRM, you can't just save the video easily.
- dbcooper 1y agoDoes this work for videos flagged as age-restricted, or paid subscriber content? IIRC, yt-dlp needs to be passed browser cookies to do so?
- fcpguru 1y agoAh good point. yt-dlp already has this, just need to prompt for auth and pass it thru. I made a ticket: https://github.com/andrewarrow/starchive/issues/4 https://github.com/andrewarrow/starchive/issues/4