7 ms·

Self hosted YouTube media server

245 points by 037 3y ago

codetrotter 3y ago
Does it save the video thumbnail as well? Video description? Comments? Channel name? Channel avatar? etc Currently I use yt-dlp to manually download individual videos that I want to keep. At the moment I only save the video itself. And most of the time I then also paste the URL of the video into archive.is save page and web.archive.org/save so that there is a snapshot of what the video page itself looked like at the time. But this is still incomplete, and relies on those services continuing to exist. Locally saving a snapshot of the page like that, and then also saving the thumbnail and perhaps more of the comments would be nice.
- CrampusDestrus 3y ago
  You can embed both the thumbnail and the description of a video with ytdl, if you use the appropriate containers
  - tjfl 3y ago
    I didn't know this, neat! Going to try this out: https://www.reddit.com/r/youtubedl/comments/mcvgmr/download_the_description_and_automatically_put/ https://www.reddit.com/r/youtubedl/comments/mcvgmr/download_...
- submeta 3y ago
  Check this code fragment: youtube_details = { "youtube_id": vid_id, "channel_name": vid["channel"], "vid_thumb_url": vid["thumbnail"], "title": vid["title"], "channel_id": vid["channel_id"], "duration": duration_str, "published": published, "timestamp": int(datetime.now().timestamp()), # Pulling enum value out so it is serializable "vid_type": vid_type.value, }
- brettwilcox 3y ago
  You should also check out https://archivebox.io/ https://archivebox.io/
- moritzruth 3y ago
  It saves all the things you mentioned plus the number of views and likes.
- abracadaniel 3y ago
  I’ve been using the ytdl-sub docker container for managing automated download of channels and importing them into Plex https://github.com/jmbannon/ytdl-sub https://github.com/jmbannon/ytdl-sub
- snthd 3y ago
  https://github.com/TheFrenchGhosty/TheFrenchGhostys-Ultimate-YouTube-DL-Scripts-Collection https://github.com/TheFrenchGhosty/TheFrenchGhostys-Ultimate...
- progman32 3y ago
  Yes to all of those. It even indexes most of those so you can do full text searches if you want.
freefaler 3y ago
Looks great, I will try it, since YouTube broke my scirpts a while a go... The way I was using them was to create a playlist named "save" and pulling from it once a day. It worked for a while, but YT started to ban somehow my script. Tube Archivist looks like would be ideal for that. Thanks for sharing this!
- Adverblessly 3y ago
  It might just be that you were banned because you were doing it exactly once a day. I use YT's RSS feature to follow channels and playlists I'm interested in and discovered that (somewhat ironically) if I have it query the RSS periodically Google will decide that I am a bot, will return errors for all reads and force me to pass a captcha next time I try to use any Google product (presumably connecting the two activities via ip). So now my RSS reader does not periodically query YT and instead I manually click the update button when I'm interested...
bt4u 3y ago
[dead]
nullcipher 3y ago
I couldn't find docs for installing from source. Is docker really mandatory ? Also, "Tube Archivist depends on Elasticsearch 8." . Wow, why?
- tibbydudeza 3y ago
  Why ???.
- moritzruth 3y ago
  IIRC it uses Elasticsearch as its only database.
  - pkulak 3y ago
    Seems to require Redis too. Maybe just for cache.
- Jolter 3y ago
  Probably because it’s fast, scales well in terms of size, and decently easy to build apps around. Shouldn’t the question be, why not?
  - hkt 3y ago
    Why not? Well, if you're self hosting for yourself, friends, and family.. it isn't likely to be a thing you'll want to care about fixing when it eventually breaks in mysterious ways. Better to use sqlite or just a blob of yaml in a file if self hosting might be involved.
    - Jolter 3y ago
      Sure, you’re hosting a few hundred or thousand videos, a blob of yaml can be good enough. I have no history if this project, but I suspect they started out like that. Something like Elasticsearch usually comes into play when you are ingesting a lot of data point very quickly, so I would assume the authors wanted to archive fairly huge amounts of videos.
  - 3y ago
MaikaDiHaika 3y ago
I tried installing it half a year ago and the setup and documentation was really bad. Maybe I'll try it again sometime
indus 3y ago
Interesting idea. I always dream of writing a proxy server—-where all videos—-irrespective of device—-get stored in a local cache and served without going outside on subsequent requests. Gonna try this one, and gonna take that direction.
- WirelessGigabit 3y ago
  You can. And you can do fun stuff. Look at the upside-down-ternet: http://www.ex-parrot.com/pete/upside-down-ternet.html http://www.ex-parrot.com/pete/upside-down-ternet.html But here are the problems you'll run into today: Cache hit rate. How big is your cache? Large enough to get a hit rate that economically saves you money vs the cost of the SAN? Can you cache YouTube videos? Can you intercept YouTube videos? You'll need a root cert installed on your client devices. And, here's the worst: many applications do cert pinning so they'll refuse to load, even if the signer is in the root store. They require a specific signer.
- ComputerGuru 3y ago
  https has put the kibosh on a lot of this type of stuff, unless you’re willing to set up a local CA and trust the root cert on ~every device that will use your network.
- zo1 3y ago
  I had this same idea around 2012ish. And had the naive impression that you could cache youtube videos on a proxy level (using Squid). Boy was I wrong, even then YT was employing heavy tactics to prevent people from caching videos at the native http cache header level. And it was at that point I realized something was wrong with the internet.
snthd 3y ago
As a less sophisticated alternative there's a metadata plugin for jellyfin https://github.com/ankenyr/jellyfin-youtube-metadata-plugin https://github.com/ankenyr/jellyfin-youtube-metadata-plugin
c0brac0bra 3y ago
I've had significant problems running this for extended periods. It will crash and then restoration will fail internally with corruption errors, requiring reading through docker logs or just starting over from scratch completely.
EGreg 3y ago
How does it download videos? I thought YouTube blocked ripping videos?
- simonw 3y ago
  https://github.com/ytdl-org/youtube-dl https://github.com/ytdl-org/youtube-dl and https://github.com/yt-dlp/yt-dlp https://github.com/yt-dlp/yt-dlp (which this project uses, see https://github.com/tubearchivist/tubearchivist/blob/f848e73251d59de75671aeb97e08f064601a6042/tubearchivist/requirements.txt#L12 https://github.com/tubearchivist/tubearchivist/blob/f848e732... ) are great at ripping videos, and not just from YouTube, they support 100+ different media providers. They are actively updated every time a new blocking technique comes along.
  - EGreg 3y ago
    OK but this is self-hosted
    - simonw 3y ago
      Those tools work on a laptop - I use them all the time.
DCKing 3y ago
Tube Archivist is quite heavyweight as it's meant to do heavy full archiving of YouTube channels and search through positively huge libraries. I'm getting the sense that it's a data hoarding tool, not a casual web video watching tool. I found that I just want to add a few channels to my media library, for which I use Jellyfin already. For people looking for a more lightweight option of that kind, I run the following script hourly [1]. This script uses yt-dlp to go through a text file full of YouTube RSS urls (either a channel RSS or a playlist RSS works for channels where you're only interested in a subset of videos) [2] and downloads the latest 5 videos organized in folders based on channel name. I watch these files by adding the output folder in a Jellyfin "Movies" type library sorted by most recent. The script contains a bunch of flags to make sure Jellyfin can display video metadata and thumbnails without any further plugins, and repackages videos in a format that is 1080p yet plays efficiently even in web browsers on devices released in at least the last 10 years. It uses yt-dlp's "archive" functionality to keep track of videos it's already downloaded such that it only downloads a video once, and I use a separate script to clean out files older than two weeks once in a while. Running the script depends on ffmpeg (just used for repackaging videos, not transcoding!), xq (usually comes packaged with jq or yq) and yt-dlp being installed. You sometimes will need to update yt-dlp if a YouTube side change breaks it. For my personal usage it's been honed for a little while and now runs reliably for my purposes at least. Hope it's useful to more people. [1]: https://pastebin.com/s6kSzXrL https://pastebin.com/s6kSzXrL [2]: E.g. https://danielmiessler.com/p/rss-feed-youtube-channel/ https://danielmiessler.com/p/rss-feed-youtube-channel/
- aftergibson 3y ago
  Looks great, but I couldn't get it working, what version of xq are you using this against? 1.2.1 does not seem to have a -r argument.
  - jchw 3y ago
    You need to use yq, which has an xq binary. (edit: removed my "plug" since I mentioned it elsewhere.)
    - 3y ago
deleted 3y ago
[deleted]
simonw 3y ago
I saw this was a Django app so I dug around to look at their models. As far as I can tell this is all they have: https://github.com/tubearchivist/tubearchivist/blob/master/tubearchivist/home/models.py https://github.com/tubearchivist/tubearchivist/blob/master/t... - just a `Account` model. It looks like Django + SQLite is used for user accounts, but all other data storage happens in Elasticsearch. It's an interesting design decision. I would have gone all-in on the database, and used SQLite FTS in place of Elasticsearch for simplicity, but that's my own personal favourite stack. Not saying their design is bad, just different.
- 037 3y ago
  Perhaps Elasticsearch was chosen because they also index video comments and subtitles, making full-text search a key feature. But I agree, SQLite FTS might suffice, and much of the metadata could be better managed using a traditional Django structure. It would be great to add embeddings to the index, possibly using one of your Python tools.
renegat0x0 3y ago
Ha, I have also wrote something similar https://github.com/rumca-js/Django-link-archive https://github.com/rumca-js/Django-link-archive I support not only youtube, but also any RSS source. It functions as link aggregation software. I can also fetch meta for all videos in channel, and download videos, audios. I am using standard Django auth module. It still lacks polish, and it is under development. I am not a webdev, so I am still struggling with overall architecture
ocdtrekkie 3y ago
I like prologic's Tube. Way simpler, single Golang binary. Hosts video, not much else.

245 points by 037 3y ago

Does it save the video thumbnail as well? Video description? Comments? Channel name? Channel avatar? etc Currently I use yt-dlp to manually download individual videos that I want to keep. At the moment I only save the video itself. And most of the time I then also paste the URL of the video into archive.is save page and web.archive.org/save so that there is a snapshot of what the video page itself looked like at the time. But this is still incomplete, and relies on those services continuing to exist. Locally saving a snapshot of the page like that, and then also saving the thumbnail and perhaps more of the comments would be nice.

CrampusDestrus 3y ago
You can embed both the thumbnail and the description of a video with ytdl, if you use the appropriate containers
- tjfl 3y ago
  I didn't know this, neat! Going to try this out: https://www.reddit.com/r/youtubedl/comments/mcvgmr/download_the_description_and_automatically_put/ https://www.reddit.com/r/youtubedl/comments/mcvgmr/download_...
submeta 3y ago
Check this code fragment: youtube_details = { "youtube_id": vid_id, "channel_name": vid["channel"], "vid_thumb_url": vid["thumbnail"], "title": vid["title"], "channel_id": vid["channel_id"], "duration": duration_str, "published": published, "timestamp": int(datetime.now().timestamp()), # Pulling enum value out so it is serializable "vid_type": vid_type.value, }
brettwilcox 3y ago
You should also check out https://archivebox.io/ https://archivebox.io/
moritzruth 3y ago
It saves all the things you mentioned plus the number of views and likes.
abracadaniel 3y ago
I’ve been using the ytdl-sub docker container for managing automated download of channels and importing them into Plex https://github.com/jmbannon/ytdl-sub https://github.com/jmbannon/ytdl-sub
snthd 3y ago
https://github.com/TheFrenchGhosty/TheFrenchGhostys-Ultimate-YouTube-DL-Scripts-Collection https://github.com/TheFrenchGhosty/TheFrenchGhostys-Ultimate...
progman32 3y ago
Yes to all of those. It even indexes most of those so you can do full text searches if you want.

freefaler 3y ago

Looks great, I will try it, since YouTube broke my scirpts a while a go... The way I was using them was to create a playlist named "save" and pulling from it once a day. It worked for a while, but YT started to ban somehow my script. Tube Archivist looks like would be ideal for that. Thanks for sharing this!

Adverblessly 3y ago
It might just be that you were banned because you were doing it exactly once a day. I use YT's RSS feature to follow channels and playlists I'm interested in and discovered that (somewhat ironically) if I have it query the RSS periodically Google will decide that I am a bot, will return errors for all reads and force me to pass a captcha next time I try to use any Google product (presumably connecting the two activities via ip). So now my RSS reader does not periodically query YT and instead I manually click the update button when I'm interested...

bt4u 3y ago

[dead]

nullcipher 3y ago

I couldn't find docs for installing from source. Is docker really mandatory ? Also, "Tube Archivist depends on Elasticsearch 8." . Wow, why?

tibbydudeza 3y ago
Why ???.
moritzruth 3y ago
IIRC it uses Elasticsearch as its only database.
- pkulak 3y ago
  Seems to require Redis too. Maybe just for cache.
Jolter 3y ago
Probably because it’s fast, scales well in terms of size, and decently easy to build apps around. Shouldn’t the question be, why not?
- hkt 3y ago
  Why not? Well, if you're self hosting for yourself, friends, and family.. it isn't likely to be a thing you'll want to care about fixing when it eventually breaks in mysterious ways. Better to use sqlite or just a blob of yaml in a file if self hosting might be involved.
  - Jolter 3y ago
    Sure, you’re hosting a few hundred or thousand videos, a blob of yaml can be good enough. I have no history if this project, but I suspect they started out like that. Something like Elasticsearch usually comes into play when you are ingesting a lot of data point very quickly, so I would assume the authors wanted to archive fairly huge amounts of videos.
- 3y ago

MaikaDiHaika 3y ago

I tried installing it half a year ago and the setup and documentation was really bad. Maybe I'll try it again sometime

indus 3y ago

Interesting idea. I always dream of writing a proxy server—-where all videos—-irrespective of device—-get stored in a local cache and served without going outside on subsequent requests. Gonna try this one, and gonna take that direction.

WirelessGigabit 3y ago
You can. And you can do fun stuff. Look at the upside-down-ternet: http://www.ex-parrot.com/pete/upside-down-ternet.html http://www.ex-parrot.com/pete/upside-down-ternet.html But here are the problems you'll run into today: Cache hit rate. How big is your cache? Large enough to get a hit rate that economically saves you money vs the cost of the SAN? Can you cache YouTube videos? Can you intercept YouTube videos? You'll need a root cert installed on your client devices. And, here's the worst: many applications do cert pinning so they'll refuse to load, even if the signer is in the root store. They require a specific signer.
ComputerGuru 3y ago
https has put the kibosh on a lot of this type of stuff, unless you’re willing to set up a local CA and trust the root cert on ~every device that will use your network.
zo1 3y ago
I had this same idea around 2012ish. And had the naive impression that you could cache youtube videos on a proxy level (using Squid). Boy was I wrong, even then YT was employing heavy tactics to prevent people from caching videos at the native http cache header level. And it was at that point I realized something was wrong with the internet.

snthd 3y ago

As a less sophisticated alternative there's a metadata plugin for jellyfin https://github.com/ankenyr/jellyfin-youtube-metadata-plugin https://github.com/ankenyr/jellyfin-youtube-metadata-plugin

c0brac0bra 3y ago

I've had significant problems running this for extended periods. It will crash and then restoration will fail internally with corruption errors, requiring reading through docker logs or just starting over from scratch completely.

EGreg 3y ago

How does it download videos? I thought YouTube blocked ripping videos?

simonw 3y ago
https://github.com/ytdl-org/youtube-dl https://github.com/ytdl-org/youtube-dl and https://github.com/yt-dlp/yt-dlp https://github.com/yt-dlp/yt-dlp (which this project uses, see https://github.com/tubearchivist/tubearchivist/blob/f848e73251d59de75671aeb97e08f064601a6042/tubearchivist/requirements.txt#L12 https://github.com/tubearchivist/tubearchivist/blob/f848e732... ) are great at ripping videos, and not just from YouTube, they support 100+ different media providers. They are actively updated every time a new blocking technique comes along.
- EGreg 3y ago
  OK but this is self-hosted
  - simonw 3y ago
    Those tools work on a laptop - I use them all the time.

DCKing 3y ago

Tube Archivist is quite heavyweight as it's meant to do heavy full archiving of YouTube channels and search through positively huge libraries. I'm getting the sense that it's a data hoarding tool, not a casual web video watching tool. I found that I just want to add a few channels to my media library, for which I use Jellyfin already. For people looking for a more lightweight option of that kind, I run the following script hourly [1]. This script uses yt-dlp to go through a text file full of YouTube RSS urls (either a channel RSS or a playlist RSS works for channels where you're only interested in a subset of videos) [2] and downloads the latest 5 videos organized in folders based on channel name. I watch these files by adding the output folder in a Jellyfin "Movies" type library sorted by most recent. The script contains a bunch of flags to make sure Jellyfin can display video metadata and thumbnails without any further plugins, and repackages videos in a format that is 1080p yet plays efficiently even in web browsers on devices released in at least the last 10 years. It uses yt-dlp's "archive" functionality to keep track of videos it's already downloaded such that it only downloads a video once, and I use a separate script to clean out files older than two weeks once in a while. Running the script depends on ffmpeg (just used for repackaging videos, not transcoding!), xq (usually comes packaged with jq or yq) and yt-dlp being installed. You sometimes will need to update yt-dlp if a YouTube side change breaks it. For my personal usage it's been honed for a little while and now runs reliably for my purposes at least. Hope it's useful to more people. [1]: https://pastebin.com/s6kSzXrL https://pastebin.com/s6kSzXrL [2]: E.g. https://danielmiessler.com/p/rss-feed-youtube-channel/ https://danielmiessler.com/p/rss-feed-youtube-channel/

aftergibson 3y ago
Looks great, but I couldn't get it working, what version of xq are you using this against? 1.2.1 does not seem to have a -r argument.
- jchw 3y ago
  You need to use yq, which has an xq binary. (edit: removed my "plug" since I mentioned it elsewhere.)
  - 3y ago

deleted 3y ago

[deleted]

simonw 3y ago

I saw this was a Django app so I dug around to look at their models. As far as I can tell this is all they have: https://github.com/tubearchivist/tubearchivist/blob/master/tubearchivist/home/models.py https://github.com/tubearchivist/tubearchivist/blob/master/t... - just a `Account` model. It looks like Django + SQLite is used for user accounts, but all other data storage happens in Elasticsearch. It's an interesting design decision. I would have gone all-in on the database, and used SQLite FTS in place of Elasticsearch for simplicity, but that's my own personal favourite stack. Not saying their design is bad, just different.

037 3y ago
Perhaps Elasticsearch was chosen because they also index video comments and subtitles, making full-text search a key feature. But I agree, SQLite FTS might suffice, and much of the metadata could be better managed using a traditional Django structure. It would be great to add embeddings to the index, possibly using one of your Python tools.

renegat0x0 3y ago

Ha, I have also wrote something similar https://github.com/rumca-js/Django-link-archive https://github.com/rumca-js/Django-link-archive I support not only youtube, but also any RSS source. It functions as link aggregation software. I can also fetch meta for all videos in channel, and download videos, audios. I am using standard Django auth module. It still lacks polish, and it is under development. I am not a webdev, so I am still struggling with overall architecture

ocdtrekkie 3y ago

I like prologic's Tube. Way simpler, single Golang binary. Hosts video, not much else.