6 ms·

For some time I've been tinkering with a script which manages my own fine-grained youtube subscription preferences, periodically scrapes various pages for metad

For some time I've been tinkering with a script which manages my own fine-grained youtube subscription preferences, periodically scrapes various pages for metadata, and then as necessary extracts videos by using the youtube-dl utility. The end result is a directory on my machine which always has a few interesting videos to watch, ad-free, tracking-free, network connection free, using my favorite full-featured video playback tools. I think others seeking a minimalist youtube experience might enjoy youtube-dl: https://rg3.github.io/youtube-dl/ https://rg3.github.io/youtube-dl/

striking 9y ago
Could you release some part of that script too? It sounds really interesting.
lozf 9y ago
I've been a youtube-dl fan for a long time -- it was essential whilst traveling a few years ago with suboptimal 3G bandwidth... these days I usually just paste the URL in to a terminal after typing `mpv` for a minimal experience but still with all the control you might want for playback speed, brightness, screenshots, & skipping around etc. Those who prefer a GUI could use SMplayer.
goda90 9y ago
How well does youtube-dl deal with video playback being restricted to play only from youtube.com? For instance this Tube site can't really play most music videos.
- seba_dos1 9y ago
  Well.
chriswarbo 9y ago
youtube-dl is really cool, and supports loads of sites as well as just youtube. VLC can also play youtube URLs directly, although buffering can be annoying. There's also a Python package called "whitey" which provides an ncurses interface for searching youtube, and sending videos to e.g. mplayer or youtube-dl. If you want to "subscribe" without needing an account, every youtube channel provides an RSS feed (look in the <head> of the page source).
- iamdave 9y ago
  VLC can also play youtube URLs directly I legitimately did not know this, thanks for the knowledge share! Can it follow playlist s of videos? There's a few video game streamers whose commentary and remarks are so funny and well written I find myself listening to them more like a podcast than watching the video, so I'm often queuing up playlists and having the background noise while doing house chores and the like
  - vatueil 9y ago
    VLC can't play YouTube playlists natively, but there's a VLC add-on that supports doing so: https://addons.videolan.org/content/show.php/+Youtube+playlist?content=149909 https://addons.videolan.org/content/show.php/+Youtube+playli... Also, mpv can use youtube-dl if both are installed to play YouTube playlists, e.g.: mpv https://www.youtube.com/playlist?list=PLhyKYa0YJ_5DfJnFBZwz8xd37MA6I16oJ Note that you need to pass it the link to the playlist itself and not a video in the playlist: https://github.com/mpv-player/mpv/wiki/FAQ#How_can_I_play_YouTube_playlists https://github.com/mpv-player/mpv/wiki/FAQ#How_can_I_play_Yo...
    - iamdave 9y ago
      Excellent, thanks!
  - foobazzy 9y ago
    Well, if you're on Mac and if you want to forgo the playback speed controls (like for, music) I can also recommend Vox earlier versions (2.8.26)[0] It is an audio player with the youtube-list-as-playlist feature. In my experience, it generally delivers a better sound quality of the same videos than youtube. Copy the url (make sure it's /?list=<id>). Hit Cmd + U, paste, <Enter>. Then optionally download the extension for using keyboard buttons[1] and control playback from there. Or you can always use `youtube-dl -ix /?list=` then open the resulting files with .opus extension with your favorite player. [0]: https://vox.rocks/mac-music-player/old-versions https://vox.rocks/mac-music-player/old-versions [1]: https://vox.rocks/mac-music-player/control-extension-download https://vox.rocks/mac-music-player/control-extension-downloa... Disclaimers: No commercial interests. Just a happy user.
doomrobo 9y ago
Not sure how much you'd benefit from it, given that you've already got a good setup, but I just made a website called HookRSSTube[0] that essentially proxies YouTube RSS feeds and replaces the video playback with HookTube[1] instead. I'd love to know if you have any comments or suggestions. [0] https://hookrsstube.com https://hookrsstube.com [1] https://hooktube.com https://hooktube.com
- craftyguy 9y ago
  +1 for hooktube! I now use a greasemonkey script to auto-redirect all youtube.com links to hooktube instead.
  - barathvutukuri 9y ago
    Could you please share the script :)
- barathvutukuri 9y ago
  +1 for hooktube, very useful. Now I can watch blocked videos in my company network :)
ehPReth 9y ago
Do you have a way to ensure you get the highest quality? I do similar with cron but often get videos as they're still processing (and thus only at lower qualities). One issue is the channels I 'subscribe' to don't always output at the same resolution and frame rate.. so I can't do a simple 'is video 4k60' test.
- deleted 9y ago
  [deleted]
- rcthompson 9y ago
  I don't think there's any way to know what qualities are currently being prepared. Your best bet might be to just figure out what time that creator typically uploads, and make sure your script doesn't run during that time and for several hours afterward, to give YouTube the time it needs to prepare the high-quality ones.
- wingerlang 9y ago
  Could you look at the upload date and wait a certain amount until the processing is more likely to be done?
  - rcthompson 9y ago
    That's an excellent idea. You can work out a reasonable amount of time to allow YouTube to do the video processing, and then use youtube-dl's "--datebefore" option to only download videos that are at least that old. You'll have to manually compute the date and time for X hours in the past, I suppose.
    - ehPReth 9y ago
      Thanks for the hint wrt --datebefore. I'll give it a go :)
  - ehPReth 9y ago
    Ah! That could work indeed. Thanks :)
rcthompson 9y ago
I actually have a script running on a cron job on my personal server that does approximately this. It downloads into a Syncthing folder that automatically syncs to whatever devices I might want to watch videos on. The difference is I just have a set of playlists for it to download, so I'm relying on the YouTube creators to organize their videos into appropriate playlists, which means it's not exactly "fine-grained". It's basically an 80% solution, and for the remaining 20%, I just invoke youtube-dl manually.
- iggg 9y ago
  Uploads is a playlist.
  - rcthompson 9y ago
    For the few YouTube channels where I actually want to watch every single video uploaded by the creator, I do use this. For the rest, I have to hope that they create playlists for the subset of their videos that I am interested in. Many creators are actually quite diligent about organizing their uploads into playlists.
spapas82 9y ago
For anybody interested I've written a tutorial for youtube-dl: https://spapas.github.io/2018/03/06/easy-youtube-mp3-downloading/ https://spapas.github.io/2018/03/06/easy-youtube-mp3-downloa...
- nishparadox 9y ago
  I have a project that kinda acts as music assistant: "Search and play any song from terminal" The project is somewhat inactive right now due to personal priorities. http://github.com/NISH1001/playx http://github.com/NISH1001/playx
arca_vorago 9y ago
If you like youtube-dl you should check out mps-youtube (https://github.com/mps-youtube/mps-youtube https://github.com/mps-youtube/mps-youtube) which has the ability to use youtuble-dl as the backend I'm pretty sure (i prefer to use the native one personally) It's also gpl vs youtube-dl's public domain. I'm sure eventually I'll figure out a way to just do it in emacs. (I'm mostly joking... kinda... maybe.)
- app4soft 9y ago
  > mps-youtube Funny that state[0] on PyPi[1] is [downloads|no longer available] [0] https://img.shields.io/pypi/dm/mps-youtube.svg https://img.shields.io/pypi/dm/mps-youtube.svg [1] https://pypi.python.org/pypi/mps-youtube https://pypi.python.org/pypi/mps-youtube
  - akx 9y ago
    PyPI no longer tracks download counts on its own. Instead, they pipe access logs to Google BigTable, so you can BigQuery the heck out of them.
- app4soft 9y ago
  > (i prefer to use the native one personally) Then you should try - SMTube[0]. SMTube allow use "mpv + youtube-dl" for play videos.[1] [0] http://www.smtube.org/ http://www.smtube.org/ [1] https://blog.smplayer.info/how-to-download-videos-with-the-new-smtube/ https://blog.smplayer.info/how-to-download-videos-with-the-n...
enz 9y ago
You don't need to scrap the page, YouTube provides the RSS stream for each channel: `https://www.youtube.com/feeds/videos.xml?channel_id=[ID]` https://www.youtube.com/feeds/videos.xml?channel_id=[ID]`. The RSS contains the "normal" link of each video, so you can invoke youtube-dl or mpv on it.
- latexr 9y ago
  > You don't need to scrap the page They do if they are scraping playlists. Youtube assumes that when creators make a playlist, they’ll add videos to the top. As such, the RSS for a playlist[1] always returns just the top 15 results. But a lot of creators — typically the ones that take greater care into organising their playlists, such as Crash Course[2] — add videos to the bottom of their playlists. Those RSS feeds will be stuck at the same 15 items forever, and your only option is to scrape the page. [1]: https://www.youtube.com/feeds/videos.xml?playlist_id={playlist_id} https://www.youtube.com/feeds/videos.xml?playlist_id={playli... [2]: https://www.youtube.com/user/crashcourse https://www.youtube.com/user/crashcourse
  - zaarn 9y ago
    You can use youtube-dl's archiving feature for that. I use it regularly on my fairly large music playlist and it goes through all 20 pages just fine despite adding stuff on the bottom occasionally (I sorted it by creation date)
endisukaj 9y ago
> ad-free You don't want the people who make the content you enjoy to earn a living doing that?
dredmorbius 9y ago
See also mps-youtube, a Python utility installable via pip.

striking 9y ago

Could you release some part of that script too? It sounds really interesting.

lozf 9y ago

I've been a youtube-dl fan for a long time -- it was essential whilst traveling a few years ago with suboptimal 3G bandwidth... these days I usually just paste the URL in to a terminal after typing `mpv` for a minimal experience but still with all the control you might want for playback speed, brightness, screenshots, & skipping around etc. Those who prefer a GUI could use SMplayer.

goda90 9y ago

How well does youtube-dl deal with video playback being restricted to play only from youtube.com? For instance this Tube site can't really play most music videos.

seba_dos1 9y ago
Well.

chriswarbo 9y ago

youtube-dl is really cool, and supports loads of sites as well as just youtube. VLC can also play youtube URLs directly, although buffering can be annoying. There's also a Python package called "whitey" which provides an ncurses interface for searching youtube, and sending videos to e.g. mplayer or youtube-dl. If you want to "subscribe" without needing an account, every youtube channel provides an RSS feed (look in the <head> of the page source).

iamdave 9y ago
VLC can also play youtube URLs directly I legitimately did not know this, thanks for the knowledge share! Can it follow playlist s of videos? There's a few video game streamers whose commentary and remarks are so funny and well written I find myself listening to them more like a podcast than watching the video, so I'm often queuing up playlists and having the background noise while doing house chores and the like
- vatueil 9y ago
  VLC can't play YouTube playlists natively, but there's a VLC add-on that supports doing so: https://addons.videolan.org/content/show.php/+Youtube+playlist?content=149909 https://addons.videolan.org/content/show.php/+Youtube+playli... Also, mpv can use youtube-dl if both are installed to play YouTube playlists, e.g.: mpv https://www.youtube.com/playlist?list=PLhyKYa0YJ_5DfJnFBZwz8xd37MA6I16oJ Note that you need to pass it the link to the playlist itself and not a video in the playlist: https://github.com/mpv-player/mpv/wiki/FAQ#How_can_I_play_YouTube_playlists https://github.com/mpv-player/mpv/wiki/FAQ#How_can_I_play_Yo...
  - iamdave 9y ago
    Excellent, thanks!
- foobazzy 9y ago
  Well, if you're on Mac and if you want to forgo the playback speed controls (like for, music) I can also recommend Vox earlier versions (2.8.26)[0] It is an audio player with the youtube-list-as-playlist feature. In my experience, it generally delivers a better sound quality of the same videos than youtube. Copy the url (make sure it's /?list=<id>). Hit Cmd + U, paste, <Enter>. Then optionally download the extension for using keyboard buttons[1] and control playback from there. Or you can always use `youtube-dl -ix /?list=` then open the resulting files with .opus extension with your favorite player. [0]: https://vox.rocks/mac-music-player/old-versions https://vox.rocks/mac-music-player/old-versions [1]: https://vox.rocks/mac-music-player/control-extension-download https://vox.rocks/mac-music-player/control-extension-downloa... Disclaimers: No commercial interests. Just a happy user.

doomrobo 9y ago

Not sure how much you'd benefit from it, given that you've already got a good setup, but I just made a website called HookRSSTube[0] that essentially proxies YouTube RSS feeds and replaces the video playback with HookTube[1] instead. I'd love to know if you have any comments or suggestions. [0] https://hookrsstube.com https://hookrsstube.com [1] https://hooktube.com https://hooktube.com

craftyguy 9y ago
+1 for hooktube! I now use a greasemonkey script to auto-redirect all youtube.com links to hooktube instead.
- barathvutukuri 9y ago
  Could you please share the script :)
barathvutukuri 9y ago
+1 for hooktube, very useful. Now I can watch blocked videos in my company network :)

ehPReth 9y ago

Do you have a way to ensure you get the highest quality? I do similar with cron but often get videos as they're still processing (and thus only at lower qualities). One issue is the channels I 'subscribe' to don't always output at the same resolution and frame rate.. so I can't do a simple 'is video 4k60' test.

deleted 9y ago
[deleted]
rcthompson 9y ago
I don't think there's any way to know what qualities are currently being prepared. Your best bet might be to just figure out what time that creator typically uploads, and make sure your script doesn't run during that time and for several hours afterward, to give YouTube the time it needs to prepare the high-quality ones.
wingerlang 9y ago
Could you look at the upload date and wait a certain amount until the processing is more likely to be done?
- rcthompson 9y ago
  That's an excellent idea. You can work out a reasonable amount of time to allow YouTube to do the video processing, and then use youtube-dl's "--datebefore" option to only download videos that are at least that old. You'll have to manually compute the date and time for X hours in the past, I suppose.
  - ehPReth 9y ago
    Thanks for the hint wrt --datebefore. I'll give it a go :)
- ehPReth 9y ago
  Ah! That could work indeed. Thanks :)

rcthompson 9y ago

I actually have a script running on a cron job on my personal server that does approximately this. It downloads into a Syncthing folder that automatically syncs to whatever devices I might want to watch videos on. The difference is I just have a set of playlists for it to download, so I'm relying on the YouTube creators to organize their videos into appropriate playlists, which means it's not exactly "fine-grained". It's basically an 80% solution, and for the remaining 20%, I just invoke youtube-dl manually.

iggg 9y ago
Uploads is a playlist.
- rcthompson 9y ago
  For the few YouTube channels where I actually want to watch every single video uploaded by the creator, I do use this. For the rest, I have to hope that they create playlists for the subset of their videos that I am interested in. Many creators are actually quite diligent about organizing their uploads into playlists.

spapas82 9y ago

For anybody interested I've written a tutorial for youtube-dl: https://spapas.github.io/2018/03/06/easy-youtube-mp3-downloading/ https://spapas.github.io/2018/03/06/easy-youtube-mp3-downloa...

nishparadox 9y ago
I have a project that kinda acts as music assistant: "Search and play any song from terminal" The project is somewhat inactive right now due to personal priorities. http://github.com/NISH1001/playx http://github.com/NISH1001/playx

arca_vorago 9y ago

If you like youtube-dl you should check out mps-youtube (https://github.com/mps-youtube/mps-youtube https://github.com/mps-youtube/mps-youtube) which has the ability to use youtuble-dl as the backend I'm pretty sure (i prefer to use the native one personally) It's also gpl vs youtube-dl's public domain. I'm sure eventually I'll figure out a way to just do it in emacs. (I'm mostly joking... kinda... maybe.)

app4soft 9y ago
> mps-youtube Funny that state[0] on PyPi[1] is [downloads|no longer available] [0] https://img.shields.io/pypi/dm/mps-youtube.svg https://img.shields.io/pypi/dm/mps-youtube.svg [1] https://pypi.python.org/pypi/mps-youtube https://pypi.python.org/pypi/mps-youtube
- akx 9y ago
  PyPI no longer tracks download counts on its own. Instead, they pipe access logs to Google BigTable, so you can BigQuery the heck out of them.
app4soft 9y ago
> (i prefer to use the native one personally) Then you should try - SMTube[0]. SMTube allow use "mpv + youtube-dl" for play videos.[1] [0] http://www.smtube.org/ http://www.smtube.org/ [1] https://blog.smplayer.info/how-to-download-videos-with-the-new-smtube/ https://blog.smplayer.info/how-to-download-videos-with-the-n...

enz 9y ago

You don't need to scrap the page, YouTube provides the RSS stream for each channel: `https://www.youtube.com/feeds/videos.xml?channel_id=[ID]` https://www.youtube.com/feeds/videos.xml?channel_id=[ID]`. The RSS contains the "normal" link of each video, so you can invoke youtube-dl or mpv on it.

latexr 9y ago
> You don't need to scrap the page They do if they are scraping playlists. Youtube assumes that when creators make a playlist, they’ll add videos to the top. As such, the RSS for a playlist[1] always returns just the top 15 results. But a lot of creators — typically the ones that take greater care into organising their playlists, such as Crash Course[2] — add videos to the bottom of their playlists. Those RSS feeds will be stuck at the same 15 items forever, and your only option is to scrape the page. [1]: https://www.youtube.com/feeds/videos.xml?playlist_id={playlist_id} https://www.youtube.com/feeds/videos.xml?playlist_id={playli... [2]: https://www.youtube.com/user/crashcourse https://www.youtube.com/user/crashcourse
- zaarn 9y ago
  You can use youtube-dl's archiving feature for that. I use it regularly on my fairly large music playlist and it goes through all 20 pages just fine despite adding stuff on the bottom occasionally (I sorted it by creation date)

endisukaj 9y ago

> ad-free You don't want the people who make the content you enjoy to earn a living doing that?

dredmorbius 9y ago

See also mps-youtube, a Python utility installable via pip.