7 ms·

Online streaming sites and URL obfuscation

67 points by Jonhoo 11y ago

hackerboos 11y ago
rtmpdump [1] and livestreamer [2] are other good tools for this type of thing. I've always had trouble trying to rip JWPlayer videos that sit behind a login. I did manage it but I had to stream (second-by-second) the entire video. [1] - https://rtmpdump.mplayerhq.hu/ https://rtmpdump.mplayerhq.hu/ [2] - http://docs.livestreamer.io/ http://docs.livestreamer.io/
- Pengwin 11y ago
  Youtube-dl [1] is my go to tool for pulling video content out of a page, and its source [2] is a nice lesson in de-obfuscating the sites it supports [1] = https://rg3.github.io/youtube-dl/ https://rg3.github.io/youtube-dl/ [2] - https://github.com/rg3/youtube-dl/tree/master/youtube_dl/extractor https://github.com/rg3/youtube-dl/tree/master/youtube_dl/ext...
tlow 11y ago
For a long time I've used the by hand method of URL extraction either inspecting source code and searching for .flv or .mp4 filetypes (or simply "file:*"). The other method I use is the built in developer tools "network" tab of chrome under "inspect element". So far, I've only encountered one host (sorry I cannot remember the name) which I wasn't able to extract the source file via one of these two methods.
- Jonhoo 11y ago
  Yes, the second method I used to use quite frequently, but it means you have to open your browser, click play to get the video to load (which will spawn a torrent of ad pop-ups and captcha prompts and whatnot). This way I can get the download straight from the command line.
NickHaflinger 11y ago
How do you use streamsh? https://github.com/jonhoo/streamsh/blob/master/tricks/jwplayer.sh https://github.com/jonhoo/streamsh/blob/master/tricks/jwplay...
- Jonhoo 11y ago
  stream.sh <video URL (preferably the embed URL)> <filename to download to (without extension)>
tobltobs 11y ago
At the end he says something like "We need Encrypted Media Extensions to get ad free streaming...". Hmm, never thought about that this way. Maybe he is right, but couldn't it be that with EME the ads will not be sitting next to the player, instead the streamed video will contain the ads? Or does anybody believe (enough) people will pay for streaming on youtube or similar pages?
- madeofpalk 11y ago
  > the ads will not be sitting next to the player, instead the streamed video will contain the ads? This tech already exists, and it's called 'Server Side Ad Insertion', with Brightcove, The Platform, Adobe and Google all offering solutions. It's actually pretty cool - it works by instead of streaming a baked file, the client gets a 'live stream' URL and ads are inserted into that stream, dynamically, just like regular video. It's like having `badge.svg` image that's actually a PHP script that dynamically creates an SVG. I'm not overly familiar with video tech, so I've definitely butchered the exclamation.
  - TheLoneWolfling 11y ago
    And then ad blockers will snip out the ad portion of the stream and the cat and mouse game will continue.
    - userbinator 11y ago
      VCRs did that long before streaming became popular, as in those days the only way ads could appear were as directly in the video signal. https://en.wikipedia.org/wiki/Commercial_skipping https://en.wikipedia.org/wiki/Commercial_skipping Not surprisingly there was some legal action, but I think it's rather unenforceable in practice, and saying it's illegal to do so is nearly equivalent to saying that you do not have the freedom to change the channel, look away, or even close your eyes - a rather scary situation.
anseljh 11y ago
"streaming freedom" is a really clever euphemism.
stabilo 11y ago
Is a shorter solution possible? Is decompiling SWF and/or de-obfuscating Javascript really necessary? Years ago, I approached these sites like the author in the blog post. I spent hours reading about Flash and figuring out the schemes websites used. I decompiled swf. I also tried rtmpdump, livestreamer and other all-in-one solutions. I used early youtube downloaders (clive, etc) and later quvi. I also used /dev/bpf to capture HTTP requests. I tried it all. I can relate to what the author is trying to do. I never settle for "streaming". Download only. However I found over the years either the websites have made things easier or I was simply trying too hard. Or maybe I just am not interested in the type of video that is served in this way. For some reason it takes much less effort now. To make sure I am not imagining this, I decided to try one of the author's examples. I chose projectfreetv.so 1. I followed a link on the main page for some TV show I have never seen. 2. I read the HTML. 3. I noticed /watch/?aff_id= URL's. 4. I chose the first one, 493165, and followed it. 5. I read the HTML. 6. I noticed a URL pointing to a website that serves TV shows and followed it. 7. I read the HTML. 8. I noticed a URL pointing to an MP4 file. 9. I downloaded the file. I assume I got the TV episode on offer. The screen size is a little small but the file was about 89MB and appeared to be the full episode. Total time: less than 10min What did I use? sed, netcat, less and ftp for the download. The entire process is a one-liner. If anyone has some more examples to try, please list them. I would welcome the challenge.
- Jonhoo 11y ago
  Author here: for some sites, that approach works well, and this is what I originally did (and why I decided to write streamsh in bash). Have a look in the "tricks" directory and you'll see some of the much simpler solutions that are tried first. However, an increasing number of sites try to hide the URL, and through increasingly complicated means. The article is about vidbull.com, but there are others. They often also layer multiple obfuscators (vidbull uses both JS obfuscation and URL encryption for example). While it's true you can get pretty far just by scanning for mp4s on some mirrors, they are becoming fewer and farther between.
  - stabilo 11y ago
    vidbull looks more like the old "file storage locker" sites that were popular before the megaupload crackdown. Isn't this the same content you can find using torrent sites? Is it likely that vidbull would have something for which no torrent would ever exist? Sorry for my ignorance.
wodenokoto 11y ago
Many of these sites have started supporting iOS. Wouldn't it be easier to spoof iOS Safari and then only have to dig around in JavaScript?

67 points by Jonhoo 11y ago

rtmpdump [1] and livestreamer [2] are other good tools for this type of thing. I've always had trouble trying to rip JWPlayer videos that sit behind a login. I did manage it but I had to stream (second-by-second) the entire video. [1] - https://rtmpdump.mplayerhq.hu/ https://rtmpdump.mplayerhq.hu/ [2] - http://docs.livestreamer.io/ http://docs.livestreamer.io/

Pengwin 11y ago
Youtube-dl [1] is my go to tool for pulling video content out of a page, and its source [2] is a nice lesson in de-obfuscating the sites it supports [1] = https://rg3.github.io/youtube-dl/ https://rg3.github.io/youtube-dl/ [2] - https://github.com/rg3/youtube-dl/tree/master/youtube_dl/extractor https://github.com/rg3/youtube-dl/tree/master/youtube_dl/ext...

tlow 11y ago

For a long time I've used the by hand method of URL extraction either inspecting source code and searching for .flv or .mp4 filetypes (or simply "file:*"). The other method I use is the built in developer tools "network" tab of chrome under "inspect element". So far, I've only encountered one host (sorry I cannot remember the name) which I wasn't able to extract the source file via one of these two methods.

Jonhoo 11y ago
Yes, the second method I used to use quite frequently, but it means you have to open your browser, click play to get the video to load (which will spawn a torrent of ad pop-ups and captcha prompts and whatnot). This way I can get the download straight from the command line.

NickHaflinger 11y ago

How do you use streamsh? https://github.com/jonhoo/streamsh/blob/master/tricks/jwplayer.sh https://github.com/jonhoo/streamsh/blob/master/tricks/jwplay...

Jonhoo 11y ago
stream.sh <video URL (preferably the embed URL)> <filename to download to (without extension)>

tobltobs 11y ago

At the end he says something like "We need Encrypted Media Extensions to get ad free streaming...". Hmm, never thought about that this way. Maybe he is right, but couldn't it be that with EME the ads will not be sitting next to the player, instead the streamed video will contain the ads? Or does anybody believe (enough) people will pay for streaming on youtube or similar pages?

madeofpalk 11y ago
> the ads will not be sitting next to the player, instead the streamed video will contain the ads? This tech already exists, and it's called 'Server Side Ad Insertion', with Brightcove, The Platform, Adobe and Google all offering solutions. It's actually pretty cool - it works by instead of streaming a baked file, the client gets a 'live stream' URL and ads are inserted into that stream, dynamically, just like regular video. It's like having `badge.svg` image that's actually a PHP script that dynamically creates an SVG. I'm not overly familiar with video tech, so I've definitely butchered the exclamation.
- TheLoneWolfling 11y ago
  And then ad blockers will snip out the ad portion of the stream and the cat and mouse game will continue.
  - userbinator 11y ago
    VCRs did that long before streaming became popular, as in those days the only way ads could appear were as directly in the video signal. https://en.wikipedia.org/wiki/Commercial_skipping https://en.wikipedia.org/wiki/Commercial_skipping Not surprisingly there was some legal action, but I think it's rather unenforceable in practice, and saying it's illegal to do so is nearly equivalent to saying that you do not have the freedom to change the channel, look away, or even close your eyes - a rather scary situation.

anseljh 11y ago

"streaming freedom" is a really clever euphemism.

stabilo 11y ago

Is a shorter solution possible? Is decompiling SWF and/or de-obfuscating Javascript really necessary? Years ago, I approached these sites like the author in the blog post. I spent hours reading about Flash and figuring out the schemes websites used. I decompiled swf. I also tried rtmpdump, livestreamer and other all-in-one solutions. I used early youtube downloaders (clive, etc) and later quvi. I also used /dev/bpf to capture HTTP requests. I tried it all. I can relate to what the author is trying to do. I never settle for "streaming". Download only. However I found over the years either the websites have made things easier or I was simply trying too hard. Or maybe I just am not interested in the type of video that is served in this way. For some reason it takes much less effort now. To make sure I am not imagining this, I decided to try one of the author's examples. I chose projectfreetv.so 1. I followed a link on the main page for some TV show I have never seen. 2. I read the HTML. 3. I noticed /watch/?aff_id= URL's. 4. I chose the first one, 493165, and followed it. 5. I read the HTML. 6. I noticed a URL pointing to a website that serves TV shows and followed it. 7. I read the HTML. 8. I noticed a URL pointing to an MP4 file. 9. I downloaded the file. I assume I got the TV episode on offer. The screen size is a little small but the file was about 89MB and appeared to be the full episode. Total time: less than 10min What did I use? sed, netcat, less and ftp for the download. The entire process is a one-liner. If anyone has some more examples to try, please list them. I would welcome the challenge.

Jonhoo 11y ago
Author here: for some sites, that approach works well, and this is what I originally did (and why I decided to write streamsh in bash). Have a look in the "tricks" directory and you'll see some of the much simpler solutions that are tried first. However, an increasing number of sites try to hide the URL, and through increasingly complicated means. The article is about vidbull.com, but there are others. They often also layer multiple obfuscators (vidbull uses both JS obfuscation and URL encryption for example). While it's true you can get pretty far just by scanning for mp4s on some mirrors, they are becoming fewer and farther between.
- stabilo 11y ago
  vidbull looks more like the old "file storage locker" sites that were popular before the megaupload crackdown. Isn't this the same content you can find using torrent sites? Is it likely that vidbull would have something for which no torrent would ever exist? Sorry for my ignorance.

wodenokoto 11y ago

Many of these sites have started supporting iOS. Wouldn't it be easier to spoof iOS Safari and then only have to dig around in JavaScript?