4 ms·

Not sure if DMCA is invoked to take them down, but a friend of mine recently challenged me to scrape a video file from one of these shady free streaming sites.

by RajT88 3y ago

Not sure if DMCA is invoked to take them down, but a friend of mine recently challenged me to scrape a video file from one of these shady free streaming sites. I promise this has something to do with GitHub. Gamely, I accepted the challenge. Browser dev mode to sniff things out was very difficult - they put a lot of work into jamming that up. Turns out it is trivially easy with Fiddler. What I learned in the process was interesting, trying on a few video files. What these sites do is slice up their video files into thousands of chunks, and then rename the files as innocuous file extensions (like *.jpg, *.html, *.csv, etc.) and stick them on various hosting services where you can do a GET on the file for free with no authentication. They spread those thousands of files for each video file all over the internet. Usually across multiple accounts / repos on a service like GitHub but not just GitHub - any place they can manage. A lot I looked at while testing had files on GitHub though. Then, once the thousands of files are distributed, they stitch them together via an *.m3u8 playlist file which the in-browser player can handle buffering to make it all seamless.

Grimblewald 3y ago
Thanks for sharing, that was enormously intriguing. Do you have a bigger write-up on this? I'd love to read it.
- deleted 3y ago
  [deleted]
- EMIRELADERO 3y ago
  https://github.com/movie-web https://github.com/movie-web covers this quite well
  - notso411 3y ago
    [dead]
  - cosmojg 3y ago
    Oh wow, this is quite practical! Thanks for sharing! In the past, I've typically relied on a great deal of google-fu and considerable time investment (~10 minutes! *gasp*) whenever I couldn't find a show on one of the big streaming services, and it seems like the backend sources made available by the movie-web extension are a superset of those used by the various shady and ephemeral middleman-of-the-month streaming websites I would typically land on. It's also quite nice how they've neatly open-sourced and atomized all the little tools that go into making the service work such that one could, in theory, build other services on top of these tools. The quality of the documentation is outstanding, too.
    - EMIRELADERO 3y ago
      > It's also quite nice how they've neatly open-sourced and atomized all the little tools that go into making the service work such that one could, in theory, build other services on top of these tools. The quality of the documentation is outstanding, too. Astonishing is the right word. It's like "neat sparkling new JS-based startup" meets "mid-2000s underground piracy", and I can't get enough of it.
- Retr0id 3y ago
  I wrote a twitter thread about doing a similar trick with twitter's image CDN: https://twitter.com/David3141593/status/1587978423120666625 https://twitter.com/David3141593/status/1587978423120666625 Unfortunately I'm unable to access twitter threads now so ymmv - I think it still works. The TL;DR is that the playlist metadata file (mpd or m3u8) references multiple fragments, and arbitrary file offsets within those fragments. Slice your video into fragments (using standard video processing tools), embed them into PNGs using standard twitter-compatible file format polyglot tricks, and upload them to twitter.
treyd 3y ago
Why wouldn't they do some basic erasure coding (or even just xor with a repeating pattern) to get around the trivial DMCA takedowns that's inviting?
- RajT88 3y ago
  If they were that sophisticated, they probably wouldn't be involved in movie piracy. I'm sure there's a certain amount of laziness that also comes with how keen each site is on enforcement. The most recent one I looked at, all the files were on the same site, in the same folder.
  - xp84 3y ago
    > If they were that sophisticated, they probably wouldn't be involved That's an odd assertion. Some of the smartest and most sophisticated people I've ever interacted with were very actively involved in the piracy scene. These types love a challenge, and that's why they get involved in piracy. In some cases they aren't even all that into the content itself. I don't unilaterally hate all IP (other than patents) btw, so this is not an endorsement of piracy itself, just a frank acknowledgement that a lot of these people are very smart.
    - RajT88 3y ago
      Let me clarify what I'm getting at: These streaming sites are laden with ads, and some even have offer subscriptions, so there's a profit motivation for the people that run them. I'm not talking about the people who upload torrents (there's not much in a profit motive there, since that stuff is hard to monetize), or people who crack software. That sort of stuff. Interviews digging into their motivations have been going around for a long time now. If you're hacking on running a pirate streaming site to make money off it, if you have skills enough to stay ahead of the people constantly trying to take your content down, at some point you're going to realize you can make a lot more money using those skills working for a legit company. My take on the profit motive for malware authors is not the same; a malware dev can make a lot of money if the group manages to hit the right victims. The ones who are state sponsored probably make decent money too.
mdrzn 3y ago
Very interesting, would love to know more about it!
swozey 3y ago
I worked in this space on the CDN non-pirate side and it's basically the same thing but everyone (including the major ones you know) distribute all of their content like that over various cdns and they're all competitively tested against one another for errors/latency/etc and traffic auto-shifted based on live current cost + that magic. If you're interested in some of the tools used that essentially re-unites from endless sources, auto determines bitrate, allows you to interject interstitial ads based on geography/user tokens etc, check out things like bitmovin. No relation, I've just used it. Then cedexis for the latency/error-rate magic that makes your cdns compete against one another. There's also jwplayer which a lot of video platforms like Univision and Fox use. https://bitmovin.com/ https://bitmovin.com/ https://github.com/bitmovin https://github.com/bitmovin Another interesting thing is how the DRM is handled, the video chunks that are all muxed/reunited/demuxed, etc on the fly- https://developer.bitmovin.com/encoding/docs/how-to-protect-your-content-with-bitmovin-and-vualto-drm#creating-manifest-for-dash https://developer.bitmovin.com/encoding/docs/how-to-protect-... I started out in video stuff back in RTSP days (2005ish), took a LONG break in that area (2016ish) and the systems today are so cool. IIRC back then everything was rtsp. I actually have no idea beyond what you just mentioned how the pirate side does things so that's neat. I wonder what tools they're using. https://jwplayer.com/ https://jwplayer.com/ https://www.cedexis.com/ https://www.cedexis.com/ .. which I guess was bought by citrix so I have no idea whats up with that now.
IncRnd 3y ago
That's like RAID0 for streaming - or in a different way like people who backup their drive onto gmail messages, where each file block fits into a gmail message.
- RajT88 3y ago
  Heh. That's how the warez scene used to operate back in the AOL chatroom days. You would request a bot to send you a file, and it'd forward the whole series of messages to you, which you'd add to your download manager, and then un-rar once done. I've heard.
johndough 3y ago
I noticed that many of those websites abuse the debugger functionality, freezing the browser whenever someone tries to open the developer console. Did you find a way around that?
- password4321 3y ago
  The post mentioned using Fiddler instead. https://docs.telerik.com/fiddler/configure-fiddler/tasks/installfiddler https://docs.telerik.com/fiddler/configure-fiddler/tasks/ins... If read-only access is all that is needed, SSLKEYLOGFILE might help. https://my.f5.com/manage/s/article/K50557518 https://my.f5.com/manage/s/article/K50557518
- RajT88 3y ago
  Yes. You can disable breaking in dev mode, which gets you a bit further. But they have some script in there as well that checks for if you're in dev mode somehow, and as well on top of that misbehaves if you're in private mode. At that point I got tired of fighting it, and realized none of this Javascript nonsense was going to work against Fiddler or other proxy-based solution.
  - Invertigrate 3y ago
    You don't even need to do that, just get the video to start playing and then press F12 to open the dev tools as the video will keep streaming and you'll still be able to see the network requests despite the messages.
    - RajT88 3y ago
      That approach will miss the transmission of the playlist file. Chasing the individual fragments of video is not the right way to go about it.
tamimio 3y ago
> Then, once the thousands of files are distributed, they stitch them together via an *.m3u8 playlist file which the in-browser player can handle buffering I did try streaming one time with that method, besides the high latency (~5seconds), it was CPU intensive for both the server and the client(browser), so not so sure how viable this would be.
RajT88 3y ago
Wow, this comment got a lot of attention! If anyone is wondering the other piece of this, which is re-unifying all the thousands of video fragments, the game is over once you have the URI to the playlist file. You feed that URI into VLC's "Convert/Save" functionality and it handles it like a champ. No need to script out anything to grab the pieces and work FFMPEG magic. At the end of the day, it is a 2 step process I could document for my friend who is more or less a layman. A skilled user (decent with Excel), but not a power user (can't script).
Invertigrate 3y ago
Just looked this up for myself and it's indeed partially correct (mostly webp, jpg, png, etc though). You left out an important part which is that they're proxying these requests through their own servers to avoid the source being found. This way they could be placing the files on servers that actually require some sort of authentication such as a login page and once that's done the sky is their limit as you could even be hosting files on Discord and it would be very difficult to find the source. User -> Pirate proxy server -> Github (or some other site)

by RajT88 3y ago

Grimblewald 3y ago

Thanks for sharing, that was enormously intriguing. Do you have a bigger write-up on this? I'd love to read it.

deleted 3y ago
[deleted]
EMIRELADERO 3y ago
https://github.com/movie-web https://github.com/movie-web covers this quite well
- notso411 3y ago
  [dead]
- cosmojg 3y ago
  Oh wow, this is quite practical! Thanks for sharing! In the past, I've typically relied on a great deal of google-fu and considerable time investment (~10 minutes! *gasp*) whenever I couldn't find a show on one of the big streaming services, and it seems like the backend sources made available by the movie-web extension are a superset of those used by the various shady and ephemeral middleman-of-the-month streaming websites I would typically land on. It's also quite nice how they've neatly open-sourced and atomized all the little tools that go into making the service work such that one could, in theory, build other services on top of these tools. The quality of the documentation is outstanding, too.
  - EMIRELADERO 3y ago
    > It's also quite nice how they've neatly open-sourced and atomized all the little tools that go into making the service work such that one could, in theory, build other services on top of these tools. The quality of the documentation is outstanding, too. Astonishing is the right word. It's like "neat sparkling new JS-based startup" meets "mid-2000s underground piracy", and I can't get enough of it.
Retr0id 3y ago
I wrote a twitter thread about doing a similar trick with twitter's image CDN: https://twitter.com/David3141593/status/1587978423120666625 https://twitter.com/David3141593/status/1587978423120666625 Unfortunately I'm unable to access twitter threads now so ymmv - I think it still works. The TL;DR is that the playlist metadata file (mpd or m3u8) references multiple fragments, and arbitrary file offsets within those fragments. Slice your video into fragments (using standard video processing tools), embed them into PNGs using standard twitter-compatible file format polyglot tricks, and upload them to twitter.

treyd 3y ago

Why wouldn't they do some basic erasure coding (or even just xor with a repeating pattern) to get around the trivial DMCA takedowns that's inviting?

RajT88 3y ago
If they were that sophisticated, they probably wouldn't be involved in movie piracy. I'm sure there's a certain amount of laziness that also comes with how keen each site is on enforcement. The most recent one I looked at, all the files were on the same site, in the same folder.
- xp84 3y ago
  > If they were that sophisticated, they probably wouldn't be involved That's an odd assertion. Some of the smartest and most sophisticated people I've ever interacted with were very actively involved in the piracy scene. These types love a challenge, and that's why they get involved in piracy. In some cases they aren't even all that into the content itself. I don't unilaterally hate all IP (other than patents) btw, so this is not an endorsement of piracy itself, just a frank acknowledgement that a lot of these people are very smart.
  - RajT88 3y ago
    Let me clarify what I'm getting at: These streaming sites are laden with ads, and some even have offer subscriptions, so there's a profit motivation for the people that run them. I'm not talking about the people who upload torrents (there's not much in a profit motive there, since that stuff is hard to monetize), or people who crack software. That sort of stuff. Interviews digging into their motivations have been going around for a long time now. If you're hacking on running a pirate streaming site to make money off it, if you have skills enough to stay ahead of the people constantly trying to take your content down, at some point you're going to realize you can make a lot more money using those skills working for a legit company. My take on the profit motive for malware authors is not the same; a malware dev can make a lot of money if the group manages to hit the right victims. The ones who are state sponsored probably make decent money too.

mdrzn 3y ago

Very interesting, would love to know more about it!

swozey 3y ago

I worked in this space on the CDN non-pirate side and it's basically the same thing but everyone (including the major ones you know) distribute all of their content like that over various cdns and they're all competitively tested against one another for errors/latency/etc and traffic auto-shifted based on live current cost + that magic. If you're interested in some of the tools used that essentially re-unites from endless sources, auto determines bitrate, allows you to interject interstitial ads based on geography/user tokens etc, check out things like bitmovin. No relation, I've just used it. Then cedexis for the latency/error-rate magic that makes your cdns compete against one another. There's also jwplayer which a lot of video platforms like Univision and Fox use. https://bitmovin.com/ https://bitmovin.com/ https://github.com/bitmovin https://github.com/bitmovin Another interesting thing is how the DRM is handled, the video chunks that are all muxed/reunited/demuxed, etc on the fly- https://developer.bitmovin.com/encoding/docs/how-to-protect-your-content-with-bitmovin-and-vualto-drm#creating-manifest-for-dash https://developer.bitmovin.com/encoding/docs/how-to-protect-... I started out in video stuff back in RTSP days (2005ish), took a LONG break in that area (2016ish) and the systems today are so cool. IIRC back then everything was rtsp. I actually have no idea beyond what you just mentioned how the pirate side does things so that's neat. I wonder what tools they're using. https://jwplayer.com/ https://jwplayer.com/ https://www.cedexis.com/ https://www.cedexis.com/ .. which I guess was bought by citrix so I have no idea whats up with that now.

IncRnd 3y ago

That's like RAID0 for streaming - or in a different way like people who backup their drive onto gmail messages, where each file block fits into a gmail message.

RajT88 3y ago
Heh. That's how the warez scene used to operate back in the AOL chatroom days. You would request a bot to send you a file, and it'd forward the whole series of messages to you, which you'd add to your download manager, and then un-rar once done. I've heard.

johndough 3y ago

I noticed that many of those websites abuse the debugger functionality, freezing the browser whenever someone tries to open the developer console. Did you find a way around that?

password4321 3y ago
The post mentioned using Fiddler instead. https://docs.telerik.com/fiddler/configure-fiddler/tasks/installfiddler https://docs.telerik.com/fiddler/configure-fiddler/tasks/ins... If read-only access is all that is needed, SSLKEYLOGFILE might help. https://my.f5.com/manage/s/article/K50557518 https://my.f5.com/manage/s/article/K50557518
RajT88 3y ago
Yes. You can disable breaking in dev mode, which gets you a bit further. But they have some script in there as well that checks for if you're in dev mode somehow, and as well on top of that misbehaves if you're in private mode. At that point I got tired of fighting it, and realized none of this Javascript nonsense was going to work against Fiddler or other proxy-based solution.
- Invertigrate 3y ago
  You don't even need to do that, just get the video to start playing and then press F12 to open the dev tools as the video will keep streaming and you'll still be able to see the network requests despite the messages.
  - RajT88 3y ago
    That approach will miss the transmission of the playlist file. Chasing the individual fragments of video is not the right way to go about it.

tamimio 3y ago

> Then, once the thousands of files are distributed, they stitch them together via an *.m3u8 playlist file which the in-browser player can handle buffering I did try streaming one time with that method, besides the high latency (~5seconds), it was CPU intensive for both the server and the client(browser), so not so sure how viable this would be.

RajT88 3y ago

Wow, this comment got a lot of attention! If anyone is wondering the other piece of this, which is re-unifying all the thousands of video fragments, the game is over once you have the URI to the playlist file. You feed that URI into VLC's "Convert/Save" functionality and it handles it like a champ. No need to script out anything to grab the pieces and work FFMPEG magic. At the end of the day, it is a 2 step process I could document for my friend who is more or less a layman. A skilled user (decent with Excel), but not a power user (can't script).

Invertigrate 3y ago

Just looked this up for myself and it's indeed partially correct (mostly webp, jpg, png, etc though). You left out an important part which is that they're proxying these requests through their own servers to avoid the source being found. This way they could be placing the files on servers that actually require some sort of authentication such as a login page and once that's done the sky is their limit as you could even be hosting files on Discord and it would be very difficult to find the source. User -> Pirate proxy server -> Github (or some other site)