5 ms·

If you’re YouTube or any site, and want to stop these sort of wrappers - what’s the easiest way to do so without breaking your own site? I find this task to be

If you’re YouTube or any site, and want to stop these sort of wrappers - what’s the easiest way to do so without breaking your own site? I find this task to be an interesting engineering problem. A related question is if there’s an unspoofable way to detect a client.

sschueller 5y ago
Provide a public API (charge for it if you have too). The videos on YouTube are the property of the creators, not YouTube.
- charcircuit 5y ago
  YouTube already has a public API
samisupset 5y ago
Keep changing the implementation, keep changing names, keep changing the API formats. I'm definitely curious if there's a way to do a rotation that resists easy automatic code analysis.
- FrenchDevRemote 5y ago
  you don't even have to look at the script, just at the network requests you'd basically have to make your own stealthy video format, otherwise you can just catch network requests
- comprev 5y ago
  Facebook does something similar to combat adblockers. They mangle the names of div elements to make sponsored posts indistinguishable from friends/group posts. I'm not aware of any browser plugins which are effective at blocking FB ads. Anyone know if other websites put as much effort into anti-adblock engineering?
  - lostmsu 5y ago
    FB Purity seems to be keeping up. But it is not a general purpose adblocker.
  - wantlotsofcurry 5y ago
    I think Workday does the same. Trying to scrape data from there was a nightmare (team didn't have api access).
  - mgdlbp 5y ago
    Could you not base a filter on whether a div contains text matching your list of groups and friends?
  - oefrha 5y ago
    A vision (ML)-based blocker should easily blow rigid rule-based cosmetic blockers out of the water and render div soup obfuscation completely useless. Not sure anyone’s investing in that space though. Once you’ve defeated obfuscation you’ll be dealing with DOM integrity-based defenses.
FrenchDevRemote 5y ago
Given the power of Google they may be able to force you to sign in after watching a few videos. Or only serve videos if they detect some kind of physical input(mouse/keyboard) But they don't seem to care that much for non-mainstream tools, or this would be blocked already For small scale sites it's definitely not easy to block They may be some paid libraries that promises to do it, but I've never seen one that seemed really unbreakable Even if you go all the way and block EVERY possible way of doing this, you can make a puppeteer script that could watch and record sound/video directly on the screen in like 15 lines of code, even if it would be really slow to get long videos
Nextgrid 5y ago
The only bulletproof way is to change your business model to make these useless, irrelevant or no longer threatening. Yours and the user’s incentives should be aligned. If you serve content, put it behind a paywall and rate-limit based on the maximum amount a human can reasonably consume and stop caring whether the user uses your own client or something like this - after all you’re getting paid either way. The only businesses that are threatened by unofficial API clients are cancerous “growth and engagement” crap where the “value” is the wasting of the user’s time. Don’t be such a business and you’ll be fine.
- motoxpro 5y ago
  What about something like Craigslist? I would rather that not be behind a paywall. I would think they would also enjoy not having their site scraped and re-created.
xyst 5y ago
So a form of DRM on the APIs themselves? It wouldn't be easy. How would you determine if the request is originating from an official YT developed app (iOS, Android, ...) or client (YouTube.com); or a non-approved client? I suppose you can have a system which works like TOTP except for machine-to-machine. Although it would probably be broken since anything on the client side can be disassembled. The UX would likely suffer as a result as well.
- freedomben 5y ago
  I think the best you could do is attempt detection/fingerprinting, but I don't think it's possible to stop it since you never have full control of the client. I once saw a certificate-based implementation once where the server issued a temporary x509 cert to the client and then used mTLS[1]. It did reduce "unauthorized" clients by raising the bar, but it ended up making life way harder for their devs and the people they really wanted to stop just implemented the cert strategy and moved on. [1]: https://freedomben.medium.com/what-is-mtls-and-how-does-it-work-9dcdbf6c1e41?sk=45af139ec181252e711590b6ca73dc2f https://freedomben.medium.com/what-is-mtls-and-how-does-it-w...
deleted 5y ago
[deleted]
ec109685 5y ago
The demos have Node.JS examples. If that's the case, it doesn't seem possible to block. If it's running in a browser, Google can simply disallow those domains to make API calls. Also, at least Apple will block apps that make unauthorized API calls to third parties.
- Bilal_io 5y ago
  The agent can be faked. There are CORS rules, but those are enforced by the browser, a backend cannot prevent you from calling it, except by requiring an access token or something similar.
freedomben 5y ago
It's probably impossible to stop it, but you could make it pretty painful by aggressively breaking/changing the API contract. You could maximally engage human ID and fingerprinting techniques and CAPTCHAs, or do machine learning on usage patterns to find likely bots and ban them if you're willing to accept that false positives will hurt innocent customers. Whatever you decide to do, it won't be free because it will also hurt your own devs (such as making it harder and more complex to work on). Obviously your own devs can have advance notice for changes, but it's still a pain to deal with. Overall though, I would seriously ask why? Anti-cheat for a game maybe? It will cost you time/money to prevent, and it will hurt people that like your product enough to hack on it. As a user who rejoices in having APIs I can use to automate products I like, I'd be far more likely to pay you if you have an API I can use.
smrtinsert 5y ago
Require request tokens. Authorized clients would have some sort of generator, anything else would fail to execute.
mediocregopher 5y ago
There's no perfect solution, but you can make it painful. One solution I've seen, which only works in a server-side rendered site, is for the server to generate a random name for each form field being rendered. The mapping of random id to real field name is kept in the user's session information server-side, so the translation is then done server-side as well whenever the user performs an action. At that point anyone writing a library like this would need to actually pull in the rendered page on which the user is supposed to be navigated, scrape the field names off of that (which won't be easy), and only _then_ could they perform the form action. But if you're a big enough site, someone will likely still take the time to do it.
guessbest 5y ago
Every few requests put up a catchpa. That is how google did it for their search https://support.google.com/websearch/answer/86640?hl=en https://support.google.com/websearch/answer/86640?hl=en
- nomel 5y ago
  Wouldn't there be all sorts of human detection that they could do, similar to how game cheat engines work? A human is going to move the mouse across elements, drag, poke the screen, be slow, etc, and all in fairly predictable ways. Some API calls almost certainly require human interaction, where some interaction graph could be feed as a key to the API. It's cat and mouse, but at some point the mouse is going to get tired.
  - ziziyO 5y ago
    This is close to how recaptcha v3 works. It can look at the users behavior on the site and classify normal users vs bot users. You have to do some setup to feed your own set of user action data into recaptcha though.
heavyset_go 5y ago
Seems like a use case for remote attestation.
- no_time 5y ago
  Indeed. Yet another reason why we must rally against such efforts.
  - heavyset_go 5y ago
    Yeah, it's not a future that I'm looking forward to.
inlined 5y ago
Firebase app check does this and I wouldn’t be surprised if YouTube adopts it eventually.
jszymborski 5y ago
I mean, offering your own API gateway is pretty effective.
tptacek 5y ago
Youtube in particular sort of famously had content protection code, which took in part the form of a VM implemented in Javascript that probed its runtime to detect non-browser or otherwise headless clients. I think Mike Hearn worked on it. Found it: https://moderncrypto.org/mail-archive/messaging/2014/000780.html https://moderncrypto.org/mail-archive/messaging/2014/000780....
- tedunangst 5y ago
  There doesn't seem to be a whole lot in there on YouTube js obfuscation techniques?

sschueller 5y ago

Provide a public API (charge for it if you have too). The videos on YouTube are the property of the creators, not YouTube.

charcircuit 5y ago
YouTube already has a public API

samisupset 5y ago

Keep changing the implementation, keep changing names, keep changing the API formats. I'm definitely curious if there's a way to do a rotation that resists easy automatic code analysis.

FrenchDevRemote 5y ago
you don't even have to look at the script, just at the network requests you'd basically have to make your own stealthy video format, otherwise you can just catch network requests
comprev 5y ago
Facebook does something similar to combat adblockers. They mangle the names of div elements to make sponsored posts indistinguishable from friends/group posts. I'm not aware of any browser plugins which are effective at blocking FB ads. Anyone know if other websites put as much effort into anti-adblock engineering?
- lostmsu 5y ago
  FB Purity seems to be keeping up. But it is not a general purpose adblocker.
- wantlotsofcurry 5y ago
  I think Workday does the same. Trying to scrape data from there was a nightmare (team didn't have api access).
- mgdlbp 5y ago
  Could you not base a filter on whether a div contains text matching your list of groups and friends?
- oefrha 5y ago
  A vision (ML)-based blocker should easily blow rigid rule-based cosmetic blockers out of the water and render div soup obfuscation completely useless. Not sure anyone’s investing in that space though. Once you’ve defeated obfuscation you’ll be dealing with DOM integrity-based defenses.

FrenchDevRemote 5y ago

Given the power of Google they may be able to force you to sign in after watching a few videos. Or only serve videos if they detect some kind of physical input(mouse/keyboard) But they don't seem to care that much for non-mainstream tools, or this would be blocked already For small scale sites it's definitely not easy to block They may be some paid libraries that promises to do it, but I've never seen one that seemed really unbreakable Even if you go all the way and block EVERY possible way of doing this, you can make a puppeteer script that could watch and record sound/video directly on the screen in like 15 lines of code, even if it would be really slow to get long videos

Nextgrid 5y ago

The only bulletproof way is to change your business model to make these useless, irrelevant or no longer threatening. Yours and the user’s incentives should be aligned. If you serve content, put it behind a paywall and rate-limit based on the maximum amount a human can reasonably consume and stop caring whether the user uses your own client or something like this - after all you’re getting paid either way. The only businesses that are threatened by unofficial API clients are cancerous “growth and engagement” crap where the “value” is the wasting of the user’s time. Don’t be such a business and you’ll be fine.

motoxpro 5y ago
What about something like Craigslist? I would rather that not be behind a paywall. I would think they would also enjoy not having their site scraped and re-created.

xyst 5y ago

So a form of DRM on the APIs themselves? It wouldn't be easy. How would you determine if the request is originating from an official YT developed app (iOS, Android, ...) or client (YouTube.com); or a non-approved client? I suppose you can have a system which works like TOTP except for machine-to-machine. Although it would probably be broken since anything on the client side can be disassembled. The UX would likely suffer as a result as well.

freedomben 5y ago
I think the best you could do is attempt detection/fingerprinting, but I don't think it's possible to stop it since you never have full control of the client. I once saw a certificate-based implementation once where the server issued a temporary x509 cert to the client and then used mTLS[1]. It did reduce "unauthorized" clients by raising the bar, but it ended up making life way harder for their devs and the people they really wanted to stop just implemented the cert strategy and moved on. [1]: https://freedomben.medium.com/what-is-mtls-and-how-does-it-work-9dcdbf6c1e41?sk=45af139ec181252e711590b6ca73dc2f https://freedomben.medium.com/what-is-mtls-and-how-does-it-w...

deleted 5y ago

[deleted]

ec109685 5y ago

The demos have Node.JS examples. If that's the case, it doesn't seem possible to block. If it's running in a browser, Google can simply disallow those domains to make API calls. Also, at least Apple will block apps that make unauthorized API calls to third parties.

Bilal_io 5y ago
The agent can be faked. There are CORS rules, but those are enforced by the browser, a backend cannot prevent you from calling it, except by requiring an access token or something similar.

freedomben 5y ago

It's probably impossible to stop it, but you could make it pretty painful by aggressively breaking/changing the API contract. You could maximally engage human ID and fingerprinting techniques and CAPTCHAs, or do machine learning on usage patterns to find likely bots and ban them if you're willing to accept that false positives will hurt innocent customers. Whatever you decide to do, it won't be free because it will also hurt your own devs (such as making it harder and more complex to work on). Obviously your own devs can have advance notice for changes, but it's still a pain to deal with. Overall though, I would seriously ask why? Anti-cheat for a game maybe? It will cost you time/money to prevent, and it will hurt people that like your product enough to hack on it. As a user who rejoices in having APIs I can use to automate products I like, I'd be far more likely to pay you if you have an API I can use.

smrtinsert 5y ago

Require request tokens. Authorized clients would have some sort of generator, anything else would fail to execute.

mediocregopher 5y ago

There's no perfect solution, but you can make it painful. One solution I've seen, which only works in a server-side rendered site, is for the server to generate a random name for each form field being rendered. The mapping of random id to real field name is kept in the user's session information server-side, so the translation is then done server-side as well whenever the user performs an action. At that point anyone writing a library like this would need to actually pull in the rendered page on which the user is supposed to be navigated, scrape the field names off of that (which won't be easy), and only _then_ could they perform the form action. But if you're a big enough site, someone will likely still take the time to do it.

guessbest 5y ago

Every few requests put up a catchpa. That is how google did it for their search https://support.google.com/websearch/answer/86640?hl=en https://support.google.com/websearch/answer/86640?hl=en

nomel 5y ago
Wouldn't there be all sorts of human detection that they could do, similar to how game cheat engines work? A human is going to move the mouse across elements, drag, poke the screen, be slow, etc, and all in fairly predictable ways. Some API calls almost certainly require human interaction, where some interaction graph could be feed as a key to the API. It's cat and mouse, but at some point the mouse is going to get tired.
- ziziyO 5y ago
  This is close to how recaptcha v3 works. It can look at the users behavior on the site and classify normal users vs bot users. You have to do some setup to feed your own set of user action data into recaptcha though.

heavyset_go 5y ago

Seems like a use case for remote attestation.

no_time 5y ago
Indeed. Yet another reason why we must rally against such efforts.
- heavyset_go 5y ago
  Yeah, it's not a future that I'm looking forward to.

inlined 5y ago

Firebase app check does this and I wouldn’t be surprised if YouTube adopts it eventually.

jszymborski 5y ago

I mean, offering your own API gateway is pretty effective.

tptacek 5y ago

Youtube in particular sort of famously had content protection code, which took in part the form of a VM implemented in Javascript that probed its runtime to detect non-browser or otherwise headless clients. I think Mike Hearn worked on it. Found it: https://moderncrypto.org/mail-archive/messaging/2014/000780.html https://moderncrypto.org/mail-archive/messaging/2014/000780....

tedunangst 5y ago
There doesn't seem to be a whole lot in there on YouTube js obfuscation techniques?