2 ms·

> For the web it requires that you run a snippet of javascript code (the challenge) in the browser to prove that you are not a bot. How does this prove you are

> For the web it requires that you run a snippet of javascript code (the challenge) in the browser to prove that you are not a bot. How does this prove you are not a bot. How does this code not work in a headless Chromimum if it's just client side JS?

Andrews54757 1y ago
Good question! Indeed you can run the challenge code using headless Chromium and it will function [1]. They are constantly updating the challenge however, and may add additional checks in the future. I suppose Google wants to make it more expensive overall to scrape Youtube to deter the most egregious bots. [1] https://github.com/LuanRT/BgUtils https://github.com/LuanRT/BgUtils
- toomuchtodo 1y ago
  LLMs solve challenges. Can we not solve these challenges with sufficiently advanced LLMs? Gemini even, if you're feeling lulz-y.
  - balder1991 1y ago
    Yes, by spending money.
    - toomuchtodo 1y ago
      I agree, in some cases and depending on LLM endpoint, some money may need to be spent to enable ripping. But is it cheaper than paying Youtube/Google? That is the question.
      - dylan604 1y ago
        sometimes, it's not about the cost. it's about who/where the money is being spent.
Beretta_Vexee 1y ago
Once JavaScript is running, it can perform complex fingerprinting operations that are difficult to circumvent effectively. I have a little experience with Selenium headless on Facebook. Facebook tests fonts, SVG rendering, CSS support, screen resolution, clock and geographical settings, and hundreds of other things that give it a very good idea of whether it's a normal client or Selenium headless. Since it picks a certain number of checks more or less at random and they can modify the JS each time it loads, it is very, very complicated to simulate. Facebook and Instagram know this and allow it below a certain limit because it is more about bot protection than content protection. This is the case when you have a real web browser running in the background. Here we are talking about standalone software written in Python.
- dylan604 1y ago
  why can a bot dev not just get all of these values from the laptop's settings and hardwire the headless version to have the same values?
  - Beretta_Vexee 1y ago
    Because the expected values are not fixed, it is possible to measure response times and errors to check whether something is in the cache or not, etc. There are a whole host of tricks relating to rendering and positioning at the edge of the display window and canvas rather than the window, which allow you to detect execution without rendering. To simulate all this correctly, you end up with a standard browser, standard execution times, full rendering in the background, etc. No one wants to download their YouTube video at 1x speed and wait for the adverts to finish.
- cylemons 1y ago
  How does testing rendering work? Can javascript get pixel data from the DOM
  - Beretta_Vexee 1y ago
    https://www.w3schools.com/tags/canvas_getimagedata.asp https://www.w3schools.com/tags/canvas_getimagedata.asp

Andrews54757 1y ago

Good question! Indeed you can run the challenge code using headless Chromium and it will function [1]. They are constantly updating the challenge however, and may add additional checks in the future. I suppose Google wants to make it more expensive overall to scrape Youtube to deter the most egregious bots. [1] https://github.com/LuanRT/BgUtils https://github.com/LuanRT/BgUtils

toomuchtodo 1y ago
LLMs solve challenges. Can we not solve these challenges with sufficiently advanced LLMs? Gemini even, if you're feeling lulz-y.
- balder1991 1y ago
  Yes, by spending money.
  - toomuchtodo 1y ago
    I agree, in some cases and depending on LLM endpoint, some money may need to be spent to enable ripping. But is it cheaper than paying Youtube/Google? That is the question.
    - dylan604 1y ago
      sometimes, it's not about the cost. it's about who/where the money is being spent.

Beretta_Vexee 1y ago

Once JavaScript is running, it can perform complex fingerprinting operations that are difficult to circumvent effectively. I have a little experience with Selenium headless on Facebook. Facebook tests fonts, SVG rendering, CSS support, screen resolution, clock and geographical settings, and hundreds of other things that give it a very good idea of whether it's a normal client or Selenium headless. Since it picks a certain number of checks more or less at random and they can modify the JS each time it loads, it is very, very complicated to simulate. Facebook and Instagram know this and allow it below a certain limit because it is more about bot protection than content protection. This is the case when you have a real web browser running in the background. Here we are talking about standalone software written in Python.

dylan604 1y ago
why can a bot dev not just get all of these values from the laptop's settings and hardwire the headless version to have the same values?
- Beretta_Vexee 1y ago
  Because the expected values are not fixed, it is possible to measure response times and errors to check whether something is in the cache or not, etc. There are a whole host of tricks relating to rendering and positioning at the edge of the display window and canvas rather than the window, which allow you to detect execution without rendering. To simulate all this correctly, you end up with a standard browser, standard execution times, full rendering in the background, etc. No one wants to download their YouTube video at 1x speed and wait for the adverts to finish.
cylemons 1y ago
How does testing rendering work? Can javascript get pixel data from the DOM
- Beretta_Vexee 1y ago
  https://www.w3schools.com/tags/canvas_getimagedata.asp https://www.w3schools.com/tags/canvas_getimagedata.asp