10 ms·

It seems he's doing something with header detection. I used Puppeteer to play around with the site and various configurations I use when scraping. In headless

by jadell 7y ago

It seems he's doing something with header detection. I used Puppeteer to play around with the site and various configurations I use when scraping. In headless Chrome, the "Accept-Language" header is not sent. In Puppeteer, one can force the header to be sent by doing: page.setExtraHTTPHeaders({ 'Accept-Language': 'en-US,en;q=0.9' }) However, Puppeteer sends that header as lowercase: accept-language: en-US,en;q=0.9 So it seems the detection is as simply: if there is no 'Accept-Language' header (case-sensitive), then "Headless Chrome"; else, "Not Headless Chrome". This is a completely server-side check, which is why he can say the fpcollect client-side javascript library isn't involved. Here are some curl commands that demonstrate: Detected: not headless curl 'https://arh.antoinevastel.com/bots/areyouheadless' \ -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3803.0 Safari/537.36' \ -H 'Accept-Language: en-US,en;q=0.9' Detected: headless curl 'https://arh.antoinevastel.com/bots/areyouheadless' \ -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3803.0 Safari/537.36' Detected: headless curl 'https://arh.antoinevastel.com/bots/areyouheadless' \ -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3803.0 Safari/537.36' \ -H 'accept-language: en-US,en;q=0.9'

jadell 7y ago
As a followup, if you have the ability to modify the Chrome/Chromium command line arguments, using the following option completely fools the detection: --lang=en-US,en;q=0.9 You can prove this with the following Puppeteer script: (async () => { const puppeteer = require('puppeteer'); const browserOpts = { headless: true, args: [ '--no-sandbox', '--disable-setuid-sandbox', '--user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3803.0 Safari/537.36', // THIS IS THE KEY BIT! '--lang=en-US,en;q=0.9', ], }; const browser = await puppeteer.launch(browserOpts); const page = await browser.newPage(); await page.goto('https://arh.antoinevastel.com/bots/areyouheadless'); await page.screenshot({ path: 'areyouheadless.png' }); await browser.close(); })();

by jadell 7y ago

jadell 7y ago

As a followup, if you have the ability to modify the Chrome/Chromium command line arguments, using the following option completely fools the detection: --lang=en-US,en;q=0.9 You can prove this with the following Puppeteer script: (async () => { const puppeteer = require('puppeteer'); const browserOpts = { headless: true, args: [ '--no-sandbox', '--disable-setuid-sandbox', '--user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3803.0 Safari/537.36', // THIS IS THE KEY BIT! '--lang=en-US,en;q=0.9', ], }; const browser = await puppeteer.launch(browserOpts); const page = await browser.newPage(); await page.goto('https://arh.antoinevastel.com/bots/areyouheadless'); await page.screenshot({ path: 'areyouheadless.png' }); await browser.close(); })();