2 ms·
Phantom still gets blocked, as it reveals itself in the header. I've had success with a headless Chrome instance in a virtual display (xvfb) driven with Seleni
by dhruvkar 9y ago
Phantom still gets blocked, as it reveals itself in the header.
I've had success with a headless Chrome instance in a virtual display (xvfb) driven with Selenium, backed by Postgres. It's as close you can get to scripting a real browser.
- rectangletangle 9y agoYou can set the user-agent with Phantom var webPage = require('webpage'); var page = webPage.create(); page.settings.userAgent = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36'; http://phantomjs.org/api/webpage/property/settings.html http://phantomjs.org/api/webpage/property/settings.html
- dhruvkar 9y agoWhile that's true, user-agent isn't the only thing in the header that reveals PhantomJS[0]. You could take the time to build in spoofs for these issues. But for testing (and scraping), you're going to be better off if your headless browser is the same as your GUI browser. 0: https://blog.shapesecurity.com/2015/01/22/detecting-phantomjs-based-visitors/ https://blog.shapesecurity.com/2015/01/22/detecting-phantomj...
- eriknstr 9y agoI think I read recently that Chrome / Chromium is now able to run without having to use xvfb, so now truly headless.