3 ms·

AFAIK mostly details, news, stories and stuff related to public knowledge about a company (which they factor in to the real HFT data) due to - as you already sa

AFAIK mostly details, news, stories and stuff related to public knowledge about a company (which they factor in to the real HFT data) due to - as you already said - high latencies in mobile networks. The issue that arises is mostly cloudflare-related, due to them having a huge influence on hosting, and the forced recaptchas when they detect anomalies in traffic behaviour makes a web scraping workflow real shitty. From an algotrader's point of view it's a fix to make their web scrapers work again. I'm not sure how Chrome/ium headless could fix this (if it could). I'm a bit sceptical as it's just yet another cat and mouse game. But as of late I've seen a huge scene start to develop around extension-building for headless Chrome specifically, so that they can run headless and still get the data as they want it to by integrating a content script that sends the data to another service.

KirillPanov 6y ago
Ah, thanks, now I get it: it doesn't bother them that fingerprinting lets them be tracked; it bothers them that fingerprinting (and VPN IPs) mean their scrapers get hit with captchas. FWIW, it takes very little effort to completely conceal the fact that firefox is running headless under marionette/webdriver/geckodriver. Chrom(ium) takes more effort, but these guys have solved it (and built a business around it): https://intoli.com/blog/making-chrome-headless-undetectable/ https://intoli.com/blog/making-chrome-headless-undetectable/ https://github.com/intoli/remote-browser https://github.com/intoli/remote-browser Of course neither of these address fingerprinting -- all your scrape requests will have the same or similar fingerprint, which will lead to captchas pretty quickly. This might help (and might even be part of the reason for buying piles of cellphones): https://news.ycombinator.com/item?id=25379342 https://news.ycombinator.com/item?id=25379342
- nextaccountic 6y ago
  How to make headless firefox undetectable to fingerprinting?

KirillPanov 6y ago

Ah, thanks, now I get it: it doesn't bother them that fingerprinting lets them be tracked; it bothers them that fingerprinting (and VPN IPs) mean their scrapers get hit with captchas. FWIW, it takes very little effort to completely conceal the fact that firefox is running headless under marionette/webdriver/geckodriver. Chrom(ium) takes more effort, but these guys have solved it (and built a business around it): https://intoli.com/blog/making-chrome-headless-undetectable/ https://intoli.com/blog/making-chrome-headless-undetectable/ https://github.com/intoli/remote-browser https://github.com/intoli/remote-browser Of course neither of these address fingerprinting -- all your scrape requests will have the same or similar fingerprint, which will lead to captchas pretty quickly. This might help (and might even be part of the reason for buying piles of cellphones): https://news.ycombinator.com/item?id=25379342 https://news.ycombinator.com/item?id=25379342

nextaccountic 6y ago
How to make headless firefox undetectable to fingerprinting?