4 ms·

As I'm repeating myself too often regarding this: the statistical norm is to be trackable. Every browser that isn't will get flagged. Privacy does only exist i

As I'm repeating myself too often regarding this: the statistical norm is to be trackable. Every browser that isn't will get flagged. Privacy does only exist if you seem to look exactly like any other person visiting the website. That is why user agent, web apis, asset loading order and behaviours, http stack behaviours, tcp fingerprints all have to look like they came from the Browser Engine you're trying to identify with. If you don't, you're watched. It's as simple as that. Don't kid yourself into safety if you think that the Content-Security-Policy hack of adblockers work to prevent tracking and fingerprinting. Sure, you send less data, but less data means more obvious than the norm. As there was not a single concept that could fix this (in regards to looking like another Browser) I started to build my own "web filtering browser" [1] that is able to emulate fingerprinting, and sticks them to specific domains and their originating uncached CDN requests so that if you have the same IP for the same website, they cannot correlate it to ptevious visits. The most effective way to not get tracked is to not visit the website. And I think that there's a huge potential in offloading traffic to peers that have the same URL already in their caches. Peer-to-peer Browser Caches would fix so many issues on their own, I don't understand why nobody is doing that. To the end user it doesn't matter where the assets come from, as long as the website is rendered in the same way afterwards. [1] https://github.com/tholian-network/stealth https://github.com/tholian-network/stealth

sixothree 6y ago
I still think there is an opportunity for a hosted browser vpn type service or a generic contained browser. It would look like every other person because every other person uses the same setup.
- cookiengineer 6y ago
  Except tracking services know about centralized VPN IP ranges that do not rotate. That's why everybody in algotrading switched to mobile apps and 3G/4G proxies that are installed on actual smartphones, and usually they have dozens of SIM cards laying around.
  - KirillPanov 6y ago
    Er, wait, your comment sounds really interesting, but what is the threat model for algorithmic traders? Who are they trying to hide from and why? Almost none of the web breaks simply because you're using a (very well known, very old) VPN IP. Purchases using a credit card are pretty much the only thing that is likely to get blocked. And you face a few more captchas sometimes. Do algorithmic traders need to use something on the web that blocks VPN IPs more aggressively than this? They must, because juggling all those SIM cards sounds like a huge headache, and cell phone data has awful latency. I'm wondering what they're scraping that the average person doesn't use, and why they want to look like an average person if that's the case.
    - cookiengineer 6y ago
      AFAIK mostly details, news, stories and stuff related to public knowledge about a company (which they factor in to the real HFT data) due to - as you already said - high latencies in mobile networks. The issue that arises is mostly cloudflare-related, due to them having a huge influence on hosting, and the forced recaptchas when they detect anomalies in traffic behaviour makes a web scraping workflow real shitty. From an algotrader's point of view it's a fix to make their web scrapers work again. I'm not sure how Chrome/ium headless could fix this (if it could). I'm a bit sceptical as it's just yet another cat and mouse game. But as of late I've seen a huge scene start to develop around extension-building for headless Chrome specifically, so that they can run headless and still get the data as they want it to by integrating a content script that sends the data to another service.
      - KirillPanov 6y ago
        Ah, thanks, now I get it: it doesn't bother them that fingerprinting lets them be tracked; it bothers them that fingerprinting (and VPN IPs) mean their scrapers get hit with captchas. FWIW, it takes very little effort to completely conceal the fact that firefox is running headless under marionette/webdriver/geckodriver. Chrom(ium) takes more effort, but these guys have solved it (and built a business around it): https://intoli.com/blog/making-chrome-headless-undetectable/ https://intoli.com/blog/making-chrome-headless-undetectable/ https://github.com/intoli/remote-browser https://github.com/intoli/remote-browser Of course neither of these address fingerprinting -- all your scrape requests will have the same or similar fingerprint, which will lead to captchas pretty quickly. This might help (and might even be part of the reason for buying piles of cellphones): https://news.ycombinator.com/item?id=25379342 https://news.ycombinator.com/item?id=25379342
t0astbread 6y ago
How does a Peer-to-Peer cache work in regards to trust?
- cookiengineer 6y ago
  A peer to peer consent algorithm always has to be trustless, meaning that you have to be able to verify things cryptographically so that there can be no 51% attack for each peer bucket. How that cryptography can be "safe" in the sense that nobody is able to fake it is a different story, because in the web lots of things can be statistically true or untrue, but stochastically the opposite (e.g. a website being down right now or unreachable or rendered differently for each country etc). In my case I'm using existing TLS infrastructure so that each peer can use their own certificates to communicate with other peers directly.

sixothree 6y ago

I still think there is an opportunity for a hosted browser vpn type service or a generic contained browser. It would look like every other person because every other person uses the same setup.

cookiengineer 6y ago
Except tracking services know about centralized VPN IP ranges that do not rotate. That's why everybody in algotrading switched to mobile apps and 3G/4G proxies that are installed on actual smartphones, and usually they have dozens of SIM cards laying around.
- KirillPanov 6y ago
  Er, wait, your comment sounds really interesting, but what is the threat model for algorithmic traders? Who are they trying to hide from and why? Almost none of the web breaks simply because you're using a (very well known, very old) VPN IP. Purchases using a credit card are pretty much the only thing that is likely to get blocked. And you face a few more captchas sometimes. Do algorithmic traders need to use something on the web that blocks VPN IPs more aggressively than this? They must, because juggling all those SIM cards sounds like a huge headache, and cell phone data has awful latency. I'm wondering what they're scraping that the average person doesn't use, and why they want to look like an average person if that's the case.
  - cookiengineer 6y ago
    AFAIK mostly details, news, stories and stuff related to public knowledge about a company (which they factor in to the real HFT data) due to - as you already said - high latencies in mobile networks. The issue that arises is mostly cloudflare-related, due to them having a huge influence on hosting, and the forced recaptchas when they detect anomalies in traffic behaviour makes a web scraping workflow real shitty. From an algotrader's point of view it's a fix to make their web scrapers work again. I'm not sure how Chrome/ium headless could fix this (if it could). I'm a bit sceptical as it's just yet another cat and mouse game. But as of late I've seen a huge scene start to develop around extension-building for headless Chrome specifically, so that they can run headless and still get the data as they want it to by integrating a content script that sends the data to another service.
    - KirillPanov 6y ago
      Ah, thanks, now I get it: it doesn't bother them that fingerprinting lets them be tracked; it bothers them that fingerprinting (and VPN IPs) mean their scrapers get hit with captchas. FWIW, it takes very little effort to completely conceal the fact that firefox is running headless under marionette/webdriver/geckodriver. Chrom(ium) takes more effort, but these guys have solved it (and built a business around it): https://intoli.com/blog/making-chrome-headless-undetectable/ https://intoli.com/blog/making-chrome-headless-undetectable/ https://github.com/intoli/remote-browser https://github.com/intoli/remote-browser Of course neither of these address fingerprinting -- all your scrape requests will have the same or similar fingerprint, which will lead to captchas pretty quickly. This might help (and might even be part of the reason for buying piles of cellphones): https://news.ycombinator.com/item?id=25379342 https://news.ycombinator.com/item?id=25379342

t0astbread 6y ago

How does a Peer-to-Peer cache work in regards to trust?

cookiengineer 6y ago
A peer to peer consent algorithm always has to be trustless, meaning that you have to be able to verify things cryptographically so that there can be no 51% attack for each peer bucket. How that cryptography can be "safe" in the sense that nobody is able to fake it is a different story, because in the web lots of things can be statistically true or untrue, but stochastically the opposite (e.g. a website being down right now or unreachable or rendered differently for each country etc). In my case I'm using existing TLS infrastructure so that each peer can use their own certificates to communicate with other peers directly.