4 ms·
I have a considerable amount of experience in the industry. Some of these so-called "advanced" techniques: * We use our own mobile emulation software (simil
by gzer0 5y ago
I have a considerable amount of experience in the industry.
Some of these so-called "advanced" techniques:
* We use our own mobile emulation software (similiar to bluestacks). Turns out, mobile helps with a lot of things (below).
* We use mobile IPs only. Mobile LTE data users are behind CGNATfor IPV4. You can't block one ip without possibly blocking hundreds of innocent IPs using the same exit point.
* All you need is a new useragent and browser fingerprint; combined with emulation + mobile IPs, there's really no easy way for companies to block this.
* With the advent and ease of virtualization; we avoid using any headless browsers. Seriously, if you can, never use headless. This should be close to rule number one for anyone looking to operate any kind of scrapers. All of our scrapers are run in isolated virtual instances with full mobile browsers.
* We can easily reset our device identifier, device carrier, simulated SIM information, and especially important is the Google advertising ID that is set per device; the list goes on. The key here is #1, our mobile emulation software.
* Our automation scripts are a combination of human recorded set of actions which we then perfected and can run in certain loops (for some of our data).
- AlexanderTheGr8 5y agoThis list is pretty interesting. If you don't mind me asking, what do you work on that you requires such sophisticated stuff? Also, does this work only for browsers or also for mobile apps? I have always assumed that it is always theoretically possible to get data from browsers (very extreme resort is save the browser page / (screenshot + computer vision)); but it can be impossible to get data from apps (especially ios). Are my assumptions correct? Also can you explain mobile IPs more? If they are such a big vulnerability, why is there no potential solution to them?
- kingcharles 5y agoMobile IPs will be a problem until the entire Internet is IPv6. The issue is that there are not enough IPv4 addresses for everyone to get their own IP every time their phone connects to the Internet. So the mobile networks use one IP for many handsets. Block the IP, block dozens of different (innocent) people. Once we're all on IPv6 we can go back to blocking IPs. But then IPv6 creates its own problems.
- AlexanderTheGr8 5y agoDoesn't the same idea apply for normal IPs (non-mobile IPs)? Our computers also don't get their separate IPv4 address. We use NATs. So how can someone block our computer specifically without blocking dozens of innocent people using the same public IPv4 address? Basically what's different about mobile and laptop IPs?
- mike_d 5y agoCGNAT is done at a ridiculously large scale in mobile networks. Your home or local coffee shop might have 10-20 users max behind a single IP. A mobile network on the other hand might put most a small-medium city behind 3 or 4 IPs.
- matheusmoreira 5y ago> Mobile LTE data users are behind CGNATfor IPV4. > You can't block one ip without possibly blocking hundreds of innocent IPs using the same exit point. This is similar how Tor is supposed to work in practice. Make everyone look like the same user so these companies can't tell who's who.