7 ms·

I'm a lead engineer on the search team of a publicly traded company who's bread and butter is this domain. I was curious about this list, it candidly misses the

I'm a lead engineer on the search team of a publicly traded company who's bread and butter is this domain. I was curious about this list, it candidly misses the mark- the tech mentioned in this blog is what you might get if you hired a competent consultant to build out a service without having domain knowledge. In my experience, what's being used on the bleeding edge is two steps ahead of this.

ryandrake 5y ago
Do you have any factual corrections? Your post reminds me of those "I'm getting a kick out of these replies" copypasta--declaring someone wrong and claiming authoritative knowledge, but without actually correcting any of the errors of fact.
- ComputerGuru 5y ago
  Yes, but it’s also understandable why they wouldn’t want to expand.
- abadger9 5y ago
  Sorry about that, we're unable to discuss our projects with adjacent teams within the company. What you're saying is a valid frustration, the motivation behind the original comment was to put a thermometer on the repo.
  - gwbas1c 5y ago
    Well, don't go getting our hopes up!
  - fragmede 5y ago
    Adjacent teams? It makes sense not to discuss it with totally unrelated teams, but a large Internet company wants to have its cake and eat it too - scrape all of the worlds websites, while simultaneously denying others from scraping its results. Limiting communication between adjacent teams (ie the scraper team and the bot blocker team) really seems like it would be a hinder rather than help.
- jsnell 5y ago
  Abuse is the kind of problem area where anyone seriously working on either side will be unlikely to go into the details. It's the same whether it's blocking scraping, stopping spam, preventing account takeovers, or detecting payment fraud. In this specific case, the people wanting to detect bots want to avoid having their signals burned, the scrapers don't want the defenders to know which signals they're able to cloak since it will spur new signals development. So what gets disclosed publicly is just the really simple stuff. It's kind of sad. There's a huge pipeline for getting people up to speed on security engineering, since there's a lot of incentive for everyone in the ecosystem to share information as publicly as possible except for relatively short responsible disclosure windows. In contrast, the only way to learn abuse engineering is to happen to work in an organization with an abuse problem (and live with the frustration of an abnormally long ramp up period), or to go black hat. And likewise it's quite hard for the good guys to actually learn from each other, since they're spread across so many companies and it's thus hard for them to exchange information on what works and what doesn't.
- ChuckMcM 5y ago
  Not the OP but when I was running Operations at Blekko (a search engine) I spent part of my time dealing with scrapers. When treated like a puzzle it can be really interesting. So I thought I'd share a few tidbits. 1) We did a simple 'speed' test, how many queries per second were coming from an IP, and auto-ban on the limit being exceeded, started at 100qps and watched as the traffic moved down to 99.5qps. Pushed to 10qps and watched the traffic follow it down. Even at 3qps you would get traffic at bit over 2 qps trying to limbo in under the limit. 2) At that time, lots of people who highjacked browsers with toolbars sold scraping as a service to third parties. Their toolbar would check in to see if it should do a query and it would launch a query and return the results without the user even knowing. One company, 80 legs, was pretty up front about their "service", SEO types would use it to scrape Google results to see how their SEO campaigns were doing. 3) The majority of the traffic had criminal intent, looking for metadata on web pages to indicate they were running an unpatched version of some store software or had sql injection bugs. These would often come from PCs that had been compromised for other purposes or "zombie" PCs. We could rapidly map out these networks when we got 100 queries from 100 different IPs looking for "joomla version x.y",p=1 through "joomla version x.y",p=100. We briefly played around with sending them official looking SERPs but all the links went to fbi.gov though an obfuscator. One of more effective strategies was to field a "black hole" server, basically it was an http server that answered like you had gotten hold of it but then it never sent any data. With some simple kernel mods these TCP connections were silently removed on our end so they took no resources and the client would wait basically forever. We ack'd all keep alive packets with "Yup, we're here." so they just kept waiting and waiting. It really was a never ending game. We mass banned an entire Ukranian ISP because out of billions of queries not a single one was legitimate.
  - Scoundreller 5y ago
    > One of more effective strategies was to field a "black hole" server, basically it was an http server that answered like you had gotten hold of it but then it never sent any data. With some simple kernel mods these TCP connections were silently removed on our end so they took no resources and the client would wait basically forever. We ack'd all keep alive packets with "Yup, we're here." so they just kept waiting and waiting. Mailinator would do a similar thing with their custom email server hardware. Since they didn't really use sockets in the traditional sense, they were happy to give slooow replies and never disconnect "bad" connections.
gzer0 5y ago
I have a considerable amount of experience in the industry. Some of these so-called "advanced" techniques: * We use our own mobile emulation software (similiar to bluestacks). Turns out, mobile helps with a lot of things (below). * We use mobile IPs only. Mobile LTE data users are behind CGNATfor IPV4. You can't block one ip without possibly blocking hundreds of innocent IPs using the same exit point. * All you need is a new useragent and browser fingerprint; combined with emulation + mobile IPs, there's really no easy way for companies to block this. * With the advent and ease of virtualization; we avoid using any headless browsers. Seriously, if you can, never use headless. This should be close to rule number one for anyone looking to operate any kind of scrapers. All of our scrapers are run in isolated virtual instances with full mobile browsers. * We can easily reset our device identifier, device carrier, simulated SIM information, and especially important is the Google advertising ID that is set per device; the list goes on. The key here is #1, our mobile emulation software. * Our automation scripts are a combination of human recorded set of actions which we then perfected and can run in certain loops (for some of our data).
- AlexanderTheGr8 5y ago
  This list is pretty interesting. If you don't mind me asking, what do you work on that you requires such sophisticated stuff? Also, does this work only for browsers or also for mobile apps? I have always assumed that it is always theoretically possible to get data from browsers (very extreme resort is save the browser page / (screenshot + computer vision)); but it can be impossible to get data from apps (especially ios). Are my assumptions correct? Also can you explain mobile IPs more? If they are such a big vulnerability, why is there no potential solution to them?
  - kingcharles 5y ago
    Mobile IPs will be a problem until the entire Internet is IPv6. The issue is that there are not enough IPv4 addresses for everyone to get their own IP every time their phone connects to the Internet. So the mobile networks use one IP for many handsets. Block the IP, block dozens of different (innocent) people. Once we're all on IPv6 we can go back to blocking IPs. But then IPv6 creates its own problems.

ryandrake 5y ago

Do you have any factual corrections? Your post reminds me of those "I'm getting a kick out of these replies" copypasta--declaring someone wrong and claiming authoritative knowledge, but without actually correcting any of the errors of fact.

ComputerGuru 5y ago
Yes, but it’s also understandable why they wouldn’t want to expand.
abadger9 5y ago
Sorry about that, we're unable to discuss our projects with adjacent teams within the company. What you're saying is a valid frustration, the motivation behind the original comment was to put a thermometer on the repo.
- gwbas1c 5y ago
  Well, don't go getting our hopes up!
- fragmede 5y ago
  Adjacent teams? It makes sense not to discuss it with totally unrelated teams, but a large Internet company wants to have its cake and eat it too - scrape all of the worlds websites, while simultaneously denying others from scraping its results. Limiting communication between adjacent teams (ie the scraper team and the bot blocker team) really seems like it would be a hinder rather than help.
jsnell 5y ago
Abuse is the kind of problem area where anyone seriously working on either side will be unlikely to go into the details. It's the same whether it's blocking scraping, stopping spam, preventing account takeovers, or detecting payment fraud. In this specific case, the people wanting to detect bots want to avoid having their signals burned, the scrapers don't want the defenders to know which signals they're able to cloak since it will spur new signals development. So what gets disclosed publicly is just the really simple stuff. It's kind of sad. There's a huge pipeline for getting people up to speed on security engineering, since there's a lot of incentive for everyone in the ecosystem to share information as publicly as possible except for relatively short responsible disclosure windows. In contrast, the only way to learn abuse engineering is to happen to work in an organization with an abuse problem (and live with the frustration of an abnormally long ramp up period), or to go black hat. And likewise it's quite hard for the good guys to actually learn from each other, since they're spread across so many companies and it's thus hard for them to exchange information on what works and what doesn't.
ChuckMcM 5y ago
Not the OP but when I was running Operations at Blekko (a search engine) I spent part of my time dealing with scrapers. When treated like a puzzle it can be really interesting. So I thought I'd share a few tidbits. 1) We did a simple 'speed' test, how many queries per second were coming from an IP, and auto-ban on the limit being exceeded, started at 100qps and watched as the traffic moved down to 99.5qps. Pushed to 10qps and watched the traffic follow it down. Even at 3qps you would get traffic at bit over 2 qps trying to limbo in under the limit. 2) At that time, lots of people who highjacked browsers with toolbars sold scraping as a service to third parties. Their toolbar would check in to see if it should do a query and it would launch a query and return the results without the user even knowing. One company, 80 legs, was pretty up front about their "service", SEO types would use it to scrape Google results to see how their SEO campaigns were doing. 3) The majority of the traffic had criminal intent, looking for metadata on web pages to indicate they were running an unpatched version of some store software or had sql injection bugs. These would often come from PCs that had been compromised for other purposes or "zombie" PCs. We could rapidly map out these networks when we got 100 queries from 100 different IPs looking for "joomla version x.y",p=1 through "joomla version x.y",p=100. We briefly played around with sending them official looking SERPs but all the links went to fbi.gov though an obfuscator. One of more effective strategies was to field a "black hole" server, basically it was an http server that answered like you had gotten hold of it but then it never sent any data. With some simple kernel mods these TCP connections were silently removed on our end so they took no resources and the client would wait basically forever. We ack'd all keep alive packets with "Yup, we're here." so they just kept waiting and waiting. It really was a never ending game. We mass banned an entire Ukranian ISP because out of billions of queries not a single one was legitimate.
- Scoundreller 5y ago
  > One of more effective strategies was to field a "black hole" server, basically it was an http server that answered like you had gotten hold of it but then it never sent any data. With some simple kernel mods these TCP connections were silently removed on our end so they took no resources and the client would wait basically forever. We ack'd all keep alive packets with "Yup, we're here." so they just kept waiting and waiting. Mailinator would do a similar thing with their custom email server hardware. Since they didn't really use sockets in the traditional sense, they were happy to give slooow replies and never disconnect "bad" connections.

gzer0 5y ago

I have a considerable amount of experience in the industry. Some of these so-called "advanced" techniques: * We use our own mobile emulation software (similiar to bluestacks). Turns out, mobile helps with a lot of things (below). * We use mobile IPs only. Mobile LTE data users are behind CGNATfor IPV4. You can't block one ip without possibly blocking hundreds of innocent IPs using the same exit point. * All you need is a new useragent and browser fingerprint; combined with emulation + mobile IPs, there's really no easy way for companies to block this. * With the advent and ease of virtualization; we avoid using any headless browsers. Seriously, if you can, never use headless. This should be close to rule number one for anyone looking to operate any kind of scrapers. All of our scrapers are run in isolated virtual instances with full mobile browsers. * We can easily reset our device identifier, device carrier, simulated SIM information, and especially important is the Google advertising ID that is set per device; the list goes on. The key here is #1, our mobile emulation software. * Our automation scripts are a combination of human recorded set of actions which we then perfected and can run in certain loops (for some of our data).

AlexanderTheGr8 5y ago
This list is pretty interesting. If you don't mind me asking, what do you work on that you requires such sophisticated stuff? Also, does this work only for browsers or also for mobile apps? I have always assumed that it is always theoretically possible to get data from browsers (very extreme resort is save the browser page / (screenshot + computer vision)); but it can be impossible to get data from apps (especially ios). Are my assumptions correct? Also can you explain mobile IPs more? If they are such a big vulnerability, why is there no potential solution to them?
- kingcharles 5y ago
  Mobile IPs will be a problem until the entire Internet is IPv6. The issue is that there are not enough IPv4 addresses for everyone to get their own IP every time their phone connects to the Internet. So the mobile networks use one IP for many handsets. Block the IP, block dozens of different (innocent) people. Once we're all on IPv6 we can go back to blocking IPs. But then IPv6 creates its own problems.