4 ms·

It's a B2B ecommerce site. Our annual revenue from the site would put us on the list of top 100 ecommerce sites [1] (we're not listed because ecommerce isn't th

It's a B2B ecommerce site. Our annual revenue from the site would put us on the list of top 100 ecommerce sites [1] (we're not listed because ecommerce isn't the only businesss we do. With that much potential revenue to steal from us, perhaps the stakes are higher. As described elsewhere, rate limiting doesn't work. The bots come from hundreds to thousands of separate IPs simultaneously, cooperating in a distributed fashion. Any one of them is within reasonable behavioral ranges. Also, caching, even through a CDN doesn't help. As a B2B site, all our pricing is custom as negotiated with each customer. (What's ironic is that this means that the pricing data that the bots are scraping isn't even representative - it only shows what we offer walkup, non-contract customers.) And because the pricing is dynamic, it also means that the scraping to get these prices is one of the more computationally expensive activities they could do. To be fair, there is some low-hanging fruit in blocking many of them. Like, it's easy to detect those that are flooding from a single address, or sending SQL injection attacks, or just plain coming from Russia. I assume those are just the script kiddies and stuff. The problem is that it still leaves a whole lot of bad actors once these are skimmed off the top. [1] https://en.wikipedia.org/wiki/List_of_largest_Internet_companies https://en.wikipedia.org/wiki/List_of_largest_Internet_compa...

YPCrumble 5y ago
Thanks for the explanation! The thing I still don’t understand is why (edit server not cdn) caching doesn’t work - you have to identify customers somehow, and provide everyone else a cached response at the server level. For that matter, rate limit non-customers also.
- CWuestefeld 5y ago
  The pages getting most of the bot action are search and product details. Search results obviously can't be cached, as it's completely ad hoc. Product details can't be cached either, or more precisely, there are parts of each product page that can't be cached because * different customers have different products in the catalog * different products have different prices for a given product * different products have customer-specific aliases * there's a huge number of products (low millions) and many thousands of distinct catalogs (many customers have effectively identical catalogs, and we've already got logic that collapses those in the backend) * prices are also based on costs from upstream suppliers, which are themselves changing dynamically. Putting all this together, the number of times a given [product,customer] tuple will be requested in a reasonable cache TTL isn't very much greater than 1. The exception being for walk-up pricing for non-contract users, and we've been talking about how we might optimize that particular cases.
  - YPCrumble 5y ago
    Ahhhhh, search results makes a whole lot more sense! Thank you. Search can't be cached and the people who want to use your search functionality as a high availability API endpoint use different IP addresses to get around rate limiting. The low millions of products also makes some sense I suppose but it's hard to imagine why this doesn't simply take a login for the customer to see the products if they're unique to each customer. On the other hand, I suspect the price this company is paying to mitigate scrapers is akin to a drop of water in the ocean, no? As a percent of the development budget it might seem high and therefore seem big to the developer, but I suspect the CEO of the company doesn't even know that scrapers are scraping the site. Maybe I'm wrong. Thanks again for the multiple explanations in any case, it opened my eyes to a way scrapers could be problematic that I hadn't thought about.
  - pdimitar 5y ago
    Good explanation, thank you. I would think that artificially slowing down search results can discourage part of the bots. Humans don't care much it a starch finishes in 5 seconds and not 2 AFAIK. Especially on backends where each request is a relatively cheap operations wise (especially when each request is a green thread like in Erlang/Elixir), I think you can score a win against the bots. Have you attempted something like this?
    - YPCrumble 5y ago
      This is really interesting but they’re using a network of bots already - even if you put a spinner that makes them wait a couple seconds the scrapers would just make more parallel requests no?
      - pdimitar 5y ago
        Yes, they absolutely will, but that's the strength of certain runtimes: green threads (i.e. HTTP request/response sessions in this case) cost almost nothing so you can hold onto 5-10 million of them on a VPS with 16-32 GB RAM, easily. I haven't had to defend against extensive bot scraping operations -- only against simpler ones -- but I've utilized such a practice in my admittedly much more limited experience, and was actually successful. Not that the bots gave up but their authors realized they can't accelerate the process of scraping data so they dialed down their instances, likely to save money from their own hosting bills. Win-win. Apologies, I don't mention to lecture you, just sharing a small piece of experience. Granted that's very specific to the backend tech but what the heck, maybe you'll find the tidbit valuable.
mindslight 5y ago
> As a B2B site, all our pricing is custom as negotiated with each customer ... the pricing is dynamic So your company is deliberately trying to frustrate the market, and doesn't like the result of third parties attempting to help market efficiency? It seems like this is the exact kind of scraping that we generally want more of! I'm sorry about your personal technical predicament, but it doesn't sound like your perspective is really coming from the moral high ground here.
- YPCrumble 5y ago
  This makes a lot of sense to me. What do you think about their other assertion that the search page is getting a gigantic number of hits that a/ cannot be cached and b/ cannot be rate limited because they're using a botnet?
  - mindslight 5y ago
    I'm guessing the bots are hitting the search page because it contains the most amount of information per hit, and that the caching problems are exactly due to these dynamically generated prices or other such nonsense. After all, the fundamental goal of scraping is to straightforwardly enumerate the entire dataset. The scale of the botnet sounds like an awfully determined and entrenched adversary, likely arising because this company has been frustrating the market for quite some time. A good faith API wouldn't make the bots change behavior tomorrow, but they certainly would if there were breaking page format changes containing a comment linking to the API.
- CWuestefeld 5y ago
  So your company is deliberately trying to frustrate the market, and doesn't like the result of third parties attempting to help market efficiency? No. First, we as a middleman resseller MUST provide custom prices, at least to a certain degree. Consider that it's typical for manufacturers to offer different prices to, e.g., schools. This is reflected by offering to us (the middleman) a lower cost, which we pass on to applicable customers. Further, the costs and prices vary similarly from one country to another. Less obviously, many manufacturers (e.g., Microsoft, Adobe, HP) offer licensing programs that entitle those enrolled to purchase their products at a lower cost. So if nothing else, the business terms of the manufacturers whose products we sell necessitates a certain degree of custom pricing. Second, it seems strange to characterize as "frustrating the market" what we're doing when we cooperate with customers who want to structure their expenses in different ways - say, getting a better deal on expensive products that can be classified as "capital expenses" while allowing us to recover some of that revenue by charging them somewhat more for the products that they'd classify as operational expenses.
runlevel1 5y ago
If the queries are expensive because of custom negotiated prices and these bots are scraping the walkup prices, can you not just shard out the walkup prices and cache them? Being on that list puts the company's revenue at over $1 billion USD. At a certain point it becomes cheaper and easier to fix the system to handle the load.
- CWuestefeld 5y ago
  Indeed - this is one of the strategies we're considering.

YPCrumble 5y ago

Thanks for the explanation! The thing I still don’t understand is why (edit server not cdn) caching doesn’t work - you have to identify customers somehow, and provide everyone else a cached response at the server level. For that matter, rate limit non-customers also.

CWuestefeld 5y ago
The pages getting most of the bot action are search and product details. Search results obviously can't be cached, as it's completely ad hoc. Product details can't be cached either, or more precisely, there are parts of each product page that can't be cached because * different customers have different products in the catalog * different products have different prices for a given product * different products have customer-specific aliases * there's a huge number of products (low millions) and many thousands of distinct catalogs (many customers have effectively identical catalogs, and we've already got logic that collapses those in the backend) * prices are also based on costs from upstream suppliers, which are themselves changing dynamically. Putting all this together, the number of times a given [product,customer] tuple will be requested in a reasonable cache TTL isn't very much greater than 1. The exception being for walk-up pricing for non-contract users, and we've been talking about how we might optimize that particular cases.
- YPCrumble 5y ago
  Ahhhhh, search results makes a whole lot more sense! Thank you. Search can't be cached and the people who want to use your search functionality as a high availability API endpoint use different IP addresses to get around rate limiting. The low millions of products also makes some sense I suppose but it's hard to imagine why this doesn't simply take a login for the customer to see the products if they're unique to each customer. On the other hand, I suspect the price this company is paying to mitigate scrapers is akin to a drop of water in the ocean, no? As a percent of the development budget it might seem high and therefore seem big to the developer, but I suspect the CEO of the company doesn't even know that scrapers are scraping the site. Maybe I'm wrong. Thanks again for the multiple explanations in any case, it opened my eyes to a way scrapers could be problematic that I hadn't thought about.
- pdimitar 5y ago
  Good explanation, thank you. I would think that artificially slowing down search results can discourage part of the bots. Humans don't care much it a starch finishes in 5 seconds and not 2 AFAIK. Especially on backends where each request is a relatively cheap operations wise (especially when each request is a green thread like in Erlang/Elixir), I think you can score a win against the bots. Have you attempted something like this?
  - YPCrumble 5y ago
    This is really interesting but they’re using a network of bots already - even if you put a spinner that makes them wait a couple seconds the scrapers would just make more parallel requests no?
    - pdimitar 5y ago
      Yes, they absolutely will, but that's the strength of certain runtimes: green threads (i.e. HTTP request/response sessions in this case) cost almost nothing so you can hold onto 5-10 million of them on a VPS with 16-32 GB RAM, easily. I haven't had to defend against extensive bot scraping operations -- only against simpler ones -- but I've utilized such a practice in my admittedly much more limited experience, and was actually successful. Not that the bots gave up but their authors realized they can't accelerate the process of scraping data so they dialed down their instances, likely to save money from their own hosting bills. Win-win. Apologies, I don't mention to lecture you, just sharing a small piece of experience. Granted that's very specific to the backend tech but what the heck, maybe you'll find the tidbit valuable.

mindslight 5y ago

> As a B2B site, all our pricing is custom as negotiated with each customer ... the pricing is dynamic So your company is deliberately trying to frustrate the market, and doesn't like the result of third parties attempting to help market efficiency? It seems like this is the exact kind of scraping that we generally want more of! I'm sorry about your personal technical predicament, but it doesn't sound like your perspective is really coming from the moral high ground here.

YPCrumble 5y ago
This makes a lot of sense to me. What do you think about their other assertion that the search page is getting a gigantic number of hits that a/ cannot be cached and b/ cannot be rate limited because they're using a botnet?
- mindslight 5y ago
  I'm guessing the bots are hitting the search page because it contains the most amount of information per hit, and that the caching problems are exactly due to these dynamically generated prices or other such nonsense. After all, the fundamental goal of scraping is to straightforwardly enumerate the entire dataset. The scale of the botnet sounds like an awfully determined and entrenched adversary, likely arising because this company has been frustrating the market for quite some time. A good faith API wouldn't make the bots change behavior tomorrow, but they certainly would if there were breaking page format changes containing a comment linking to the API.
CWuestefeld 5y ago
So your company is deliberately trying to frustrate the market, and doesn't like the result of third parties attempting to help market efficiency? No. First, we as a middleman resseller MUST provide custom prices, at least to a certain degree. Consider that it's typical for manufacturers to offer different prices to, e.g., schools. This is reflected by offering to us (the middleman) a lower cost, which we pass on to applicable customers. Further, the costs and prices vary similarly from one country to another. Less obviously, many manufacturers (e.g., Microsoft, Adobe, HP) offer licensing programs that entitle those enrolled to purchase their products at a lower cost. So if nothing else, the business terms of the manufacturers whose products we sell necessitates a certain degree of custom pricing. Second, it seems strange to characterize as "frustrating the market" what we're doing when we cooperate with customers who want to structure their expenses in different ways - say, getting a better deal on expensive products that can be classified as "capital expenses" while allowing us to recover some of that revenue by charging them somewhat more for the products that they'd classify as operational expenses.

runlevel1 5y ago

If the queries are expensive because of custom negotiated prices and these bots are scraping the walkup prices, can you not just shard out the walkup prices and cache them? Being on that list puts the company's revenue at over $1 billion USD. At a certain point it becomes cheaper and easier to fix the system to handle the load.

CWuestefeld 5y ago
Indeed - this is one of the strategies we're considering.