4 ms·

I worked for a competitor, and the amount of time+money we spent blocking scrapers was insane. The reason we spent all that time+money? It affected the metrics

I worked for a competitor, and the amount of time+money we spent blocking scrapers was insane. The reason we spent all that time+money? It affected the metrics on which we were paid, AND consumed a huge amount of compute resources of ourselves and partners. It's also (at least in part) why places are doing the "Sign up to see special offers" thing - because the offer isn't for display on the open site, and they don't want that value to appear on scrapers. So... logged in users only, and validate that user isn't doing scraping.

wolco 7y ago
Why not offer a public api? Logging in seems like a small hurdle for a bot but enough of a hassel that customers lose interest.
- rocqua 7y ago
  Login with re-captcha, and banning of accounts that are obviously scraping could be quite effective I guess.
- paranoidrobot 7y ago
  We did have an API, but it wasn't public - because search traffic is expensive. Both, as mentioned, in compute resources, but also in pissing off and/or outright crashing vendors. Logging in - requiring an email validation loop - would be simple for bots, but again, anti-bot tech stops them from doing it en-masse.
GordonS 7y ago
> AND consumed a huge amount of compute resources of ourselves and partners Wow, I'm interested to know more here! How much bot/scraper traffic was there that it cost you big on compute?
- paranoidrobot 7y ago
  I don't want to give away too much detail - Company IP, self identifying, etc. However, lets just say that if normal peak traffic is 1X, then bot traffic can come in at 100-10000X. And in most cases it's not a smooth rise, it'll be a case of suddenly all this new traffic came in, no warning, no rise. How many non-trivially complex sites do you know that can suddenly take 100-10000X their normal peak in under a minute? Those that do - are their managers happy about spending on idle infrastructure? How fast do you think even services like AWS can scale? (hint: it's not fast enough). Normal traffic is, in aggregate, quite predictable in load. Outside of very unusual situations, you don't get hundreds of thousands of people suddenly hitting your site. Plus, because that traffic is nice and predictable, you can have decent caching rules, reducing the cost of the traffic again. Along comes Mr Botnet and they want to scrape everything: every hotel in every city, every checkin/out tuple for stays of 1-7 days for the next 12 months, and a bunch of room configurations. Kiss your caching goodbye, because now not only is your cache hit ratio for their searches going to be effectively zero, but now it's being filled with other shit that nobody else wants. And this is just our infrastructure. There's the third party OTAs that we hit, and they have the same issue, often with smaller and less experienced teams running on crappier infrastructure. So, you get angry calls from them that their shared hosting provider is cutting them off and you're ruining their business. Because of course, this shared hosting is not only hosting their room availability search API, but their hotel check-in and sales applications - so nobody in 30 hotels can check-in/out.
  - GordonS 7y ago
    Thanks for the detail. It's just mind-boggling to me that you could have a peak of (for example) 1K requests per second, then a bot rises that to 10M requests per second. If nothing else, it seems incredible that bot authors would be so willfully harmful - they must know that kind of behaviour is going to prompt a reaction
    - ddorian43 7y ago
      I don't think even google crawls the whole www with 10M rps.
  - chatmasta 7y ago
    This wouldn’t be a problem if these companies offered an API to access their data. But of course, that wouldn’t be good for a business which ultimately depends on information asymmetry and walled garden lock-in. Personally I’m not a big fan of these business models (see LinkedIn for another example), but I at least understand where they’re coming from. I also understand it’s sometimes out of your hands, i.e. the hotel partners don’t want you to make the data easily available. Similar incentive structures exist in flight booking and geoblocked Netflix content. FWIW, I’ve been on the other side of this (not in the travel industry), and written scrapers. I never scraped the HTML, though. My strategy was to MITM the mobile app and use the same API. I also made sure to throttle the traffic, if not to be respectful, at least to blend in without setting off alarms...
    - paranoidrobot 7y ago
      > This wouldn’t be a problem if these companies offered an API to access their data. Sorry, but you're talking about something you have no idea of. The impact on that platform of someone using the API vs crawling the site is approximately the same. Whether someone scraped us via the API or via the HTML made no difference in terms of impact on the platform.
      - chatmasta 7y ago
        > Sorry, but you're talking about something you have no idea of. This kind of personal attack seems unnecessary, and by the way, I probably know a lot more than you assume about this particular topic. Regardless, if you had an API, you could charge for it. It wouldn't cover 100% of the scrapers, but it would cover some.
jacquesm 7y ago
> Sign up to see special offers Another reason why they do that is because their partners require they don't advertise special pricing to the world at large, they're contractually forbidden from doing so.

wolco 7y ago

Why not offer a public api? Logging in seems like a small hurdle for a bot but enough of a hassel that customers lose interest.

rocqua 7y ago
Login with re-captcha, and banning of accounts that are obviously scraping could be quite effective I guess.
paranoidrobot 7y ago
We did have an API, but it wasn't public - because search traffic is expensive. Both, as mentioned, in compute resources, but also in pissing off and/or outright crashing vendors. Logging in - requiring an email validation loop - would be simple for bots, but again, anti-bot tech stops them from doing it en-masse.

GordonS 7y ago

> AND consumed a huge amount of compute resources of ourselves and partners Wow, I'm interested to know more here! How much bot/scraper traffic was there that it cost you big on compute?

paranoidrobot 7y ago
I don't want to give away too much detail - Company IP, self identifying, etc. However, lets just say that if normal peak traffic is 1X, then bot traffic can come in at 100-10000X. And in most cases it's not a smooth rise, it'll be a case of suddenly all this new traffic came in, no warning, no rise. How many non-trivially complex sites do you know that can suddenly take 100-10000X their normal peak in under a minute? Those that do - are their managers happy about spending on idle infrastructure? How fast do you think even services like AWS can scale? (hint: it's not fast enough). Normal traffic is, in aggregate, quite predictable in load. Outside of very unusual situations, you don't get hundreds of thousands of people suddenly hitting your site. Plus, because that traffic is nice and predictable, you can have decent caching rules, reducing the cost of the traffic again. Along comes Mr Botnet and they want to scrape everything: every hotel in every city, every checkin/out tuple for stays of 1-7 days for the next 12 months, and a bunch of room configurations. Kiss your caching goodbye, because now not only is your cache hit ratio for their searches going to be effectively zero, but now it's being filled with other shit that nobody else wants. And this is just our infrastructure. There's the third party OTAs that we hit, and they have the same issue, often with smaller and less experienced teams running on crappier infrastructure. So, you get angry calls from them that their shared hosting provider is cutting them off and you're ruining their business. Because of course, this shared hosting is not only hosting their room availability search API, but their hotel check-in and sales applications - so nobody in 30 hotels can check-in/out.
- GordonS 7y ago
  Thanks for the detail. It's just mind-boggling to me that you could have a peak of (for example) 1K requests per second, then a bot rises that to 10M requests per second. If nothing else, it seems incredible that bot authors would be so willfully harmful - they must know that kind of behaviour is going to prompt a reaction
  - ddorian43 7y ago
    I don't think even google crawls the whole www with 10M rps.
- chatmasta 7y ago
  This wouldn’t be a problem if these companies offered an API to access their data. But of course, that wouldn’t be good for a business which ultimately depends on information asymmetry and walled garden lock-in. Personally I’m not a big fan of these business models (see LinkedIn for another example), but I at least understand where they’re coming from. I also understand it’s sometimes out of your hands, i.e. the hotel partners don’t want you to make the data easily available. Similar incentive structures exist in flight booking and geoblocked Netflix content. FWIW, I’ve been on the other side of this (not in the travel industry), and written scrapers. I never scraped the HTML, though. My strategy was to MITM the mobile app and use the same API. I also made sure to throttle the traffic, if not to be respectful, at least to blend in without setting off alarms...
  - paranoidrobot 7y ago
    > This wouldn’t be a problem if these companies offered an API to access their data. Sorry, but you're talking about something you have no idea of. The impact on that platform of someone using the API vs crawling the site is approximately the same. Whether someone scraped us via the API or via the HTML made no difference in terms of impact on the platform.
    - chatmasta 7y ago
      > Sorry, but you're talking about something you have no idea of. This kind of personal attack seems unnecessary, and by the way, I probably know a lot more than you assume about this particular topic. Regardless, if you had an API, you could charge for it. It wouldn't cover 100% of the scrapers, but it would cover some.

jacquesm 7y ago

> Sign up to see special offers Another reason why they do that is because their partners require they don't advertise special pricing to the world at large, they're contractually forbidden from doing so.