6 ms·

Funny, I saw this HN headline just after banning another scraper's IP range You're welcome to scrape my sites but please do it ethically. Idk how to define tha

by lucb1e 1y ago

Funny, I saw this HN headline just after banning another scraper's IP range You're welcome to scrape my sites but please do it ethically. Idk how to define that but some examples of things I consider not cool: - Scraping without a contact method, or at least some unique identifier (like your project's codename), in the user agent string. This is common practice, see e.g.: <https://en.wikipedia.org/wiki/User-Agent_header#Format_for_automated_agents_(bots) https://en.wikipedia.org/wiki/User-Agent_header#Format_for_a...>. Many sites mention in public API guidelines to include an email address so you can be contacted in case of problems. If you don't include this and you're causing trouble, all I can do is ban your IP address altogether (or entire ranges: if you hop between several IPs I'll have to assume you have access to the whole range). Nobody likes IP bans: you have to get a new IP, your provider has a burned IP address, the next customer runs into issues... don't be this person, include an identifier. - Timing out the request after a few seconds. Some pages on my site involve number crunching and take 20 seconds to load. I could add complexity to do this async instead, but, by having it live, the regular users get the latest info and they know to just wait a few seconds and everybody is happy. Even the scrapers can get the info, I'm fine computing those pages for you. But if you ask for me to do work and then walk away, that's just rude. It shows up in my logs as HTTP status 499 and I'll ban scrapers that I notice doing this regularly - Ignoring robots.txt. I have exactly 1 entry in there, and that's a caching proxy for another site that is struggling with load. If you ignore the robots file and just crawl the thing from A to Z at a high rate, that causes a lot of requests to the upstream site for updating stale caches. You can obviously expect a ban because it's again just a waste of resources

edoceo 1y ago
What do you have for log analytics and ban automation? Could you say more about how to identify these bad-bots?
- lucb1e 1y ago
  There is no automation, I use `tail -f access.log` I just look at what's happening on my server every now and then. Sometimes not for months, but then when I set up a project like that caching proxy, I'm currently keeping a more regular eye to see that crawlers aren't bothering the upstream via me. Most respect the robots policy, most of the ones that don't set a user agent string that include the word 'bot' and so I know not to refresh the cache based on that request. So far it has mostly been Huawei who pretend to be a regular user but request millions of pages (from 12 separate IP ranges so far, some of them bigger than /16, some of them a handful of /24s). > Could you say more about how to identify these bad-bots? Many requests per day to random pages from either the same IP address (range), or ranges owned by the same corporation
  - reconnecting 1y ago
    Interesting. Our open-source platform [1] has the capacity to help with all of this through a GUI and rule engine, but I'm still concerned about whether we should present this way of bot hunting as a feature. I worry that this approach may be irrelevant in today's context. [1] https://github.com/TirrenoTechnologies/tirreno https://github.com/TirrenoTechnologies/tirreno
    - tough 1y ago
      I mean if anything with AI data and main sources are becoming the actual precious resource again. So i'd expect an uptick in bots as everyone races to try and compete with google on data hoarding
      - reconnecting 1y ago
        As I can see, there is already a heavy wave of new AI/startup/VC etc. data companies that goes beyond the data consumption expectations of websites in the pre-AI era. However, I see the development of new bot types that tackle security in more aggressive ways. It's not just simple SQL injection as it was before, but more sophisticated and custom bots that not only request but also push a lot. Or just a couple of days ago, I found a new type of bot that "brute-forces" website folder structure. ~205,000 requests in a couple of days. These new bots are probably not directly the work of AI, but they seem to be a consequence of it.
- reconnecting 1y ago
  We use tirreno [1] to manually and automatically analyze traffic and block unwanted bots. Although bot management is not currently listed as an official feature, it works well and is particularly helpful in complex bot hunting. [1] https://github.com/TirrenoTechnologies/tirreno https://github.com/TirrenoTechnologies/tirreno
VladVladikoff 1y ago
What sort of pages require 20 seconds to generate? This is extremely slow by most web standards and even your users would be frustrated by this. It sounds like poorly designed database queries with unindexed joins. Google will also abandon page loads that take too long, and will demote rankings for that page (or the entire site!)
- lucb1e 1y ago
  > It sounds like poorly designed database queries with unindexed joins Neither of those assumptions are correct. As an example, one page needs to look through 2.5 million records to find where the world record holder changed because it provides stats on who held the most records, held them for the greatest cumulative time, etc. The only thing to do would be introducing caching layers for parts of the computation, but for the number of users this system has, it's just not worth spending more development time than I already have. Also keep in mind it's a free web service and I don't run ads or anything, it's just a fan project for a game > Google will ... demote rankings for that page (or the entire site!) Google employs anticompetitive practices to maintain the search monopoly. We need more diversity in search engines, I don't know how else to encourage people to use something instead of, or at least in addition to, Google, besides by making Google Search just not competitive anymore. Google's crawler cannot access my site in the first place (but their other crawlers can; I'm pretty selective about this). My sites never show up in Google searches, on purpose It's also not the whole site that's slow, it's when you click on a handful of specific pages. If that makes those pages not appear in search results, that's fine. Besides that it's not my loss, it's not like any other site has the info so people will find their way to the main page and click on what they want to see
  - VladVladikoff 1y ago
    Like I said then, you need indexes on those columns which you filter on in this table. Search a table of 2.5 million records for a value is still blazing fast if you use indexes correctly. I’m talking about 0.01 seconds or less. Even with tables much larger. I agree about Google being shit. However, my website makes my living, and feeds and clothes my children, so I have to play along to their rules, or suffer. Please take your slowest performing query and run it with EXPLAIN in front. And share that (or dump it into an LLM and it will tell you have to fix it)
whazor 1y ago
I find it unethical for a website robots.txt to allow-list particular search engines and ban all others. Essentially you are colluding with established search providers.
- Loic 1y ago
  Not necessarily, I have a website with 95% (maybe even more) of the traffic generated by crawlers. If some of them are behaving badly, it is fair to exclude them with my robots.txt. But of course, the ones behaving badly tend to not respect the robots.txt, so you end up banning the IP or IP block. And here, I am a nice guy, the crawler must really be a piece of crap for me to start to block.
  - throawayonthe 1y ago
    the parent comment is talking about allow-listing (aka 'whitelisting') just a few crawlers from like, google
  - whazor 1y ago
    Deny-listing/banning bad crawlers is fine. Especially if they ignore the robots.txt. But allow-listing particular crawlers only is collusion.
- ToucanLoucan 1y ago
  This rather bluntly runs up against the fact that permitting crawling is an expense the web operator is taking on, ergo, receiving that content is by definition a privilege not a right.
  - p3rls 1y ago
    I think my team has spent about a quarter of this year's dev time on defending against scrapers. Their arrogance is the stuff of ancient greek dramas.
- lucb1e 1y ago
  I don't know if that's a reply at me or a general remark, but yes, I never understood why you'd include a few big names and ban the rest for example. That's just screaming for anticompetitiveness. I don't know if my mention of robots.txt sounded like I do this, but I do not
Fokamul 1y ago
Who cares, IP ranges are cheap. You're just banning datacenters.
datatrashfire 1y ago
> - Scraping without a contact method, or at least some unique identifier (like your project's codename), in the user agent string. This is a very effective way to make sure you won't get any scraping done!
- lucb1e 1y ago
  Tell that to Googlebot, Bingbot, Petalbot, SemrushBot, MJ12bot, MojeekBot, DotBot, YandexBot, SeznamBot, Barkrowler, AhrefsBot, DuckDuckBot, AcademicBotRTU, Bytespider, Applebot, ZoominfoBot, TelegramBot, TwitterBot, SemanticScholarBot, redditbot, Pinterestbot... From a quick peek at my access log, all include either a link (most) or an email address (zoom, tiktok/bytedance, dotbot, and that academic bot) Very few individual bots don't follow this good practice. Most of the IP ranges of violating bots are owned by Huawei (a few is Huawei Cloud so it could be anyone, but the majority seems to be Huawei themselves) and the remainder is all small beans as far as I remember (few thousand accesses in a day and then disappear forever, for example)
  - datatrashfire 1y ago
    None of the institutional market intelligence products I’ve ever worked on in nearly a decade of doing this do. Why? Cause they wouldn’t otherwise work. Many APIs require specific user agents. Tools like curl impersonate require specific user agents.

by lucb1e 1y ago

edoceo 1y ago

What do you have for log analytics and ban automation? Could you say more about how to identify these bad-bots?

lucb1e 1y ago
There is no automation, I use `tail -f access.log` I just look at what's happening on my server every now and then. Sometimes not for months, but then when I set up a project like that caching proxy, I'm currently keeping a more regular eye to see that crawlers aren't bothering the upstream via me. Most respect the robots policy, most of the ones that don't set a user agent string that include the word 'bot' and so I know not to refresh the cache based on that request. So far it has mostly been Huawei who pretend to be a regular user but request millions of pages (from 12 separate IP ranges so far, some of them bigger than /16, some of them a handful of /24s). > Could you say more about how to identify these bad-bots? Many requests per day to random pages from either the same IP address (range), or ranges owned by the same corporation
- reconnecting 1y ago
  Interesting. Our open-source platform [1] has the capacity to help with all of this through a GUI and rule engine, but I'm still concerned about whether we should present this way of bot hunting as a feature. I worry that this approach may be irrelevant in today's context. [1] https://github.com/TirrenoTechnologies/tirreno https://github.com/TirrenoTechnologies/tirreno
  - tough 1y ago
    I mean if anything with AI data and main sources are becoming the actual precious resource again. So i'd expect an uptick in bots as everyone races to try and compete with google on data hoarding
    - reconnecting 1y ago
      As I can see, there is already a heavy wave of new AI/startup/VC etc. data companies that goes beyond the data consumption expectations of websites in the pre-AI era. However, I see the development of new bot types that tackle security in more aggressive ways. It's not just simple SQL injection as it was before, but more sophisticated and custom bots that not only request but also push a lot. Or just a couple of days ago, I found a new type of bot that "brute-forces" website folder structure. ~205,000 requests in a couple of days. These new bots are probably not directly the work of AI, but they seem to be a consequence of it.
reconnecting 1y ago
We use tirreno [1] to manually and automatically analyze traffic and block unwanted bots. Although bot management is not currently listed as an official feature, it works well and is particularly helpful in complex bot hunting. [1] https://github.com/TirrenoTechnologies/tirreno https://github.com/TirrenoTechnologies/tirreno

VladVladikoff 1y ago

What sort of pages require 20 seconds to generate? This is extremely slow by most web standards and even your users would be frustrated by this. It sounds like poorly designed database queries with unindexed joins. Google will also abandon page loads that take too long, and will demote rankings for that page (or the entire site!)

lucb1e 1y ago
> It sounds like poorly designed database queries with unindexed joins Neither of those assumptions are correct. As an example, one page needs to look through 2.5 million records to find where the world record holder changed because it provides stats on who held the most records, held them for the greatest cumulative time, etc. The only thing to do would be introducing caching layers for parts of the computation, but for the number of users this system has, it's just not worth spending more development time than I already have. Also keep in mind it's a free web service and I don't run ads or anything, it's just a fan project for a game > Google will ... demote rankings for that page (or the entire site!) Google employs anticompetitive practices to maintain the search monopoly. We need more diversity in search engines, I don't know how else to encourage people to use something instead of, or at least in addition to, Google, besides by making Google Search just not competitive anymore. Google's crawler cannot access my site in the first place (but their other crawlers can; I'm pretty selective about this). My sites never show up in Google searches, on purpose It's also not the whole site that's slow, it's when you click on a handful of specific pages. If that makes those pages not appear in search results, that's fine. Besides that it's not my loss, it's not like any other site has the info so people will find their way to the main page and click on what they want to see
- VladVladikoff 1y ago
  Like I said then, you need indexes on those columns which you filter on in this table. Search a table of 2.5 million records for a value is still blazing fast if you use indexes correctly. I’m talking about 0.01 seconds or less. Even with tables much larger. I agree about Google being shit. However, my website makes my living, and feeds and clothes my children, so I have to play along to their rules, or suffer. Please take your slowest performing query and run it with EXPLAIN in front. And share that (or dump it into an LLM and it will tell you have to fix it)

whazor 1y ago

I find it unethical for a website robots.txt to allow-list particular search engines and ban all others. Essentially you are colluding with established search providers.

Loic 1y ago
Not necessarily, I have a website with 95% (maybe even more) of the traffic generated by crawlers. If some of them are behaving badly, it is fair to exclude them with my robots.txt. But of course, the ones behaving badly tend to not respect the robots.txt, so you end up banning the IP or IP block. And here, I am a nice guy, the crawler must really be a piece of crap for me to start to block.
- throawayonthe 1y ago
  the parent comment is talking about allow-listing (aka 'whitelisting') just a few crawlers from like, google
- whazor 1y ago
  Deny-listing/banning bad crawlers is fine. Especially if they ignore the robots.txt. But allow-listing particular crawlers only is collusion.
ToucanLoucan 1y ago
This rather bluntly runs up against the fact that permitting crawling is an expense the web operator is taking on, ergo, receiving that content is by definition a privilege not a right.
- p3rls 1y ago
  I think my team has spent about a quarter of this year's dev time on defending against scrapers. Their arrogance is the stuff of ancient greek dramas.
lucb1e 1y ago
I don't know if that's a reply at me or a general remark, but yes, I never understood why you'd include a few big names and ban the rest for example. That's just screaming for anticompetitiveness. I don't know if my mention of robots.txt sounded like I do this, but I do not

Fokamul 1y ago

Who cares, IP ranges are cheap. You're just banning datacenters.

datatrashfire 1y ago

> - Scraping without a contact method, or at least some unique identifier (like your project's codename), in the user agent string. This is a very effective way to make sure you won't get any scraping done!

lucb1e 1y ago
Tell that to Googlebot, Bingbot, Petalbot, SemrushBot, MJ12bot, MojeekBot, DotBot, YandexBot, SeznamBot, Barkrowler, AhrefsBot, DuckDuckBot, AcademicBotRTU, Bytespider, Applebot, ZoominfoBot, TelegramBot, TwitterBot, SemanticScholarBot, redditbot, Pinterestbot... From a quick peek at my access log, all include either a link (most) or an email address (zoom, tiktok/bytedance, dotbot, and that academic bot) Very few individual bots don't follow this good practice. Most of the IP ranges of violating bots are owned by Huawei (a few is Huawei Cloud so it could be anyone, but the majority seems to be Huawei themselves) and the remainder is all small beans as far as I remember (few thousand accesses in a day and then disappear forever, for example)
- datatrashfire 1y ago
  None of the institutional market intelligence products I’ve ever worked on in nearly a decade of doing this do. Why? Cause they wouldn’t otherwise work. Many APIs require specific user agents. Tools like curl impersonate require specific user agents.