4 ms·

Is there support for robots.txt so service operators can opt out of your mass scraping?

by xena 2y ago

soulofmischief 2y ago
Would you like to explain how directing your user agent to use the internet just as you would in order to complete a task or solve a problem is "mass scraping"?
- ranger_danger 2y ago
  Intent.
  - soulofmischief 2y ago
    Care to elaborate? That is not a substantial argument.
- dkh 2y ago
  > to use the internet just as you would Your premise is flawed, or at the very least, far from guaranteed. One could use tools like this to use to browse the internet only as they would otherwise. One could also use it for mass scraping, and many do. If you've looked at HN on previous days this week, there's been a front-page story nearly every day about problems resulting from exactly that. The parent comment was perhaps a bit too snarky with the assumption that this could/would only be used for malicious behavior on a large scale, but your assumption that it wouldn't be is not any better, and also runs contrary to what people have been experiencing and discussing in this arena in recent days
shrisukhani 2y ago
No, we don't enforce any robots.txt restrictions ourselves. We also don't do any scraping ourselves. We provide browser infrastructure that operates like any normal browser would - what users choose to do with it is up to them. We're building tools that give AI agents the same web access capabilities that humans have, don't think it's our place to impose any additional limitations.
- raggi 2y ago
  It is 100% your responsibility what your servers do to other peoples servers in this context, and wanton negligence is not an excuse that will stop your servers from being evicted by hosting companies.
- xena 2y ago
  Do you publish an IP range?
- dkh 2y ago
  You make the tools, what people do with them isn't up to you. I can tolerate some form of that opinion/argument on some level, but it is at the very least short-sighted on your part to not have been better-equipped for how to respond to concerns people have about potential misuse. If what has been said elsewhere in this thread is true about providing documentation on how to circumvent attempts to detect/block your service and your resistance to providing helpful information such as IP ranges used and how user agents are set, then you have strayed far from being neutral and hands-off. "it's not our place" is not actual neutrality, it's performative or complicit neutrality. Actual neutrality would be perhaps not providing ways to counter your service, but also not documenting how to circumvent people from trying. And if this is what your POV is, fine! You are far from alone--given the state of the automated browsing/scraping ecosystem right now, plenty of people feel this way. Be honest about it! Don't deflect questions. Don't give misleading answers/information. That's what carries this into sketchy territory
olivia-l 2y ago
Not only do they not respect robots.txt, but they publish an entire page[1] in their docs dedicated to circumventing scraping countermeasures. I pointed their scraper at a url on my server to test it's behavior. It made four separate requests to the same page, three with the UA "udici"[2], and one with the UA "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36" and a different IP. The first three requests were all made within the span of 1 second, and the fourth 27 seconds later. I emailed their published support address asking for an IP range and UA. They gave me the entire IP range of google cloud, and ignored the UA question. This goes well beyond the "it's up to our users to implement responsible scraping practices" implication from the developer's other comment[3]. Instead, their service behaves maliciously by default, and they have implemented and documented switches that users can toggle for additional malicious scraping behavior. As far as I can tell, it is not even possible to implement a robots.txt-respecting scraper on top of this, because I couldn't find any mechanism for users to set a specific UA string. [1]: https://docs.hyperbrowser.ai/sessions/advanced-privacy-and-anti-detection https://docs.hyperbrowser.ai/sessions/advanced-privacy-and-a... (archived: https://web.archive.org/web/20250322045952/https://docs.hyperbrowser.ai/sessions/advanced-privacy-and-anti-detection https://web.archive.org/web/20250322045952/https://docs.hype..., http://archive.today/2025.03.22-050029/https://docs.hyperbrowser.ai/sessions/advanced-privacy-and-anti-detection http://archive.today/2025.03.22-050029/https://docs.hyperbro...) [2]: https://github.com/nodejs/undici https://github.com/nodejs/undici [3]: https://news.ycombinator.com/item?id=43442116 https://news.ycombinator.com/item?id=43442116
- dkh 2y ago
  Nice work. I feel like there's a lot to unpack here, and still much to discuss in the broader context. There's a few things that can be excused or at least reasonably argued. Multiple IPs makes some sense since requests are being made through random proxies, and I don't think on its own demonstrates intention of bad behavior. If it was, I think you would've seen all four requests as coming from different IPs and with different user agents all posing as legitimate user browser sessions. But the rest is inexcusable. Providing documentation on how to circumvent countermeasures without even acknowledging, or presenting a viewpoint on, the concern many have with the potential malicious use of these tools. Taking the time to respond to your inquiry about the IP range and UA but giving an answer that is somewhere between intentionally incomplete and intentionally misleading. (Is the actual IP range even within the large Google IP range given the source of the proxies as mentioned elsewhere in this thread?) Just very poor decisions on how to handle the extremely predictable resistance they were very obviously going to encounter

Is there support for robots.txt so service operators can opt out of your mass scraping?

by xena 2y ago

Is there support for robots.txt so service operators can opt out of your mass scraping?

Would you like to explain how directing your user agent to use the internet just as you would in order to complete a task or solve a problem is "mass scraping"?

ranger_danger 2y ago
Intent.
- soulofmischief 2y ago
  Care to elaborate? That is not a substantial argument.
dkh 2y ago
> to use the internet just as you would Your premise is flawed, or at the very least, far from guaranteed. One could use tools like this to use to browse the internet only as they would otherwise. One could also use it for mass scraping, and many do. If you've looked at HN on previous days this week, there's been a front-page story nearly every day about problems resulting from exactly that. The parent comment was perhaps a bit too snarky with the assumption that this could/would only be used for malicious behavior on a large scale, but your assumption that it wouldn't be is not any better, and also runs contrary to what people have been experiencing and discussing in this arena in recent days

shrisukhani 2y ago

No, we don't enforce any robots.txt restrictions ourselves. We also don't do any scraping ourselves. We provide browser infrastructure that operates like any normal browser would - what users choose to do with it is up to them. We're building tools that give AI agents the same web access capabilities that humans have, don't think it's our place to impose any additional limitations.

raggi 2y ago
It is 100% your responsibility what your servers do to other peoples servers in this context, and wanton negligence is not an excuse that will stop your servers from being evicted by hosting companies.
xena 2y ago
Do you publish an IP range?
dkh 2y ago
You make the tools, what people do with them isn't up to you. I can tolerate some form of that opinion/argument on some level, but it is at the very least short-sighted on your part to not have been better-equipped for how to respond to concerns people have about potential misuse. If what has been said elsewhere in this thread is true about providing documentation on how to circumvent attempts to detect/block your service and your resistance to providing helpful information such as IP ranges used and how user agents are set, then you have strayed far from being neutral and hands-off. "it's not our place" is not actual neutrality, it's performative or complicit neutrality. Actual neutrality would be perhaps not providing ways to counter your service, but also not documenting how to circumvent people from trying. And if this is what your POV is, fine! You are far from alone--given the state of the automated browsing/scraping ecosystem right now, plenty of people feel this way. Be honest about it! Don't deflect questions. Don't give misleading answers/information. That's what carries this into sketchy territory

olivia-l 2y ago

Not only do they not respect robots.txt, but they publish an entire page[1] in their docs dedicated to circumventing scraping countermeasures. I pointed their scraper at a url on my server to test it's behavior. It made four separate requests to the same page, three with the UA "udici"[2], and one with the UA "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36" and a different IP. The first three requests were all made within the span of 1 second, and the fourth 27 seconds later. I emailed their published support address asking for an IP range and UA. They gave me the entire IP range of google cloud, and ignored the UA question. This goes well beyond the "it's up to our users to implement responsible scraping practices" implication from the developer's other comment[3]. Instead, their service behaves maliciously by default, and they have implemented and documented switches that users can toggle for additional malicious scraping behavior. As far as I can tell, it is not even possible to implement a robots.txt-respecting scraper on top of this, because I couldn't find any mechanism for users to set a specific UA string. [1]: https://docs.hyperbrowser.ai/sessions/advanced-privacy-and-anti-detection https://docs.hyperbrowser.ai/sessions/advanced-privacy-and-a... (archived: https://web.archive.org/web/20250322045952/https://docs.hyperbrowser.ai/sessions/advanced-privacy-and-anti-detection https://web.archive.org/web/20250322045952/https://docs.hype..., http://archive.today/2025.03.22-050029/https://docs.hyperbrowser.ai/sessions/advanced-privacy-and-anti-detection http://archive.today/2025.03.22-050029/https://docs.hyperbro...) [2]: https://github.com/nodejs/undici https://github.com/nodejs/undici [3]: https://news.ycombinator.com/item?id=43442116 https://news.ycombinator.com/item?id=43442116

dkh 2y ago
Nice work. I feel like there's a lot to unpack here, and still much to discuss in the broader context. There's a few things that can be excused or at least reasonably argued. Multiple IPs makes some sense since requests are being made through random proxies, and I don't think on its own demonstrates intention of bad behavior. If it was, I think you would've seen all four requests as coming from different IPs and with different user agents all posing as legitimate user browser sessions. But the rest is inexcusable. Providing documentation on how to circumvent countermeasures without even acknowledging, or presenting a viewpoint on, the concern many have with the potential malicious use of these tools. Taking the time to respond to your inquiry about the IP range and UA but giving an answer that is somewhere between intentionally incomplete and intentionally misleading. (Is the actual IP range even within the large Google IP range given the source of the proxies as mentioned elsewhere in this thread?) Just very poor decisions on how to handle the extremely predictable resistance they were very obviously going to encounter