4 ms·

How do I block your service? Do you read robots.txt and have an identifiable user agent?

by xena 1y ago

calebhwin 1y ago
Good point, we should probably integrate that. Feel free to submit a PR! BLAST can also be used to add automation to your own site/app FWIW.
- dejobaan 1y ago
  > Feel free to submit a PR! I think it's cool that you're experimenting in this area, but I'm not a huge fan of this as answer to a question about responsible/respectful web crawling. This stuff seems like it should be table stakes (even if you wanted to make it optional for the end user), but "yeah probably; learn the codebase, fork it, make changes, then we'll review it" really puts the onus onto the original poster.
  - lostmsu 1y ago
    Curious: if a user has an ad blocker, are they browsing responsibly?
    - subscribed 1y ago
      Very. Malvertising is a thing. Adtech surveillance is a thing. Ad blocker is the least user can do.
      - lostmsu 1y ago
        That's not why most users use ad blockers.
  - calebhwin 1y ago
    Ah you're right, my bad. Hope I didn't sound dismissive because I think some sort of robots.txt needs to exist for AI that's scraping the web both at train or test time. I'm really not excited at all about the "scrape other people's data" use case for BLAST and if we can prevent it then awesome. I'm excited about BLAST automating science, legacy web apps, internal tools, adding AI automation to your own app, etc.
- xena 1y ago
  I will for my typical hourly rate plus the consulting bonus.
- BrenBarn 1y ago
  Feel free to take down your project until you comply with basic ethical standards.
pal9000i 1y ago
The whole point of AI browser automation is mimicking human behaviour, fighting the anti-bot detection systems. If the point is interacting with systems, we'd be using APIs
- croemer 1y ago
  ...when there is an API. Often there isn't.
- jlpom 1y ago
  No, it's to automate tasks that can't be done using an API, like RPA.
  - nikisweeting 1y ago
    you're saying essentiall the same thing as the parent comment, both are true anti-bot fingerprinting exists to prevent you from using the human-facing site like an API
- subscribed 1y ago
  How can I find for myself an affordable accommodation in Iceland across several websites aggregating cheap accommodation but without signing API access contract with all of them and without building a middleware to abstract them? I don't have API for _that_.
diggan 1y ago
Seems Blast uses browser-use (https://github.com/browser-use/browser-use https://github.com/browser-use/browser-use) which seems to be some client specifically for AIs to connect to/run browser runtimes. Unfortunately, it seems like browser-use tries to hide that it's controlled by an AI, and uses a typical browser user-agent: https://github.com/browser-use/browser-use/blob/d8c4d03d9ea90775a69fbce4deb8738e8c79fefe/browser_use/browser/chrome.py#L1-L3 https://github.com/browser-use/browser-use/blob/d8c4d03d9ea9... I'm guessing because of the amount of flags, you could probably come up with a unique fingerprint for browser-use, based on available features, screen/canvas size and so on, that could be reused for blocking everyone using Blast/browser-use. If calebhwin wanted to make Blast easier to identify, they could set a custom user-agent for browser-use that makes it clear it's Blast doing the browsing for the user.
- ATechGuy 1y ago
  Can browser-use be blocked using Anubis or other anti-bot measures?
  - razemio 1y ago
    I think these kind of requests can not be blocked. It is like asking if Claude computer control can be blocked from visiting websites. It is not detectable. You could ofc display captchas which are difficult to solve for an AI.
    - diggan 1y ago
      As mentioned, browser-use doesn't seem "out of this world" hard to fingerprint since all instances would be using the exact same settings.
      - ZeroTalent 1y ago
        Browser-use just switched to patchright https://github.com/Kaliiiiiiiiii-Vinyzu/patchright https://github.com/Kaliiiiiiiiii-Vinyzu/patchright which is an “undetectable” version of Playwright and can rotate/randomize fingerprints. Another extension uses residential proxies and pulls real fingerprints of real people.

How do I block your service? Do you read robots.txt and have an identifiable user agent?

by xena 1y ago

How do I block your service? Do you read robots.txt and have an identifiable user agent?

Good point, we should probably integrate that. Feel free to submit a PR! BLAST can also be used to add automation to your own site/app FWIW.

dejobaan 1y ago
> Feel free to submit a PR! I think it's cool that you're experimenting in this area, but I'm not a huge fan of this as answer to a question about responsible/respectful web crawling. This stuff seems like it should be table stakes (even if you wanted to make it optional for the end user), but "yeah probably; learn the codebase, fork it, make changes, then we'll review it" really puts the onus onto the original poster.
- lostmsu 1y ago
  Curious: if a user has an ad blocker, are they browsing responsibly?
  - subscribed 1y ago
    Very. Malvertising is a thing. Adtech surveillance is a thing. Ad blocker is the least user can do.
    - lostmsu 1y ago
      That's not why most users use ad blockers.
- calebhwin 1y ago
  Ah you're right, my bad. Hope I didn't sound dismissive because I think some sort of robots.txt needs to exist for AI that's scraping the web both at train or test time. I'm really not excited at all about the "scrape other people's data" use case for BLAST and if we can prevent it then awesome. I'm excited about BLAST automating science, legacy web apps, internal tools, adding AI automation to your own app, etc.
xena 1y ago
I will for my typical hourly rate plus the consulting bonus.
BrenBarn 1y ago
Feel free to take down your project until you comply with basic ethical standards.

pal9000i 1y ago

The whole point of AI browser automation is mimicking human behaviour, fighting the anti-bot detection systems. If the point is interacting with systems, we'd be using APIs

croemer 1y ago
...when there is an API. Often there isn't.
jlpom 1y ago
No, it's to automate tasks that can't be done using an API, like RPA.
- nikisweeting 1y ago
  you're saying essentiall the same thing as the parent comment, both are true anti-bot fingerprinting exists to prevent you from using the human-facing site like an API
subscribed 1y ago
How can I find for myself an affordable accommodation in Iceland across several websites aggregating cheap accommodation but without signing API access contract with all of them and without building a middleware to abstract them? I don't have API for _that_.

diggan 1y ago

Seems Blast uses browser-use (https://github.com/browser-use/browser-use https://github.com/browser-use/browser-use) which seems to be some client specifically for AIs to connect to/run browser runtimes. Unfortunately, it seems like browser-use tries to hide that it's controlled by an AI, and uses a typical browser user-agent: https://github.com/browser-use/browser-use/blob/d8c4d03d9ea90775a69fbce4deb8738e8c79fefe/browser_use/browser/chrome.py#L1-L3 https://github.com/browser-use/browser-use/blob/d8c4d03d9ea9... I'm guessing because of the amount of flags, you could probably come up with a unique fingerprint for browser-use, based on available features, screen/canvas size and so on, that could be reused for blocking everyone using Blast/browser-use. If calebhwin wanted to make Blast easier to identify, they could set a custom user-agent for browser-use that makes it clear it's Blast doing the browsing for the user.

ATechGuy 1y ago
Can browser-use be blocked using Anubis or other anti-bot measures?
- razemio 1y ago
  I think these kind of requests can not be blocked. It is like asking if Claude computer control can be blocked from visiting websites. It is not detectable. You could ofc display captchas which are difficult to solve for an AI.
  - diggan 1y ago
    As mentioned, browser-use doesn't seem "out of this world" hard to fingerprint since all instances would be using the exact same settings.
    - ZeroTalent 1y ago
      Browser-use just switched to patchright https://github.com/Kaliiiiiiiiii-Vinyzu/patchright https://github.com/Kaliiiiiiiiii-Vinyzu/patchright which is an “undetectable” version of Playwright and can rotate/randomize fingerprints. Another extension uses residential proxies and pulls real fingerprints of real people.