3 ms·

What measures are you using to make sure you're not bombarding websites with a ton of requests, since you seem to automatically "scale up" the concurrency to cr

by diggan 2y ago

What measures are you using to make sure you're not bombarding websites with a ton of requests, since you seem to automatically "scale up" the concurrency to create even more requests/second? Does it read any of the rate-limit headers from the responses or do something else to back-off in case what it's trying to visit suddenly becomes offline or starts having slower response times? Slightly broader question: Do you feel like there is any ethical considerations one should think about before using something like this?

calebhwin 2y ago
The main sort of parallelism we exploit is across distinct websites. For example "find me the cheapest rental" spawning tasks to look at many different websites. There is another level of parallelism that could be exploited within a web site/app. And yes we would have to make our planner rate limit aware for that. Absolutely agree there are ethical considerations with web browsing AI in general. (And the whole general ongoing shift from using websites to using chatgpt/perplexity)
- diggan 2y ago
  > Absolutely agree there are ethical considerations with web browsing AI in general. I'm personally not sure there are, but I'm curious to hear what those are for you :)
  - calebhwin 2y ago
    Maybe more of a legal than ethical consideration but web browsing AI makes scraping trivial. You could use that for surveillance, profiling (get a full picture of a user's whole online life before they even hit Sign Up), cutting egress cost in certain cases. Right now CAPTCHA is actually holding up pretty well against web browsing AI for sites that really want to protect their IP but it will be interesting to see if that devolves into yet another instance of an AI vs AI "arms race".
    - sieabahlpark 2y ago
      [dead]
- rollcat 2y ago
  > There is another level of parallelism that could be exploited within a web site/app. And yes we would have to make our planner rate limit aware for that. People are already deploying tools like Anubis[1] or go-away[2] to cope with the insane load that bots put on their server infrastructure. This is an arms race. In the end, the users lose. [1]: https://anubis.techaro.lol https://anubis.techaro.lol [2]: https://git.gammaspectra.live/git/go-away https://git.gammaspectra.live/git/go-away
  - calebhwin 2y ago
    IMO it depends on how this tech is deployed. One way I see this being extremely useful is for developers to quickly build AI automation for their own sites. E.g. if I'm the developer of a workforce management app (e.g. https://WhenIWork.com https://WhenIWork.com) I could deploy BLAST to quickly provide automation for users of my app.
    - rollcat 2y ago
      That's my point. You can use a knife to slice bread or to stab your neighbor. We're seeing an unprecedented amount of stabbings. People are getting away with murder, there's no accountability. Refining the stilettos doesn't help the problem.
      - spiderfarmer 2y ago
        I just block every AI bot that doesn’t give me traffic or other benefits. I know I can’t block them all, but it won’t be for a lack of trying.
  - lostmsu 2y ago
    > In the end, the users lose. I think it would take more substantiation to claim this. Maybe 10 out of 1000 websites will get closed, but the users will be able to use AI tools to use the remaining 990. Not sure about you, but sounds like a win for users to me.
    - 2y ago

by diggan 2y ago

calebhwin 2y ago

The main sort of parallelism we exploit is across distinct websites. For example "find me the cheapest rental" spawning tasks to look at many different websites. There is another level of parallelism that could be exploited within a web site/app. And yes we would have to make our planner rate limit aware for that. Absolutely agree there are ethical considerations with web browsing AI in general. (And the whole general ongoing shift from using websites to using chatgpt/perplexity)

diggan 2y ago
> Absolutely agree there are ethical considerations with web browsing AI in general. I'm personally not sure there are, but I'm curious to hear what those are for you :)
- calebhwin 2y ago
  Maybe more of a legal than ethical consideration but web browsing AI makes scraping trivial. You could use that for surveillance, profiling (get a full picture of a user's whole online life before they even hit Sign Up), cutting egress cost in certain cases. Right now CAPTCHA is actually holding up pretty well against web browsing AI for sites that really want to protect their IP but it will be interesting to see if that devolves into yet another instance of an AI vs AI "arms race".
  - sieabahlpark 2y ago
    [dead]
rollcat 2y ago
> There is another level of parallelism that could be exploited within a web site/app. And yes we would have to make our planner rate limit aware for that. People are already deploying tools like Anubis[1] or go-away[2] to cope with the insane load that bots put on their server infrastructure. This is an arms race. In the end, the users lose. [1]: https://anubis.techaro.lol https://anubis.techaro.lol [2]: https://git.gammaspectra.live/git/go-away https://git.gammaspectra.live/git/go-away
- calebhwin 2y ago
  IMO it depends on how this tech is deployed. One way I see this being extremely useful is for developers to quickly build AI automation for their own sites. E.g. if I'm the developer of a workforce management app (e.g. https://WhenIWork.com https://WhenIWork.com) I could deploy BLAST to quickly provide automation for users of my app.
  - rollcat 2y ago
    That's my point. You can use a knife to slice bread or to stab your neighbor. We're seeing an unprecedented amount of stabbings. People are getting away with murder, there's no accountability. Refining the stilettos doesn't help the problem.
    - spiderfarmer 2y ago
      I just block every AI bot that doesn’t give me traffic or other benefits. I know I can’t block them all, but it won’t be for a lack of trying.
- lostmsu 2y ago
  > In the end, the users lose. I think it would take more substantiation to claim this. Maybe 10 out of 1000 websites will get closed, but the users will be able to use AI tools to use the remaining 990. Not sure about you, but sounds like a win for users to me.
  - 2y ago