3 ms·

I wish someone made a similar command line tool that does screen scraping, is powered by AI and could simulate clicks as a real user.

guessmyname 3y ago
I can build it. Out of curiosity, how much would you be willing to pay for such a tool?
- lelanthran 3y ago
  > I can build it. Out of curiosity, how much would you be willing to pay for such a tool? You beat me to it by just a few seconds! :-)
- jakozaur 3y ago
  $20-50 / month Though I'm a picky and very demanding user who likes to use API, etc, you have been warned :-).
lelanthran 3y ago
> I wish someone made a similar command line tool that does screen scraping, is powered by AI and could simulate clicks as a real user. How much are you, personally, willing to pay for such a tool? Because, minus the AI[1], I'm pretty certain that such tools exists at different price points. I'm not sure what the AI would do for such a tool, but I'm sure that the companies making such a thing are busy working hard to incorporate some form of AI into it. [1] What would the AI in such a tool do?
- nextaccountic 3y ago
  > [1] What would the AI in such a tool do? Solving captchas. In particular, not only solving the challenge itself (recognizing crosswalks, etc) but also handling input (moving the mouse, alt+tabbing, etc) as if it were a real human. I was recently reading the Cloudflare turnstile announcement https://blog.cloudflare.com/turnstile-ga https://blog.cloudflare.com/turnstile-ga and it has the following links about this https://uncaptcha.cs.umd.edu/papers/uncaptcha_woot17.pdf https://uncaptcha.cs.umd.edu/papers/uncaptcha_woot17.pdf https://www.vox.com/22436832/captchas-getting-harder-ai-artificial-intelligence https://www.vox.com/22436832/captchas-getting-harder-ai-arti... https://arxiv.org/abs/1903.01003 https://arxiv.org/abs/1903.01003 In those excerpts: > The prevalence of free speech-to-text services has made it easy for bots to solve audio CAPTCHAs as well, with a recent study showing bots can accurately solve audio CAPTCHAs in over 85% of attempts. We’re proud to state that Turnstile is WCAG 2.1 Level AA compliant, while eliminating the need for audio CAPTCHAs as well as visual ones. And > With all of our emphasis on how easy it is to pass a Turnstile challenge, you would be right to ask how it can stop a bot. If a bot can find all images with crosswalks in grainy photos faster than we can, surely it can check a box as well. Bots definitely can check a box, and they can even mimic the erratic path of human mouse movement while doing so. For Turnstile, the actual act of checking a box isn’t important, it’s the background data we’re analyzing while the box is checked that matters. We find and stop bots by running a series of in-browser tests, checking browser characteristics, native browser APIs, and asking the browser to pass lightweight tests (ex: proof-of-work tests, proof-of-space tests) to prove that it’s an actual browser. The current deployment of Turnstile checks billions of visitors every day, and we are able to identify browser abnormalities that bots exhibit while attempting to pass those tests. So, this is a cat and mouse game, the cat is using AI, the mouse is probably using AI in some form or another too.
  - eddd-ddde 3y ago
    We really are gonna end up with servers and clients evolving AI challenges and solvers respectively to the point where humans won't be able to solve them anymore haha
- jakozaur 3y ago
  The brainer price would be $20 / month. $50 / month if it was terrific. The trick has to be really useful without too much work on my end. Combine screen scraping and simulating real users with some mix of AI. My use cases: 1. Fill any form with intelligent data (from orders, through plane tickets to tax forms). Think of ChatGPT but for forms. 2. Research any person and company using various tools (e.g. LinkedIn, Google, job boards). Do it at human speed so I won't get blocked and can observe it on a screen. I can provide a script on how to do it (e.g. check LinkedIn, this board for that, etc.) Provide me with a summary of the text. 3. I want to be able to script it and hack it to whatever I need. API and CLI.
kordlessagain 3y ago
I just finished up a Grub crawler for https://mitta.ai https://mitta.ai. The result is that you can screenshot a page and then run it through OCR, then push the text into an LLM, or embed chunks of it: https://www.youtube.com/watch?v=QadlImAKzCE https://www.youtube.com/watch?v=QadlImAKzCE While it won't do clicking, the system also has a processor that can fetch the HTML (or other files). It would be an afternoon's work to add something like CurlyQ to it. I was kicking around adding an ffmpeg processor to it...
dbish 3y ago
We’re working on a tool that “screen shares” with AI and allows both looking at what is on screen when you’re sharing and acting on it (simulate clicks and other). What’s your use case? Happy to share more about where we’re at with the project. It’s quite a lot to build to get right for many kinds of apps and to be multi-os but we’re pretty close. No CLI though, would love to hear more about your use case there. Feel free to email me diamond@augmend.com
- jakozaur 3y ago
  My use cases: 1. Fill any form with intelligent data (from orders, through plane tickets to tax forms). Think of ChatGPT but for forms. 2. Research any person and company using various tools (e.g. LinkedIn, Google, job boards). Do it at human speed so I won't get blocked and can observe it on a screen. Provide me with a summary of text and list. 3. I want to be be able to script it and hack to whatever I need. API and CLI.

guessmyname 3y ago

I can build it. Out of curiosity, how much would you be willing to pay for such a tool?

lelanthran 3y ago
> I can build it. Out of curiosity, how much would you be willing to pay for such a tool? You beat me to it by just a few seconds! :-)
jakozaur 3y ago
$20-50 / month Though I'm a picky and very demanding user who likes to use API, etc, you have been warned :-).

lelanthran 3y ago

> I wish someone made a similar command line tool that does screen scraping, is powered by AI and could simulate clicks as a real user. How much are you, personally, willing to pay for such a tool? Because, minus the AI[1], I'm pretty certain that such tools exists at different price points. I'm not sure what the AI would do for such a tool, but I'm sure that the companies making such a thing are busy working hard to incorporate some form of AI into it. [1] What would the AI in such a tool do?

nextaccountic 3y ago
> [1] What would the AI in such a tool do? Solving captchas. In particular, not only solving the challenge itself (recognizing crosswalks, etc) but also handling input (moving the mouse, alt+tabbing, etc) as if it were a real human. I was recently reading the Cloudflare turnstile announcement https://blog.cloudflare.com/turnstile-ga https://blog.cloudflare.com/turnstile-ga and it has the following links about this https://uncaptcha.cs.umd.edu/papers/uncaptcha_woot17.pdf https://uncaptcha.cs.umd.edu/papers/uncaptcha_woot17.pdf https://www.vox.com/22436832/captchas-getting-harder-ai-artificial-intelligence https://www.vox.com/22436832/captchas-getting-harder-ai-arti... https://arxiv.org/abs/1903.01003 https://arxiv.org/abs/1903.01003 In those excerpts: > The prevalence of free speech-to-text services has made it easy for bots to solve audio CAPTCHAs as well, with a recent study showing bots can accurately solve audio CAPTCHAs in over 85% of attempts. We’re proud to state that Turnstile is WCAG 2.1 Level AA compliant, while eliminating the need for audio CAPTCHAs as well as visual ones. And > With all of our emphasis on how easy it is to pass a Turnstile challenge, you would be right to ask how it can stop a bot. If a bot can find all images with crosswalks in grainy photos faster than we can, surely it can check a box as well. Bots definitely can check a box, and they can even mimic the erratic path of human mouse movement while doing so. For Turnstile, the actual act of checking a box isn’t important, it’s the background data we’re analyzing while the box is checked that matters. We find and stop bots by running a series of in-browser tests, checking browser characteristics, native browser APIs, and asking the browser to pass lightweight tests (ex: proof-of-work tests, proof-of-space tests) to prove that it’s an actual browser. The current deployment of Turnstile checks billions of visitors every day, and we are able to identify browser abnormalities that bots exhibit while attempting to pass those tests. So, this is a cat and mouse game, the cat is using AI, the mouse is probably using AI in some form or another too.
- eddd-ddde 3y ago
  We really are gonna end up with servers and clients evolving AI challenges and solvers respectively to the point where humans won't be able to solve them anymore haha
jakozaur 3y ago
The brainer price would be $20 / month. $50 / month if it was terrific. The trick has to be really useful without too much work on my end. Combine screen scraping and simulating real users with some mix of AI. My use cases: 1. Fill any form with intelligent data (from orders, through plane tickets to tax forms). Think of ChatGPT but for forms. 2. Research any person and company using various tools (e.g. LinkedIn, Google, job boards). Do it at human speed so I won't get blocked and can observe it on a screen. I can provide a script on how to do it (e.g. check LinkedIn, this board for that, etc.) Provide me with a summary of the text. 3. I want to be able to script it and hack it to whatever I need. API and CLI.

kordlessagain 3y ago

I just finished up a Grub crawler for https://mitta.ai https://mitta.ai. The result is that you can screenshot a page and then run it through OCR, then push the text into an LLM, or embed chunks of it: https://www.youtube.com/watch?v=QadlImAKzCE https://www.youtube.com/watch?v=QadlImAKzCE While it won't do clicking, the system also has a processor that can fetch the HTML (or other files). It would be an afternoon's work to add something like CurlyQ to it. I was kicking around adding an ffmpeg processor to it...

dbish 3y ago

We’re working on a tool that “screen shares” with AI and allows both looking at what is on screen when you’re sharing and acting on it (simulate clicks and other). What’s your use case? Happy to share more about where we’re at with the project. It’s quite a lot to build to get right for many kinds of apps and to be multi-os but we’re pretty close. No CLI though, would love to hear more about your use case there. Feel free to email me diamond@augmend.com

jakozaur 3y ago
My use cases: 1. Fill any form with intelligent data (from orders, through plane tickets to tax forms). Think of ChatGPT but for forms. 2. Research any person and company using various tools (e.g. LinkedIn, Google, job boards). Do it at human speed so I won't get blocked and can observe it on a screen. Provide me with a summary of text and list. 3. I want to be be able to script it and hack to whatever I need. API and CLI.