6 ms·

The examples in the paper are pretty impressive. There is an example of a windows 11 dialog image. The computer can figure out which button to press given the d

The examples in the paper are pretty impressive. There is an example of a windows 11 dialog image. The computer can figure out which button to press given the desired outcome of the user. If one where to take this model and scale it, I can see an advanced bot in <5 years navigating the web and doing work based on a text input of a human purely by visual means. Interesting times.

IanCal 4y ago
You might be interested in this: https://www.adept.ai/blog/act-1 https://www.adept.ai/blog/act-1
- freakynit 4y ago
  Holy tckin' cow!!! Is this real? Captchas be damned now. Beating AI with AI. What a time to be alive.
- moritonal 4y ago
  We can't stop it, but giving an AI unbridled access to the Internet is a terrible idea. Whether it's a misphrased question or an clever prompt hack; entire sites will be crushed by the sheer superhuman performance of it. Hackernews will be just robots chatting to each other nudging towards the latest product-hunt.
  - flangola7 4y ago
    https://www.youtube.com/watch?v=efPrtcLdcdM https://www.youtube.com/watch?v=efPrtcLdcdM The internet will be saturated with fake people soon. Someone already did a PoC of this on 4chan as a joke with just a small GPT2 model. In a few years you won't be able to tell if you're talking to a human unless they're physically in front of you.
    - deleted 4y ago
      [deleted]
  - comboy 4y ago
    We will develop web of trust [1]. I assign weighted(!) trust to my friends, they assign to their friends. Small world [2]. This also fixes fake reviews, dependencies security, politics and whole lot of other things. The idea is too good not to happen. I repeat it from time to time on HN. I'm currently not in the position to implement it (not sure if I ever will be, it is hard). I just hope it's created in some decentralized form before some corpo does it. When controlled by single entity it is useless. 1. http://comboy.pl/wot.html http://comboy.pl/wot.html (not really worth your time, long time ago too verbose, but feel free) 2. https://en.wikipedia.org/wiki/Six_degrees_of_separation https://en.wikipedia.org/wiki/Six_degrees_of_separation
    - furyofantares 4y ago
      I've idly and very-casually thought about this for a long time, ever since decentralized filesharing really. If you ever do feel that you're in a position to take a stab at it, I'd be interested as well.
    - sangnoir 4y ago
      > This also fixes fake reviews, dependencies security, politics and whole lot of other things. All you have to do is lookup some batshit-crazy things people in your social circle already share on Facebook (or LinkedIn) to know this won't solve most of those problems. I may trust my friend to thoroughly vet information on Disc Golf, but they may be out of their element when it comes to "Revolutionary cold fusion breakthrough", which I may get through them if there's a single generic weight for trust.
      - comboy 4y ago
        Good observation. I thought at first that it should be multidimensional, but categories are hard. Very hard. I think general trust could work. People I trust won't give strong opinions about thing they have no idea about. You choose these people and you assign weights. It's not a random family circle. I have some vague ideas about wallet of personalities which would serve the same purpose as categories of trust.
reset-password 4y ago
I've been following tech long enough to know that as soon as the computer can figure out which button to press it's only going to click on ads, I guarantee it.
- joshspankit 4y ago
  and then that’s going to be met with MS making it “impossible” for bots to automate clicking on ads which will have the unintended consequence of making it harder to use for power users.
- sebzim4500 4y ago
  Isn't it trivial to make a computer click on ads though? Just run selenium, apply the filtering rules from adblock and then click on a random element which would be blocked.
  - 13of40 4y ago
    I think their point is the opposite - it's *not* trivial to make the computer click the correct "Download Now!" button to get Minecraft versus the other 4 that lead to malware.
    - sebzim4500 4y ago
      Isn't it? Just run it on a machine with ublock origin.
- duringwork12 4y ago
  That explains googles Ux understanding AI. It can tell in words what the next step on a form is.
ren_engineer 4y ago
this seems like the future to me, a huge chunk of work will be able to be done by just talking to your computer and then automating the task. Society is really going to need to adapt, knowledge workers being replaced will be as big a change as the industrial revolution replacing many manual laborers what's interesting is how will these systems be maintained when all the junior tier engineering work is replaced by AI? Companies don't like hiring junior engineers now, will be an even bigger gap before a junior engineer becomes net productive now. Plus people building stuff using AI without understanding how things work under the hood. Seems ripe for some 40K tier situation where we have tech priests running systems that nobody knows how to build from scratch anymore
- novaRom 4y ago
  Talking to your computer? This will not last too long until another disruptive change will happen. How about computers will overtake those decisions you think you freely able to do right now. It is all running fast, accelerating actually.
ivanvanderbyl 4y ago
I worked at a company a few years back that used YoLO trained on billions of UI screenshots to navigate the UI of any desktop application based on plain English instructions. It already exists.

IanCal 4y ago

You might be interested in this: https://www.adept.ai/blog/act-1 https://www.adept.ai/blog/act-1

freakynit 4y ago
Holy tckin' cow!!! Is this real? Captchas be damned now. Beating AI with AI. What a time to be alive.
moritonal 4y ago
We can't stop it, but giving an AI unbridled access to the Internet is a terrible idea. Whether it's a misphrased question or an clever prompt hack; entire sites will be crushed by the sheer superhuman performance of it. Hackernews will be just robots chatting to each other nudging towards the latest product-hunt.
- flangola7 4y ago
  https://www.youtube.com/watch?v=efPrtcLdcdM https://www.youtube.com/watch?v=efPrtcLdcdM The internet will be saturated with fake people soon. Someone already did a PoC of this on 4chan as a joke with just a small GPT2 model. In a few years you won't be able to tell if you're talking to a human unless they're physically in front of you.
  - deleted 4y ago
    [deleted]
- comboy 4y ago
  We will develop web of trust [1]. I assign weighted(!) trust to my friends, they assign to their friends. Small world [2]. This also fixes fake reviews, dependencies security, politics and whole lot of other things. The idea is too good not to happen. I repeat it from time to time on HN. I'm currently not in the position to implement it (not sure if I ever will be, it is hard). I just hope it's created in some decentralized form before some corpo does it. When controlled by single entity it is useless. 1. http://comboy.pl/wot.html http://comboy.pl/wot.html (not really worth your time, long time ago too verbose, but feel free) 2. https://en.wikipedia.org/wiki/Six_degrees_of_separation https://en.wikipedia.org/wiki/Six_degrees_of_separation
  - furyofantares 4y ago
    I've idly and very-casually thought about this for a long time, ever since decentralized filesharing really. If you ever do feel that you're in a position to take a stab at it, I'd be interested as well.
  - sangnoir 4y ago
    > This also fixes fake reviews, dependencies security, politics and whole lot of other things. All you have to do is lookup some batshit-crazy things people in your social circle already share on Facebook (or LinkedIn) to know this won't solve most of those problems. I may trust my friend to thoroughly vet information on Disc Golf, but they may be out of their element when it comes to "Revolutionary cold fusion breakthrough", which I may get through them if there's a single generic weight for trust.
    - comboy 4y ago
      Good observation. I thought at first that it should be multidimensional, but categories are hard. Very hard. I think general trust could work. People I trust won't give strong opinions about thing they have no idea about. You choose these people and you assign weights. It's not a random family circle. I have some vague ideas about wallet of personalities which would serve the same purpose as categories of trust.

reset-password 4y ago

I've been following tech long enough to know that as soon as the computer can figure out which button to press it's only going to click on ads, I guarantee it.

joshspankit 4y ago
and then that’s going to be met with MS making it “impossible” for bots to automate clicking on ads which will have the unintended consequence of making it harder to use for power users.
sebzim4500 4y ago
Isn't it trivial to make a computer click on ads though? Just run selenium, apply the filtering rules from adblock and then click on a random element which would be blocked.
- 13of40 4y ago
  I think their point is the opposite - it's *not* trivial to make the computer click the correct "Download Now!" button to get Minecraft versus the other 4 that lead to malware.
  - sebzim4500 4y ago
    Isn't it? Just run it on a machine with ublock origin.
duringwork12 4y ago
That explains googles Ux understanding AI. It can tell in words what the next step on a form is.

ren_engineer 4y ago

this seems like the future to me, a huge chunk of work will be able to be done by just talking to your computer and then automating the task. Society is really going to need to adapt, knowledge workers being replaced will be as big a change as the industrial revolution replacing many manual laborers what's interesting is how will these systems be maintained when all the junior tier engineering work is replaced by AI? Companies don't like hiring junior engineers now, will be an even bigger gap before a junior engineer becomes net productive now. Plus people building stuff using AI without understanding how things work under the hood. Seems ripe for some 40K tier situation where we have tech priests running systems that nobody knows how to build from scratch anymore

novaRom 4y ago
Talking to your computer? This will not last too long until another disruptive change will happen. How about computers will overtake those decisions you think you freely able to do right now. It is all running fast, accelerating actually.

ivanvanderbyl 4y ago

I worked at a company a few years back that used YoLO trained on billions of UI screenshots to navigate the UI of any desktop application based on plain English instructions. It already exists.