4 ms·

ChatGPT (and its future offspring) can navigate regular websites the same way humans do It seems to be at least a not-yet-true claim, but let’s ignore that for

by wruza 3y ago

ChatGPT (and its future offspring) can navigate regular websites the same way humans do It seems to be at least a not-yet-true claim, but let’s ignore that for now. It’s interesting if LLMs actually could do this. As I understand it, LLMs are trained on texts and source code among other things. But lacking… let’s name it a reasoning apparatus, can they really look at a DOM tree and tell what it is/does? It’s not a text, and barely can be a well-structured source code that was ever discussed (it’s a result of either bundling or componentization). This is almost on par with “LLMs will look at any .exe and be able to integrate with it immediately”.

delusional 3y ago
> “LLMs will look at any .exe and be able to integrate with it immediately”. The LLM will look at any .exe and determine if it halts.
- KronisLV 3y ago
  > The LLM will look at any .exe and determine if it halts. At that point you'd need an AGI that can figure out something we can't. Edit: but yeah, sarcasm. Nowadays I can't even tell sometimes.
  - wruza 3y ago
    I believe it was a positive sarcasm in gp.
  - kaba0 3y ago
    I am 100% sure that parent poster was being sarcastic at the hype around LLMs and how many think they can solve everything, even if said thing is impossible to do, e.g. the halting problem. Do note that the halting problem is fundamentally true, no AGI will realize some new way around, unless are mathematics are flawed to the core.
    - mythhabit 3y ago
      There is some nuances here. While the general halting problem for a general Turing machine is undecidable, and with a fairly easy to understand proof as well, the computers we run today are not a general Turing machine. They are of a weaker class called Linear Bounded Automatons and for the programs they can run, the halting problem is fact decidable, on a theoretical level due to their finite nature. So we will probably never practically solve the halting for LBAs, but the quest for this AGI that should be able to solve this, is not just day dreaming, it's rooted in the theory.
      - kaba0 3y ago
        I have to admit my knowledge of complexity theory doesn’t extend too far, but isn’t the solution to LBAs just.. brute forcing? Also, is it even decidable a priori whether a program is LBA vs requiring a tape that is not only linear function of its input?
        vidarh 3y ago
        The worst case solution is just brute forcing, yes. An LBA is effectively "just" a Turing machine that has a finite tape. A typical current computer is an LBA only if you disallow all IO of any sort or bound that IO and include it as part of the system you analyse and so fix the values which will be provided as IO, which of course is a very unusual situation, and so that constraint does not really make the halting problem more tractable in situations we usually care about.
        mythhabit 3y ago
        The naive way of deciding it it halts is executing it and keep track of all state between steps. If the machine halts, you're done, if it repeats a state, it's looping. If you can represent a program with an LBA, it is by definition a context sensitive language and decidable. You could also show it by making a equivalent gramma for the language, that accepts the same input as the program. This gramma must be constructed in a certain way, and then you know it is a context sensitive language.
      - 3y ago
  - reichstein 3y ago
    I think KronisLV's point, presented ironically, is that not even an AI superior to human reasoning, which can figure out things we cannot, can decide the undecidable. Being undecidable is a hard limit, not something that requires better algorithms. Another phrasing of it could be: No finite program can decide it. (And what even is an infinite program? Not something we can run on any current computer.) If an AI is itself a finite-sized program, something we run on computers, it cannot possibly solve the halting problem. And _any_ non-trivial property of programs is undecidable, so an AI "integrating with any .exe file" isn't really meaningful. It's just words.
    - vidarh 3y ago
      > Another phrasing of it could be: No finite program can decide it. (And what even is an infinite program? Not something we can run on any current computer.) The complexity of the program trying to decide is not really the issue. An "infinite program" could add an infinite number of extra rules to try to cut down on the time taken to determine if a decidable program halts, but the issue with the halting problem is that there is an infinite set of undecidable programs where the size of your detector will make no difference to your ability to decide them. E.g. "while next_symbol() == some_arbitrary_symbol {}" is undecidable unless you add constraints on the length or contents of the input. Even if you haven infinite-sized program you can't decide whether or not the unconstrained version of that halts, because deciding it is equivalent to being able to determine if an infinite tape contains a given symbol, and no matter how long you scan the tape the symbol can always be the next one on the tape after the last symbol you scan. > And _any_ non-trivial property of programs is undecidable I don't think I agree with this without adding the qualifier "in general". There are a whole lot of useful properties we can decide, but often the properties will have constraints. E.g. for a whole lot of programs where we can't decide whether or not they will halt, we can still decide whether or not they will halt assuming certain properties of their inputs. E.g. we can decide the property of my pseudo-code above that it will halt IF "some_arbitrary_symbol" is in the input. We can also decide the property that whether or not it will halt in general is undecidable, and that is itself useful to know, because for many programs knowing what makes them undecidable is useful in order to suggest e.g. adding timeouts, or ensuring there are ways to bail early from certain actions without restarting the machine or killing the program. For a whole lot of problems we also do not really care whether or not a given property is decidable. We care whether they're decidable often enough within certain time constraints, and that's a very different ballgame.
- abwizz 3y ago
  i suspect this is because they can divide by zero
- chromoblob 3y ago
  .exe's (for the usual architectures) run on finite-state machines, not Turing machines. So, indeed, that is possible. If the .exe connects to a service that runs on something more powerful than finite-state machine, though, I don't know.
  - kaba0 3y ago
    That’s still a stupid pedantry, and is even false. A Turing machine only makes use of at most n cells of its tape after n steps - so running it for a finite number of steps is possible even in finite memory. Especially that modern computers can do arbitrary side effects, having access to the whole universe as tape, which is still finite, but so is time. There is no way to differentiate between a magical Turing machine with infinite tape and a “fake” one that has n-sized memory under any program that takes n steps, so for all practical purposes they are identical.
    - chromoblob 3y ago
      Why are they identical? A program may use more steps than memory.
      - kaba0 3y ago
        A Turing machine can either go left or right on its tape (some versions have a stay step as well, doesn’t matter). If all your program does is step right forever, than it will use the maximum amount of memory, but only ever a finite amount, that is equal to the number of steps taken. So if you don’t have infinite time (you don’t have), and you have big enough memory for the particular use case so that you don’t get OOMKiller involved, then you have a Turing machine for all complexity theoretical and practical purposes, especially that RAM is not analogous to the Turing machine tape - your computer has much more state than only its memory, if it has network access, you can basically make use of a cloud vendors whole army of servers as storage, just as an example.
        deleted 3y ago
        [deleted]
kevinstubbs 3y ago
> can they really look at a DOM tree and tell what it is/does Yes, if you encode the DOM as a list of options for ChatGPT to choose from. In fact I developed a proof of concept of this for a client. https://jarvys.ai/ https://jarvys.ai/ although they seem to have pivoted from automating just the browser to automating all software.
- wruza 3y ago
  if you encode the DOM as a list of options for ChatGPT to choose from Not sure if I understand this, does it mean you have to pre-cook DOM in a specific way? If yes, then isn’t the answer to my question “no”, like “no, it can’t take any site and use it as is”?
  - kevinstubbs 3y ago
    You have to give GPT an objective, like "find an apartment in Florida" and then say something like "given the following options, which one would you interact with to get closer to your objective." So if you assume that you start on google.com, then your options are like 1.) Input with name "search", placeholder "search anything", value "" 2.) Button with label "I'm feeling lucky" 3.) Button with label "search" Obviously, doing just one of these doesn't achieve the objective - it just needs to pick which one it thinks has the most "value" for completing the objective. If you repeat that enough times, then it can actually do what your overall goal of the session was. I'm just giving a simplistic answer, and if you implemented only what I've written, then it's going to get stuck in a loop more often than not. But that's the gist of how you could encode the DOM into something that GPT can interpret and make decisions/take actions based on.
    - wruza 3y ago
      Got it, thanks!
    - TeMPOraL 3y ago
      Remember HATEOAS? I have a feeling LLMs would excel at navigating proper REST (not "RESTful") APIs - HATEOAS is, in principle, just what you did here: providing a list of possible/useful next steps along with the response. In fact, the problem of HATEOAS is exactly what LLMs seem to be good at - inferring the interface at runtime, from dynamically received metadata. This should even be easy to try in practice today - HATEOAS can be trivially mapped to the "function calling" feature of OpenAI's GPT-3.5/GPT-4 APIs.
3cats-in-a-coat 3y ago
LLM would find it easier to browse through a browser. Like us. Implying image model linked.
tyingq 3y ago
"ChatGPT (and its future offspring) can navigate regular websites the same way humans do" I don't know how much deeper it goes, but it does have some context: Prompt: Given this html, what does an end user see? <div style="display:none">hello</div><div>you</div> ChatGPT: An end user would see only the text "you" on the webpage. The first <div> element has the inline style display:none, which means it is set to be hidden, and its content "hello" will not be visible to the user. On the other hand, the second <div> element has no specific display style, so it will be visible, and its content "you" will be displayed on the webpage.
datagram 3y ago
There's already companies working on this: https://axiom.ai/ https://axiom.ai/ I would guess that it's just a matter of converting the DOM accessibility tree into text descriptions, e.g. "There is a button that says 'Start'" And then converting text like "Click the Start button" back into an actual action on the page.

by wruza 3y ago

delusional 3y ago

> “LLMs will look at any .exe and be able to integrate with it immediately”. The LLM will look at any .exe and determine if it halts.

KronisLV 3y ago
> The LLM will look at any .exe and determine if it halts. At that point you'd need an AGI that can figure out something we can't. Edit: but yeah, sarcasm. Nowadays I can't even tell sometimes.
- wruza 3y ago
  I believe it was a positive sarcasm in gp.
- kaba0 3y ago
  I am 100% sure that parent poster was being sarcastic at the hype around LLMs and how many think they can solve everything, even if said thing is impossible to do, e.g. the halting problem. Do note that the halting problem is fundamentally true, no AGI will realize some new way around, unless are mathematics are flawed to the core.
  - mythhabit 3y ago
    There is some nuances here. While the general halting problem for a general Turing machine is undecidable, and with a fairly easy to understand proof as well, the computers we run today are not a general Turing machine. They are of a weaker class called Linear Bounded Automatons and for the programs they can run, the halting problem is fact decidable, on a theoretical level due to their finite nature. So we will probably never practically solve the halting for LBAs, but the quest for this AGI that should be able to solve this, is not just day dreaming, it's rooted in the theory.
    - kaba0 3y ago
      I have to admit my knowledge of complexity theory doesn’t extend too far, but isn’t the solution to LBAs just.. brute forcing? Also, is it even decidable a priori whether a program is LBA vs requiring a tape that is not only linear function of its input?
      - vidarh 3y ago
        The worst case solution is just brute forcing, yes. An LBA is effectively "just" a Turing machine that has a finite tape. A typical current computer is an LBA only if you disallow all IO of any sort or bound that IO and include it as part of the system you analyse and so fix the values which will be provided as IO, which of course is a very unusual situation, and so that constraint does not really make the halting problem more tractable in situations we usually care about.
      - mythhabit 3y ago
        The naive way of deciding it it halts is executing it and keep track of all state between steps. If the machine halts, you're done, if it repeats a state, it's looping. If you can represent a program with an LBA, it is by definition a context sensitive language and decidable. You could also show it by making a equivalent gramma for the language, that accepts the same input as the program. This gramma must be constructed in a certain way, and then you know it is a context sensitive language.
    - 3y ago
- reichstein 3y ago
  I think KronisLV's point, presented ironically, is that not even an AI superior to human reasoning, which can figure out things we cannot, can decide the undecidable. Being undecidable is a hard limit, not something that requires better algorithms. Another phrasing of it could be: No finite program can decide it. (And what even is an infinite program? Not something we can run on any current computer.) If an AI is itself a finite-sized program, something we run on computers, it cannot possibly solve the halting problem. And _any_ non-trivial property of programs is undecidable, so an AI "integrating with any .exe file" isn't really meaningful. It's just words.
  - vidarh 3y ago
    > Another phrasing of it could be: No finite program can decide it. (And what even is an infinite program? Not something we can run on any current computer.) The complexity of the program trying to decide is not really the issue. An "infinite program" could add an infinite number of extra rules to try to cut down on the time taken to determine if a decidable program halts, but the issue with the halting problem is that there is an infinite set of undecidable programs where the size of your detector will make no difference to your ability to decide them. E.g. "while next_symbol() == some_arbitrary_symbol {}" is undecidable unless you add constraints on the length or contents of the input. Even if you haven infinite-sized program you can't decide whether or not the unconstrained version of that halts, because deciding it is equivalent to being able to determine if an infinite tape contains a given symbol, and no matter how long you scan the tape the symbol can always be the next one on the tape after the last symbol you scan. > And _any_ non-trivial property of programs is undecidable I don't think I agree with this without adding the qualifier "in general". There are a whole lot of useful properties we can decide, but often the properties will have constraints. E.g. for a whole lot of programs where we can't decide whether or not they will halt, we can still decide whether or not they will halt assuming certain properties of their inputs. E.g. we can decide the property of my pseudo-code above that it will halt IF "some_arbitrary_symbol" is in the input. We can also decide the property that whether or not it will halt in general is undecidable, and that is itself useful to know, because for many programs knowing what makes them undecidable is useful in order to suggest e.g. adding timeouts, or ensuring there are ways to bail early from certain actions without restarting the machine or killing the program. For a whole lot of problems we also do not really care whether or not a given property is decidable. We care whether they're decidable often enough within certain time constraints, and that's a very different ballgame.
abwizz 3y ago
i suspect this is because they can divide by zero
chromoblob 3y ago
.exe's (for the usual architectures) run on finite-state machines, not Turing machines. So, indeed, that is possible. If the .exe connects to a service that runs on something more powerful than finite-state machine, though, I don't know.
- kaba0 3y ago
  That’s still a stupid pedantry, and is even false. A Turing machine only makes use of at most n cells of its tape after n steps - so running it for a finite number of steps is possible even in finite memory. Especially that modern computers can do arbitrary side effects, having access to the whole universe as tape, which is still finite, but so is time. There is no way to differentiate between a magical Turing machine with infinite tape and a “fake” one that has n-sized memory under any program that takes n steps, so for all practical purposes they are identical.
  - chromoblob 3y ago
    Why are they identical? A program may use more steps than memory.
    - kaba0 3y ago
      A Turing machine can either go left or right on its tape (some versions have a stay step as well, doesn’t matter). If all your program does is step right forever, than it will use the maximum amount of memory, but only ever a finite amount, that is equal to the number of steps taken. So if you don’t have infinite time (you don’t have), and you have big enough memory for the particular use case so that you don’t get OOMKiller involved, then you have a Turing machine for all complexity theoretical and practical purposes, especially that RAM is not analogous to the Turing machine tape - your computer has much more state than only its memory, if it has network access, you can basically make use of a cloud vendors whole army of servers as storage, just as an example.
      - deleted 3y ago
        [deleted]

kevinstubbs 3y ago

> can they really look at a DOM tree and tell what it is/does Yes, if you encode the DOM as a list of options for ChatGPT to choose from. In fact I developed a proof of concept of this for a client. https://jarvys.ai/ https://jarvys.ai/ although they seem to have pivoted from automating just the browser to automating all software.

wruza 3y ago
if you encode the DOM as a list of options for ChatGPT to choose from Not sure if I understand this, does it mean you have to pre-cook DOM in a specific way? If yes, then isn’t the answer to my question “no”, like “no, it can’t take any site and use it as is”?
- kevinstubbs 3y ago
  You have to give GPT an objective, like "find an apartment in Florida" and then say something like "given the following options, which one would you interact with to get closer to your objective." So if you assume that you start on google.com, then your options are like 1.) Input with name "search", placeholder "search anything", value "" 2.) Button with label "I'm feeling lucky" 3.) Button with label "search" Obviously, doing just one of these doesn't achieve the objective - it just needs to pick which one it thinks has the most "value" for completing the objective. If you repeat that enough times, then it can actually do what your overall goal of the session was. I'm just giving a simplistic answer, and if you implemented only what I've written, then it's going to get stuck in a loop more often than not. But that's the gist of how you could encode the DOM into something that GPT can interpret and make decisions/take actions based on.
  - wruza 3y ago
    Got it, thanks!
  - TeMPOraL 3y ago
    Remember HATEOAS? I have a feeling LLMs would excel at navigating proper REST (not "RESTful") APIs - HATEOAS is, in principle, just what you did here: providing a list of possible/useful next steps along with the response. In fact, the problem of HATEOAS is exactly what LLMs seem to be good at - inferring the interface at runtime, from dynamically received metadata. This should even be easy to try in practice today - HATEOAS can be trivially mapped to the "function calling" feature of OpenAI's GPT-3.5/GPT-4 APIs.

3cats-in-a-coat 3y ago

LLM would find it easier to browse through a browser. Like us. Implying image model linked.

tyingq 3y ago

"ChatGPT (and its future offspring) can navigate regular websites the same way humans do" I don't know how much deeper it goes, but it does have some context: Prompt: Given this html, what does an end user see? <div style="display:none">hello</div><div>you</div> ChatGPT: An end user would see only the text "you" on the webpage. The first <div> element has the inline style display:none, which means it is set to be hidden, and its content "hello" will not be visible to the user. On the other hand, the second <div> element has no specific display style, so it will be visible, and its content "you" will be displayed on the webpage.

datagram 3y ago

There's already companies working on this: https://axiom.ai/ https://axiom.ai/ I would guess that it's just a matter of converting the DOM accessibility tree into text descriptions, e.g. "There is a button that says 'Start'" And then converting text like "Click the Start button" back into an actual action on the page.