5 ms·

Building a ChatGPT-enhanced Python REPL

103 points by synergy20 4y ago

asah 4y ago
Cool! Neptyne spreadsheet has a GPT-enhanced python repl as well: https://neptyne.com/ https://neptyne.com/
wpietri 4y ago
I am a big casual user of anthropomorphic language for what computers are up to. I do it not because it's true, but because it's more engaging to say that "the computer is confused about X and so it's mad". But I'm wondering if that's dangerous when it comes to things that produce pseudo-human output. E.g.: > I think that the LLM thinks that it cannot generate code for the simple task. Unlike every other computer API in existence we can prompt the LLM to tell us why it responded in the way it does. I agree the LLM is going to produce text that is going to be narratively consistent with the assumptions of the question. But we already know LLMs are giant bullshitters. Is there any reason to think they are doing actual introspection and description of internal state? Or are they just going to give plausible-sounding words?
- isthisit 4y ago
  Author here. Fully agree. > I agree the LLM is going to produce text that is going to be narratively consistent with the assumptions of the question. But we already know LLMs are giant bullshitters. Is there any reason to think they are doing actual introspection and description of internal state? Or are they just going to give plausible-sounding words? This sounds like something we could test by inverting the LLM-provided reason of why it can/cannot do something and playing that back through the model. The example I gave in the blog was, unprompted, the LLM thought it should not generate code that it thought had already been executed. If we take it at face value, I could add a line in the code generation prompt to instruct it to generate code even if it "thinks" it has already executed it. We then assert the outcome. Not very scientific, but might give us some insight when trialed across thousands of prompts.
- vba616 4y ago
  >Is there any reason to think they are doing actual introspection and description of internal state? Or are they just going to give plausible-sounding words? Someone needs to call out the double standard applied to AI. If machines that bullshit are philosophically indistinguishable from intelligent humans, how is it that humans that bullshit are distinguishable from intelligent humans?
  - wpietri 4y ago
    Sorry, I'm not following your point.
    - pulvinar 4y ago
      From GTP-4 (or maybe this is bullshitting too?): I believe vba616 is trying to point out that if we consider AI systems that generate plausible-sounding responses to be "bullshitting," then we should also acknowledge that humans who do the same are not necessarily distinguishable from "intelligent" humans. In other words, both AI systems and humans can produce seemingly coherent responses without necessarily having a deep understanding of the subject matter. It raises the question of whether we should hold AI systems to the same standard as humans when it comes to evaluating their output or understanding. If we accept that humans can sometimes provide plausible yet shallow answers, should we not also accept that AI systems, like LLMs, can do the same?
      - wpietri 4y ago
        Sorry, I do not give LLM-generated copy-pastes any attention. If you personally have something to say here, I'll read it with interest. But I have no interest in automated bullshit.
        nullsense 4y ago
        As an AI language model, I fully agree.
        pulvinar 4y ago
- famouswaffles 4y ago
  humans don't always give faithful reasoning either just so you know. even removing any willful obfuscation, we simply don't know all the biases that can influence a certain action. even more than that, Your brain will happily make up a reason for an action. This is demonstrated in the split brain experiments. Knowing that, if what it says in response to a given problem can meaningfully change output for the better following its directive later then what else ? might as well go with it
  - wpietri 4y ago
    You could make exactly the same argument for consulting psychics, astrologers, and tarot readers. Indeed, I think what a lot of people are doing with LLM output is the same thing customers of paranormal scams do. Faced with artfully constructed ambiguity, they fill in the blanks themselves, not even noticing how much they're supplying the work. Some bit go well and some poorly; the successes are proof that the psychic has paranormal powers, and the rest are quickly forgotten about.
    - famouswaffles 4y ago
      The point is that a distinction without a difference is a meaningless distinction. Shouting "This is not real [insert property]" is beyond meaningless. Either this supposed difference is important enough to manifest in reality/output or it functionally doesn't exist. If rationalizations work as well as human ones do then treat them as you would human ones.
      - wpietri 4y ago
        Yes, ergo I believe there are differences. But you didn't address my point. Do you believe that explanations produced via tarot or zodiac should be treated with that same level of deference? That explanations from ouija boards are things we "might as well go with"?
        famouswaffles 4y ago
        GPT-4 doesn't not perform at the same level on tasks as a ouija board. The whole point is that the level of deference is based on the level of performance it exhibits on whatever task you're having it do.
- dragonwriter 4y ago
  > If machines that bullshit are philosophically indistinguishable from intelligent humans, how is it that humans that bullshit are distinguishable from intelligent humans? Well, if we use the ReAct model, the flow of action involves the LLM narrating its reasoning and then acting on it. Which can still be bullshit, of course, but its at least a different thing than a post hoc rationalization when questioned. (Of course, we know that when questioned about past actions, humans often construct post hoc rationalizations, too.)
- drvdevd 4y ago
  Perhaps this is one of the greatest paradigm shifts for traditional software engineers as “AI” becomes more commonly integrated: maybe we will have to use anthropomorphic language and maybe it will be considered the most technically correct way of talking about these systems due not only to their non-deterministic nature but also their emergent properties. (This was a great post btw - highly recommend if you’re just perusing the comments)
  - usrbinbash 4y ago
    Meeeeeehhhh...I don't think that is likely to happen any time soon. Could be wrong, but I don't think it's likely. The reason being: All software engineers are trained to read and write code, aka. precise formal languages. All feedback from systems about errors in that are also precise and formal languages. What will likely happen, is that we will use anthropomorphic language more and more to generate code, especially code that is easy to describe but just a boring hassle to type (aka. boilerplate or things like simple unit tests).
- bheadmaster 4y ago
  Is there any reason to think humans are doing actual introspection and description of internal state? Or are they just giving plausible-sounding words?
  - spherebroken 4y ago
    People with separated brain hemispheres, for example sight and speak capabilities, try to make plausible reasoning. There are some well known physiological experiments to prove this. This proof that our brain try to provide a plausible explanation that has nothing to do with what is happening in reality. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/ (1)
  - spherebroken 4y ago
    People with brain-split separated brain hemispheres, for example sight and speak capabilities, try to make plausible reasoning that are not grounded. Here is an example From (1) about brain-splitted subject called Joe: In a particularly dramatic recorded demonstration, the famous patient “Joe” was able to draw a cowboy hat with his left hand in response to the word “Texas” presented in his left visual half field. His commentary (produced by the verbal left hemisphere) showed a complete absence of insight into why his left hand had drawn this cowboy hat. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/ (1)
- simonw 4y ago
  Asking LLMs why they made a specific decision is a very common misconception. The state that lead to their previous generation was GBs of floating point calculations which were discarded as soon as the output had been generated. If you ask them why they said something they will generate a brand new explanation based on the previous text they generated, but any relationship to their actual decision process will be entirely coincidental. Update: I just noticed that the article uses this: > If you cannot return executable python code return set the reason why in the description and return no code. That DOES work - getting it to generate justification during a single response isn't affected by the lost state issue I'm describing here.
  - heisenzombie 4y ago
    People also can’t really recreate an earlier mental state, and have to resort to post-hoc rationalisation. E.g. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ I feel like the answer to the question “give me a possible reason that a person/llm would give this answer given this situation” is all that’s ever really possible!
  - IanCal 4y ago
    Hang on I think you're assuming there's some internal state that I don't think exists. Each time it just generates one more token until it hits a stop token, using the previous input. It shouldn't matter if it created a token or you did.
  - OrderlyTiamat 4y ago
    You are right on fhe first point, I believe the second needs to be corrected. LLMs have no state. Each word- each token- is made anew, the network being oblivious to the process that led to the previous token. It cannot even distinguish between text it generated and text you added unless you tell it that. So any relationship between output and explanation is 'coincidental'. Actually, any relationship between two tokens next to each other is 'coincidental' in the exact same way, so it cannot explain its reasoning. If you prompt it to comply or explain why not, the network may generate an explanation of why it can't comply, but it is doing that simply as a good continuation of the input text. I'm not saying you can't call that an explanation if you want to use that word, I'm trying to make a distinction between the generated output and an explanation of that output. All of this is the former. The latter is not possible unless you analyse the model as it is running, and we don't really have a good understanding of that process yet. There was a cool paper where researchers analysed a finetuned model playing othello, and they found some neurons that clearly represented the game state- if they changed those neuron values, the network responded correctly, even if that game position was impossible to get to normally [1] I'd argue work like that is getting us closer to getting explanations for llm behaviour. [1] https://www.joyk.com/dig/detail/1674380676520322 https://www.joyk.com/dig/detail/1674380676520322
- lynx23 4y ago
  I am pretty convinced a large majority of humans don't do anything differently, they also just generate "plausible-sounding words".
westurner 4y ago
"Show HN: IPython-GPT, a Jupyter/IPython Interface to Chat GPT" https://news.ycombinator.com/item?id=35580959 https://news.ycombinator.com/item?id=35580959 lists a few REPLs
alfons_foobar 4y ago
> As the technology evolves I see there being a greater demand for deterministic responses from LLMs. I think we will see demand for it, but I also think it completely misses the nature of LLMs / for what they are most useful. Basically trying to shoehorn "old" patterns into a new paradigm.
kgeist 4y ago
What if LLM hallucinates some destructive operation, such as deleting local files, and runs it? I think the project should automatically run everything in a container, just in case.
- 9dev 4y ago
  Yeah, that's a shared problem among these "let an algorithm type commands for me" projects. What I would thoroughly enjoy, however, would be ChatGPT-assisted autocomplete and suggestions in the shell. Somewhat similar to Copilot, but for the terminal, suggesting commands I might use next, directories I could want to navigate to, and so on. Maybe with a key binding to cycle through different suggestions. Is there something like that yet?
  - tomviner 4y ago
    Copilot CLI is exactly this https://www.npmjs.com/package/@githubnext/github-copilot-cli https://www.npmjs.com/package/@githubnext/github-copilot-cli
ogogmad 4y ago
This might combine well with Sympy, especially to help with the var('x y z') nonsense. I think I'll install this.
lee101 4y ago
[dead]

103 points by synergy20 4y ago

asah 4y ago

Cool! Neptyne spreadsheet has a GPT-enhanced python repl as well: https://neptyne.com/ https://neptyne.com/

I am a big casual user of anthropomorphic language for what computers are up to. I do it not because it's true, but because it's more engaging to say that "the computer is confused about X and so it's mad". But I'm wondering if that's dangerous when it comes to things that produce pseudo-human output. E.g.: > I think that the LLM thinks that it cannot generate code for the simple task. Unlike every other computer API in existence we can prompt the LLM to tell us why it responded in the way it does. I agree the LLM is going to produce text that is going to be narratively consistent with the assumptions of the question. But we already know LLMs are giant bullshitters. Is there any reason to think they are doing actual introspection and description of internal state? Or are they just going to give plausible-sounding words?

isthisit 4y ago
Author here. Fully agree. > I agree the LLM is going to produce text that is going to be narratively consistent with the assumptions of the question. But we already know LLMs are giant bullshitters. Is there any reason to think they are doing actual introspection and description of internal state? Or are they just going to give plausible-sounding words? This sounds like something we could test by inverting the LLM-provided reason of why it can/cannot do something and playing that back through the model. The example I gave in the blog was, unprompted, the LLM thought it should not generate code that it thought had already been executed. If we take it at face value, I could add a line in the code generation prompt to instruct it to generate code even if it "thinks" it has already executed it. We then assert the outcome. Not very scientific, but might give us some insight when trialed across thousands of prompts.
vba616 4y ago
>Is there any reason to think they are doing actual introspection and description of internal state? Or are they just going to give plausible-sounding words? Someone needs to call out the double standard applied to AI. If machines that bullshit are philosophically indistinguishable from intelligent humans, how is it that humans that bullshit are distinguishable from intelligent humans?
- wpietri 4y ago
  Sorry, I'm not following your point.
  - pulvinar 4y ago
    From GTP-4 (or maybe this is bullshitting too?): I believe vba616 is trying to point out that if we consider AI systems that generate plausible-sounding responses to be "bullshitting," then we should also acknowledge that humans who do the same are not necessarily distinguishable from "intelligent" humans. In other words, both AI systems and humans can produce seemingly coherent responses without necessarily having a deep understanding of the subject matter. It raises the question of whether we should hold AI systems to the same standard as humans when it comes to evaluating their output or understanding. If we accept that humans can sometimes provide plausible yet shallow answers, should we not also accept that AI systems, like LLMs, can do the same?
    - wpietri 4y ago
      Sorry, I do not give LLM-generated copy-pastes any attention. If you personally have something to say here, I'll read it with interest. But I have no interest in automated bullshit.
      - nullsense 4y ago
        As an AI language model, I fully agree.
      - pulvinar 4y ago
famouswaffles 4y ago
humans don't always give faithful reasoning either just so you know. even removing any willful obfuscation, we simply don't know all the biases that can influence a certain action. even more than that, Your brain will happily make up a reason for an action. This is demonstrated in the split brain experiments. Knowing that, if what it says in response to a given problem can meaningfully change output for the better following its directive later then what else ? might as well go with it
- wpietri 4y ago
  You could make exactly the same argument for consulting psychics, astrologers, and tarot readers. Indeed, I think what a lot of people are doing with LLM output is the same thing customers of paranormal scams do. Faced with artfully constructed ambiguity, they fill in the blanks themselves, not even noticing how much they're supplying the work. Some bit go well and some poorly; the successes are proof that the psychic has paranormal powers, and the rest are quickly forgotten about.
  - famouswaffles 4y ago
    The point is that a distinction without a difference is a meaningless distinction. Shouting "This is not real [insert property]" is beyond meaningless. Either this supposed difference is important enough to manifest in reality/output or it functionally doesn't exist. If rationalizations work as well as human ones do then treat them as you would human ones.
    - wpietri 4y ago
      Yes, ergo I believe there are differences. But you didn't address my point. Do you believe that explanations produced via tarot or zodiac should be treated with that same level of deference? That explanations from ouija boards are things we "might as well go with"?
      - famouswaffles 4y ago
        GPT-4 doesn't not perform at the same level on tasks as a ouija board. The whole point is that the level of deference is based on the level of performance it exhibits on whatever task you're having it do.
dragonwriter 4y ago
> If machines that bullshit are philosophically indistinguishable from intelligent humans, how is it that humans that bullshit are distinguishable from intelligent humans? Well, if we use the ReAct model, the flow of action involves the LLM narrating its reasoning and then acting on it. Which can still be bullshit, of course, but its at least a different thing than a post hoc rationalization when questioned. (Of course, we know that when questioned about past actions, humans often construct post hoc rationalizations, too.)
drvdevd 4y ago
Perhaps this is one of the greatest paradigm shifts for traditional software engineers as “AI” becomes more commonly integrated: maybe we will have to use anthropomorphic language and maybe it will be considered the most technically correct way of talking about these systems due not only to their non-deterministic nature but also their emergent properties. (This was a great post btw - highly recommend if you’re just perusing the comments)
- usrbinbash 4y ago
  Meeeeeehhhh...I don't think that is likely to happen any time soon. Could be wrong, but I don't think it's likely. The reason being: All software engineers are trained to read and write code, aka. precise formal languages. All feedback from systems about errors in that are also precise and formal languages. What will likely happen, is that we will use anthropomorphic language more and more to generate code, especially code that is easy to describe but just a boring hassle to type (aka. boilerplate or things like simple unit tests).
bheadmaster 4y ago
Is there any reason to think humans are doing actual introspection and description of internal state? Or are they just giving plausible-sounding words?
- spherebroken 4y ago
  People with separated brain hemispheres, for example sight and speak capabilities, try to make plausible reasoning. There are some well known physiological experiments to prove this. This proof that our brain try to provide a plausible explanation that has nothing to do with what is happening in reality. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/ (1)
- spherebroken 4y ago
  People with brain-split separated brain hemispheres, for example sight and speak capabilities, try to make plausible reasoning that are not grounded. Here is an example From (1) about brain-splitted subject called Joe: In a particularly dramatic recorded demonstration, the famous patient “Joe” was able to draw a cowboy hat with his left hand in response to the word “Texas” presented in his left visual half field. His commentary (produced by the verbal left hemisphere) showed a complete absence of insight into why his left hand had drawn this cowboy hat. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/ (1)
simonw 4y ago
Asking LLMs why they made a specific decision is a very common misconception. The state that lead to their previous generation was GBs of floating point calculations which were discarded as soon as the output had been generated. If you ask them why they said something they will generate a brand new explanation based on the previous text they generated, but any relationship to their actual decision process will be entirely coincidental. Update: I just noticed that the article uses this: > If you cannot return executable python code return set the reason why in the description and return no code. That DOES work - getting it to generate justification during a single response isn't affected by the lost state issue I'm describing here.
- heisenzombie 4y ago
  People also can’t really recreate an earlier mental state, and have to resort to post-hoc rationalisation. E.g. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ I feel like the answer to the question “give me a possible reason that a person/llm would give this answer given this situation” is all that’s ever really possible!
- IanCal 4y ago
  Hang on I think you're assuming there's some internal state that I don't think exists. Each time it just generates one more token until it hits a stop token, using the previous input. It shouldn't matter if it created a token or you did.
- OrderlyTiamat 4y ago
  You are right on fhe first point, I believe the second needs to be corrected. LLMs have no state. Each word- each token- is made anew, the network being oblivious to the process that led to the previous token. It cannot even distinguish between text it generated and text you added unless you tell it that. So any relationship between output and explanation is 'coincidental'. Actually, any relationship between two tokens next to each other is 'coincidental' in the exact same way, so it cannot explain its reasoning. If you prompt it to comply or explain why not, the network may generate an explanation of why it can't comply, but it is doing that simply as a good continuation of the input text. I'm not saying you can't call that an explanation if you want to use that word, I'm trying to make a distinction between the generated output and an explanation of that output. All of this is the former. The latter is not possible unless you analyse the model as it is running, and we don't really have a good understanding of that process yet. There was a cool paper where researchers analysed a finetuned model playing othello, and they found some neurons that clearly represented the game state- if they changed those neuron values, the network responded correctly, even if that game position was impossible to get to normally [1] I'd argue work like that is getting us closer to getting explanations for llm behaviour. [1] https://www.joyk.com/dig/detail/1674380676520322 https://www.joyk.com/dig/detail/1674380676520322
lynx23 4y ago
I am pretty convinced a large majority of humans don't do anything differently, they also just generate "plausible-sounding words".

westurner 4y ago

"Show HN: IPython-GPT, a Jupyter/IPython Interface to Chat GPT" https://news.ycombinator.com/item?id=35580959 https://news.ycombinator.com/item?id=35580959 lists a few REPLs

alfons_foobar 4y ago

> As the technology evolves I see there being a greater demand for deterministic responses from LLMs. I think we will see demand for it, but I also think it completely misses the nature of LLMs / for what they are most useful. Basically trying to shoehorn "old" patterns into a new paradigm.

kgeist 4y ago

What if LLM hallucinates some destructive operation, such as deleting local files, and runs it? I think the project should automatically run everything in a container, just in case.

9dev 4y ago
Yeah, that's a shared problem among these "let an algorithm type commands for me" projects. What I would thoroughly enjoy, however, would be ChatGPT-assisted autocomplete and suggestions in the shell. Somewhat similar to Copilot, but for the terminal, suggesting commands I might use next, directories I could want to navigate to, and so on. Maybe with a key binding to cycle through different suggestions. Is there something like that yet?
- tomviner 4y ago
  Copilot CLI is exactly this https://www.npmjs.com/package/@githubnext/github-copilot-cli https://www.npmjs.com/package/@githubnext/github-copilot-cli

ogogmad 4y ago

This might combine well with Sympy, especially to help with the var('x y z') nonsense. I think I'll install this.

lee101 4y ago

[dead]