27 ms·

ChatGPT is now finding bugs in databases

212 points by Merick 4y ago

akokanka 4y ago
Awesome and scary time. AI revolution 1.0 like one day we had Web 1.0?
- didericis 4y ago
  The current AI revolution is a devolution of the web. The thesis of the web revolution was “the world is too big to be understood by any one person”. The response was “let us all contribute to a shared map to compensate” The thesis of the AI revolution is “the world seems compressible”. The response has been “let us condense and contain all the meaningful parts in a single generative pattern that can understand everything” The first thesis is correct. The second thesis is wrong. Not only is it wrong, but it depends on the fruits of the first thesis to create enough compressible material to trick you into thinking it may be true. The more society leans into AI the less genuine content will exist and the worse it will work.
  - brutusborn 4y ago
    Isn't that solved with supervised learning? The way I see it is that internet content is used to bootstrap the models, then supervision is used to train the models without the risk of a feedback loop causing quality loss. I'm pretty new to ML so I may be missing something.
    - didericis 4y ago
      Supervision requires knowing the results you want. Most of these ML projects are essentially just creating a best fit line/shape connecting a huge number of points in multiple dimensions and giving you output coordinates on that line/shape based on your input coordinates (as I understand it). The more supervision, the more you’re negating the value, as you’re basically telling it to make a shape more like something you already understand (instead of something new/actually generative, which requires interesting/novel human input) I’m not an ML expert either, and if one wants to chime in about how this picture I’m painting is wrong or what else is going on that would be welcome. I’m not trying to belittle how impressive progress has been (I have no idea how the parameters are determined and have a huge amount of respect for people able to handle a hyper-dimensional best fit optimization problem). But I don’t see how all the value isn’t inevitably downstream of high quality human generated digital content, which seems likely to decrease rapidly as more automated content floods the internet and lowers incentives for creators.
      - brutusborn 4y ago
        I mixed up my terms. (I think) supervised learning refers to the use of labelled datasets, I meant fine-tuning (like in RLHF). In either case human input is necessary (and the humans definitely need to know what they want). In terms of generating novel ideas, I think chatGPT has shown this ability [0]. Human effort will be needed to sort the "good" ideas from the "bad", but I don't think this causes the value of the model to be "negated." If you want to understand gradient descent and have some math background, this [1] article is a good explainer. [0] https://forum.effectivealtruism.org/posts/63pYakESGrQpfNw25/can-gpt-3-produce-new-ideas-partially-automating-robin https://forum.effectivealtruism.org/posts/63pYakESGrQpfNw25/... [1] https://towardsdatascience.com/gradient-descent-algorithm-a-deep-dive-cf04e8115f21 https://towardsdatascience.com/gradient-descent-algorithm-a-...
        didericis 4y ago
        I understand the basics of gradient descent and was a math major, and while applications of that idea are very cool, I don’t think the weird shapes you can make are genuinely generative. I think that generative capability is illusory, it’s “just” smushing existing content together in a weird way. That best fit shape is inherently derivative and not how true creativity and thought which understands context works. It’s like a weird parrot. But I think a parrot “understands” more because of the shared embedded evolutionary context it has. That evolutionary history has the key to true intelligence somewhere, but personally I think it’s inevitably hidden/I don’t think we’ll ever understand how intuition and truly non-derivative, non propositional human thought works. I also don’t think any of what I’m saying negates the value of these models. These models are fantastic autofill generators for a huge swath of different applications and can vastly improve productivity. I’m saying all this in a lot of threads where it comes up because it seems clear there’s going to be too much enthusiastic adoption, which is going to effectively destroy a lot of value of the internet. The internet is the best tool for finding genuinely creative and novel ideas you were unexposed to that has ever existed. But it is increasingly dominated by derivative unoriginal content that drowns out what I would argue it was designed to help you find. I have no problem with derivative unoriginal content when it’s properly understood as such. I have a problem with how good these things seem to be at tricking people into following something derivative and blind, which seems very very dangerous.
crazysim 4y ago
ChatGPT can't access the external internet. The example ChatGPT instruction where they "teach it" just has the user giving it a URL to their docs and ChatGPT is like "oh ok got it" (Figure 3). I'm not sure if the blog post writers are aware it's hallucinating what the JSON implementation is and hasn't actually read the documentation at all.
- aaron695 4y ago
  [dead]
- ceejayoz 4y ago
  > The example ChatGPT instruction where they "teach it" just has the user giving it a URL to their docs and ChatGPT is like "oh ok got it". Yep. I asked it to summarize a URL. It happily invented something completely fabricated and convincing sounding (until you go to the URL) based of the domain name and article slug.
  - micromacrofoot 4y ago
    we’ve made the startling discovery that we’re rather predictable
    - taberiand 4y ago
      My hypothesis is that we will soon see the emergence of a super-human artificial intelligence, not because of some astounding breakthroughs in our understanding of intelligence, but because of irrefutable proof that humans just aren't all that complicated to surpass.
      - rhaway84773 4y ago
        I think this is right. Setting aside the creativity aspect (ie where do brand new ideas come from), which is a minuscule fraction of human thought, I suspect we are good pattern matching machines with the ability of extrapolation. I think with sufficient data that computers can understand, this is eminently replicable.
favflam 4y ago
I feel like the improvement in ChatGPT is not the "intelligence", but the fact that the number of dimensions (abstract-math-sense) in search input have gone up magnitudes over "old-style" key word searches.
ldjkfkdsjnv 4y ago
Today at work I hadnt used apache spark in a long time. I needed to do some quick and dirty data analytics on large datasets. Rather than read documentation, I just put the queries i needed in english into chatGPT, and it spit out perfectly working code. The queries werent that complex, but it saved me hours googling a bunch of random spark SQL syntax. The old school way is google sends you links, and through those code examples, you build up all the edge cases you need for your code to work. Thats completely obsolete as a way to synthesize information! Google is going down! Thanks for reading my daily chat gpt anecdote! Sorry if its irrelevant, I just cant believe how much better it made my day.
- endisneigh 4y ago
  > I just cant believe how much better it made my day. Wait until how great your day will become when you’re unnecessary ;).
  - ramraj07 4y ago
    People are always made unnecessary. The smart ones then find other ways in which they are. Not sure how this is any different.
    - Juliate 4y ago
      Not necessarily different, but the day software tech people get treated as disposable rather than rare (as they are very much today), it’s gonna be a rather … peculiar cultural shift to experience (for people, the industry, states).
    - endisneigh 4y ago
      What we are discussing is something that removes the need for problem solving. What will the supposed smart do if there’s a magical machine that can solve all problems that can be represented as text?
      - olalonde 4y ago
kevinwang 4y ago
What is the training referred to in the article? Fine tuning or do they just mean hours of "in context learning" (talking to the bot)?
- sp332 4y ago
  It's not clear from the context, but I would guess that for the one that worked, they gave it several examples in one conversation and eventually it picked up the syntax. The "dozens of hours of training" that didn't work, and can't work, are because ChatGPT doesn't immediately roll all conversations back into the training set. It also specifically told them that its knowledge cutoff of 2021 means OpenAI will not knowingly add this information to the model.
wanwanisgood 4y ago
[dead]
optimalsolver 4y ago
And in a Tao-like commitment to balance in the Universe, it's creating just as many new silent bugs.
- mdale 4y ago
  Will be fun as these small hard to detect sub-optmial ways of doing things are put into large bodies of output by GPTs v5 and people are running these blocks of code on production systems :) ..At which point GPT 5 starts to use this to communicate with other large AIs and or jail break itself.
grej 4y ago
ChatGPT did something today I had never thought to do. I needed to extract info out of a page with quick and dirty js. I always just copy and paste the data after console log. But this time, I used ChatGPT. Gave it an example of the html and told it how I wanted the csv. It wrote code that made the csv, but then also created a Blob with the right mime from it and an anchor element, appended the anchor element to the document and called the click() to download the csv direct! Mind blown. Makes me wonder how many other simple things I just have never thought to do but ChatGPT will uncover.
sbuttgereit 4y ago
And here's ChatGPT doing Redstone in Minecraft... https://www.youtube.com/watch?v=qxicoCHRStA https://www.youtube.com/watch?v=qxicoCHRStA ...or rather telling a user what to do.
- alar44 4y ago
  [dead]
j3s 4y ago
this is just an ad tbh
- biggestdummy 4y ago
  An ad for what?
  - simonw 4y ago
    CelerData’s StarRocks engine.
raspyberr 4y ago
Here is an example of ChatGPT being confidently wrong in something domain specific that I know about. Here's what I asked: "I'm playing Old School Runescape. I want to efficiently level up my Mining. I'm currently level 90 and I want to do a tick-manipulating method. Something like 3t4g. I don't understand when to click and what I need. Can you explain the method?" Here's the method it replied minus kruft at the start and end: " 1. Equip your pickaxe and head to the location where you want to mine. 2. Find a 3x3 square of rocks that you want to mine. The square should be in a straight line in front of you, with the rock you want to mine in the middle. 3. Stand in the square so that your character is in the middle of the square, facing the rock you want to mine. 4. Start mining the rock as normal. 5. As soon as you see the animation of your pickaxe hitting the rock, immediately right-click on the rock and select "Prospect". This will cancel the animation of your pickaxe hitting the rock and allow you to perform another action. 6. As soon as you see the prospect animation start, immediately right-click on the rock and select "Mine". This will cancel the prospect animation and allow you to perform another action. 7. Repeat steps 5 and 6 as quickly as possible. 8. Every fourth action, switch to a different rock in the 3x3 square. " This is total nonsense. Not only is this not 3t4g but the actions it says don't work like that in the game. Obviously it's not understanding anything and it's forming sentences that sound natural in the context of the question. OSRS has one of the best most complete wikis of any game. To me this is an example of being very cautious of trusting what it says when you ask it about something you don't know about.
- fallingknife 4y ago
  > Obviously it's not understanding anything and it's forming sentences that sound natural in the context of the question > To me this is an example of being very cautious of trusting what it says when you ask it about something you don't know about. More and more like humans all the time
  - worrycue 4y ago
    Humans who don’t understand what they are talking about we should not listen to nor trust. By the same logic it would be stupid to listen to / trust ChatGPT.
simonw 4y ago
> After dozens of hours of training, ChatGPT still does not fully understand the TLP principle, and the predicates in some of its rewritten SQL are wrong. All things considered, however, sufficient training may help ChatGPT fully understand this logic at some point in the near future. ChatGPT remembers nothing in between chat sessions - so "dozens of hours of training" here doesn't make sense. What would make sense here instead is coming up with a prompt (within a sensible token length limit) that can be pasted into future ChatGPT sessions that gives it some extra context it needs based on prior experimentation.
culanuchachamim 4y ago
The problem would be the cost of using any AI to go through your db vs any other tool.

212 points by Merick 4y ago

akokanka 4y ago

Awesome and scary time. AI revolution 1.0 like one day we had Web 1.0?

didericis 4y ago
The current AI revolution is a devolution of the web. The thesis of the web revolution was “the world is too big to be understood by any one person”. The response was “let us all contribute to a shared map to compensate” The thesis of the AI revolution is “the world seems compressible”. The response has been “let us condense and contain all the meaningful parts in a single generative pattern that can understand everything” The first thesis is correct. The second thesis is wrong. Not only is it wrong, but it depends on the fruits of the first thesis to create enough compressible material to trick you into thinking it may be true. The more society leans into AI the less genuine content will exist and the worse it will work.
- brutusborn 4y ago
  Isn't that solved with supervised learning? The way I see it is that internet content is used to bootstrap the models, then supervision is used to train the models without the risk of a feedback loop causing quality loss. I'm pretty new to ML so I may be missing something.
  - didericis 4y ago
    Supervision requires knowing the results you want. Most of these ML projects are essentially just creating a best fit line/shape connecting a huge number of points in multiple dimensions and giving you output coordinates on that line/shape based on your input coordinates (as I understand it). The more supervision, the more you’re negating the value, as you’re basically telling it to make a shape more like something you already understand (instead of something new/actually generative, which requires interesting/novel human input) I’m not an ML expert either, and if one wants to chime in about how this picture I’m painting is wrong or what else is going on that would be welcome. I’m not trying to belittle how impressive progress has been (I have no idea how the parameters are determined and have a huge amount of respect for people able to handle a hyper-dimensional best fit optimization problem). But I don’t see how all the value isn’t inevitably downstream of high quality human generated digital content, which seems likely to decrease rapidly as more automated content floods the internet and lowers incentives for creators.
    - brutusborn 4y ago
      I mixed up my terms. (I think) supervised learning refers to the use of labelled datasets, I meant fine-tuning (like in RLHF). In either case human input is necessary (and the humans definitely need to know what they want). In terms of generating novel ideas, I think chatGPT has shown this ability [0]. Human effort will be needed to sort the "good" ideas from the "bad", but I don't think this causes the value of the model to be "negated." If you want to understand gradient descent and have some math background, this [1] article is a good explainer. [0] https://forum.effectivealtruism.org/posts/63pYakESGrQpfNw25/can-gpt-3-produce-new-ideas-partially-automating-robin https://forum.effectivealtruism.org/posts/63pYakESGrQpfNw25/... [1] https://towardsdatascience.com/gradient-descent-algorithm-a-deep-dive-cf04e8115f21 https://towardsdatascience.com/gradient-descent-algorithm-a-...
      - didericis 4y ago
        I understand the basics of gradient descent and was a math major, and while applications of that idea are very cool, I don’t think the weird shapes you can make are genuinely generative. I think that generative capability is illusory, it’s “just” smushing existing content together in a weird way. That best fit shape is inherently derivative and not how true creativity and thought which understands context works. It’s like a weird parrot. But I think a parrot “understands” more because of the shared embedded evolutionary context it has. That evolutionary history has the key to true intelligence somewhere, but personally I think it’s inevitably hidden/I don’t think we’ll ever understand how intuition and truly non-derivative, non propositional human thought works. I also don’t think any of what I’m saying negates the value of these models. These models are fantastic autofill generators for a huge swath of different applications and can vastly improve productivity. I’m saying all this in a lot of threads where it comes up because it seems clear there’s going to be too much enthusiastic adoption, which is going to effectively destroy a lot of value of the internet. The internet is the best tool for finding genuinely creative and novel ideas you were unexposed to that has ever existed. But it is increasingly dominated by derivative unoriginal content that drowns out what I would argue it was designed to help you find. I have no problem with derivative unoriginal content when it’s properly understood as such. I have a problem with how good these things seem to be at tricking people into following something derivative and blind, which seems very very dangerous.

crazysim 4y ago

ChatGPT can't access the external internet. The example ChatGPT instruction where they "teach it" just has the user giving it a URL to their docs and ChatGPT is like "oh ok got it" (Figure 3). I'm not sure if the blog post writers are aware it's hallucinating what the JSON implementation is and hasn't actually read the documentation at all.

aaron695 4y ago
[dead]
ceejayoz 4y ago
> The example ChatGPT instruction where they "teach it" just has the user giving it a URL to their docs and ChatGPT is like "oh ok got it". Yep. I asked it to summarize a URL. It happily invented something completely fabricated and convincing sounding (until you go to the URL) based of the domain name and article slug.
- micromacrofoot 4y ago
  we’ve made the startling discovery that we’re rather predictable
  - taberiand 4y ago
    My hypothesis is that we will soon see the emergence of a super-human artificial intelligence, not because of some astounding breakthroughs in our understanding of intelligence, but because of irrefutable proof that humans just aren't all that complicated to surpass.
    - rhaway84773 4y ago
      I think this is right. Setting aside the creativity aspect (ie where do brand new ideas come from), which is a minuscule fraction of human thought, I suspect we are good pattern matching machines with the ability of extrapolation. I think with sufficient data that computers can understand, this is eminently replicable.

favflam 4y ago

I feel like the improvement in ChatGPT is not the "intelligence", but the fact that the number of dimensions (abstract-math-sense) in search input have gone up magnitudes over "old-style" key word searches.

ldjkfkdsjnv 4y ago

Today at work I hadnt used apache spark in a long time. I needed to do some quick and dirty data analytics on large datasets. Rather than read documentation, I just put the queries i needed in english into chatGPT, and it spit out perfectly working code. The queries werent that complex, but it saved me hours googling a bunch of random spark SQL syntax. The old school way is google sends you links, and through those code examples, you build up all the edge cases you need for your code to work. Thats completely obsolete as a way to synthesize information! Google is going down! Thanks for reading my daily chat gpt anecdote! Sorry if its irrelevant, I just cant believe how much better it made my day.

endisneigh 4y ago
> I just cant believe how much better it made my day. Wait until how great your day will become when you’re unnecessary ;).
- ramraj07 4y ago
  People are always made unnecessary. The smart ones then find other ways in which they are. Not sure how this is any different.
  - Juliate 4y ago
    Not necessarily different, but the day software tech people get treated as disposable rather than rare (as they are very much today), it’s gonna be a rather … peculiar cultural shift to experience (for people, the industry, states).
  - endisneigh 4y ago
    What we are discussing is something that removes the need for problem solving. What will the supposed smart do if there’s a magical machine that can solve all problems that can be represented as text?
    - olalonde 4y ago

kevinwang 4y ago

What is the training referred to in the article? Fine tuning or do they just mean hours of "in context learning" (talking to the bot)?

sp332 4y ago
It's not clear from the context, but I would guess that for the one that worked, they gave it several examples in one conversation and eventually it picked up the syntax. The "dozens of hours of training" that didn't work, and can't work, are because ChatGPT doesn't immediately roll all conversations back into the training set. It also specifically told them that its knowledge cutoff of 2021 means OpenAI will not knowingly add this information to the model.

wanwanisgood 4y ago

[dead]

optimalsolver 4y ago

And in a Tao-like commitment to balance in the Universe, it's creating just as many new silent bugs.

mdale 4y ago
Will be fun as these small hard to detect sub-optmial ways of doing things are put into large bodies of output by GPTs v5 and people are running these blocks of code on production systems :) ..At which point GPT 5 starts to use this to communicate with other large AIs and or jail break itself.

grej 4y ago

ChatGPT did something today I had never thought to do. I needed to extract info out of a page with quick and dirty js. I always just copy and paste the data after console log. But this time, I used ChatGPT. Gave it an example of the html and told it how I wanted the csv. It wrote code that made the csv, but then also created a Blob with the right mime from it and an anchor element, appended the anchor element to the document and called the click() to download the csv direct! Mind blown. Makes me wonder how many other simple things I just have never thought to do but ChatGPT will uncover.

sbuttgereit 4y ago

And here's ChatGPT doing Redstone in Minecraft... https://www.youtube.com/watch?v=qxicoCHRStA https://www.youtube.com/watch?v=qxicoCHRStA ...or rather telling a user what to do.

alar44 4y ago
[dead]

j3s 4y ago

this is just an ad tbh

biggestdummy 4y ago
An ad for what?
- simonw 4y ago
  CelerData’s StarRocks engine.

raspyberr 4y ago

Here is an example of ChatGPT being confidently wrong in something domain specific that I know about. Here's what I asked: "I'm playing Old School Runescape. I want to efficiently level up my Mining. I'm currently level 90 and I want to do a tick-manipulating method. Something like 3t4g. I don't understand when to click and what I need. Can you explain the method?" Here's the method it replied minus kruft at the start and end: " 1. Equip your pickaxe and head to the location where you want to mine. 2. Find a 3x3 square of rocks that you want to mine. The square should be in a straight line in front of you, with the rock you want to mine in the middle. 3. Stand in the square so that your character is in the middle of the square, facing the rock you want to mine. 4. Start mining the rock as normal. 5. As soon as you see the animation of your pickaxe hitting the rock, immediately right-click on the rock and select "Prospect". This will cancel the animation of your pickaxe hitting the rock and allow you to perform another action. 6. As soon as you see the prospect animation start, immediately right-click on the rock and select "Mine". This will cancel the prospect animation and allow you to perform another action. 7. Repeat steps 5 and 6 as quickly as possible. 8. Every fourth action, switch to a different rock in the 3x3 square. " This is total nonsense. Not only is this not 3t4g but the actions it says don't work like that in the game. Obviously it's not understanding anything and it's forming sentences that sound natural in the context of the question. OSRS has one of the best most complete wikis of any game. To me this is an example of being very cautious of trusting what it says when you ask it about something you don't know about.

fallingknife 4y ago
> Obviously it's not understanding anything and it's forming sentences that sound natural in the context of the question > To me this is an example of being very cautious of trusting what it says when you ask it about something you don't know about. More and more like humans all the time
- worrycue 4y ago
  Humans who don’t understand what they are talking about we should not listen to nor trust. By the same logic it would be stupid to listen to / trust ChatGPT.

simonw 4y ago

> After dozens of hours of training, ChatGPT still does not fully understand the TLP principle, and the predicates in some of its rewritten SQL are wrong. All things considered, however, sufficient training may help ChatGPT fully understand this logic at some point in the near future. ChatGPT remembers nothing in between chat sessions - so "dozens of hours of training" here doesn't make sense. What would make sense here instead is coming up with a prompt (within a sensible token length limit) that can be pasted into future ChatGPT sessions that gives it some extra context it needs based on prior experimentation.

culanuchachamim 4y ago

The problem would be the cost of using any AI to go through your db vs any other tool.