3 ms·

The paper shows several distinct improvements to a sorting algorithm, and presents evidence that the process is generally applicable. This tweet points GPT-4 at

The paper shows several distinct improvements to a sorting algorithm, and presents evidence that the process is generally applicable. This tweet points GPT-4 at 20 instructions and asks if any can be removed, and it finds one optimisation. That's good to see from GPT-4, but the comparison seems disingenuous, and I'd expect more from someone in academia.

deelowe 3y ago
Agreed. They pointed GPT-4 at the solution and GPT-4 found it. Still somewhat impressive, but not exactly the same.
hackerlight 3y ago
> but the comparison seems disingenuous, and I'd expect more from someone in academia. And delivered with sarcasm and condescension. Not good to see, but par for the course on social media.
- wilg 3y ago
  I think this is a fair critique! The original tweeter was being condescending, probably to Increase Engagement. This is something social platforms should systemically discourage!
lupire 3y ago
Why does Twitter not show any rebuttals? Just Elon saying "Interesting".
- mrguyorama 3y ago
  You have answered your own question.
vidarh 3y ago
I just tried feeding GPT-4 some assembler as well, and while I did get it to make some optimisations, I also had to point out several mistakes it made (e.g. it moved one instruction to the other side of a label that'd cause a register to have the wrong value if a preceding branch was taken). It made mistakes that I've seen people new to assembler make, but happened to get some things right. When I pointed out an obvious optimisation it had missed, it applied it "half right" (I suggested it might look at collapsing multiple stack allocations because they didn't overlap, but while it collapsed them into a single allocation, it applied the sum of the allocations instead of just the largest one, which was a pointless increase in stack use). Also the kind of misunderstanding I could see with a junior dev with little assembly experience fall for if they recognise what you point to but don't quite understand the implications. It's "close enough" to be worth trying to see if it hits on something useful, but also bad enough that you really need to carefully scrutinise what it comes up with, and probably need to write a much more comprehensive prompt about strategies it should and shouldn't try to apply to make it more generally usable.
- astrobe_ 3y ago
  > I just tried feeding GPT-4 some assembler as well, and while I did get it to make some optimisations, I also had to point out several mistakes it made My point of view on this is that this type of AI cannot reason, so it seems to me quite audacious to use it for programming. Following the AI's suggestion is like examining a PR from someone you don't know: don't they try to insert a backdoor or a subtle bug? Even if you use it as some kind of "rubber duck", examining its suggestions - doing the necessary reasoning and the double checks - is pretty time consuming. Wouldn't the long term result be better if one spent some time to acquire a deeper knowledge of the field, rather than play the "prompt engineering" game (of which the rules might change next month)? It feels like we are in the equivalent of programming language's "honey moon" [1] phase with this kind of technology. People will eventually get over the "wow" effect and use it for what it is good at. [1] https://earthly.dev/blog/brown-green-language/ https://earthly.dev/blog/brown-green-language/
  - eropple 3y ago
    For making new stuff, I agree with you that it's pretty nutty. I'd never ask it to try to invent something novel, the way this does, because it's pretty high-noise. I do think it has significant programming advantages insofar as it can analyze existing code and make reasonable permutations of it, though, especially for stuff that's able to effectively analyze your current set of code. "Would it be better to acquire a deeper knowledge"--for the cases described above, actually no? I already have that deeper knowledge. I don't have the attention span (both because this stuff can be boring and because I am frequently randomized) and for a lot of this stuff the gate is actually not solving the problem but the physical act of typing, or even copying and changing. Copilot, at least, absolutely does create bugs, but in my experience they're mostly bugs of the form "this actually isn't typesafe in the way that it's written" and that makes isolating and fixing them clear. (Rarely are those bugs logic bugs, just a burp in the output.) There is a very clear future where this turns into incredible copy-paste garbage because it's not capable of refactoring, and that's a different problem. Avoiding it is why they pay us the medium bucks.
  - chii 3y ago
    > My point of view on this is that this type of AI cannot reason but it can answer the glass door push or pull question (see https://news.ycombinator.com/item?id=35506666 https://news.ycombinator.com/item?id=35506666), albeit a bit off (it didn't understand the mirrored writing). But some other models had gotten the right answer. This points to some evidence of being able to derive logical reasoning. Just because the model is "doing statistical guessing of the next word", doesn't mean this process does not produce logical reasoning as an outcome.
timdiggerm 3y ago
His comment "Sparks of AGI" ought to tell us a lot
schnable 3y ago
> That's good to see from GPT-4, but the comparison seems disingenuous, and I'd expect more from someone in academia. Apparently you haven't been following many academics on Twitter.
sfn42 3y ago
A lot of people in academia are fairly average and so is the work they do.

deelowe 3y ago

Agreed. They pointed GPT-4 at the solution and GPT-4 found it. Still somewhat impressive, but not exactly the same.

hackerlight 3y ago

> but the comparison seems disingenuous, and I'd expect more from someone in academia. And delivered with sarcasm and condescension. Not good to see, but par for the course on social media.

wilg 3y ago
I think this is a fair critique! The original tweeter was being condescending, probably to Increase Engagement. This is something social platforms should systemically discourage!

lupire 3y ago

Why does Twitter not show any rebuttals? Just Elon saying "Interesting".

mrguyorama 3y ago
You have answered your own question.

vidarh 3y ago

I just tried feeding GPT-4 some assembler as well, and while I did get it to make some optimisations, I also had to point out several mistakes it made (e.g. it moved one instruction to the other side of a label that'd cause a register to have the wrong value if a preceding branch was taken). It made mistakes that I've seen people new to assembler make, but happened to get some things right. When I pointed out an obvious optimisation it had missed, it applied it "half right" (I suggested it might look at collapsing multiple stack allocations because they didn't overlap, but while it collapsed them into a single allocation, it applied the sum of the allocations instead of just the largest one, which was a pointless increase in stack use). Also the kind of misunderstanding I could see with a junior dev with little assembly experience fall for if they recognise what you point to but don't quite understand the implications. It's "close enough" to be worth trying to see if it hits on something useful, but also bad enough that you really need to carefully scrutinise what it comes up with, and probably need to write a much more comprehensive prompt about strategies it should and shouldn't try to apply to make it more generally usable.

astrobe_ 3y ago
> I just tried feeding GPT-4 some assembler as well, and while I did get it to make some optimisations, I also had to point out several mistakes it made My point of view on this is that this type of AI cannot reason, so it seems to me quite audacious to use it for programming. Following the AI's suggestion is like examining a PR from someone you don't know: don't they try to insert a backdoor or a subtle bug? Even if you use it as some kind of "rubber duck", examining its suggestions - doing the necessary reasoning and the double checks - is pretty time consuming. Wouldn't the long term result be better if one spent some time to acquire a deeper knowledge of the field, rather than play the "prompt engineering" game (of which the rules might change next month)? It feels like we are in the equivalent of programming language's "honey moon" [1] phase with this kind of technology. People will eventually get over the "wow" effect and use it for what it is good at. [1] https://earthly.dev/blog/brown-green-language/ https://earthly.dev/blog/brown-green-language/
- eropple 3y ago
  For making new stuff, I agree with you that it's pretty nutty. I'd never ask it to try to invent something novel, the way this does, because it's pretty high-noise. I do think it has significant programming advantages insofar as it can analyze existing code and make reasonable permutations of it, though, especially for stuff that's able to effectively analyze your current set of code. "Would it be better to acquire a deeper knowledge"--for the cases described above, actually no? I already have that deeper knowledge. I don't have the attention span (both because this stuff can be boring and because I am frequently randomized) and for a lot of this stuff the gate is actually not solving the problem but the physical act of typing, or even copying and changing. Copilot, at least, absolutely does create bugs, but in my experience they're mostly bugs of the form "this actually isn't typesafe in the way that it's written" and that makes isolating and fixing them clear. (Rarely are those bugs logic bugs, just a burp in the output.) There is a very clear future where this turns into incredible copy-paste garbage because it's not capable of refactoring, and that's a different problem. Avoiding it is why they pay us the medium bucks.
- chii 3y ago
  > My point of view on this is that this type of AI cannot reason but it can answer the glass door push or pull question (see https://news.ycombinator.com/item?id=35506666 https://news.ycombinator.com/item?id=35506666), albeit a bit off (it didn't understand the mirrored writing). But some other models had gotten the right answer. This points to some evidence of being able to derive logical reasoning. Just because the model is "doing statistical guessing of the next word", doesn't mean this process does not produce logical reasoning as an outcome.

timdiggerm 3y ago

His comment "Sparks of AGI" ought to tell us a lot

schnable 3y ago

> That's good to see from GPT-4, but the comparison seems disingenuous, and I'd expect more from someone in academia. Apparently you haven't been following many academics on Twitter.

sfn42 3y ago

A lot of people in academia are fairly average and so is the work they do.