3 ms·

LLMs are not worthless for programming. You just cannot expect it to ship a full programm for you, but for generating functions with limited scope, I found it v

LLMs are not worthless for programming. You just cannot expect it to ship a full programm for you, but for generating functions with limited scope, I found it very useful. How to make use of a new and common libary for example. But of course you have to check and test. And for text I know people who use it succesfully (professionally) to generate texts for them as a summary from some data. They still have to proof read, but it saves them time, so it is valuable.

Roark66 3y ago
>LLMs are not worthless for programming. They can be worse than worthless. They can sabotage your work if you let them making you spend even more time fixing it afterwards. For an example. I've used Gpt4 as a sort of Google on steroids with prompts like "do subnets in gcloud span azs" and ", "in gcloud secret manager can you access secrets across regions". I very quickly learned to ask "is it true" after every answer and to never rely on a given answer too much(verify it quickly, don't let misinformation get you too far down the wrong route). So is it useful? Yes, but can it lead you down the wrong path? It very well can. The least experience you have in the field the easier it will happen. >You just cannot expect it to ship a full programm for you, but for generating functions with limited scope, I found it very useful Entire functions? Wow. I found it useful for generating skeletons I then have to fill by hand or tweak. I don't think I ever got anything out of Gpt4 that is useful as is (maybe except short snippets 3 lines long). However, I found it extremely useful in parsing emails received from people or writing nice sounding replies. For that it is really good (in English).
- gumballindie 3y ago
  > They can be worse than worthless. They can sabotage your work if you let them making you spend even more time fixing it afterwards. I basically gave up on llms because i was spending more time figuring out what it did wrong than actually getting value. People without programming skill are still impressed by them. But they yet have to learn or deliver anything of value even with the help of chat bots.
  - simonw 3y ago
    I have twenty years of programming experience and LLMs give me a significant productivity boost for programming on a daily basis: https://simonwillison.net/2023/Sep/29/llms-podcast/#code-interpreter-as-a-weird-kind-of-intern https://simonwillison.net/2023/Sep/29/llms-podcast/#code-int...
    - zare_st 3y ago
      Sorry, I may have gotten something wrong by skimming over your link. Is this the "significant project" you have been assisted by LLMs? https://github.com/simonw/sqlite-history https://github.com/simonw/sqlite-history
      - simonw 3y ago
        That's one of about a dozen at this point - but yeah, that's the one that I used LLMs to research the initial triggers and schema design for. Here's the transcript (it pre-dates the ChatGPT share feature): https://gist.github.com/simonw/1aa4050f3f7d92b048ae414a40cddbd5 https://gist.github.com/simonw/1aa4050f3f7d92b048ae414a40cdd... I wrote more about it here: https://simonwillison.net/2023/Apr/15/sqlite-history/ https://simonwillison.net/2023/Apr/15/sqlite-history/ Here's another one I built using AppleScript: https://github.com/dogsheep/apple-notes-to-sqlite https://github.com/dogsheep/apple-notes-to-sqlite - I wrote about that here: https://til.simonwillison.net/gpt3/chatgpt-applescript https://til.simonwillison.net/gpt3/chatgpt-applescript
        gumballindie 3y ago
        While it is impressive that an ai can generate all this, the code is anything but significant. Using triggers for history is one sure way to bring a scalable system down fast and one of the first lessons a junior will learn.
        simonw 3y ago
        Are you sure that holds with SQLite? My benchmarks so far have shown it to add a pretty inconsequential overhead. Also: not every system has to be a scalable system. That's another lesson junior engineers (should) learn.
- hutzlibu 3y ago
  "They can be worse than worthless." But that is the same, when you blindly follow some stackoverflow answer. And yes, I always have to tweak and I use it only rarely. But when I did, it was faster than googling and parsing the results.
  - marcosdumay 3y ago
    Nobody ever made a code editor plugin that reads random SO answers and automatically pastes them over your code. The amount of fighting I needed against MS development tools mingling my code recently is absurd. (Also, who the fuck decided that autocomplete on space and enter was a reasonable thing? Was that person high?)
- FrustratedMonky 3y ago
  >"I found it useful for generating skeletons I then have to fill by hand or tweak". Even this can be a big time saver, that increases productivity. Just like others have said, it isn't going to write a Pynchon novel, but it does do a great job at the other 99% of general writing that is done. Same for computers, the average programmer isn't creating some new Dijkstra Algorithm every day, they are really just cranking out connecting things together and doing the equivalent of 'generic boiler plate'.
matheusmoreira 3y ago
I've been using it for code review. I just paste some of my code in and ask the AI to critique it, suggest ideas and improvements. Makes for a less lonely coding experience. Wish I could point it to my git repositories and have it review the entire projects. I've had mixed experiences with getting it to generate new code. It produced good node.js command line application code. It didn't do so well at writing a program that creates 16 bit PCM audio file. I asked it to explain the WAV file format and things like lengths of structures got so confusing I had to research the stuff to figure out the truth.
- airstrike 3y ago
  This mirrors my experience. Very helpful writing node.js application code, but struggles to walk through simple operations in assembly. My hunch is that the tokenization process really hurts keeping the 1s and 0s straight. It's been hit or miss with rust. It's super helpful in decrypting compilation errors, decent with "core rust" and less helpful with 3rd party libraries like the cursive TUI crate Which comes as no surprise, really, as there's certainly less training data on the cursive crate than, say, expressjs Also FWIW I have actually pointed it at entire git repos with the WebPilot plugin within ChatGPT and it could explain what the repo did, but getting it to actually incorporate the source files as it wrote new code didn't work quite so well (I pointed it to https://github.com/kean/Get https://github.com/kean/Get and it would frequently fall back to writing native Swift code for HTTP requests instead of using the library)

Roark66 3y ago

>LLMs are not worthless for programming. They can be worse than worthless. They can sabotage your work if you let them making you spend even more time fixing it afterwards. For an example. I've used Gpt4 as a sort of Google on steroids with prompts like "do subnets in gcloud span azs" and ", "in gcloud secret manager can you access secrets across regions". I very quickly learned to ask "is it true" after every answer and to never rely on a given answer too much(verify it quickly, don't let misinformation get you too far down the wrong route). So is it useful? Yes, but can it lead you down the wrong path? It very well can. The least experience you have in the field the easier it will happen. >You just cannot expect it to ship a full programm for you, but for generating functions with limited scope, I found it very useful Entire functions? Wow. I found it useful for generating skeletons I then have to fill by hand or tweak. I don't think I ever got anything out of Gpt4 that is useful as is (maybe except short snippets 3 lines long). However, I found it extremely useful in parsing emails received from people or writing nice sounding replies. For that it is really good (in English).

gumballindie 3y ago
> They can be worse than worthless. They can sabotage your work if you let them making you spend even more time fixing it afterwards. I basically gave up on llms because i was spending more time figuring out what it did wrong than actually getting value. People without programming skill are still impressed by them. But they yet have to learn or deliver anything of value even with the help of chat bots.
- simonw 3y ago
  I have twenty years of programming experience and LLMs give me a significant productivity boost for programming on a daily basis: https://simonwillison.net/2023/Sep/29/llms-podcast/#code-interpreter-as-a-weird-kind-of-intern https://simonwillison.net/2023/Sep/29/llms-podcast/#code-int...
  - zare_st 3y ago
    Sorry, I may have gotten something wrong by skimming over your link. Is this the "significant project" you have been assisted by LLMs? https://github.com/simonw/sqlite-history https://github.com/simonw/sqlite-history
    - simonw 3y ago
      That's one of about a dozen at this point - but yeah, that's the one that I used LLMs to research the initial triggers and schema design for. Here's the transcript (it pre-dates the ChatGPT share feature): https://gist.github.com/simonw/1aa4050f3f7d92b048ae414a40cddbd5 https://gist.github.com/simonw/1aa4050f3f7d92b048ae414a40cdd... I wrote more about it here: https://simonwillison.net/2023/Apr/15/sqlite-history/ https://simonwillison.net/2023/Apr/15/sqlite-history/ Here's another one I built using AppleScript: https://github.com/dogsheep/apple-notes-to-sqlite https://github.com/dogsheep/apple-notes-to-sqlite - I wrote about that here: https://til.simonwillison.net/gpt3/chatgpt-applescript https://til.simonwillison.net/gpt3/chatgpt-applescript
      - gumballindie 3y ago
        While it is impressive that an ai can generate all this, the code is anything but significant. Using triggers for history is one sure way to bring a scalable system down fast and one of the first lessons a junior will learn.
        simonw 3y ago
        Are you sure that holds with SQLite? My benchmarks so far have shown it to add a pretty inconsequential overhead. Also: not every system has to be a scalable system. That's another lesson junior engineers (should) learn.
hutzlibu 3y ago
"They can be worse than worthless." But that is the same, when you blindly follow some stackoverflow answer. And yes, I always have to tweak and I use it only rarely. But when I did, it was faster than googling and parsing the results.
- marcosdumay 3y ago
  Nobody ever made a code editor plugin that reads random SO answers and automatically pastes them over your code. The amount of fighting I needed against MS development tools mingling my code recently is absurd. (Also, who the fuck decided that autocomplete on space and enter was a reasonable thing? Was that person high?)
FrustratedMonky 3y ago
>"I found it useful for generating skeletons I then have to fill by hand or tweak". Even this can be a big time saver, that increases productivity. Just like others have said, it isn't going to write a Pynchon novel, but it does do a great job at the other 99% of general writing that is done. Same for computers, the average programmer isn't creating some new Dijkstra Algorithm every day, they are really just cranking out connecting things together and doing the equivalent of 'generic boiler plate'.

matheusmoreira 3y ago

I've been using it for code review. I just paste some of my code in and ask the AI to critique it, suggest ideas and improvements. Makes for a less lonely coding experience. Wish I could point it to my git repositories and have it review the entire projects. I've had mixed experiences with getting it to generate new code. It produced good node.js command line application code. It didn't do so well at writing a program that creates 16 bit PCM audio file. I asked it to explain the WAV file format and things like lengths of structures got so confusing I had to research the stuff to figure out the truth.

airstrike 3y ago
This mirrors my experience. Very helpful writing node.js application code, but struggles to walk through simple operations in assembly. My hunch is that the tokenization process really hurts keeping the 1s and 0s straight. It's been hit or miss with rust. It's super helpful in decrypting compilation errors, decent with "core rust" and less helpful with 3rd party libraries like the cursive TUI crate Which comes as no surprise, really, as there's certainly less training data on the cursive crate than, say, expressjs Also FWIW I have actually pointed it at entire git repos with the WebPilot plugin within ChatGPT and it could explain what the repo did, but getting it to actually incorporate the source files as it wrote new code didn't work quite so well (I pointed it to https://github.com/kean/Get https://github.com/kean/Get and it would frequently fall back to writing native Swift code for HTTP requests instead of using the library)