3 ms·

> Does it seem like there's a pattern to the failures? Personally, trying to use it to write code (primarily Elixir backend and Rust systems/CLI), I tend to ru

by notamy 3y ago

> Does it seem like there's a pattern to the failures? Personally, trying to use it to write code (primarily Elixir backend and Rust systems/CLI), I tend to run into: - Hallucinating APIs that don't exist - Hallucinating entire libraries, despite being told repeatedly they don't exist - Saying it will make requested changes and not doing so - Not being anywhere close to idiomatic code - Not being able to explain code it writes - Running out of "memory" (I can't remember the right term. Context?) in the middle of generating code, then telling me I never prompted it when I ask it to continue On the other hand, I've found that it's good at cleaning up ugly data. I can copy/paste in a table with bad formatting, ask it to turn it into code, and it does it near-perfectly. That's been the best use-case for it I've found so far. I use boring normal free ChatGPT so maybe it's on me for not using GPT-4 or some other model, but either way, imo it's not been very impressive in the problem spaces I find myself in.

mrtranscendence 3y ago
> Not being able to explain code it writes I've actually found it (ChatGPT-4) reasonably good at explaining other people's code, for what it's worth. It does require providing some context for the code it's explaining. ("This function is part of library X that does Y, and it seems to be about Z. I can't figure out the purpose of the loop in the middle, though. Can you explain it?") I'm with you on it writing code, though. It makes things up too frequently to be really useful.
simonw 3y ago
Yeah, GPT-4 is so, so much better at not doing the things you're describing there. ChatGPT is also much better for Python and JavaScript than Rust and Elixir, presumably because it's seen an order of magnitude more example code for those languages.
gwd 3y ago
> I use boring normal free ChatGPT so maybe it's on me for not using GPT-4 or some other model, but either way, imo it's not been very impressive in the problem spaces I find myself in. You can use GPT-4 fairly cheap if you sign up for a developer account on platform.openai.com and then use their playground. There you pay per usage; even though I've been using it fairly heavily, my typical monthly usage is still way under $20. Here's an example that both impressed me and saved me a load of time just yesterday: https://gitlab.com/-/snippets/2535443 https://gitlab.com/-/snippets/2535443

by notamy 3y ago

mrtranscendence 3y ago

> Not being able to explain code it writes I've actually found it (ChatGPT-4) reasonably good at explaining other people's code, for what it's worth. It does require providing some context for the code it's explaining. ("This function is part of library X that does Y, and it seems to be about Z. I can't figure out the purpose of the loop in the middle, though. Can you explain it?") I'm with you on it writing code, though. It makes things up too frequently to be really useful.

simonw 3y ago

Yeah, GPT-4 is so, so much better at not doing the things you're describing there. ChatGPT is also much better for Python and JavaScript than Rust and Elixir, presumably because it's seen an order of magnitude more example code for those languages.

gwd 3y ago

> I use boring normal free ChatGPT so maybe it's on me for not using GPT-4 or some other model, but either way, imo it's not been very impressive in the problem spaces I find myself in. You can use GPT-4 fairly cheap if you sign up for a developer account on platform.openai.com and then use their playground. There you pay per usage; even though I've been using it fairly heavily, my typical monthly usage is still way under $20. Here's an example that both impressed me and saved me a load of time just yesterday: https://gitlab.com/-/snippets/2535443 https://gitlab.com/-/snippets/2535443