3 ms·
Latest Chat-GPT4 release (May 12th 2023) sure is buggy
May 12th release, already hit some real junk responses when asking it write some Go code for me (I'm the creator of rqlite[1] and have been using GPT to help write some code[2]).
Here is the latest sample of nonsense output when I asked it to suggest some small refactoring changes to the lowest level of rqlite[3]:
if ro {
rows, opErr := db.queryStmtWithConn(stmt, xTime, queryer)
eqResponse = append(eqResponse, createEQQueryResponse(rows, opErr))
} else {
result either a map is not'sup K LRB, Jud case of system failure
result, opErr := db.executeStmtWithConn for the "executeStmt that you , you can manipulate the picture ', you ... i = t given, and it' . If, -212<|endoftext|>
Screenshot: https://storage.googleapis.com/bucket-vallified/rqlite/GPT-4Bugs.png
[1] https://rqlite.io/
[2] https://www.philipotoole.com/what-did-gpt-4-find-wrong-with-the-rqlite-source-code/
[3] https://github.com/rqlite/rqlite/blob/master/db/db.go
- otoolep 3y agoThe release prior to this one was quite good, no issues at all. Very noticeable problems in the last few hours.
- cushpush 3y ago3.5 might be better tuned to Go
- otoolep 3y agoCould be, but I've been using 4 up until tonight without a single problem. 4 was magic until this evening, TBH. Maybe if it continues I'll do as you suggest.
- bumbledraven 3y agoDo you have API access? If so, are you able to find a prompt/response pair where the model `gpt-4-0314` (supposedly the original publicly released version) does worse than `gpt-4` (the current model)? If you don't have API access, you can share some transcripts where the latest version of `gpt-4` seems to have performed worse than it did in the past, and I will try them myself against `gpt-4-0314` and post the results.
- cushpush 3y agoInteresting, please keep up abreast of your encounters with the good GPT.
- verdverm 3y agoI have tried 3.5, 4, and Bard, all are pretty bad at Go. The hallucinations are quite troublesome in all cases. They create unused vars and then use vars that aren't defined. They all seem to have trouble with this. Even in Python, I find them importing and then never using something.
- kleer001 3y agoI found the code more reliable if I kept the function as atomic and low level as possible and to start a new chat if it got off the rails and loopy. I'd start with a better and more conscience queery to get a mlre thoughtful answer.
- Aperocky 3y agoIn your experience, did the chatbot makes coding tasks faster or slower? I always feel like they will slow thing down as they introduce the additional debugging requirement which imo is worse than me thinking about how to write some of these code.
- otoolep 3y agoYes, it's saved me hours of time -- with certain narrow tasks. For example, it generated these files for me: https://github.com/rqlite/rqlite/blob/master/auto/backup/sum https://github.com/rqlite/rqlite/blob/master/auto/backup/sum* It took a few minutes, might have taken me 1+ hour of some research, and mechanically typing it all in.
- otoolep 3y agoAs in, the two files in that directory that start with "sum". This is one example, it's also found bugs in my code. See the blog post linked in the original post for more examples.