I tested out 7 different LLMs today with a seemingly simple task: Write a Python script that takes a protobuf binary and produce a template proto file with plac

I tested out 7 different LLMs today with a seemingly simple task: Write a Python script that takes a protobuf binary and produce a template proto file with placeholder names. Gemini did the worst, with even the import being straight out wrong. GPT4 got slightly closer but still had a bunch of just straight up wrong usage. Surprisingly, Yi (Chinese or Japanese?) was the only one other than Claude Opus that gave code which actually ran on the first try (but still slightly wrong). I ended up writing the code myself in Golang (which I’m more familiar with) in about half the time it took me to debug the scripts pumped out by the LLMs. There is still a long way to go until we can be certain they save more time than waste.