Some of the best tests come from taking something mundane and twisting it just a little off the beaten path in a way that LLMs cannot handle. The usual example

Some of the best tests come from taking something mundane and twisting it just a little off the beaten path in a way that LLMs cannot handle. The usual example is the river crossing puzzle which has a standard format and solution but can be modified to make that solution incorrect. For code, an example would be parsing log lines to compile insights but switching it up so that the timestamp means something other than the time the log was collected, etc. You can completely change the complexity of a bog standard leetcode question with the slightest of modifications, and I’m certain no extant LLM will catch such a nuance if your problem is a very common one. A fun exercise is to ask chatGPT what the river crossing goat puzzle is and then say “do the same puzzle but with two goats and a cabbage please”. It can’t get it right even with multiple corrections. Might stop working at some point now that I posted this though!