4 ms·

I'm not sure what you mean. It seems to me less of an issue that LLMs produce code that aren't compliant with language specs, but rather that LLMs produce code

I'm not sure what you mean. It seems to me less of an issue that LLMs produce code that aren't compliant with language specs, but rather that LLMs produce code that do not meet the stated requirements, or that they break from earlier stated requirements as new requirements are requested. And of course, code not meeting requirements is not a problem native to LLMs, but to all code production agents everywhere (including humans).

aarondia 3y ago
> or that they break from earlier stated requirements as new requirements are requested. This nails one of the biggest issues that we're trying to help Mito users with -- code that meets __changing__ requirements. ie: the input data column headers change. The best approach we have right now is giving users flexibility to parametrize their script (ie: tell the script the right column header or reference columns by position instead of name) It turns out that teams that have moved their data to Snowflake tend to avoid these issues primarily because the schema of their data changes much less frequently.
DarthNebo 3y ago
That's the problem with humans as well, since we don't exactly memorise the spec, we just check & update basis errors & looking up references. But since this is an LLM we are talking about, it should be able to infer the same once given the spec of a language without needing a compiler or atleast being able to check with one via plugin or api call.