3 ms·
A couple observations from attempting to use both GPT-3.5 and GPT-4 via the web interface for coding tasks: - The model's ability to respond accurately drops d
by dangerlibrary 3y ago
A couple observations from attempting to use both GPT-3.5 and GPT-4 via the web interface for coding tasks:
- The model's ability to respond accurately drops drastically when asked questions of the form "is there a different way to accomplish X, using Y?" or "is there a way to accomplish X that runs in O(log(n)) time instead?" Example: I wanted to upsert an integer value using a SQLite db using "INSERT ... RETURNING..." ChatGPT repeatedly told me that sqlite doesn't support "RETURNING" (it does, since March 2021). It insisted I would need two DB round trips from my application to accomplish this. When asked "can this be done in one round trip, instead?" it repeatedly wrote code that would return the number of rows modified instead of the integer column value.
- ChatGPT's limited standard library knowledge means that the solutions it produces, even when correct, are often lower-level and less idiomatic. Problems that would be trivially solved with e.g. a Java.String.replaceAll or .codePointCount will instead loop over each character, often splitting the string into an intermediate array and implementing special cases for first/last character edge cases. The code winds up being mostly correct, but also (for lack of a better word) weird. No human I've ever worked with would do things the way ChatGPT sometimes does, which means the code will likely be much harder to maintain and debug over time.
- ch4s3 3y ago> are often lower-level and less idiomatic I would go as far as to say it's adept at producing technically working spaghetti.
- jstarfish 3y ago> ChatGPT repeatedly told me that sqlite doesn't support "RETURNING" (it does, since March 2021). Its dataset cuts off around 2021. There's a little footer message warning you not to expect knowledge of recent events.
- dangerlibrary 3y agoIt cuts off in September 2021, supposedly - that's why I pointed out the month of the relevant sqlite release.
- tedsanders 3y agoThink of GPT like a document simulator. If 90% of documents on the web talk about an old version of a library, then GPT-4 is likely to use that old version. When I asked it write me a Chrome extension, it used Manifest V2, which was already slated for deprecation. When I asked for a V3 extension specifically, it was happy to comply. So even if a new release came out before Sep 2021, if it hasn't had time to become the dominant version in code examples around the web, GPT-4 may still use older versions.
- dangerlibrary 3y agoMy point (which may not have been clear) was that when I explicitly told it that the versions of SQLite I'm using supports "returning," it still returned incorrect code. It did not use "returning", and repeatedly, even after being corrected, grabbed the row count instead of the column value. I understand that it's a language model - my point was that in my experience, the depth of training data just isn't there for alternative implementations. It can do the things you ask one way, maybe two or three, but if you know how you actually want something done you're probably better off writing it yourself (for now).
- chillfox 3y agoDid you try copy pasting the relevant Sqlite docs into ChatGPT?