3 ms·
I figured that out, the first time I submitted a Swift question to ChatGPT, where I would normally have used StackOverflow. It hallucinated an Apple API call.
by ChrisMarshallNY 2y ago
I figured that out, the first time I submitted a Swift question to ChatGPT, where I would normally have used StackOverflow.
It hallucinated an Apple API call. Not just that, but that nonexistent call was the entire fulcrum of its "solution."
- desumeku 2y agoDon't use "hallucinated". Use "bullshitted". https://link.springer.com/article/10.1007/s10676-024-09775-5 https://link.springer.com/article/10.1007/s10676-024-09775-5
- viraptor 2y agoThat's a problem with lots of less common languages. The accuracy seems to scale with the number of github repos and SO questions. Most Swift code being in proprietary iOS/MacOS apps doesn't help it.
- ChrisMarshallNY 2y agoAlso, Swift allows extensions of fundamental types. I often declare extensions of Int and Array. It’s fairly common to add computed properties and functions to basic APIs. I’m pretty sure that a number of folks extended the API with this property (which is what I did, once I figured out how to address the issue. It just made sense to do so), and ChatGPT read that as a standard system call. That’s likely to be an issue with more mainstream languages, as extension is a fairly common pattern, these days. I think I may start tagging my extensions (like a naming prefix), to signal they are added after the fact. I do a lot of open-source work, so it’s likely to be used as training data.
- surgical_fire 2y agoEh, I had the same experience asking questions on Java and Python. The more strict your requirements, the less reliable LLMs are.
- viraptor 2y agoUsing agents improves this dramatically if you want to try again. Putting a linter, test runner and self reflection in the loop fixes a ridiculous number of LLM issues.