4 ms·
The problem is that there are no tasks that LLMs are reliably good at. I believe that's what OP is getting at. I fixed a production issue earlier this year tha
by NateEag 2y ago
The problem is that there are no tasks that LLMs are reliably good at. I believe that's what OP is getting at.
I fixed a production issue earlier this year that turned out to be a naive infinite loop - it was trying to load all data from a paginated API endpoint, but there was no logic to update the page number being fetched.
There was a test for it. Alas, the test didn't actually cover this scenario.
I mention this because it was committed by a co-worker whose work is historically excellent, but who started using Copilot / ChatGPT. I'm pretty sure it was an LLM-generated function and test, and they were deeply broken.
Mostly they've been working great for this co-worker.
But not reliably.
- margalabargala 2y agoI understand that, the point I'm making is that reliability is not a requirement for utility. One does not need to be reliable to be reliably useful :) A very similar example is StackOverflow. If you copy/paste answers verbatim from SO, you will have problems. Some top answers are deeply broken or have obvious bugs. Frequently, SO answers are only related to your question, but do not explicitly answer it. SO is useful to the industry in the same way LLMs are.
- bawolff 2y agoSure, there is a range. If it works 100% of the time its clearly useful. If it works 0% then it clearly isn't. LLMs are in the middle. Its unclear which side of the line they are on. Some ancedotes say one thing some say another. That's why studies would be great. Its also why syntax highlighting is a bad comparison since that is not in the greyzone.