4 ms·

> get an LLM that can write production code reliably it depends on what that user wanted the code to do, and how important it is. For example, an average, non

by chii 2y ago

> get an LLM that can write production code reliably it depends on what that user wanted the code to do, and how important it is. For example, an average, non-technical user could use this to generate a script to sort out their email, or a script for automation in MS Office VBA. Just because it's not perfect, doesn't mean it doesn't have a good use, and won't improve. Tom Scott's video[0] makes a very good argument: we don't know where we are on the technology curve. [0]: https://www.youtube.com/watch?v=jPhJbKBuNnA https://www.youtube.com/watch?v=jPhJbKBuNnA

from-nibly 2y ago
The comment you replied to said production code.
- chii 2y ago
  production code is code that is not just for testing, but for actual use. Nothing about production code says critical.
fhdsgbbcaA 2y ago
I mean systems critical infrastructure code where error tolerance is zero. For one off scripts or sketching a concept quickly it’s good enough, and for language reference it’s generally useful. However, one thing I’ve noticed with Claude in particular is it tends to overweight the top answers in stack overflow. The problem there is top answers are rarely the best answer - rather they tend to be overly verbose and long, whereas the best answer is usually the second one that just tells you what function to call. On multiple occasions I’ve had Claude answer a simple prompt with horribly verbose and complicated code. Then I’ll say “what about this single call?” (Eg the type of SO answer than gets the second most votes), and it’s says “You’re right! That’s a much better answer”. Likewise I suggest to anybody to take the domain you are most knowledgeable in and pepper your LLM of choice with lots of questions and see how much it knows. You’ll get a feel for how much you can trust it in other domains - which is “not much”.
- chii 2y ago
  > where error tolerance is zero. which is quite niche, and you'd be correct that no one would trust GPT generated code blindly for that! But this is a spectrum, and while i think today's GPT models don't quite make it there, i argue that we're closer to success with this than auto-driving cars; mainly due to the larger tolerance for bad code, rather than actual tech improvements.

by chii 2y ago

from-nibly 2y ago

The comment you replied to said production code.

chii 2y ago
production code is code that is not just for testing, but for actual use. Nothing about production code says critical.

fhdsgbbcaA 2y ago

I mean systems critical infrastructure code where error tolerance is zero. For one off scripts or sketching a concept quickly it’s good enough, and for language reference it’s generally useful. However, one thing I’ve noticed with Claude in particular is it tends to overweight the top answers in stack overflow. The problem there is top answers are rarely the best answer - rather they tend to be overly verbose and long, whereas the best answer is usually the second one that just tells you what function to call. On multiple occasions I’ve had Claude answer a simple prompt with horribly verbose and complicated code. Then I’ll say “what about this single call?” (Eg the type of SO answer than gets the second most votes), and it’s says “You’re right! That’s a much better answer”. Likewise I suggest to anybody to take the domain you are most knowledgeable in and pepper your LLM of choice with lots of questions and see how much it knows. You’ll get a feel for how much you can trust it in other domains - which is “not much”.

chii 2y ago
> where error tolerance is zero. which is quite niche, and you'd be correct that no one would trust GPT generated code blindly for that! But this is a spectrum, and while i think today's GPT models don't quite make it there, i argue that we're closer to success with this than auto-driving cars; mainly due to the larger tolerance for bad code, rather than actual tech improvements.