3 ms·

It's quite amazing - I often find that I read quite positive comments towards LLM tools for coding. Yet, an "Ask HN" I posted a while ago (and which admittedly

by kleiba 3y ago

It's quite amazing - I often find that I read quite positive comments towards LLM tools for coding. Yet, an "Ask HN" I posted a while ago (and which admittedly didn't gain much traction) seemed to mirror mostly negative/pessimistic responses. https://news.ycombinator.com/item?id=38803836 https://news.ycombinator.com/item?id=38803836 Was it just that my submission didn't find enough / more balanced commenters?

simonw 3y ago
You only got comments from six people so yeah, definitely not representative.
yorwba 3y ago
You got two positive and two negative responses. You replied only to the negative responses. Now you think that the responses were mostly negative. I blame salience bias. Anyways, there's also a difference between "are you excited about this new thing becoming available" and "now that you've used it, do you like the experience". The former is more likely to feature rosy expectations and the latter bitter disappointment. (Though it could also be the other way around, with people dismissing it at first and then discovering that it's kind of nice actually.)
Havoc 3y ago
The precise wording matters. How has it changed your work life leads people down the rabbit hole of will coding jobs be safe. This one is a lot more neutral/technical.
weebull 3y ago
If somebody can show me a coding task that LLMs have successfully done that isn't an interview question or a documentation snippet, I might start to value it. Spending huge amount of resource to be a bit better at autocompleting code doesn't have value to me. I want it to solve significant problems, and it's looking like it can't do it and scaling it to be able to is totally impractical. > In aggregate, training all 9 Code Llama models required 400K GPU hours of computation on hardware of type A100-80GB (TDP of 350-400W). That is: * 45⅔ GPU years * 160 MWh or... * 45 average UK homes annual electric consumption * 18 average US homes * 64 average drivers annual milage in an EV. ...and that's just the GPUs. Add on all the rest of the system (s).
- regularfry 3y ago
  In the grand scheme of things it's ancient history, but https://code-as-policies.github.io/ https://code-as-policies.github.io/ works by generating code then executing it. That's worth running at. The code generation in that paper was done on code-davinci-002, which is (or rather was - it's deprecated) a 15B GPT-3 model. I've not done it yet, but I'd expect the open source 7B code completion models to be able to replicate it by now.

by kleiba 3y ago

simonw 3y ago

You only got comments from six people so yeah, definitely not representative.

yorwba 3y ago

You got two positive and two negative responses. You replied only to the negative responses. Now you think that the responses were mostly negative. I blame salience bias. Anyways, there's also a difference between "are you excited about this new thing becoming available" and "now that you've used it, do you like the experience". The former is more likely to feature rosy expectations and the latter bitter disappointment. (Though it could also be the other way around, with people dismissing it at first and then discovering that it's kind of nice actually.)

Havoc 3y ago

The precise wording matters. How has it changed your work life leads people down the rabbit hole of will coding jobs be safe. This one is a lot more neutral/technical.

weebull 3y ago

If somebody can show me a coding task that LLMs have successfully done that isn't an interview question or a documentation snippet, I might start to value it. Spending huge amount of resource to be a bit better at autocompleting code doesn't have value to me. I want it to solve significant problems, and it's looking like it can't do it and scaling it to be able to is totally impractical. > In aggregate, training all 9 Code Llama models required 400K GPU hours of computation on hardware of type A100-80GB (TDP of 350-400W). That is: * 45⅔ GPU years * 160 MWh or... * 45 average UK homes annual electric consumption * 18 average US homes * 64 average drivers annual milage in an EV. ...and that's just the GPUs. Add on all the rest of the system (s).

regularfry 3y ago
In the grand scheme of things it's ancient history, but https://code-as-policies.github.io/ https://code-as-policies.github.io/ works by generating code then executing it. That's worth running at. The code generation in that paper was done on code-davinci-002, which is (or rather was - it's deprecated) a 15B GPT-3 model. I've not done it yet, but I'd expect the open source 7B code completion models to be able to replicate it by now.