3 ms·

I feel that a lot of the "here's what LLMs can't do, and why humans are still king" posts are missing: What kind of humans are we comparing LLMs against? Juni

I feel that a lot of the "here's what LLMs can't do, and why humans are still king" posts are missing: What kind of humans are we comparing LLMs against? Junior devs? Intermediate devs? Senior devs? Experts? I think few jr. devs would be able to apply the optimizations that were in the blog post. So maybe already LLMs are performing on a level that is beyond human jr. devs, and quickly catching up with the next tier.

th0ma5 2y ago
For me, the show stopper is that every single token is suspect. Adding layers of more of it compounds this suspicion, leading to fatigue and automation bias, amplifying the speed of error or technical debt while possibly even acting against the understanding and control of stakeholders. They are actually in my mind far above human skill, actually, and far below basic microbes in basic existing and engagement of course. I think the true societal task now is to give names to these non-alive qualities and risks, the echo chamber aspects, what the total information of a model plus a generated result is, and things like X/Y problems and unknown unknowns.
n4r9 2y ago
> LLMs are performing on a level that is beyond human jr. devs, and quickly catching up with the next tier. I am a little more pessimistic. I would say that LLMs have been hovering above junior for a year or two, with some progress towards intermediate in the context of a few methods/classes, but not much evidence of the wider architectural/systems-level thinking that I'd expect from an intermediate.
dgacmu 2y ago
It's a good question. I want to think that most of the students we graduate could do this (once they learned Rust, as it's not part of our curriculum, but they could do it in C or Java or something). But our (CMU CS) students get a fair bit of algorithmic experience, and I'm not sure how you're defining "junior dev" (and the experience of a junior dev probably varies widely given the many different paths to get there).
- cma 2y ago
  CMU is a top school in CS, like tied for #1 with MIT. Probably puts out 99th percentile junior devs.
onlyrealcuzzo 2y ago
I think a lot of your posts are missing that someone's got to tell LLMs what to do. What kind of humans are doing that? How much more productive is it making them?

th0ma5 2y ago

For me, the show stopper is that every single token is suspect. Adding layers of more of it compounds this suspicion, leading to fatigue and automation bias, amplifying the speed of error or technical debt while possibly even acting against the understanding and control of stakeholders. They are actually in my mind far above human skill, actually, and far below basic microbes in basic existing and engagement of course. I think the true societal task now is to give names to these non-alive qualities and risks, the echo chamber aspects, what the total information of a model plus a generated result is, and things like X/Y problems and unknown unknowns.

n4r9 2y ago

> LLMs are performing on a level that is beyond human jr. devs, and quickly catching up with the next tier. I am a little more pessimistic. I would say that LLMs have been hovering above junior for a year or two, with some progress towards intermediate in the context of a few methods/classes, but not much evidence of the wider architectural/systems-level thinking that I'd expect from an intermediate.

dgacmu 2y ago

It's a good question. I want to think that most of the students we graduate could do this (once they learned Rust, as it's not part of our curriculum, but they could do it in C or Java or something). But our (CMU CS) students get a fair bit of algorithmic experience, and I'm not sure how you're defining "junior dev" (and the experience of a junior dev probably varies widely given the many different paths to get there).

cma 2y ago
CMU is a top school in CS, like tied for #1 with MIT. Probably puts out 99th percentile junior devs.

onlyrealcuzzo 2y ago

I think a lot of your posts are missing that someone's got to tell LLMs what to do. What kind of humans are doing that? How much more productive is it making them?