3 ms·

Yeah, I've been suggesting this is likely coming since the launch of GPT-3.5. It seems clear at this point that in the future a lot of development work could be

by kypro 3y ago

Yeah, I've been suggesting this is likely coming since the launch of GPT-3.5. It seems clear at this point that in the future a lot of development work could be done by a LLM simply listening in on meetings / emails, then raising tickets based on those discussions, and then raising PRs for those tickets. For the time being humans will probably still want to approve those tickets and PRs, but I think in the future in would be quite unreasonable for a human to attempt to critic an AI. It would be like someone as incapable as Magnus Carlson critiquing StockFish's Chess moves. Whenever an AI has reached human-level competency in the past they've always exceeded it just a few years after. And GPT-4 seems to be nearing human-level today.

nmca 3y ago
It's important that humans are still able to understand the actions of these systems for safety reasons; the area focused on the problem you identify is "scalable oversight", see for example: https://arxiv.org/abs/2206.05802 https://arxiv.org/abs/2206.05802 https://arxiv.org/abs/2211.03540 https://arxiv.org/abs/2211.03540
- KyeRussell 3y ago
  Understanding them probably isn’t as important (or as likely) as someone- the right person-simply being held responsible for their actions, or for relying on them.
- kypro 3y ago
  Important for who? And what do you mean by "safety" in this context? Systems which can produce code unaided are obviously dangerous to humanity, but this isn't the concern of any single company. Businesses will do whatever makes sense for their business. If you're talking about safety in the context of cyber security, then where the stakes are high (say in the development of banking software) you will probably see slower adoption of automation, but even there it will quickly become absurd to believe that humans are more capable at producing reliable and safe software. No human can be have a real-time database of every known cyber security venerability in their head for example, but an LLM could. I'd argue GPT-4 already has super human coding knowledge, it's really just on the intelligence side of things (reasoning, planning, logical consistency) that they're lacking a bit.

by kypro 3y ago

nmca 3y ago

It's important that humans are still able to understand the actions of these systems for safety reasons; the area focused on the problem you identify is "scalable oversight", see for example: https://arxiv.org/abs/2206.05802 https://arxiv.org/abs/2206.05802 https://arxiv.org/abs/2211.03540 https://arxiv.org/abs/2211.03540

KyeRussell 3y ago
Understanding them probably isn’t as important (or as likely) as someone- the right person-simply being held responsible for their actions, or for relying on them.
kypro 3y ago
Important for who? And what do you mean by "safety" in this context? Systems which can produce code unaided are obviously dangerous to humanity, but this isn't the concern of any single company. Businesses will do whatever makes sense for their business. If you're talking about safety in the context of cyber security, then where the stakes are high (say in the development of banking software) you will probably see slower adoption of automation, but even there it will quickly become absurd to believe that humans are more capable at producing reliable and safe software. No human can be have a real-time database of every known cyber security venerability in their head for example, but an LLM could. I'd argue GPT-4 already has super human coding knowledge, it's really just on the intelligence side of things (reasoning, planning, logical consistency) that they're lacking a bit.