4 ms·

I am willing to bet it won't be solved by 2028 and the betting market is overestimating AI capabilities and progress on abstract reasoning. No current AI on the

I am willing to bet it won't be solved by 2028 and the betting market is overestimating AI capabilities and progress on abstract reasoning. No current AI on the market can consistently synthesize code according to a logical specification and that is almost certainly a requirement for solving this benchmark.

sigmoid10 2y ago
What research are you basing this on? Because in particular fill in the middle and other non-standard approaches to code generation have shown incredible capability. I'm pretty sure by 2028 LLMs will be able to write code to specification better than most human programmers. Maybe not on the level of million line monolithic codebases that certain engineers worked on for decades, but smaller, modern projects for sure.
- benchmarkist 2y ago
  It's based on my knowledge of mathematics and software engineering. I have a graduate degree in math and I have written code for more than a decade in different startups across different domains ranging from solar power plants to email marketing.
  - sigmoid10 2y ago
    I've been actively researching in this field for close to a decade now, so let me tell you: Today is nothing like when I started. Back then everyone rightly assumed this kind of AI was decades if not centuries away. Nowadays there are still some open questions regarding the path to general intelligence, but even they are more akin to technicalities that will probably be solved on a time frame of years or perhaps even months. And expert systems are basically at the point where they can start taking over.
    - benchmarkist 2y ago
      What are the technicalities?
      - sigmoid10 2y ago
        Scaling up compute, creating and curating data (be it human or synthetically sourced) and more resilient benchmarking for example. But on the algorithmic side we already have a true general purpose, arbitrarily scalable, differentiable algorithm. So training it to do the right stuff is essentially the only missing ingredient. And models are catching up fast.
        benchmarkist 2y ago
        Wishful thinking. You can revisit this comment in 2028 and decide whether you were right or not.

sigmoid10 2y ago

What research are you basing this on? Because in particular fill in the middle and other non-standard approaches to code generation have shown incredible capability. I'm pretty sure by 2028 LLMs will be able to write code to specification better than most human programmers. Maybe not on the level of million line monolithic codebases that certain engineers worked on for decades, but smaller, modern projects for sure.

benchmarkist 2y ago
It's based on my knowledge of mathematics and software engineering. I have a graduate degree in math and I have written code for more than a decade in different startups across different domains ranging from solar power plants to email marketing.
- sigmoid10 2y ago
  I've been actively researching in this field for close to a decade now, so let me tell you: Today is nothing like when I started. Back then everyone rightly assumed this kind of AI was decades if not centuries away. Nowadays there are still some open questions regarding the path to general intelligence, but even they are more akin to technicalities that will probably be solved on a time frame of years or perhaps even months. And expert systems are basically at the point where they can start taking over.
  - benchmarkist 2y ago
    What are the technicalities?
    - sigmoid10 2y ago
      Scaling up compute, creating and curating data (be it human or synthetically sourced) and more resilient benchmarking for example. But on the algorithmic side we already have a true general purpose, arbitrarily scalable, differentiable algorithm. So training it to do the right stuff is essentially the only missing ingredient. And models are catching up fast.
      - benchmarkist 2y ago
        Wishful thinking. You can revisit this comment in 2028 and decide whether you were right or not.