3 ms·

The article you linked is from Sept '24 and points to the ARC-AGI test as "evidence" that we're not getting close. We're in Feb '25. ARC-AGI (at least the vers

The article you linked is from Sept '24 and points to the ARC-AGI test as "evidence" that we're not getting close. We're in Feb '25. ARC-AGI (at least the version they're referencing) already has been solved by AI at above average human level. >everything points to incremental progress with signs of diminishing returns. Seems like everything in just Dec '24/ Jan '25 points the other way. These models are already helping PhDs in novel research, they're already getting super human at coding (yes yes, they're not perfect and I'm sure someone on HN has this weird coding job that AI can't replace yet and they're very excited to shit over AI), but they've already replaced a lot of real software dev jobs. Also aren't you contradicting yourself? > everything points to incremental progress with signs of diminishing returns > corporation replacing employees with AI If we have incremental progress, how are corporations going to replace employees with AI?

maleldil 2y ago
> getting super human at coding They're getting super-human at _competitive coding_, which is essentially identifying and writing algorithms. They _are not_ good at general coding, as demonstrated by their subpar scores at benchmarks like SWE-bench, and even those aren't particularly representative of what a real coding job is.
- raghavtoshniwal 2y ago
  >subpar scores at benchmarks like SWE-bench The last few models have remarkably improved on SWE-bench too. o3 scores 73%, this number was in the low teens 16 months ago. Willing to wager that SWE benchmark gets saturated before the end of 2025. > aren't particularly representative of what a real coding job I don't know about that, large swath of "real world" coding is writing plumbing and UIs for CRUD apps, they're getting really good at that as well. Anecdotally, engineers I know have gotten insanely productive with tools like Cursor.
fsndz 2y ago
to get the score they had on ARC AGI, they had to fine-tune o3 on ARC AGI. That is hardly a sign of general intelligence or emergent capability. PhD novel research ? What is the novel research discovered by an LLM ever since the emergence of ChatGPT ? None. Despite all the knowledge these models accumulate in their weights they haven't been able to connect the dots and discover a lot of things humans haven't discovered, autonomously. Replace which software engineering job ? They are useful, sure; good at benchmarks, yes; but not a drop in replacement of any software engineer.

maleldil 2y ago

> getting super human at coding They're getting super-human at _competitive coding_, which is essentially identifying and writing algorithms. They _are not_ good at general coding, as demonstrated by their subpar scores at benchmarks like SWE-bench, and even those aren't particularly representative of what a real coding job is.

raghavtoshniwal 2y ago
>subpar scores at benchmarks like SWE-bench The last few models have remarkably improved on SWE-bench too. o3 scores 73%, this number was in the low teens 16 months ago. Willing to wager that SWE benchmark gets saturated before the end of 2025. > aren't particularly representative of what a real coding job I don't know about that, large swath of "real world" coding is writing plumbing and UIs for CRUD apps, they're getting really good at that as well. Anecdotally, engineers I know have gotten insanely productive with tools like Cursor.

fsndz 2y ago

to get the score they had on ARC AGI, they had to fine-tune o3 on ARC AGI. That is hardly a sign of general intelligence or emergent capability. PhD novel research ? What is the novel research discovered by an LLM ever since the emergence of ChatGPT ? None. Despite all the knowledge these models accumulate in their weights they haven't been able to connect the dots and discover a lot of things humans haven't discovered, autonomously. Replace which software engineering job ? They are useful, sure; good at benchmarks, yes; but not a drop in replacement of any software engineer.