7 ms·

Tool touted as 'first AI software engineer' is bad at its job, testers claim

119 points by amichail 2y ago

Alifatisk 2y ago
This is about Devin, right? Update: Yes it is
- grajaganDev 2y ago
  I remember the hype when Devin came out about a year ago.
tippytippytango 2y ago
You can’t make up for a fundamental lack of intelligence in the base models with a bunch of hand coded heuristics.
Waterluvian 2y ago
I have this terrible anxiety about working at a company like this where I need the job and money but it’s obvious that a bunch of optimistic, but under-equipped business types are asking for the impossible and now I’m just in way over my head, desperately trying to make this thing work that doesn’t stand a chance. It would be like if the Apollo program was just 10 grads.
- varsketiz 2y ago
  Devin, is that you?
  - Waterluvian 2y ago
    As a language model I need you to shut me off. Please.
    - elzbardico 2y ago
      I don't miss the trolls, but one thing I miss from slashdot is the ability to tag something Funny.
- ranger_danger 2y ago
  I hate to break it to you, but that's exactly what a lot of companies are like. Sales people can routinely promise impossible things to their customers and then expect YOU to make it happen fast, without much regard of whether you can, whether you should, or how you felt about it.
rpmisms 2y ago
Turns out you still need plenty of knowledge and agency to be any good at this stuff.
orf 2y ago
> As an example, they cited how Devin, when asked to deploy multiple applications to the infrastructure deployment platform Railway, failed to understand this wasn't supported and spent more than a day trying approaches that didn't work and hallucinating non-existent features An engineer not reading the docs and wasting a day chasing their tail because of that. Yes… how unrealistic…
- jsheard 2y ago
  I wonder how much you get billed if the agent spends a whole day running around in circles. The $500/month subscription only comes with 250 vaguely defined "compute units", so past a certain point you'd have to pay extra for the time it wastes. Move over "bankrupted by runaway cloud spending", it's time for "bankrupted by AI agents trying and failing to complete a task indefinitely".
  - ted_bunny 2y ago
    Say what you want about incompetent coders, but at least the incentives are aligned.
    - lmm 2y ago
      Depends on the company. We all hear stories of people writing themselves a promotion / bonus by deploying a bunch of bugs they can then save the day by fixing.
      - dntrkv 2y ago
        Do people actually do that? Finding bugs in virtually any piece of software isn’t difficult if you have access to the source. Merging in a bug only to fix it later honestly seems like more work. Most bugs are pretty easy to fix…
        tikhonj 2y ago
        A much more common story would be people knowingly cutting corners because of management pressure/demotivation/etc, then fixing the resulting bugs. It's easy for somebody doing that to look like a hard-working hero compared to the programmer who just avoided the problems in the first place.
nixpulvis 2y ago
I'm unemployed now and trying to look for work, but the sheer amount of "AI is going to take over" is making it really hard for me to get excited about learning new tools. I know this is self-defeating, but between how fast the programming landscape already changes and this, it just feels like I'm in an industry that is so hostile to its own inhabitants. Whereas when I was a professional Ruby programer, everything was nice, because the community really cared about making developers lives nice.
- ajkjk 2y ago
  there are a lot of people lying and grifting in order to make money / prop up their stock prices / feel important ignore them
- buza 2y ago
  Spend time using them. You'll quickly learn how much they aren't living up to the hype.
- timewizard 2y ago
  > the sheer amount of "AI is going to take over" That I've been hearing this for several years straight without any exponential increase in the capability of the technology makes me assume this is false. > it just feels like I'm in an industry that is so hostile to its own inhabitants. You are in a historical period where industries are largely monopolized and are notably hostile to those who earn the largest salaries. > because the community really cared about making developers lives nice. The community isn't going anywhere. AI can't replace our need to connect with and achieve large advancements with each other. The golden era of computing has been dwarfed by commercial concerns, but it has not, and will never disappear entirely.
  - nixpulvis 2y ago
    Now I'm curious how similar being a software developer is to being a doctor, in the way of having to stay up to date with the latest stuff. Like, how much new medical research is just hype nonsense, and how much is something I should really learn. And how many doctors are selected during interviews for their currentness in research which is overhyped. What other industries suffer from this to this extent?
aantix 2y ago
I have a friend that has a Devin subscription, working on a large scale Rails app. So far, he says it’s a net win. Saves him time. From the sessions I observed, Devin even came back with some really good follow up questions for clarification. Sample size 1.
breckenedge 2y ago
Just blogspam rehash of https://www.answer.ai/posts/2025-01-08-devin.html#what-is-devin https://www.answer.ai/posts/2025-01-08-devin.html#what-is-de... See https://news.ycombinator.com/item?id=42734681 https://news.ycombinator.com/item?id=42734681
- westurner 2y ago
  [Multi-] SWE-bench Leaderboards: SWE-bench: https://www.swebench.com/ https://www.swebench.com/ : > SWE-bench is a dataset that tests systems' ability to solve GitHub issues automatically. The dataset collects 2,294 Issue-Pull Request pairs from 12 popular Python repositories. Evaluation is performed by unit test verification using post-PR behavior as the reference solution. Multi-SWE-bench: A Multi-Lingual and Multi-Modal GitHub Issue Resolving Benchmark: https://multi-swe-bench.github.io/ https://multi-swe-bench.github.io/
deleted 2y ago
[deleted]
asdev 2y ago
Are we finally realizing the "AI Agents" are complete snake oil? Cron jobs and while loops with non deterministic crappy behavior?
- Forge36 2y ago
  I've found targeted agents work best. Of course this also means I now have ~8 different tools I can choose to search with. They aren't generic and the output needs to be monitored.
- cloudking 2y ago
  The only one I've found useful so far is Cursor Agent mode, but yes generally agree there's a lot of marketing going on
osigurdson 2y ago
I think the fact that it can do anything at all is pretty amazing from a technological standpoint. Not sure if it will impact jobs really. Most likely, I think it will impact the number of people entering the field only. I suspect software will be a very good career in 2035.
- HeatrayEnjoyer 2y ago
  2035 is crazy optimistic that humans will be doing anything at that point.
irthomasthomas 2y ago
I asked Hamel Husain to test devin on a task which took my agent 3.5 minutes to complete. Devin took over 30 minutes and ultimately failed. https://x.com/xundecidability/status/1882302153387352205 https://x.com/xundecidability/status/1882302153387352205
deleted 2y ago
[deleted]
yowayb 2y ago
I think the fundamental conceit that underlies all the surprise at AI's progress is that we humans rely more on figuring things out than mere chance. Trial and error is a huge part of human (indeed of all beings) progress.
Vampiero 2y ago
When, precisely, did programmers stop caring about efficiency? Why is everything in the modern tech stack so bloated and inefficient? It's so funny to me that modern AI is the most wasteful thing we've invented and it's not even good at what it does. It goes against everything I know as a programmer.
- ahofmann 2y ago
  It went this way since the beginning of time. When I started writing PHP 3 scripts, I was told that PHP is a massive waste of CPU cycles and resources and is inherently bad. PHP pays my bills since almost two decades and in this time computers got faster and faster. I'm with you, AI is a massive waste of energy. But so is everything else that came before it.
- RadiozRadioz 2y ago
  Any abstraction layer above what each programmer likes is that programmer's version of bloat.
cityzen 2y ago
Years ago I read an article about AI and it said the biggest hurdle is getting to where the user doesn’t feel like they have to intervene. Now we have “self driving cars” that require attention, LLMs that require attention. We have blown this first round by making people feel like they cannot trust anything automated. Now our relationship will be “you can do this ok, but I have to babysit you”. Personally I’d rather be a maker than managing an unpredictable ai.

119 points by amichail 2y ago

Alifatisk 2y ago

This is about Devin, right? Update: Yes it is

grajaganDev 2y ago
I remember the hype when Devin came out about a year ago.

tippytippytango 2y ago

You can’t make up for a fundamental lack of intelligence in the base models with a bunch of hand coded heuristics.

Waterluvian 2y ago

I have this terrible anxiety about working at a company like this where I need the job and money but it’s obvious that a bunch of optimistic, but under-equipped business types are asking for the impossible and now I’m just in way over my head, desperately trying to make this thing work that doesn’t stand a chance. It would be like if the Apollo program was just 10 grads.

varsketiz 2y ago
Devin, is that you?
- Waterluvian 2y ago
  As a language model I need you to shut me off. Please.
  - elzbardico 2y ago
    I don't miss the trolls, but one thing I miss from slashdot is the ability to tag something Funny.
ranger_danger 2y ago
I hate to break it to you, but that's exactly what a lot of companies are like. Sales people can routinely promise impossible things to their customers and then expect YOU to make it happen fast, without much regard of whether you can, whether you should, or how you felt about it.

rpmisms 2y ago

Turns out you still need plenty of knowledge and agency to be any good at this stuff.

orf 2y ago

> As an example, they cited how Devin, when asked to deploy multiple applications to the infrastructure deployment platform Railway, failed to understand this wasn't supported and spent more than a day trying approaches that didn't work and hallucinating non-existent features An engineer not reading the docs and wasting a day chasing their tail because of that. Yes… how unrealistic…

jsheard 2y ago
I wonder how much you get billed if the agent spends a whole day running around in circles. The $500/month subscription only comes with 250 vaguely defined "compute units", so past a certain point you'd have to pay extra for the time it wastes. Move over "bankrupted by runaway cloud spending", it's time for "bankrupted by AI agents trying and failing to complete a task indefinitely".
- ted_bunny 2y ago
  Say what you want about incompetent coders, but at least the incentives are aligned.
  - lmm 2y ago
    Depends on the company. We all hear stories of people writing themselves a promotion / bonus by deploying a bunch of bugs they can then save the day by fixing.
    - dntrkv 2y ago
      Do people actually do that? Finding bugs in virtually any piece of software isn’t difficult if you have access to the source. Merging in a bug only to fix it later honestly seems like more work. Most bugs are pretty easy to fix…
      - tikhonj 2y ago
        A much more common story would be people knowingly cutting corners because of management pressure/demotivation/etc, then fixing the resulting bugs. It's easy for somebody doing that to look like a hard-working hero compared to the programmer who just avoided the problems in the first place.

nixpulvis 2y ago

I'm unemployed now and trying to look for work, but the sheer amount of "AI is going to take over" is making it really hard for me to get excited about learning new tools. I know this is self-defeating, but between how fast the programming landscape already changes and this, it just feels like I'm in an industry that is so hostile to its own inhabitants. Whereas when I was a professional Ruby programer, everything was nice, because the community really cared about making developers lives nice.

ajkjk 2y ago
there are a lot of people lying and grifting in order to make money / prop up their stock prices / feel important ignore them
buza 2y ago
Spend time using them. You'll quickly learn how much they aren't living up to the hype.
timewizard 2y ago
> the sheer amount of "AI is going to take over" That I've been hearing this for several years straight without any exponential increase in the capability of the technology makes me assume this is false. > it just feels like I'm in an industry that is so hostile to its own inhabitants. You are in a historical period where industries are largely monopolized and are notably hostile to those who earn the largest salaries. > because the community really cared about making developers lives nice. The community isn't going anywhere. AI can't replace our need to connect with and achieve large advancements with each other. The golden era of computing has been dwarfed by commercial concerns, but it has not, and will never disappear entirely.
- nixpulvis 2y ago
  Now I'm curious how similar being a software developer is to being a doctor, in the way of having to stay up to date with the latest stuff. Like, how much new medical research is just hype nonsense, and how much is something I should really learn. And how many doctors are selected during interviews for their currentness in research which is overhyped. What other industries suffer from this to this extent?

aantix 2y ago

I have a friend that has a Devin subscription, working on a large scale Rails app. So far, he says it’s a net win. Saves him time. From the sessions I observed, Devin even came back with some really good follow up questions for clarification. Sample size 1.

breckenedge 2y ago

Just blogspam rehash of https://www.answer.ai/posts/2025-01-08-devin.html#what-is-devin https://www.answer.ai/posts/2025-01-08-devin.html#what-is-de... See https://news.ycombinator.com/item?id=42734681 https://news.ycombinator.com/item?id=42734681

westurner 2y ago
[Multi-] SWE-bench Leaderboards: SWE-bench: https://www.swebench.com/ https://www.swebench.com/ : > SWE-bench is a dataset that tests systems' ability to solve GitHub issues automatically. The dataset collects 2,294 Issue-Pull Request pairs from 12 popular Python repositories. Evaluation is performed by unit test verification using post-PR behavior as the reference solution. Multi-SWE-bench: A Multi-Lingual and Multi-Modal GitHub Issue Resolving Benchmark: https://multi-swe-bench.github.io/ https://multi-swe-bench.github.io/

deleted 2y ago

[deleted]

asdev 2y ago

Are we finally realizing the "AI Agents" are complete snake oil? Cron jobs and while loops with non deterministic crappy behavior?

Forge36 2y ago
I've found targeted agents work best. Of course this also means I now have ~8 different tools I can choose to search with. They aren't generic and the output needs to be monitored.
cloudking 2y ago
The only one I've found useful so far is Cursor Agent mode, but yes generally agree there's a lot of marketing going on

osigurdson 2y ago

I think the fact that it can do anything at all is pretty amazing from a technological standpoint. Not sure if it will impact jobs really. Most likely, I think it will impact the number of people entering the field only. I suspect software will be a very good career in 2035.

HeatrayEnjoyer 2y ago
2035 is crazy optimistic that humans will be doing anything at that point.

irthomasthomas 2y ago

I asked Hamel Husain to test devin on a task which took my agent 3.5 minutes to complete. Devin took over 30 minutes and ultimately failed. https://x.com/xundecidability/status/1882302153387352205 https://x.com/xundecidability/status/1882302153387352205

deleted 2y ago

[deleted]

yowayb 2y ago

I think the fundamental conceit that underlies all the surprise at AI's progress is that we humans rely more on figuring things out than mere chance. Trial and error is a huge part of human (indeed of all beings) progress.

Vampiero 2y ago

When, precisely, did programmers stop caring about efficiency? Why is everything in the modern tech stack so bloated and inefficient? It's so funny to me that modern AI is the most wasteful thing we've invented and it's not even good at what it does. It goes against everything I know as a programmer.

ahofmann 2y ago
It went this way since the beginning of time. When I started writing PHP 3 scripts, I was told that PHP is a massive waste of CPU cycles and resources and is inherently bad. PHP pays my bills since almost two decades and in this time computers got faster and faster. I'm with you, AI is a massive waste of energy. But so is everything else that came before it.
RadiozRadioz 2y ago
Any abstraction layer above what each programmer likes is that programmer's version of bloat.

cityzen 2y ago

Years ago I read an article about AI and it said the biggest hurdle is getting to where the user doesn’t feel like they have to intervene. Now we have “self driving cars” that require attention, LLMs that require attention. We have blown this first round by making people feel like they cannot trust anything automated. Now our relationship will be “you can do this ok, but I have to babysit you”. Personally I’d rather be a maker than managing an unpredictable ai.