9 ms·

I think one of the big problems with Devin (and AI agents in general) is that they're only ever as good as they are. Sometimes their intelligence feels magical

by xmprt 2y ago

I think one of the big problems with Devin (and AI agents in general) is that they're only ever as good as they are. Sometimes their intelligence feels magical and they accomplish things within minutes that even mid level or senior software engineers would take a few hours to do. Other times, they make simple mistakes and no matter how much help you give, they run around in circles. A big quality that I value in junior engineers is coachability. If an AI agent can't be coached (and it doesn't look like it right now), then there's no way I'll ever enjoy using one.

marcyb5st 2y ago
I completely agree with you. More precisely, I feel they are useful when you have specific tasks with limited scope. For instance, just yesterday I was battling with a complex SQL query and I got halfway there. I gave our bot the query and an half assed description of what I wanted/what was missing and it got it right on the first try.
- datadrivenangel 2y ago
  Are you sure that your SQL query is correct?
  - QuadmasterXLII 2y ago
    he’s certainly sure, but lord knows if it is
  - xboxnolifes 2y ago
    Can they be sure even if they wrote it theirself?
ipnon 2y ago
My first job I spent so much time reading Python docs, and the ancient art of Stack Overflow spelunking. But I could intuitively explain a solution in seconds because of my CS background. I used to encounter a certain kind of programmer often, who did not understand algorithms well but had many years of experience with a language like Ruby, and thus was faster in completing tasks because they didn't need to do the reference work that I had to do. Now I think these kinds of programmers will slowly disappear and only the ones with the fast CS intuition will remain.
- halfmatthalfcat 2y ago
  I disagree. If anything, CS degrees have proven time and time again they aren't translatable into software development (which is why there's an entire degree field called Software Engineering emerging). If anything, my gut says that the CS concepts are very easy for LLMs to recall and will be the first things replaced (if ever) by AI. Software engineer{ing,s} (project construction, integrations, scaling, organizational/external factors, etc) will stick around for along time. There's also the meme in the industry that self-taught, non-CS degree engineers are potentially of the most capable group. Though this is anecdotal.
  - ben_w 2y ago
    > If anything, CS degrees have proven time and time again they aren't translatable into software development (which is why there's an entire degree field called Software Engineering emerging). Emerging? I graduated in 2006 with a BEng in Software Engineering. The difference between it and the BSc CompSci degree I started in, was that optional modules became mandatory — including an industrial placement year (paid internship). > Software engineer{ing,s} (project construction, integrations, scaling, organizational/external factors, etc) will stick around for along time. My gut disagrees, because LLMs are at about the same level in those things as they are in low level coding: not yet replacing humans in project level tasks any more than they do in coding tasks, but also being OK assistants for both coding and project domains. I have no reason to think either has less or more opportunity for self-training, so I expect progress to track for the foreseeable future. (That said, the foreseeable future in this case is 1-2 years).
  - viraptor 2y ago
    > the CS concepts are very easy for LLMs to recall They're easy to recall, but you have to know what to recall in the first place. Or even know enough of the territory to realise there's something to recall. Without enough background, you'll get a whole set of amazing tools that you have no idea what to do with. For example, you may be able to write a long description of your problem with some ideas how to steer the AI to give you possible solutions. And the AI may figure out what the problem is and that the hyperloglog is something that could be useful to you. And you may have the awesome programming skills to implement that. But that's a lot of maybes. It would be much faster/easier if you knew about hyperloglog ahead of time and just asked for the implementation or library recommendation. Or even if you don't know about the actual solution, you'd have enough of CS vocabulary to ask: "how do I get a fast, approximate distinct count from a multiset". It would take a long imprecise description to get the same thing for a coder with no theory background.
kkaatii 2y ago
And when working with people it's fairly easy to intervene and improve when needed. I think the current working model with LLMs is definitely suboptimal when we cannot confine their solution space AND where they should apply a solution precisely, and timely.
- llamaimperative 2y ago
  It’s also often possible to know what a human will be bad at before they start. This allows you to delegate tasks better or vary the level of pre-work you do before getting started. This is pretty unpredictable with LLMs still.
  - deleted 2y ago
    [deleted]
deleted 2y ago
[deleted]

by xmprt 2y ago

marcyb5st 2y ago

I completely agree with you. More precisely, I feel they are useful when you have specific tasks with limited scope. For instance, just yesterday I was battling with a complex SQL query and I got halfway there. I gave our bot the query and an half assed description of what I wanted/what was missing and it got it right on the first try.

datadrivenangel 2y ago
Are you sure that your SQL query is correct?
- QuadmasterXLII 2y ago
  he’s certainly sure, but lord knows if it is
- xboxnolifes 2y ago
  Can they be sure even if they wrote it theirself?

ipnon 2y ago

My first job I spent so much time reading Python docs, and the ancient art of Stack Overflow spelunking. But I could intuitively explain a solution in seconds because of my CS background. I used to encounter a certain kind of programmer often, who did not understand algorithms well but had many years of experience with a language like Ruby, and thus was faster in completing tasks because they didn't need to do the reference work that I had to do. Now I think these kinds of programmers will slowly disappear and only the ones with the fast CS intuition will remain.

halfmatthalfcat 2y ago
I disagree. If anything, CS degrees have proven time and time again they aren't translatable into software development (which is why there's an entire degree field called Software Engineering emerging). If anything, my gut says that the CS concepts are very easy for LLMs to recall and will be the first things replaced (if ever) by AI. Software engineer{ing,s} (project construction, integrations, scaling, organizational/external factors, etc) will stick around for along time. There's also the meme in the industry that self-taught, non-CS degree engineers are potentially of the most capable group. Though this is anecdotal.
- ben_w 2y ago
  > If anything, CS degrees have proven time and time again they aren't translatable into software development (which is why there's an entire degree field called Software Engineering emerging). Emerging? I graduated in 2006 with a BEng in Software Engineering. The difference between it and the BSc CompSci degree I started in, was that optional modules became mandatory — including an industrial placement year (paid internship). > Software engineer{ing,s} (project construction, integrations, scaling, organizational/external factors, etc) will stick around for along time. My gut disagrees, because LLMs are at about the same level in those things as they are in low level coding: not yet replacing humans in project level tasks any more than they do in coding tasks, but also being OK assistants for both coding and project domains. I have no reason to think either has less or more opportunity for self-training, so I expect progress to track for the foreseeable future. (That said, the foreseeable future in this case is 1-2 years).
- viraptor 2y ago
  > the CS concepts are very easy for LLMs to recall They're easy to recall, but you have to know what to recall in the first place. Or even know enough of the territory to realise there's something to recall. Without enough background, you'll get a whole set of amazing tools that you have no idea what to do with. For example, you may be able to write a long description of your problem with some ideas how to steer the AI to give you possible solutions. And the AI may figure out what the problem is and that the hyperloglog is something that could be useful to you. And you may have the awesome programming skills to implement that. But that's a lot of maybes. It would be much faster/easier if you knew about hyperloglog ahead of time and just asked for the implementation or library recommendation. Or even if you don't know about the actual solution, you'd have enough of CS vocabulary to ask: "how do I get a fast, approximate distinct count from a multiset". It would take a long imprecise description to get the same thing for a coder with no theory background.

kkaatii 2y ago

And when working with people it's fairly easy to intervene and improve when needed. I think the current working model with LLMs is definitely suboptimal when we cannot confine their solution space AND where they should apply a solution precisely, and timely.

llamaimperative 2y ago
It’s also often possible to know what a human will be bad at before they start. This allows you to delegate tasks better or vary the level of pre-work you do before getting started. This is pretty unpredictable with LLMs still.
- deleted 2y ago
  [deleted]

deleted 2y ago

[deleted]