4 ms·

This has made me wonder just how few developers have been doing even remotely interesting things. I encounter a lot of situations where the LLMs catastrophicall

This has made me wonder just how few developers have been doing even remotely interesting things. I encounter a lot of situations where the LLMs catastrophically fail, but I don’t think it’s for lack of trying to guide it through to solutions properly or sufficiently. I give as much or as little context as I can think to, give partial solutions or zero solutions, try different LLMs with the same prompts, etc. Maybe we were mostly actually doing the exact same stuff and these things are trained on a whole lot of the same. Like with react components, these things are amazing at pumping out the most basic varieties of them.

uludag 2y ago
My thoughts exactly. I honestly think that the majority of SWE work is completely unoriginal. But then there are those idiosyncrasies that every company has, a finicky build system, strange hacks to get certain things to work, extensive webs of tribal knowledge, that will be the demise for any LLM SWE application besides being a tool for skilled professionals.
- bamboozled 2y ago
  The whole world is wrinkly, that's the way it is, everyone, and everything is slightly different.
bugglebeetle 2y ago
> This has made me wonder just how few developers have been doing even remotely interesting things. There’s not much to ponder on here as the majority of human thought and action is unoriginal and irrelevant, so software development is not some special exception to this. This doesn’t mean it lacks meaning or value for those doing it, since that can be self-generated, extrinsically motivated, or both. But, by definition, most things done are within the boundaries of a normal distribution and not exceptional, transcendent, or sublime. You can strive for those latter categories, but it’s a path strewn with much failure, misery, and quite questionable rewards, even where it is achieved. Most prefer more achievable comforts or security to it, or, at most, dull-minded imitations thereof, such as amassing great wealth.
josephg 2y ago
Yeah; I've long argued that software should be divided into two separate professions. If you think about it, lots of professions are already divided in two. You often have profession of inventors / designers and a profession of creators. For example: Electrical engineers vs Electricians. Architects / civil engineers and construction workers. Chefs and line cooks. Composers and session musicians. And so on. The two professions always need somewhat different skill sets. For example, electricians need business sense and to be personable - while electrical engineers need to understand how to design circuits from scratch. Session musicians need a fantastic memory for music, and to be able to perform in lots of different musical styles. Composers need a deeper understanding of how to write compelling music - but usually only in their preferred style. I think programming should be divided in the same way. It sort of happens already: There's people who invent novel database storage engines, work on programming language design, write schedulers for operating systems and so on. And then there's people who do consulting and write almost all the software that businesses actually need to function. Eg, people who talk to actual clients and figure out their needs. The first group invents React. The second group uses react to build online stores. We sort of have these two professions already. We just don't have language to separate them. The line is a bit fuzzy, but you can tell they're two professions because each needs a slightly different skill set. If you're writing a novel full-text search engine, you need to be really good at data structures & algorithms, but you don't really need to be good at working with clients. And if you work in frontend react all day, you don't really need to understand how a B-Tree works or understand the difference between LR and recursive descent parsing. (And it doesn't make much sense to ask these sort of programmers "leetcode problems" in job interviews). ChatGPT is good at application programming. Or really, anything that its seen lots of times before. But the more novel a problem - the more its actual "software engineering" - the more it struggles to make working software. "Write a react component that does X" - easy. "Implement a B-tree with this weird quirk" - hard. It'll struggle through but the result will be buggy. "Implement this new algorithm in a paper I just read" - its more or less completely useless at that.
- mwcampbell 2y ago
  Steve Yegge wrote about this ~20 years ago, with a heaping helping of his usual snark directed at the OO fads of the time: https://sites.google.com/site/steveyegge2/age-racecar-driver https://sites.google.com/site/steveyegge2/age-racecar-driver
lm28469 2y ago
Imho the hordes of code monkeys see an incredible improvement in their productivity and keep boasting about it online which is inflating the idea that LLMs are close to agi and revolutionizing the industry Meanwhile people who work on things that are even slightly niche or complex (ie. Things which aren't answered in 20 different stack overflow threads, or things which require of few field "experts" to meet around a table and think hard for a few hours) don't understand the enthusiasm at all I'm working on fairly simple things but in a context/industry that isn't that common and I have to spoon feed LLMs to get anywhere, most of the time it doesn't even work. The next few years will be interesting, it might play out as it did for plane pilots over relying on autopilots to the point of losing key skills
- theK 2y ago
  Another aspect of this might be what I'm experiencing. I'm currently increasingly often confronted with overly bloated codebases that, with the added speed of llms / coding assistants, have reached levels of unmaintainability much faster than traditionally possible. Now, fixing those repos with LLMs fails miserably in my experience, so you have to invest hundreds of hours of senior engineer time to make the projects workable again, regardless of the novelty of the problem solved by that codebase. It also frequently makes the engineers pulled in to help miserable.
SkyBelow 2y ago
Is it that they aren't doing interesting things, or is it that developers doing interesting things tend to only ask the AI to handle the boring parts? I find I use AI quite a bit, but it is integrated with my own thinking and problem solving every step of the way. I see how others are using it and they often seem to be asking for full solutions, where as I'm using it more like a rubber duck debugger stuffed with stack overflow feedback and a knowledge of any common APIs. It mostly helps be answering things that would have otherwise been web searches, but it doesn't create any solutions from scratch.

uludag 2y ago

My thoughts exactly. I honestly think that the majority of SWE work is completely unoriginal. But then there are those idiosyncrasies that every company has, a finicky build system, strange hacks to get certain things to work, extensive webs of tribal knowledge, that will be the demise for any LLM SWE application besides being a tool for skilled professionals.

bamboozled 2y ago
The whole world is wrinkly, that's the way it is, everyone, and everything is slightly different.

bugglebeetle 2y ago

> This has made me wonder just how few developers have been doing even remotely interesting things. There’s not much to ponder on here as the majority of human thought and action is unoriginal and irrelevant, so software development is not some special exception to this. This doesn’t mean it lacks meaning or value for those doing it, since that can be self-generated, extrinsically motivated, or both. But, by definition, most things done are within the boundaries of a normal distribution and not exceptional, transcendent, or sublime. You can strive for those latter categories, but it’s a path strewn with much failure, misery, and quite questionable rewards, even where it is achieved. Most prefer more achievable comforts or security to it, or, at most, dull-minded imitations thereof, such as amassing great wealth.

josephg 2y ago

Yeah; I've long argued that software should be divided into two separate professions. If you think about it, lots of professions are already divided in two. You often have profession of inventors / designers and a profession of creators. For example: Electrical engineers vs Electricians. Architects / civil engineers and construction workers. Chefs and line cooks. Composers and session musicians. And so on. The two professions always need somewhat different skill sets. For example, electricians need business sense and to be personable - while electrical engineers need to understand how to design circuits from scratch. Session musicians need a fantastic memory for music, and to be able to perform in lots of different musical styles. Composers need a deeper understanding of how to write compelling music - but usually only in their preferred style. I think programming should be divided in the same way. It sort of happens already: There's people who invent novel database storage engines, work on programming language design, write schedulers for operating systems and so on. And then there's people who do consulting and write almost all the software that businesses actually need to function. Eg, people who talk to actual clients and figure out their needs. The first group invents React. The second group uses react to build online stores. We sort of have these two professions already. We just don't have language to separate them. The line is a bit fuzzy, but you can tell they're two professions because each needs a slightly different skill set. If you're writing a novel full-text search engine, you need to be really good at data structures & algorithms, but you don't really need to be good at working with clients. And if you work in frontend react all day, you don't really need to understand how a B-Tree works or understand the difference between LR and recursive descent parsing. (And it doesn't make much sense to ask these sort of programmers "leetcode problems" in job interviews). ChatGPT is good at application programming. Or really, anything that its seen lots of times before. But the more novel a problem - the more its actual "software engineering" - the more it struggles to make working software. "Write a react component that does X" - easy. "Implement a B-tree with this weird quirk" - hard. It'll struggle through but the result will be buggy. "Implement this new algorithm in a paper I just read" - its more or less completely useless at that.

mwcampbell 2y ago
Steve Yegge wrote about this ~20 years ago, with a heaping helping of his usual snark directed at the OO fads of the time: https://sites.google.com/site/steveyegge2/age-racecar-driver https://sites.google.com/site/steveyegge2/age-racecar-driver

lm28469 2y ago

Imho the hordes of code monkeys see an incredible improvement in their productivity and keep boasting about it online which is inflating the idea that LLMs are close to agi and revolutionizing the industry Meanwhile people who work on things that are even slightly niche or complex (ie. Things which aren't answered in 20 different stack overflow threads, or things which require of few field "experts" to meet around a table and think hard for a few hours) don't understand the enthusiasm at all I'm working on fairly simple things but in a context/industry that isn't that common and I have to spoon feed LLMs to get anywhere, most of the time it doesn't even work. The next few years will be interesting, it might play out as it did for plane pilots over relying on autopilots to the point of losing key skills

theK 2y ago
Another aspect of this might be what I'm experiencing. I'm currently increasingly often confronted with overly bloated codebases that, with the added speed of llms / coding assistants, have reached levels of unmaintainability much faster than traditionally possible. Now, fixing those repos with LLMs fails miserably in my experience, so you have to invest hundreds of hours of senior engineer time to make the projects workable again, regardless of the novelty of the problem solved by that codebase. It also frequently makes the engineers pulled in to help miserable.

SkyBelow 2y ago

Is it that they aren't doing interesting things, or is it that developers doing interesting things tend to only ask the AI to handle the boring parts? I find I use AI quite a bit, but it is integrated with my own thinking and problem solving every step of the way. I see how others are using it and they often seem to be asking for full solutions, where as I'm using it more like a rubber duck debugger stuffed with stack overflow feedback and a knowledge of any common APIs. It mostly helps be answering things that would have otherwise been web searches, but it doesn't create any solutions from scratch.