5 ms·

How to Automate Software Engineering

27 points by Tamaybes 1y ago

rvz 1y ago
Another thing to keep in mind is that it is also possible that many startups attempting to fully automate software engineering roles will die / bankrupt or shut down trying before it 'eventually' happens; whenever that is. builder.ai was the first casualty claiming to use AI to replace software engineering teams to build products. Now bankrupt. [0] It could cost a lot more money than initially estimated than other industries if (by their own admission) it is the last profession to be automated. Let's see in the next 10 years if some of these 'startups' are still around on this 'mission' and whether if software engineering will be fully automated and dead for humans in the next decade which that is the narrative that they are all hyping for more VC money. [0] https://finance.yahoo.com/news/builder-ais-shocking-450m-fall-170009323.html https://finance.yahoo.com/news/builder-ais-shocking-450m-fal...
deleted 1y ago
[deleted]
sublinear 1y ago
This "problem" is pathological and those selling solutions are misleading those who buy into it. Software engineering often isn't just answering "how" to accomplish something, but also all the five other questions starting with 'w'. Software ate the world a very long time ago, so people being allergic to code while trying to run a business is only sounding more and more absurd as time marches on. What are we really trying to solve for here? All that could be automated reasonably well has been already. In most cases you do not want a stochiastic result, but exact code to be reused. We use libraries in our code and we have reproducibility of results. The code that needs to be written for most applications is minimal and only grows large as the business refines what they want. This code is unique and virtually worthless to any other business. The code mirrors the organization. We already know all this for decades. It's very confusing to me to keep hearing about this insistence that we need to automate software engineering.
- esseph 1y ago
  Maximum movement of capital to as few individuals as possible
  - sublinear 1y ago
    That's totally achievable now and in the recent past. Write the code yourself? Several billionaires have done this.
    - codr7 1y ago
      I suspect several wannabe billionaires couldn't write code even if their life depended on it, which explains why they're so enthusiastic about AI. And why they have been so enthusiastic about every other broken promise about replacing developers that came before it.
- MangoToupe 1y ago
  > Software ate the world a very long time ago, so people being allergic to code while trying to run a business is only sounding more and more absurd as time marches on. I don't agree with this at all. Part of the pitch of code is packing really nasty semantics into a single interface. This inherently reflects in the services we provide to clients. This interface should naturally correspond to UI, if that's what you offer. If you can only express the interface for your service in code, you've failed. Unless of course you primarily operate a code-centric product, like an sdk.
  - antithesizer 1y ago
    If your car can't be owned and operated by someone with zero knowledge of how cars work you've failed. And yet to this day every car owner is aware that their lack of any such knowledge would/does cost them dearly at the mechanic and at the dealership too. And of course mechanics and car salesmen love to see such clients coming their way.
    - 1y ago
ctoth 1y ago
> Once this occurs, many software engineers could perhaps transition into adjacent positions that rely on similar expertise but are significantly harder to automate, such as software engineering management, product management, or executive leadership within software companies. > In these roles, their responsibilities would shift from writing code and debugging to higher-level oversight, decision-making, and strategic planning—until these responsibilities can be automated too. When they spoke on Dwarkesh's podcast they seemed to think this would take 30 years. Not sure why we coders are so quick to be automated but the rest aren't. https://www.dwarkesh.com/p/ege-tamay https://www.dwarkesh.com/p/ege-tamay
- rvz 1y ago
  We really have to STOP listening to these people and see that they ALL have a vested interest. In the third paragraph of the podcast Dwarkesh already told you that he already invested. >>> (disclosure - I’m an angel investor), At that point the co-founders and Dwarkesh himself will agree on everything and they will say anything to get more VC money - Even if the timelines are unrealistic. (Because that is the scam).
- gcanyon 1y ago
  > 30 years That's...ridiculous. I'm a product manager, and AI is already chipping away at my job. Two months ago I said to one of my devs, "Our dashboard here looks very bland. What if we had a more visual display of the pipeline statuses across the top of the screen?" He said he thought that was a good idea, and I went to lunch. I came back and started sketching up some ideas for how to lay out the statuses. I had barely gotten started with that when he called me over to show me what cursor had come up with when asked: it was better than what I was sketching, for sure. We're (white collar work) going to be 90% automated in less than ten years, and I feel like I'm being conservative saying ten years.
  - Amekedl 1y ago
    You occasionally do glimpse behind the curtains; depending on what you actually develop it's feasible and quick to prompt it, but attempting to go further than that across multiple components collapses so drastically that I cannot help but feel that all ai stuff is entirely incapable of replicating the real thing at the moment.
dasil003 1y ago
> The roadmap to success will most likely start with training or fine-tuning on data from human professionals performing the task, and proceed with reinforcement learning in custom environments designed to capture more of the complexity of what people do in their jobs. > [...] > We think this is essentially a data problem, not an algorithms problem. This is extremely hand-wavy. How are you going to instrument the various thought processes and non-verbal communication that goes into building successful software? A huge part of it is intuition about what makes sense to other humans. It's related to the idea of common sense, but in the software world there's this layer of unforgiving determinism and rigidity that most humans don't want to deal with. I just don't see how AI crosses that chasm.
- ege_erdil 1y ago
  how do you think humans cross that chasm?
  - AnimalMuppet 1y ago
    Pretty sure that however humans crossed it, it wasn't just "a data problem".
    - ege_erdil 1y ago
      what makes you sure about that?
      - AnimalMuppet 1y ago
        At the risk of coming across as a smart aleck, 40 years of experience building software. What experienced software engineers have is a sense of taste - this looks like good code and/or design, that doesn't. But they don't have data; they have, at best, a couple of anecdotes. It's more a sense of "that was harder to work with than it should have been; that approach seems to have drawbacks". But you only get a few examples of that in a career. And there are very few outfits compiling usable data that could shape the approaches that software engineers use. So I don't think how humans got there was primarily data.
aurizon 1y ago
I can see AI used for permutational hacking, sort of like the fuzzing types of SW and HW hacking. Where they succeed can be a very good stress test. It can be done on human code as well as AI code and I expect this to be done. It is increasingly apparent that what an AI made and then hacked, still needs the abilities that only humans have - for now
jes5199 1y ago
great, let's also record all of the daily interactions of the C-suite and see if we can replicate their job functions in simulation
jongjong 1y ago
I think the conclusion about software engineering having the potential of being both the first and last job to be automated by AI makes sense. It is an extremely unequal profession. The skill difference between a junior with 1 year of experience and a senior with 20 years of both diverse and deep experience (of relentless striving) is massive. They have very different capabilities. The level of software sophistication/complexity they can attain is very different. The code produced looks very different. This is not a profession where you've figured it all out after 5 years and can rest on your laurels. I think it takes literally 15+ years for most highly determined and intelligent people to even approach that state where they can manage complexity effectively. Most people never seem to get there.
- martin-t 1y ago
  Almost 15 years in and I feel like I am still just scratching the surface. But most of the lessons learned during that time go towards quality, not quantity or speed. The current trend with LLMs (and eventually maybe AI) seems to be doing what humans can do worse but significantly faster and cheaper. Unfortunately, not everyone needs or cares about safety, security or correctness. I am afraid software will mirror the evolution of physical products from the industrial revolution to present day. It feels like the quality of consumer grade products is constantly decreasing.
  - jongjong 1y ago
    This resonates. Quality of mass market products decrease to the upper limit of the average person's perception/expectations... Which itself seems to be declining over time.
- Amekedl 1y ago
  Even then software constantly evolves, and rot is everywhere. And we're far from having the "best possible" software solution in literally every area (if that's even possible to measure), rather just endless room for improvement. And I don't see it being improved with whatever any llm chugs out, at least not "in-depth".
hnthrow90348765 1y ago
Alright, this is it, this is the AI peak. >Today we’re announcing Mechanize, a startup focused on developing virtual work environments, benchmarks, and training data that will enable the full automation of the economy. >Compensation >$200K – $475K • 1% – 2% Just imagine for a moment you are a software engineer capable of doing what they say, rivaling the raw intellectual capabilities of every mathematician, economist, and physicist known to man, and you end up actually building something that directly leads to automating the entire fucking economy of the world. And there's still someone who says to you: "best I can do is only 1% - 2%"
- handfuloflight 1y ago
  Oh I thought the peak was going to be based on the forward looking claims, not the standard equity offer.
- martin-t 1y ago
  One of the dumbest things a smart person can do is work for an AI company. At best they don't believe actual AI will be created and they are helping a scam. At worst, they are actively working to make their own job redundant and when they're fired, they will own nothing of what they built. All the money from their work will go to the owners who fired them.
  - dsjoerg 1y ago
    > At worst, they are actively working to make their own job redundant and when they're fired, they will own nothing of what they built. All the money from their work will go to the owners who fired them. Except that's the case with literally every meaningful information-work job. Your goal is to obsolete yourself. If you do it well then your success becomes your calling card for your next job. Your career is a series of such jobs.
    - martin-t 1y ago
      1) You can and should program yourself out of a _task_, not a _job_. How many companies have a one.time need for a programmer? 2) Even if your claim was correct, then you'd amass experience on one job (task) that you could then leverage on the next job (task). If AI became reality, all that would become irrelevant. Your middle-class self would quickly realize you're only worth as much as your lower class neighbors who swing a shovel or flip burgers for a living, all the while your upper class bosses become richer thanks to your work.
nitwit005 1y ago
Never before have I seen someone list a market size in the tens of trillions, as they do on their main page: > The market potential here is absurdly large: workers in the US are paid around $18 trillion per year in aggregate. For the entire world, the number is over three times greater, around $60 trillion per year. https://www.mechanize.work/ https://www.mechanize.work/
- handfuloflight 1y ago
  It's what you do when you fancy yourself in the leagues of Berkshire Hathaway.
gerdesj 1y ago
"With every passing month, AI models get better at most tasks that a software engineer does in their job." No they don't.
satisfice 1y ago
This guy has not even begun to talk about this in a serious way. If you try to talk about automating SE and you aren’t clearly explaining how people who know nothing about engineering will directly interact with this automation to get what they need, then you aren’t saying anything. Even with vibe coding there is a certain skillset involved.
belZaah 1y ago
Y’all need some Brooks in your lives. The No Silver Bullet essay explains very clearly, why the ability to string syntactically correct commands together does not equal the ability to write code.
- owebmaster 1y ago
  Artificial intelligence. Many people expect advances in artificial intelligence to provide the revolutionary breakthrough that will give order-of-magnitude gains in software productivity and quality. I do not. To see why, we must dissect what is meant by “artificial intelligence” and then see how it applies. Parnas has clarified the terminological chaos: Two quite different definitions of AI are in common use today. AI-1: The use of computers to solve problems that previously could only be solved by applying human intelligence. AI-2: The use of a specific set of programming techniques knows as heuristic or rule-based programming. In this approach human experts are studies to determine what heuristics or rules of thumb they use in solving problems. . . . The program is designed to solve a problem the way that humans seem to solve it. The first definition has a sliding meaning. . . . Something can fit the definition of AI-1 today but, once we see how the program works and understand the problem, we will not think of it as AI anymore. . . . Unfortunately I cannot identify a body of technology that is unique to this field. . . . Most of the work is problem-specific, and some abstraction or creativity is require to see how to transfer it great read: https://worrydream.com/refs/Brooks_1986_-_No_Silver_Bullet.pdf https://worrydream.com/refs/Brooks_1986_-_No_Silver_Bullet.p...
mattgreenrocks 1y ago
OpenAI spent several years prior to ChatGPT getting AI to play Dota 2 well. They got some good results out of that, but it was a subset of the game: only a handful or two of characters. I’m not sure why they stopped; maybe that was when they pivoted to something more general? Regardless, software dev I consider way more dimensional (eg nuanced) than Dota 2, even if a lot of patterns recur on a smaller scale in the code itself. If they weren’t able to crack Dota 2, why should I believe that software eng is just around the corner?
- ege_erdil 1y ago
  we don't think it's just around the corner
jackb4040 1y ago
> The key question now is: what data do we need, exactly? What if I told you that's not the key question, and the "more data" approach has obviously and publicly hit a wall that requires causal reasoning to move past?
- ege_erdil 1y ago
  then i would disagree

27 points by Tamaybes 1y ago

rvz 1y ago

Another thing to keep in mind is that it is also possible that many startups attempting to fully automate software engineering roles will die / bankrupt or shut down trying before it 'eventually' happens; whenever that is. builder.ai was the first casualty claiming to use AI to replace software engineering teams to build products. Now bankrupt. [0] It could cost a lot more money than initially estimated than other industries if (by their own admission) it is the last profession to be automated. Let's see in the next 10 years if some of these 'startups' are still around on this 'mission' and whether if software engineering will be fully automated and dead for humans in the next decade which that is the narrative that they are all hyping for more VC money. [0] https://finance.yahoo.com/news/builder-ais-shocking-450m-fall-170009323.html https://finance.yahoo.com/news/builder-ais-shocking-450m-fal...

deleted 1y ago

[deleted]

sublinear 1y ago

This "problem" is pathological and those selling solutions are misleading those who buy into it. Software engineering often isn't just answering "how" to accomplish something, but also all the five other questions starting with 'w'. Software ate the world a very long time ago, so people being allergic to code while trying to run a business is only sounding more and more absurd as time marches on. What are we really trying to solve for here? All that could be automated reasonably well has been already. In most cases you do not want a stochiastic result, but exact code to be reused. We use libraries in our code and we have reproducibility of results. The code that needs to be written for most applications is minimal and only grows large as the business refines what they want. This code is unique and virtually worthless to any other business. The code mirrors the organization. We already know all this for decades. It's very confusing to me to keep hearing about this insistence that we need to automate software engineering.

esseph 1y ago
Maximum movement of capital to as few individuals as possible
- sublinear 1y ago
  That's totally achievable now and in the recent past. Write the code yourself? Several billionaires have done this.
  - codr7 1y ago
    I suspect several wannabe billionaires couldn't write code even if their life depended on it, which explains why they're so enthusiastic about AI. And why they have been so enthusiastic about every other broken promise about replacing developers that came before it.
MangoToupe 1y ago
> Software ate the world a very long time ago, so people being allergic to code while trying to run a business is only sounding more and more absurd as time marches on. I don't agree with this at all. Part of the pitch of code is packing really nasty semantics into a single interface. This inherently reflects in the services we provide to clients. This interface should naturally correspond to UI, if that's what you offer. If you can only express the interface for your service in code, you've failed. Unless of course you primarily operate a code-centric product, like an sdk.
- antithesizer 1y ago
  If your car can't be owned and operated by someone with zero knowledge of how cars work you've failed. And yet to this day every car owner is aware that their lack of any such knowledge would/does cost them dearly at the mechanic and at the dealership too. And of course mechanics and car salesmen love to see such clients coming their way.
  - 1y ago

ctoth 1y ago

> Once this occurs, many software engineers could perhaps transition into adjacent positions that rely on similar expertise but are significantly harder to automate, such as software engineering management, product management, or executive leadership within software companies. > In these roles, their responsibilities would shift from writing code and debugging to higher-level oversight, decision-making, and strategic planning—until these responsibilities can be automated too. When they spoke on Dwarkesh's podcast they seemed to think this would take 30 years. Not sure why we coders are so quick to be automated but the rest aren't. https://www.dwarkesh.com/p/ege-tamay https://www.dwarkesh.com/p/ege-tamay

rvz 1y ago
We really have to STOP listening to these people and see that they ALL have a vested interest. In the third paragraph of the podcast Dwarkesh already told you that he already invested. >>> (disclosure - I’m an angel investor), At that point the co-founders and Dwarkesh himself will agree on everything and they will say anything to get more VC money - Even if the timelines are unrealistic. (Because that is the scam).
gcanyon 1y ago
> 30 years That's...ridiculous. I'm a product manager, and AI is already chipping away at my job. Two months ago I said to one of my devs, "Our dashboard here looks very bland. What if we had a more visual display of the pipeline statuses across the top of the screen?" He said he thought that was a good idea, and I went to lunch. I came back and started sketching up some ideas for how to lay out the statuses. I had barely gotten started with that when he called me over to show me what cursor had come up with when asked: it was better than what I was sketching, for sure. We're (white collar work) going to be 90% automated in less than ten years, and I feel like I'm being conservative saying ten years.
- Amekedl 1y ago
  You occasionally do glimpse behind the curtains; depending on what you actually develop it's feasible and quick to prompt it, but attempting to go further than that across multiple components collapses so drastically that I cannot help but feel that all ai stuff is entirely incapable of replicating the real thing at the moment.

dasil003 1y ago

> The roadmap to success will most likely start with training or fine-tuning on data from human professionals performing the task, and proceed with reinforcement learning in custom environments designed to capture more of the complexity of what people do in their jobs. > [...] > We think this is essentially a data problem, not an algorithms problem. This is extremely hand-wavy. How are you going to instrument the various thought processes and non-verbal communication that goes into building successful software? A huge part of it is intuition about what makes sense to other humans. It's related to the idea of common sense, but in the software world there's this layer of unforgiving determinism and rigidity that most humans don't want to deal with. I just don't see how AI crosses that chasm.

ege_erdil 1y ago
how do you think humans cross that chasm?
- AnimalMuppet 1y ago
  Pretty sure that however humans crossed it, it wasn't just "a data problem".
  - ege_erdil 1y ago
    what makes you sure about that?
    - AnimalMuppet 1y ago
      At the risk of coming across as a smart aleck, 40 years of experience building software. What experienced software engineers have is a sense of taste - this looks like good code and/or design, that doesn't. But they don't have data; they have, at best, a couple of anecdotes. It's more a sense of "that was harder to work with than it should have been; that approach seems to have drawbacks". But you only get a few examples of that in a career. And there are very few outfits compiling usable data that could shape the approaches that software engineers use. So I don't think how humans got there was primarily data.

aurizon 1y ago

I can see AI used for permutational hacking, sort of like the fuzzing types of SW and HW hacking. Where they succeed can be a very good stress test. It can be done on human code as well as AI code and I expect this to be done. It is increasingly apparent that what an AI made and then hacked, still needs the abilities that only humans have - for now

jes5199 1y ago

great, let's also record all of the daily interactions of the C-suite and see if we can replicate their job functions in simulation

jongjong 1y ago

I think the conclusion about software engineering having the potential of being both the first and last job to be automated by AI makes sense. It is an extremely unequal profession. The skill difference between a junior with 1 year of experience and a senior with 20 years of both diverse and deep experience (of relentless striving) is massive. They have very different capabilities. The level of software sophistication/complexity they can attain is very different. The code produced looks very different. This is not a profession where you've figured it all out after 5 years and can rest on your laurels. I think it takes literally 15+ years for most highly determined and intelligent people to even approach that state where they can manage complexity effectively. Most people never seem to get there.

martin-t 1y ago
Almost 15 years in and I feel like I am still just scratching the surface. But most of the lessons learned during that time go towards quality, not quantity or speed. The current trend with LLMs (and eventually maybe AI) seems to be doing what humans can do worse but significantly faster and cheaper. Unfortunately, not everyone needs or cares about safety, security or correctness. I am afraid software will mirror the evolution of physical products from the industrial revolution to present day. It feels like the quality of consumer grade products is constantly decreasing.
- jongjong 1y ago
  This resonates. Quality of mass market products decrease to the upper limit of the average person's perception/expectations... Which itself seems to be declining over time.
Amekedl 1y ago
Even then software constantly evolves, and rot is everywhere. And we're far from having the "best possible" software solution in literally every area (if that's even possible to measure), rather just endless room for improvement. And I don't see it being improved with whatever any llm chugs out, at least not "in-depth".

hnthrow90348765 1y ago

Alright, this is it, this is the AI peak. >Today we’re announcing Mechanize, a startup focused on developing virtual work environments, benchmarks, and training data that will enable the full automation of the economy. >Compensation >$200K – $475K • 1% – 2% Just imagine for a moment you are a software engineer capable of doing what they say, rivaling the raw intellectual capabilities of every mathematician, economist, and physicist known to man, and you end up actually building something that directly leads to automating the entire fucking economy of the world. And there's still someone who says to you: "best I can do is only 1% - 2%"

handfuloflight 1y ago
Oh I thought the peak was going to be based on the forward looking claims, not the standard equity offer.
martin-t 1y ago
One of the dumbest things a smart person can do is work for an AI company. At best they don't believe actual AI will be created and they are helping a scam. At worst, they are actively working to make their own job redundant and when they're fired, they will own nothing of what they built. All the money from their work will go to the owners who fired them.
- dsjoerg 1y ago
  > At worst, they are actively working to make their own job redundant and when they're fired, they will own nothing of what they built. All the money from their work will go to the owners who fired them. Except that's the case with literally every meaningful information-work job. Your goal is to obsolete yourself. If you do it well then your success becomes your calling card for your next job. Your career is a series of such jobs.
  - martin-t 1y ago
    1) You can and should program yourself out of a _task_, not a _job_. How many companies have a one.time need for a programmer? 2) Even if your claim was correct, then you'd amass experience on one job (task) that you could then leverage on the next job (task). If AI became reality, all that would become irrelevant. Your middle-class self would quickly realize you're only worth as much as your lower class neighbors who swing a shovel or flip burgers for a living, all the while your upper class bosses become richer thanks to your work.

nitwit005 1y ago

Never before have I seen someone list a market size in the tens of trillions, as they do on their main page: > The market potential here is absurdly large: workers in the US are paid around $18 trillion per year in aggregate. For the entire world, the number is over three times greater, around $60 trillion per year. https://www.mechanize.work/ https://www.mechanize.work/

handfuloflight 1y ago
It's what you do when you fancy yourself in the leagues of Berkshire Hathaway.

gerdesj 1y ago

"With every passing month, AI models get better at most tasks that a software engineer does in their job." No they don't.

satisfice 1y ago

This guy has not even begun to talk about this in a serious way. If you try to talk about automating SE and you aren’t clearly explaining how people who know nothing about engineering will directly interact with this automation to get what they need, then you aren’t saying anything. Even with vibe coding there is a certain skillset involved.

belZaah 1y ago

Y’all need some Brooks in your lives. The No Silver Bullet essay explains very clearly, why the ability to string syntactically correct commands together does not equal the ability to write code.

owebmaster 1y ago
Artificial intelligence. Many people expect advances in artificial intelligence to provide the revolutionary breakthrough that will give order-of-magnitude gains in software productivity and quality. I do not. To see why, we must dissect what is meant by “artificial intelligence” and then see how it applies. Parnas has clarified the terminological chaos: Two quite different definitions of AI are in common use today. AI-1: The use of computers to solve problems that previously could only be solved by applying human intelligence. AI-2: The use of a specific set of programming techniques knows as heuristic or rule-based programming. In this approach human experts are studies to determine what heuristics or rules of thumb they use in solving problems. . . . The program is designed to solve a problem the way that humans seem to solve it. The first definition has a sliding meaning. . . . Something can fit the definition of AI-1 today but, once we see how the program works and understand the problem, we will not think of it as AI anymore. . . . Unfortunately I cannot identify a body of technology that is unique to this field. . . . Most of the work is problem-specific, and some abstraction or creativity is require to see how to transfer it great read: https://worrydream.com/refs/Brooks_1986_-_No_Silver_Bullet.pdf https://worrydream.com/refs/Brooks_1986_-_No_Silver_Bullet.p...

mattgreenrocks 1y ago

OpenAI spent several years prior to ChatGPT getting AI to play Dota 2 well. They got some good results out of that, but it was a subset of the game: only a handful or two of characters. I’m not sure why they stopped; maybe that was when they pivoted to something more general? Regardless, software dev I consider way more dimensional (eg nuanced) than Dota 2, even if a lot of patterns recur on a smaller scale in the code itself. If they weren’t able to crack Dota 2, why should I believe that software eng is just around the corner?

ege_erdil 1y ago
we don't think it's just around the corner

jackb4040 1y ago

> The key question now is: what data do we need, exactly? What if I told you that's not the key question, and the "more data" approach has obviously and publicly hit a wall that requires causal reasoning to move past?

ege_erdil 1y ago
then i would disagree