3 ms·

People keep talking about how LLMs are like a compiler from human language to code. We commit source code instead of just compiled machine code, so why should t

People keep talking about how LLMs are like a compiler from human language to code. We commit source code instead of just compiled machine code, so why should this be any different? The "source code" is the prompts

criley2 7mo ago
The prompt isn't very useful. You'd see the exact same prompt on every ticket for me. Prompt 1: "Research <X> domain, think deeply, and record a full analysis in /docs/TICKET-123-NOTES.md" Prompt 2: Based on our research, read TICKET-123 and began formulating solutions. Let's think this problem through and come up with multiple potential solutions. Document our solutions in TICKET-123-SOLUTIONS.md Prompt 3: Based on Solution X, let's formulate a complete plan to implement. Break the work into medium sized tasks that a human could complete in 5-10 hours. Write our plan in TICKET-123-PLAN.md I've often thought that some of this metadata, such as the research, solutioning and plan could be shared. I think they're valuable for code review. I've also translated these artifacts into other developer documentation paradigms. But the prompts? You're not getting a lot of value there.
- lelanthran 7mo ago
  > Prompt 1: "Research <X> domain, think deeply, and record a full analysis in /docs/TICKET-123-NOTES.md" > Prompt 2: Based on our research, read TICKET-123 and began formulating solutions. Let's think this problem through and come up with multiple potential solutions. Document our solutions in TICKET-123-SOLUTIONS.md > Prompt 3: Based on Solution X, let's formulate a complete plan to implement. Break the work into medium sized tasks that a human could complete in 5-10 hours. Write our plan in TICKET-123-PLAN.md Sounds to me that all these 10x - 100x "engineers" can be removed from the loop.
  - criley2 7mo ago
    Almost! We are certainly on the precipice of the vast majority of white collar work being removed from the loop. However, what each domain will tell you (engineering included) is that AI doesn't understand the full context of what you're doing and the point of the business and where to spend effort and where to cut corners. There is definitely still room for competent engineers to iterate here on the solutioning and plans to refine the AI work into something more sturdy. Although this is only in domains where code quality truly matters. A lot of consumer software without SLA's are just vibe coding full speed now. No code review, AI writing 100% of the code.
    - voidUpdate 7mo ago
      Judging by what I've seen recently, 100% LLM code is often buggy and not that great. I'd say code quality truly matters in all domains
      - criley2 7mo ago
        What a utopia, where code quality matters in all domains! In my opinion nearly the opposite is true: modern business solves for the "minimum viable quality". What is the absolute lowest quality the software can be and not tank the business.
        lelanthran 7mo ago
        > What a utopia, where code quality matters in all domains! It does. The degree may not, though. "We have a threshold of at least 5 hours total uptime every 24 hours" is still a quality bar, even if it is different to "We have a threshold of 99.99% uptime per year".
        voidUpdate 7mo ago
        Maybe you're different, but I prefer to write code that at least attempts to be performant, tidy and readable, as well as working at least 90% of the time. Maybe I don't achieve perfection, but I try to care about the quality of what I write
        ajam1507 7mo ago
        If you could prove what "minimum viable quality" actually was this would be true. We have standards and procedures exactly because it is unknowable. One engineers idea of "good enough" might bankrupt the business.

criley2 7mo ago

The prompt isn't very useful. You'd see the exact same prompt on every ticket for me. Prompt 1: "Research <X> domain, think deeply, and record a full analysis in /docs/TICKET-123-NOTES.md" Prompt 2: Based on our research, read TICKET-123 and began formulating solutions. Let's think this problem through and come up with multiple potential solutions. Document our solutions in TICKET-123-SOLUTIONS.md Prompt 3: Based on Solution X, let's formulate a complete plan to implement. Break the work into medium sized tasks that a human could complete in 5-10 hours. Write our plan in TICKET-123-PLAN.md I've often thought that some of this metadata, such as the research, solutioning and plan could be shared. I think they're valuable for code review. I've also translated these artifacts into other developer documentation paradigms. But the prompts? You're not getting a lot of value there.

lelanthran 7mo ago
> Prompt 1: "Research <X> domain, think deeply, and record a full analysis in /docs/TICKET-123-NOTES.md" > Prompt 2: Based on our research, read TICKET-123 and began formulating solutions. Let's think this problem through and come up with multiple potential solutions. Document our solutions in TICKET-123-SOLUTIONS.md > Prompt 3: Based on Solution X, let's formulate a complete plan to implement. Break the work into medium sized tasks that a human could complete in 5-10 hours. Write our plan in TICKET-123-PLAN.md Sounds to me that all these 10x - 100x "engineers" can be removed from the loop.
- criley2 7mo ago
  Almost! We are certainly on the precipice of the vast majority of white collar work being removed from the loop. However, what each domain will tell you (engineering included) is that AI doesn't understand the full context of what you're doing and the point of the business and where to spend effort and where to cut corners. There is definitely still room for competent engineers to iterate here on the solutioning and plans to refine the AI work into something more sturdy. Although this is only in domains where code quality truly matters. A lot of consumer software without SLA's are just vibe coding full speed now. No code review, AI writing 100% of the code.
  - voidUpdate 7mo ago
    Judging by what I've seen recently, 100% LLM code is often buggy and not that great. I'd say code quality truly matters in all domains
    - criley2 7mo ago
      What a utopia, where code quality matters in all domains! In my opinion nearly the opposite is true: modern business solves for the "minimum viable quality". What is the absolute lowest quality the software can be and not tank the business.
      - lelanthran 7mo ago
        > What a utopia, where code quality matters in all domains! It does. The degree may not, though. "We have a threshold of at least 5 hours total uptime every 24 hours" is still a quality bar, even if it is different to "We have a threshold of 99.99% uptime per year".
      - voidUpdate 7mo ago
        Maybe you're different, but I prefer to write code that at least attempts to be performant, tidy and readable, as well as working at least 90% of the time. Maybe I don't achieve perfection, but I try to care about the quality of what I write
      - ajam1507 7mo ago
        If you could prove what "minimum viable quality" actually was this would be true. We have standards and procedures exactly because it is unknowable. One engineers idea of "good enough" might bankrupt the business.