3 ms·

As a programming lead on a hobbyist video game project, we're reaping massive rewards being in category #1. I keep the architecture and important details in che

As a programming lead on a hobbyist video game project, we're reaping massive rewards being in category #1. I keep the architecture and important details in check while letting frontier models go ham. As alluded, it's a video game, not a life support system, so bugs are low-impact. But even better, the defect rate is actually the lowest it's ever been. Insidious bugs baked in by years of accumulated human error are trivial for Sol or Astra to untangle. This is the best time to be alive so far if you enjoy hobby game dev.

openfront 16d ago
I've found the same, the rate of bugs has dropped pretty dramatically after switching to ai generated code. I think it's partly because ai will write 1000s of lines of unit tests without complaining. I also have a github workflow where claude runs the /code-review command on every PR.
- sthuck 16d ago
  In the world of web apps, I find the agent's ability to write good e2e tests to be a real game changer. Turns out with enough rigor you can write pretty stable mostly not flaky e2e tests. And even flaky ones are fixed quickly due to a fuck ton of assertions at every step. Test code looks like a mess, even more than the usual LLM code. takes a while to let it go. Test report looks beautiful though.
  - bwestergard 16d ago
    When the test code "looks like a mess", how do you get assurance it is testing the right properties? Like you, I've found that LLMs can improve test coverage by decreasing the amount of developer time spent writing tests. But generally, it takes a lot of manual work to set up the initial testing framework, and even then, a lot of vigilance to ensure that what is actually tested corresponds to the description of the test.
nonameiguess 15d ago
I feel weirdly stuck on both sides of this. I haven't been in a full-time "code writing" roles for years, so my own personal usage of LLMs for anything is roughly nil right now, just because I continue to be able to get done all tasks I wanted to get done in the time I have using tools and techniques I already had available, and have seen little need to adopt new ones. The exceptions have been around things like using the text interface for diagramming tools to at least get the initial scaffolding set up without having to learn the specific quirks of that tool, then spending maybe an hour or two at the end to clean up and make it look nice and ensure it's actually coherent and has no mistakes. This is a task I have to do maybe two or three times a month, so it isn't a huge win, but it's not nothing. But that's digression. The short of it is I've seen enough to convince me these tools may as well be magic and a whole lot of tasks that break down into "produce media content of some sort" that has a well-defined goal and definition of correctness will be permanently sped up by automation. This includes a lot of software writing. At the same time, I shared the skepticism of estimates of economic impact and irrevocably changing the larger world. I'm a lot closer to the business side of the house these days, working with customers and prospective customers to identify use cases, reference architectures, pain points, feature requests, and bring this back to the development teams to attempt using real-world experience like this to inform how we design products. It's not product management as I'm focused more often on the nitty gritty technical details, not high-level user experience or roadmaps. But it gives me a great avenue into seeing what causes organizations to actually buy and/or adopt new software products, and the rate at which they can do that. And frankly, it isn't moving the needle much. They have the same budgets they always had, so they're not buying more, and our business is growing, but no faster or better than it grew before agentic coding became a thing. I always wonder because it seems the glowing success stories on Hacker News come in one of three varieties. It's the solo indie dev, usually targeting mobile app stores, who churns out dozens of roughly "will compile and doesn't immediately crash at runtime" apps in the time it used to take to complete one. It's the hobbyist, making software only they and maybe their immediate friends will ever use. Or it's startups, whose monetization model isn't monetization at all; it's just having something shiny to show investors in order to convince those with loose money to give enough to you personally that you can build up a nest egg whether or not your product ultimately ends up ever having a single paying customer. In my own business, a multi-decade, mature but not hyperscale company selling overwhelmingly self-hosted enterprise open source software, I can see the impacts on output. We have the same major products with the same release cadence. Each point release averages more new features than they used to, but also more regressions. It's overall a mixed bag. Non-technical product management staff is able to contribute code. We have a ton of new internal tools that nobody uses but they're there now. On the customer side, those that hinge decisions on wanting features that didn't exist yet are benefiting from getting those. Those that already had the features they want are losing from the greater rate at which regressions get through. The net business impact seems to be things have definitely changed qualitatively, but in purely financial quantitative terms, things are about the same as they were before. More code being committed to various git forges, but same headcount, same revenue, same margins, and same market cap.

openfront 16d ago

I've found the same, the rate of bugs has dropped pretty dramatically after switching to ai generated code. I think it's partly because ai will write 1000s of lines of unit tests without complaining. I also have a github workflow where claude runs the /code-review command on every PR.

sthuck 16d ago
In the world of web apps, I find the agent's ability to write good e2e tests to be a real game changer. Turns out with enough rigor you can write pretty stable mostly not flaky e2e tests. And even flaky ones are fixed quickly due to a fuck ton of assertions at every step. Test code looks like a mess, even more than the usual LLM code. takes a while to let it go. Test report looks beautiful though.
- bwestergard 16d ago
  When the test code "looks like a mess", how do you get assurance it is testing the right properties? Like you, I've found that LLMs can improve test coverage by decreasing the amount of developer time spent writing tests. But generally, it takes a lot of manual work to set up the initial testing framework, and even then, a lot of vigilance to ensure that what is actually tested corresponds to the description of the test.

nonameiguess 15d ago

I feel weirdly stuck on both sides of this. I haven't been in a full-time "code writing" roles for years, so my own personal usage of LLMs for anything is roughly nil right now, just because I continue to be able to get done all tasks I wanted to get done in the time I have using tools and techniques I already had available, and have seen little need to adopt new ones. The exceptions have been around things like using the text interface for diagramming tools to at least get the initial scaffolding set up without having to learn the specific quirks of that tool, then spending maybe an hour or two at the end to clean up and make it look nice and ensure it's actually coherent and has no mistakes. This is a task I have to do maybe two or three times a month, so it isn't a huge win, but it's not nothing. But that's digression. The short of it is I've seen enough to convince me these tools may as well be magic and a whole lot of tasks that break down into "produce media content of some sort" that has a well-defined goal and definition of correctness will be permanently sped up by automation. This includes a lot of software writing. At the same time, I shared the skepticism of estimates of economic impact and irrevocably changing the larger world. I'm a lot closer to the business side of the house these days, working with customers and prospective customers to identify use cases, reference architectures, pain points, feature requests, and bring this back to the development teams to attempt using real-world experience like this to inform how we design products. It's not product management as I'm focused more often on the nitty gritty technical details, not high-level user experience or roadmaps. But it gives me a great avenue into seeing what causes organizations to actually buy and/or adopt new software products, and the rate at which they can do that. And frankly, it isn't moving the needle much. They have the same budgets they always had, so they're not buying more, and our business is growing, but no faster or better than it grew before agentic coding became a thing. I always wonder because it seems the glowing success stories on Hacker News come in one of three varieties. It's the solo indie dev, usually targeting mobile app stores, who churns out dozens of roughly "will compile and doesn't immediately crash at runtime" apps in the time it used to take to complete one. It's the hobbyist, making software only they and maybe their immediate friends will ever use. Or it's startups, whose monetization model isn't monetization at all; it's just having something shiny to show investors in order to convince those with loose money to give enough to you personally that you can build up a nest egg whether or not your product ultimately ends up ever having a single paying customer. In my own business, a multi-decade, mature but not hyperscale company selling overwhelmingly self-hosted enterprise open source software, I can see the impacts on output. We have the same major products with the same release cadence. Each point release averages more new features than they used to, but also more regressions. It's overall a mixed bag. Non-technical product management staff is able to contribute code. We have a ton of new internal tools that nobody uses but they're there now. On the customer side, those that hinge decisions on wanting features that didn't exist yet are benefiting from getting those. Those that already had the features they want are losing from the greater rate at which regressions get through. The net business impact seems to be things have definitely changed qualitatively, but in purely financial quantitative terms, things are about the same as they were before. More code being committed to various git forges, but same headcount, same revenue, same margins, and same market cap.