3 ms·

> LLMs have, on the other hand, made it even easier to botch all of these and quickly roll out a half-baked MVP Compared to the status quo where people pretty

> LLMs have, on the other hand, made it even easier to botch all of these and quickly roll out a half-baked MVP Compared to the status quo where people pretty much never consider these things, like accessibility, especially not for an MVP? How many people have never added written aria attribute? I would suspect 90%+ of people touching the frontend. The difference with LLMs is that (1) they have a latent rigor for things that you weren't going to spend time caring about anyways and, more importantly, (2) you can encode these things into prompts (AGENTS.md) and processes so that they happen even when you weren't going to invest the time with or without AI. For a lot of people this only means collecting some generic "skills" they found online yet it's still much better than what they were going to do pre-AI. That's why I think AI is saving software in some ways, not leading to worse software. Or, asserting that AI will botch software might hold more weight with people who have already forgotten how dogshit software was pre-AI.

m132 5mo ago
I can somewhat see your point, but it is generally accepted that a wrong ARIA is worse than none, and LLM-assisted codebases, at least these days, only stick together thanks to testing, the more decent ones heavily emphasize in-depth human code reviews. If our hypothetical developer hasn't used any accessibility-related tags before, what chance is there that those parts of the website will receive adequate testing?
- PaulHoule 5mo ago
  0% if by testing you mean "somebody who uses a screen reader regularly was able to use the product successfully" because nobody seems to do that.
- hombre_fatal 5mo ago
  Testing is an even more powerful subject here since we barely do it. Testing is so hard that we'll agree that, e.g., TDD is great (e.g. ensure your tests actually test something, ensure your code is testable from the start) yet we never do it. And when we do write tests, we are on the hook to be eternally vigilant that they are not stale, that they test something real, that they are not redundant. And they often turn into an append-only file that you resent. Meanwhile, AI is happy to write tests, do red-green TDD cycles, refactor them, prune them, update them, justify and defend them. It will even incidentally write tests for the most aloof vibe-coder by accident because they didn't specify otherwise. Overnight, I went from never testing most of my side projects (except for, say, maybe unit tests in more straightforward things like a parser) to now everything is tested end-to-end. Every time I make a new directional / architectural decision, the tests the AI writes also encode it at the test level to reenforce the decision. It's strictly a better world for software because AI can write and maintain tests. > LLM-assisted codebases, at least these days, only stick together thanks to testing But tests also help humans and ensure human-written software is robust. We only don't test because they are so costly to write and maintain, and our software has always suffered for it. Or the tests become such an unmaintainable mess that our software is now worse because of it!
  - kakugawa 5mo ago
    a11y testing is non-trivial. axe-core can automatically detect many types of issues. However, enough compliance (to avoid being sued) needs end-to-end testing and human judgement. e.g. keyboard traps, focus restoration, alt-text, etc.
  - skydhash 5mo ago
    > Meanwhile, AI is happy to write tests, do red-green TDD cycles, refactor them, prune them, update them, justify and defend them. It will even incidentally write tests for the most aloof vibe-coder by accident because they didn't specify otherwise. I read some AI generated tests and while it looks visually impressive, ultimately it wasn’t doing anything valuable? Why? because of all the mocks and scenarios that didn’t matter. And on top of that, tests are additional code to maintain. These days, I don’t even bother with unit testing. They are a maintenance burden. I focus on integration test (whole modules) and if I have the time, on a harness to do e2e testing.
    - icedchai 5mo ago
      Many humans do the same thing. They write tests that mock so much of the actual code, the "test" is not testing much at all, except perhaps the developer's ability to basically turn the code inside out. It's often just a large volume of crap that has to be maintained (or eventually deleted.) I agree integration tests are best, with some e2e testing for common scenarios. I worked at a place that required unit tests for every new method or function. Arguments like "this other (integration) test already covers that. why do I have to add another test?" wouldn't fly. PR reviews would often degrade into arguments about testing and how all database access needed to be mocked...
    - mega_dean 5mo ago
      > I read some AI generated tests and while it looks visually impressive, ultimately it wasn’t doing anything valuable I just saw this comment yesterday about one of the tests from Bun’s rust rewrite: https://news.ycombinator.com/item?id=48314311 https://news.ycombinator.com/item?id=48314311 It reads in the raw source code and uses a regex to assert that “unsafe” is used. > These days, I don’t even bother with unit testing. They are a maintenance burden I’ve come to the same conclusion, but that’s only because I’m working on solo projects. I think they are probably worth it with multiple devs on the same project.
d0liver 5mo ago
I would much rather have software that works but lacks accessibility features than software that's broken but also has some broken accessibility features sprinkled in. The former is useful to many people, while the latter is useful to no one. But the key here is: LLMs don't have latent rigor, nor any other kind of rigor.
- hombre_fatal 5mo ago
  But software was already in a horrible state before AI, so your dichotomy doesn't work. The status quo with pre-AI Human Written software on a pedestal is that it doesn't work, and it lacks accessibility, polish, performance considerations, UX considerations, tests, and more. The built-in rigor is trivial to prove. Just put Opus 4.8 in plan mode and tell it to plan something, like a vt100 emulator. The question isn't whether you can do better than AI, because you'll put your foot on the scale and give yourself infinite time, attention, and energy just so you can say yes. It's whether AI can do as good or better than you with the same time, attention, and energy you would have given a task in the first place.
  - icedchai 5mo ago
    A couple months back, I had Sonnet build me a browser-based VT-100 terminal emulator. I then had it build me a websocket-to-telnet gateway for connecting some old retro systems to it. Both worked pretty well. Could I have done it on my own? Yes, eventually. The problem is I would've lost interest and moved on to some other useless project before getting that far.
  - d0mine 5mo ago
    In practice, you would use an already written implementation, maintained by somebody else. An option that is often ignored by LLM (copy-paste galore). For example, imagine if textual-serve author would reimplement xterm.js What effect it would make on quality. LLMs increase technical debt rapidly. It is unclear whether they can deal with the mess they create. But we'll know soon (no need to wait years, to get immovable mess). The positive side of LLMs is that they confirm experimentally the usefulness of many software engineering practices (testing,docs, adrs, design, formal specs etc)
redsocksfan45 5mo ago
[dead]
sillyfluke 5mo ago
>asserting that AI will botch software might hold more weight with people who have already forgotten how dogshit software was pre-AI. You're responding to an assertion with an assertion. It has been empirically proven that SOTA models can create more dogshit software than pre-AI software. It is also trivially known that the user is unable to predict when and how the AI will introduce dogshit into the software. We literally had a study posted on this forum claiming models give more accurate answers if you're mean to them. This is the shit we're dealing with. Stuff you couldn't make up in a dystopian Douglas Adams novel. >you can encode these things into prompts Is this satire? SOTA models randomly disobey rules in prompts all the time. When a dev drops a production db I can warn them. If they do it multiple times during their employment I can change their roles or fire them. I can count the number of companies providing SOTA models with the fingers on my hands. Imagine having an employee pool of only 5 savant coders with dementia to choose from to hire to your company. That's it. Thats the entire applicant pool. You can only fire one of them by hiring one of the other four to replace them with. And you can't really fire them for dropping production dbs if you can't prevent the other ones from making the same mistake. This is the current AI-first hellscape as it stands.
PunchyHamster 5mo ago
> That's why I think AI is saving software in some ways, not leading to worse software. So far the way AI is making software better is mostly checking for common bugs and acting as review/pair programming buddy, not the code. The code is bad. At best okay. But you can make a lot of it. And it's not bad enough to not be good enough.
selicos 5mo ago
With the right harness/agent/skill setup and requirements work, this (any specificity) can become part of the workflow from early stages. Setting that up and making sure it works, at the early stages, is not what LLMs do best, today. This sort of work is possibly incredibly valuable in the next 12-36 months, depending on what LLMs can be designed to do out of the box. An Agent that can deliver the correct legal review process for a patent and is correct often enough to validate the savings vs real humans, or at least the speed increase + manual review ala code reviews, is incredibly valuable. Are we there yet? Maybe this is where current groups like FNCR are trying to go.

m132 5mo ago

I can somewhat see your point, but it is generally accepted that a wrong ARIA is worse than none, and LLM-assisted codebases, at least these days, only stick together thanks to testing, the more decent ones heavily emphasize in-depth human code reviews. If our hypothetical developer hasn't used any accessibility-related tags before, what chance is there that those parts of the website will receive adequate testing?

PaulHoule 5mo ago
0% if by testing you mean "somebody who uses a screen reader regularly was able to use the product successfully" because nobody seems to do that.
hombre_fatal 5mo ago
Testing is an even more powerful subject here since we barely do it. Testing is so hard that we'll agree that, e.g., TDD is great (e.g. ensure your tests actually test something, ensure your code is testable from the start) yet we never do it. And when we do write tests, we are on the hook to be eternally vigilant that they are not stale, that they test something real, that they are not redundant. And they often turn into an append-only file that you resent. Meanwhile, AI is happy to write tests, do red-green TDD cycles, refactor them, prune them, update them, justify and defend them. It will even incidentally write tests for the most aloof vibe-coder by accident because they didn't specify otherwise. Overnight, I went from never testing most of my side projects (except for, say, maybe unit tests in more straightforward things like a parser) to now everything is tested end-to-end. Every time I make a new directional / architectural decision, the tests the AI writes also encode it at the test level to reenforce the decision. It's strictly a better world for software because AI can write and maintain tests. > LLM-assisted codebases, at least these days, only stick together thanks to testing But tests also help humans and ensure human-written software is robust. We only don't test because they are so costly to write and maintain, and our software has always suffered for it. Or the tests become such an unmaintainable mess that our software is now worse because of it!
- kakugawa 5mo ago
  a11y testing is non-trivial. axe-core can automatically detect many types of issues. However, enough compliance (to avoid being sued) needs end-to-end testing and human judgement. e.g. keyboard traps, focus restoration, alt-text, etc.
- skydhash 5mo ago
  > Meanwhile, AI is happy to write tests, do red-green TDD cycles, refactor them, prune them, update them, justify and defend them. It will even incidentally write tests for the most aloof vibe-coder by accident because they didn't specify otherwise. I read some AI generated tests and while it looks visually impressive, ultimately it wasn’t doing anything valuable? Why? because of all the mocks and scenarios that didn’t matter. And on top of that, tests are additional code to maintain. These days, I don’t even bother with unit testing. They are a maintenance burden. I focus on integration test (whole modules) and if I have the time, on a harness to do e2e testing.
  - icedchai 5mo ago
    Many humans do the same thing. They write tests that mock so much of the actual code, the "test" is not testing much at all, except perhaps the developer's ability to basically turn the code inside out. It's often just a large volume of crap that has to be maintained (or eventually deleted.) I agree integration tests are best, with some e2e testing for common scenarios. I worked at a place that required unit tests for every new method or function. Arguments like "this other (integration) test already covers that. why do I have to add another test?" wouldn't fly. PR reviews would often degrade into arguments about testing and how all database access needed to be mocked...
  - mega_dean 5mo ago
    > I read some AI generated tests and while it looks visually impressive, ultimately it wasn’t doing anything valuable I just saw this comment yesterday about one of the tests from Bun’s rust rewrite: https://news.ycombinator.com/item?id=48314311 https://news.ycombinator.com/item?id=48314311 It reads in the raw source code and uses a regex to assert that “unsafe” is used. > These days, I don’t even bother with unit testing. They are a maintenance burden I’ve come to the same conclusion, but that’s only because I’m working on solo projects. I think they are probably worth it with multiple devs on the same project.

d0liver 5mo ago

I would much rather have software that works but lacks accessibility features than software that's broken but also has some broken accessibility features sprinkled in. The former is useful to many people, while the latter is useful to no one. But the key here is: LLMs don't have latent rigor, nor any other kind of rigor.

hombre_fatal 5mo ago
But software was already in a horrible state before AI, so your dichotomy doesn't work. The status quo with pre-AI Human Written software on a pedestal is that it doesn't work, and it lacks accessibility, polish, performance considerations, UX considerations, tests, and more. The built-in rigor is trivial to prove. Just put Opus 4.8 in plan mode and tell it to plan something, like a vt100 emulator. The question isn't whether you can do better than AI, because you'll put your foot on the scale and give yourself infinite time, attention, and energy just so you can say yes. It's whether AI can do as good or better than you with the same time, attention, and energy you would have given a task in the first place.
- icedchai 5mo ago
  A couple months back, I had Sonnet build me a browser-based VT-100 terminal emulator. I then had it build me a websocket-to-telnet gateway for connecting some old retro systems to it. Both worked pretty well. Could I have done it on my own? Yes, eventually. The problem is I would've lost interest and moved on to some other useless project before getting that far.
- d0mine 5mo ago
  In practice, you would use an already written implementation, maintained by somebody else. An option that is often ignored by LLM (copy-paste galore). For example, imagine if textual-serve author would reimplement xterm.js What effect it would make on quality. LLMs increase technical debt rapidly. It is unclear whether they can deal with the mess they create. But we'll know soon (no need to wait years, to get immovable mess). The positive side of LLMs is that they confirm experimentally the usefulness of many software engineering practices (testing,docs, adrs, design, formal specs etc)

redsocksfan45 5mo ago

[dead]

sillyfluke 5mo ago

>asserting that AI will botch software might hold more weight with people who have already forgotten how dogshit software was pre-AI. You're responding to an assertion with an assertion. It has been empirically proven that SOTA models can create more dogshit software than pre-AI software. It is also trivially known that the user is unable to predict when and how the AI will introduce dogshit into the software. We literally had a study posted on this forum claiming models give more accurate answers if you're mean to them. This is the shit we're dealing with. Stuff you couldn't make up in a dystopian Douglas Adams novel. >you can encode these things into prompts Is this satire? SOTA models randomly disobey rules in prompts all the time. When a dev drops a production db I can warn them. If they do it multiple times during their employment I can change their roles or fire them. I can count the number of companies providing SOTA models with the fingers on my hands. Imagine having an employee pool of only 5 savant coders with dementia to choose from to hire to your company. That's it. Thats the entire applicant pool. You can only fire one of them by hiring one of the other four to replace them with. And you can't really fire them for dropping production dbs if you can't prevent the other ones from making the same mistake. This is the current AI-first hellscape as it stands.

PunchyHamster 5mo ago

> That's why I think AI is saving software in some ways, not leading to worse software. So far the way AI is making software better is mostly checking for common bugs and acting as review/pair programming buddy, not the code. The code is bad. At best okay. But you can make a lot of it. And it's not bad enough to not be good enough.

selicos 5mo ago

With the right harness/agent/skill setup and requirements work, this (any specificity) can become part of the workflow from early stages. Setting that up and making sure it works, at the early stages, is not what LLMs do best, today. This sort of work is possibly incredibly valuable in the next 12-36 months, depending on what LLMs can be designed to do out of the box. An Agent that can deliver the correct legal review process for a patent and is correct often enough to validate the savings vs real humans, or at least the speed increase + manual review ala code reviews, is incredibly valuable. Are we there yet? Maybe this is where current groups like FNCR are trying to go.