3 ms·

I can somewhat see your point, but it is generally accepted that a wrong ARIA is worse than none, and LLM-assisted codebases, at least these days, only stick to

by m132 4mo ago

I can somewhat see your point, but it is generally accepted that a wrong ARIA is worse than none, and LLM-assisted codebases, at least these days, only stick together thanks to testing, the more decent ones heavily emphasize in-depth human code reviews. If our hypothetical developer hasn't used any accessibility-related tags before, what chance is there that those parts of the website will receive adequate testing?

PaulHoule 4mo ago
0% if by testing you mean "somebody who uses a screen reader regularly was able to use the product successfully" because nobody seems to do that.
hombre_fatal 4mo ago
Testing is an even more powerful subject here since we barely do it. Testing is so hard that we'll agree that, e.g., TDD is great (e.g. ensure your tests actually test something, ensure your code is testable from the start) yet we never do it. And when we do write tests, we are on the hook to be eternally vigilant that they are not stale, that they test something real, that they are not redundant. And they often turn into an append-only file that you resent. Meanwhile, AI is happy to write tests, do red-green TDD cycles, refactor them, prune them, update them, justify and defend them. It will even incidentally write tests for the most aloof vibe-coder by accident because they didn't specify otherwise. Overnight, I went from never testing most of my side projects (except for, say, maybe unit tests in more straightforward things like a parser) to now everything is tested end-to-end. Every time I make a new directional / architectural decision, the tests the AI writes also encode it at the test level to reenforce the decision. It's strictly a better world for software because AI can write and maintain tests. > LLM-assisted codebases, at least these days, only stick together thanks to testing But tests also help humans and ensure human-written software is robust. We only don't test because they are so costly to write and maintain, and our software has always suffered for it. Or the tests become such an unmaintainable mess that our software is now worse because of it!
- kakugawa 4mo ago
  a11y testing is non-trivial. axe-core can automatically detect many types of issues. However, enough compliance (to avoid being sued) needs end-to-end testing and human judgement. e.g. keyboard traps, focus restoration, alt-text, etc.
- skydhash 4mo ago
  > Meanwhile, AI is happy to write tests, do red-green TDD cycles, refactor them, prune them, update them, justify and defend them. It will even incidentally write tests for the most aloof vibe-coder by accident because they didn't specify otherwise. I read some AI generated tests and while it looks visually impressive, ultimately it wasn’t doing anything valuable? Why? because of all the mocks and scenarios that didn’t matter. And on top of that, tests are additional code to maintain. These days, I don’t even bother with unit testing. They are a maintenance burden. I focus on integration test (whole modules) and if I have the time, on a harness to do e2e testing.
  - icedchai 4mo ago
    Many humans do the same thing. They write tests that mock so much of the actual code, the "test" is not testing much at all, except perhaps the developer's ability to basically turn the code inside out. It's often just a large volume of crap that has to be maintained (or eventually deleted.) I agree integration tests are best, with some e2e testing for common scenarios. I worked at a place that required unit tests for every new method or function. Arguments like "this other (integration) test already covers that. why do I have to add another test?" wouldn't fly. PR reviews would often degrade into arguments about testing and how all database access needed to be mocked...
  - mega_dean 4mo ago
    > I read some AI generated tests and while it looks visually impressive, ultimately it wasn’t doing anything valuable I just saw this comment yesterday about one of the tests from Bun’s rust rewrite: https://news.ycombinator.com/item?id=48314311 https://news.ycombinator.com/item?id=48314311 It reads in the raw source code and uses a regex to assert that “unsafe” is used. > These days, I don’t even bother with unit testing. They are a maintenance burden I’ve come to the same conclusion, but that’s only because I’m working on solo projects. I think they are probably worth it with multiple devs on the same project.

by m132 4mo ago

PaulHoule 4mo ago

0% if by testing you mean "somebody who uses a screen reader regularly was able to use the product successfully" because nobody seems to do that.

hombre_fatal 4mo ago

Testing is an even more powerful subject here since we barely do it. Testing is so hard that we'll agree that, e.g., TDD is great (e.g. ensure your tests actually test something, ensure your code is testable from the start) yet we never do it. And when we do write tests, we are on the hook to be eternally vigilant that they are not stale, that they test something real, that they are not redundant. And they often turn into an append-only file that you resent. Meanwhile, AI is happy to write tests, do red-green TDD cycles, refactor them, prune them, update them, justify and defend them. It will even incidentally write tests for the most aloof vibe-coder by accident because they didn't specify otherwise. Overnight, I went from never testing most of my side projects (except for, say, maybe unit tests in more straightforward things like a parser) to now everything is tested end-to-end. Every time I make a new directional / architectural decision, the tests the AI writes also encode it at the test level to reenforce the decision. It's strictly a better world for software because AI can write and maintain tests. > LLM-assisted codebases, at least these days, only stick together thanks to testing But tests also help humans and ensure human-written software is robust. We only don't test because they are so costly to write and maintain, and our software has always suffered for it. Or the tests become such an unmaintainable mess that our software is now worse because of it!

kakugawa 4mo ago
a11y testing is non-trivial. axe-core can automatically detect many types of issues. However, enough compliance (to avoid being sued) needs end-to-end testing and human judgement. e.g. keyboard traps, focus restoration, alt-text, etc.
skydhash 4mo ago
> Meanwhile, AI is happy to write tests, do red-green TDD cycles, refactor them, prune them, update them, justify and defend them. It will even incidentally write tests for the most aloof vibe-coder by accident because they didn't specify otherwise. I read some AI generated tests and while it looks visually impressive, ultimately it wasn’t doing anything valuable? Why? because of all the mocks and scenarios that didn’t matter. And on top of that, tests are additional code to maintain. These days, I don’t even bother with unit testing. They are a maintenance burden. I focus on integration test (whole modules) and if I have the time, on a harness to do e2e testing.
- icedchai 4mo ago
  Many humans do the same thing. They write tests that mock so much of the actual code, the "test" is not testing much at all, except perhaps the developer's ability to basically turn the code inside out. It's often just a large volume of crap that has to be maintained (or eventually deleted.) I agree integration tests are best, with some e2e testing for common scenarios. I worked at a place that required unit tests for every new method or function. Arguments like "this other (integration) test already covers that. why do I have to add another test?" wouldn't fly. PR reviews would often degrade into arguments about testing and how all database access needed to be mocked...
- mega_dean 4mo ago
  > I read some AI generated tests and while it looks visually impressive, ultimately it wasn’t doing anything valuable I just saw this comment yesterday about one of the tests from Bun’s rust rewrite: https://news.ycombinator.com/item?id=48314311 https://news.ycombinator.com/item?id=48314311 It reads in the raw source code and uses a regex to assert that “unsafe” is used. > These days, I don’t even bother with unit testing. They are a maintenance burden I’ve come to the same conclusion, but that’s only because I’m working on solo projects. I think they are probably worth it with multiple devs on the same project.