5 ms·

> But I’m pretty sure I can do without all that ... test cases ... Test cases? I did a Show HN [1] a couple of days back for a UI library built almost entirel

by jeswin 2y ago

> But I’m pretty sure I can do without all that ... test cases ... Test cases? I did a Show HN [1] a couple of days back for a UI library built almost entirely with AI. Gpt-o1 generated these test cases for me: https://github.com/webjsx/webjsx/tree/main/src/test https://github.com/webjsx/webjsx/tree/main/src/test - in minutes instead of days. The quality of test cases are comparable to what a human would produce. 75% of the code I've written in the last one year has been with AI. If you still see no value in it (especially with things like test cases), I'm afraid you haven't figured out how to use AI as a tool. [1]: https://news.ycombinator.com/item?id=41644099 https://news.ycombinator.com/item?id=41644099

a5c11 2y ago
That means the code you wrote must have been pretty boring and repeatable. No way AI would produce code for, for example, proprietary hardware solutions. Try AI with something which isn't already on StackOverflow. Besides, I'd rather spent hours on writing a code, than trying to explain a stupid bot what I want and reshape it later anyway.
- nicce 2y ago
  Also the most useful and expensive testcases require understanding of the whole project. You need to validate the functionality from end-to-end and also that system does not crash for unexpected things and so on. AIs don't have that level understanding as "a whole" yet. For sure, simple unit tests are easy to generate with AI.
- jeswin 2y ago
  90% of projects are boring and somewhat repeatable. I've used it for generating codegen tools (https://github.com/codespin-ai/codespin https://github.com/codespin-ai/codespin), vscode plugins (https://github.com/codespin-ai/codespin-vscode-plugin https://github.com/codespin-ai/codespin-vscode-plugin), servers in .Net (https://github.com/lesser-app/tankman https://github.com/lesser-app/tankman), and in about a dozen other work projects over the past year. > Besides, I'd rather spent hours on writing a code, than trying to explain a stupid bot what I want and reshape it later anyway. I have other things to do with my hours. If something gets me what I want in minutes, I'll take it.
righthand 2y ago
Your UI library is just a stripped down React clone. The code wasn’t generated but rather copied, these test cases and functions are identical to React. I could have done the same thing with a “build your own react” article. This is what I don’t get about the LLM hype is that 99% of the examples are people claiming they invented something new with it. We had code generators before LLM-hype took off. Now we have code generators that just steal work and repurpose it as something claimed original.
- buddhistdude 2y ago
  no programmer in my company invents things often
  - righthand 2y ago
    And so you would accept “hey I spun up a react-create-element project, but instead of React I asked an LLM to copy the parts I needed for react so we have another dependency to maintain instead of tree shaking with webpack” as a useful work?
    - buddhistdude 2y ago
      not necessarily, but it's not less creative and inventive than what I believe most programmers are doing most of the time. there are relatively few people who invent new patterns (and they actually might be overrepresented on this website). the rest learns and applies those patterns.
      - righthand 2y ago
        Right that is well understood, but having an LLM compile together functions under the guise of custom built library is hardly a software engineer applying established patterns.
        jeswin 2y ago
        It is exactly the same as applying established patterns - patterns are what the LLMs have trained on. It seems you haven't really used LLMs for coding. They're super useful and improving every month - you don't have to take my word for it. And btw - codespin (https://github.com/codespin-ai/codespin https://github.com/codespin-ai/codespin) along with the VSCode plugin is what I use for AI-assisted coding many times. That was also generated via an LLM. I wrote it last year, and at that point there weren't many projects it could copy from.
codelikeawolf 2y ago
> The quality of test cases are comparable to what a human would produce. This has actually been a problem for me. I spent a lot of time getting good at writing tests and learning the best approaches to testing things. Most devs I've worked with treat tests as second-class citizens. They either try to treat them like production code and over-abstract everything, which makes the tests difficult to navigate, or they dump a bunch of crap in a file, ignore any conventions or standards, and write superfluous test cases that don't provide any value (if I see one more "it renders successfully" test in a React project, I'm going to lose it). The tests generated by these LLMs is comparable in quality to what most humans have produced, which isn't saying much. Getting good at testing isn't like getting good at most things. It's sort of thankless, and when I point out issues in the quality of the tests, I imagine I'm getting some eye rolls. Who cares, they're just tests, at least we have them, right? But it's code you have to read and maintain, and it will break, and you'll have to fix it. I'm not saying I'm a testing wizard or anything like that. But I really sympathize with the author, because there's a lot of crappy test code coming out of these LLMs. Edit: grammar

by jeswin 2y ago

a5c11 2y ago

That means the code you wrote must have been pretty boring and repeatable. No way AI would produce code for, for example, proprietary hardware solutions. Try AI with something which isn't already on StackOverflow. Besides, I'd rather spent hours on writing a code, than trying to explain a stupid bot what I want and reshape it later anyway.

nicce 2y ago
Also the most useful and expensive testcases require understanding of the whole project. You need to validate the functionality from end-to-end and also that system does not crash for unexpected things and so on. AIs don't have that level understanding as "a whole" yet. For sure, simple unit tests are easy to generate with AI.
jeswin 2y ago
90% of projects are boring and somewhat repeatable. I've used it for generating codegen tools (https://github.com/codespin-ai/codespin https://github.com/codespin-ai/codespin), vscode plugins (https://github.com/codespin-ai/codespin-vscode-plugin https://github.com/codespin-ai/codespin-vscode-plugin), servers in .Net (https://github.com/lesser-app/tankman https://github.com/lesser-app/tankman), and in about a dozen other work projects over the past year. > Besides, I'd rather spent hours on writing a code, than trying to explain a stupid bot what I want and reshape it later anyway. I have other things to do with my hours. If something gets me what I want in minutes, I'll take it.

righthand 2y ago

Your UI library is just a stripped down React clone. The code wasn’t generated but rather copied, these test cases and functions are identical to React. I could have done the same thing with a “build your own react” article. This is what I don’t get about the LLM hype is that 99% of the examples are people claiming they invented something new with it. We had code generators before LLM-hype took off. Now we have code generators that just steal work and repurpose it as something claimed original.

buddhistdude 2y ago
no programmer in my company invents things often
- righthand 2y ago
  And so you would accept “hey I spun up a react-create-element project, but instead of React I asked an LLM to copy the parts I needed for react so we have another dependency to maintain instead of tree shaking with webpack” as a useful work?
  - buddhistdude 2y ago
    not necessarily, but it's not less creative and inventive than what I believe most programmers are doing most of the time. there are relatively few people who invent new patterns (and they actually might be overrepresented on this website). the rest learns and applies those patterns.
    - righthand 2y ago
      Right that is well understood, but having an LLM compile together functions under the guise of custom built library is hardly a software engineer applying established patterns.
      - jeswin 2y ago
        It is exactly the same as applying established patterns - patterns are what the LLMs have trained on. It seems you haven't really used LLMs for coding. They're super useful and improving every month - you don't have to take my word for it. And btw - codespin (https://github.com/codespin-ai/codespin https://github.com/codespin-ai/codespin) along with the VSCode plugin is what I use for AI-assisted coding many times. That was also generated via an LLM. I wrote it last year, and at that point there weren't many projects it could copy from.

codelikeawolf 2y ago

> The quality of test cases are comparable to what a human would produce. This has actually been a problem for me. I spent a lot of time getting good at writing tests and learning the best approaches to testing things. Most devs I've worked with treat tests as second-class citizens. They either try to treat them like production code and over-abstract everything, which makes the tests difficult to navigate, or they dump a bunch of crap in a file, ignore any conventions or standards, and write superfluous test cases that don't provide any value (if I see one more "it renders successfully" test in a React project, I'm going to lose it). The tests generated by these LLMs is comparable in quality to what most humans have produced, which isn't saying much. Getting good at testing isn't like getting good at most things. It's sort of thankless, and when I point out issues in the quality of the tests, I imagine I'm getting some eye rolls. Who cares, they're just tests, at least we have them, right? But it's code you have to read and maintain, and it will break, and you'll have to fix it. I'm not saying I'm a testing wizard or anything like that. But I really sympathize with the author, because there's a lot of crappy test code coming out of these LLMs. Edit: grammar