3 ms·

My cursor workflow for getting tests is to make the test file, import the code under test, and then type cmd-k, “unit tests for <class>”, enter. Add additional

by jitl 2y ago

My cursor workflow for getting tests is to make the test file, import the code under test, and then type cmd-k, “unit tests for <class>”, enter. Add additional cmd-k to prompt for method tests and cases as needed. Pretty basic, would adding more shenanigans get me better results? I try to write doc comments for methods with contracts and it seems cursor/claude does a good job reading and testing those.

peterldowns 2y ago
What value are the tests giving you? Asking truly naively, I haven't tried generating tests for code before, and I'm skeptical that it would be useful for me.
- jitl 2y ago
  The tests provide the same value as usual: early detection of coding mistakes, (somewhat) format specification of contracts, assurance the code does what it's supposed to do and continues to do what it's supposed to do as the codebase evolves over time.
  - peterldowns 2y ago
    Could you share an example test (if it doesn't violate your privacy)? Your answer is too vague for me to understand. I can't square my experience of LLMs (generate braindead code for well-documented situations) with the kinds of tests I write (enforcing domain-specific business logic during execution of specific scenarios) about which LLMs know nothing (not the domain-specific logic, not the scenarios, not my test helpers and mock implementations.)
    - jitl 2y ago
      This is the open-source example I have: https://github.com/justjake/eslint-seatbelt/blob/main/src/SeatbeltFile.test.ts https://github.com/justjake/eslint-seatbelt/blob/main/src/Se... Maybe you can declare this is braindead code for well-documented situations? not sure. My rough draft implementation of this class had bugs, I asked Cursor to write tests - prompt was something like "can you set up tests for SeatbeltFile.ts using node:test framework", it added "test" script to package.json and wrote 90% of the test file, then I fixed the bugs, added another case myself, committed the tests.
      - peterldowns 2y ago
        Interesting, thank you very much for sharing. I wouldn't have expected an LLM to write those tests. This makes me want to try again for some of the work I've been doing. Thank you!
arnvald 2y ago
I tried that with Cody (from Sourcegraph) using probably o1 model and I struggled. I had a long function with a number of conditions and early returns. At first Cody generated a single test case, then I asked if to consider all possible scenarios in that function (around 10), it generated 5 cases, 3 of which were correct and 2 scenarios were made up (it tried to use nonexistent enum values). In the end I used it to generate some mocks and a few minor things but then I copy pasted the test myself a few times and changed the values manually. Did it save me some time in the end? Possibly, but also it caused a lot of frustration. I still hope it gets better because generating tests should be a great use case for LLMs

by jitl 2y ago

peterldowns 2y ago

What value are the tests giving you? Asking truly naively, I haven't tried generating tests for code before, and I'm skeptical that it would be useful for me.

jitl 2y ago
The tests provide the same value as usual: early detection of coding mistakes, (somewhat) format specification of contracts, assurance the code does what it's supposed to do and continues to do what it's supposed to do as the codebase evolves over time.
- peterldowns 2y ago
  Could you share an example test (if it doesn't violate your privacy)? Your answer is too vague for me to understand. I can't square my experience of LLMs (generate braindead code for well-documented situations) with the kinds of tests I write (enforcing domain-specific business logic during execution of specific scenarios) about which LLMs know nothing (not the domain-specific logic, not the scenarios, not my test helpers and mock implementations.)
  - jitl 2y ago
    This is the open-source example I have: https://github.com/justjake/eslint-seatbelt/blob/main/src/SeatbeltFile.test.ts https://github.com/justjake/eslint-seatbelt/blob/main/src/Se... Maybe you can declare this is braindead code for well-documented situations? not sure. My rough draft implementation of this class had bugs, I asked Cursor to write tests - prompt was something like "can you set up tests for SeatbeltFile.ts using node:test framework", it added "test" script to package.json and wrote 90% of the test file, then I fixed the bugs, added another case myself, committed the tests.
    - peterldowns 2y ago
      Interesting, thank you very much for sharing. I wouldn't have expected an LLM to write those tests. This makes me want to try again for some of the work I've been doing. Thank you!

arnvald 2y ago

I tried that with Cody (from Sourcegraph) using probably o1 model and I struggled. I had a long function with a number of conditions and early returns. At first Cody generated a single test case, then I asked if to consider all possible scenarios in that function (around 10), it generated 5 cases, 3 of which were correct and 2 scenarios were made up (it tried to use nonexistent enum values). In the end I used it to generate some mocks and a few minor things but then I copy pasted the test myself a few times and changed the values manually. Did it save me some time in the end? Possibly, but also it caused a lot of frustration. I still hope it gets better because generating tests should be a great use case for LLMs