6 ms·

Can't tell you how much I love it for testing, it's basically the only thing I use it for. I now have a test suite that can rebuild my entire app from the groun

Can't tell you how much I love it for testing, it's basically the only thing I use it for. I now have a test suite that can rebuild my entire app from the ground up locally, and works in the cloud as well. It's a huge motivator actually to write a piece of code with the reward being the ability to send it to the LLM to create some tests and then seeing a nice stream of green checkmarks.

sarchertech 2y ago
[flagged]
- PaulHoule 2y ago
  I had Codeium add something to a function that added a new data value to an object. Unbidden it wrote three new tests, good tests. I wrote my own test by cutting and pasting a test it wrote with a modification, it pointed out that I didn’t edit the comment so I told it to do so. It also screwed up the imports of my tests pretty bad, some imports that worked before got changed for no good reason. It replaced the JetBrains NotNull annotation with a totally different annotation. It was able to figure out how to update a DAO object when I added a new field. It got the type of the field wrong when updating the object corresponding to a row in that database column even though it wrote the liquibase migration and should have known the type —- we had chatted plenty about that migration. It got many things right but I had to fix a lot of mistakes. It is not clear that it really saves time.
  - imp0cat 2y ago
    Let's be clear here, Codeium kinda sucks. Yeah, it's free and it works, somewhat. But I wouldn't trust it much.
  - MarcelOlsz 2y ago
    Try using Cursor with the latest claude-3-5-sonnet-20241022.
    - PaulHoule 2y ago
      Unfortunately I “think different” and use Windows. I use Microsoft Copilot and would say it is qualitatively similar to codeium in quality, a real quantitative eval would be a lot of work.
      - MarcelOlsz 2y ago
        Cursor (cursor.com) is just a vscode wrapper, should work fine with Windows. If you're already in the AI coding space I seriously urge you to at least give it a go.
        PaulHoule 2y ago
        I'll look into it. I'll add that my experience with the Codium plugin for IntelliJ is night and day different from the Windsurf editor from Codium. The first one "just doesn't work" and struggles to see files that are in my project, the second basically works.
        MarcelOlsz 2y ago
        You can also look into https://www.greptile.com/ https://www.greptile.com/ to ask codebase questions. There's so many AI coding tools out there now. I've heard good things about https://codebuddy.ca/ https://codebuddy.ca/ as well (for IntelliJ) and https://www.continue.dev/ https://www.continue.dev/ (also for IntelliJ). >The first one "just doesn't work" Haha. You're on a roll.
    - lubujackson 2y ago
      Seconding Cursor. I have a friend who used Copilot 6 mo. ago and found it vaguely helpful... but turned him on to Cursor and it's a whole new ballgame. Cross between actually useful autocomplete, personalized StackOverflow and error diagnosis (just paste and error message in chat). I know I am just scratching the usefulness and I pretty much never do changes across multiple files, but I definitely see firm net positives at this point.
- MarcelOlsz 2y ago
  It's containerized and I have a script that takes care of everything from the ground up :) I've tested this on multiple OS' and friends computers. I'm thankful to old me for writing a readme for current me lol. >Please tell me this is satire. No. I started doing TDD. It's fun to think about a piece of functionality, write out some tests, and then slowly make it pass. Removes a lot of cognitive load for me and serves as a micro todo. It's also nice that when you're working and think of something to add, you can just write out a quick test for it and add it to kanban later. I can't tell you how many times I've worked on projects that are gigantic in complexity and don't do any testing, or use typescript, or both. You're always like 25% paranoid about everything you do and it's the worst.
  - sarchertech 2y ago
    >It's a huge motivator actually to write a piece of code with the reward being the ability to send it to the LLM to create some tests and then seeing a nice stream of green checkmarks. Yeah that’s not TDD.
    - MarcelOlsz 2y ago
      Don't you have a book to get to writing instead of leaving useless comments? Haha.
      - sarchertech 2y ago
        More importantly I have a French cleat wall to finish, a Christmas present to make for my wife, and a toddler and infant to keep from killing themselves. But I also have a day job and I can’t even begin to imagine how much extra work someone doing “TDD” by writing a function and then fixing it in place with a whole suite of generated tests would cause me. I’m fine with TDD. I do it myself fairly often. I also go back in and delete the tests that I used to build it that aren’t actually going to be useful a year from now.
        MarcelOlsz 2y ago
- dang 2y ago
  "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - sarchertech 2y ago
    We continued this discussion and the OP actually meant that they write tests first and then use those tests to help them write code. However what they said was they write code then send that to an LLM to generate tests. There’s no other way to interpret what they wrote. There’s no way to get “I write tests using plain English and an LLM, then use those tests to help me write code from “I write code, I send that code to an LLM to generate tests, those tests give me a satisfying series of green checks”. This is a case of the OP writing something that is close to the opposite of what they meant and you can’t reasonably iron man that to get to a stronger position for them. I get that my wording was too harsh, but I estimated a 50% chance it was actually satire.
highfrequency 2y ago
> I now have a test suite that can rebuild my entire app from the ground up What does this mean?
- MarcelOlsz 2y ago
  Sorry, should have been more clear. Firebase is (or was) a PITA when I started the app I'm working on a few years ago. I have a lot of records in my db that I need to validate after normalizing the data. I used to have an admin page that spit out a bunch of json data with some basic filtering and self-rolled testing that I could verify at a glance. After a few years off from this project, I refactored it all, and part of that refactoring was building a test suite that I can run. When ran, it will rebuild, normalize, and verify all the data in my app (scraped data). When I deploy, it will also run these tests and then email if something breaks, but skip the seeding portion. I had plans to do this before but the firebase emulator still had a lot of issues a few years ago, and refactoring this project gave me the freedom to finally build a proper testing environment and make my entire app make full use of my local firebase emulator without issue. I like giving it my test cases in plain english. It still gets them wrong sometimes but 90% of the time they are good to go.
rr808 2y ago
I struggle to get github copilot to create any unit tests that provide any value. How to you get it to create really useful tests?
- BillyTheKing 2y ago
  Would recommend to try out anthropic sonnet 3.5 for this one - usually generates decent unit tests for reasonably sized functions
- MarcelOlsz 2y ago
  I use claude-3-5-sonnet-20241022 with a very explicit .cursorrules file with the cursor editor.
  - ponector 2y ago
    Can you share your .cursorrules? For me cursor is not much better than autocomplete, but I'm writing mostly e2e tests.
    - MarcelOlsz 2y ago
      You can find a bunch on https://cursor.directory/ https://cursor.directory/.

sarchertech 2y ago

[flagged]

PaulHoule 2y ago
I had Codeium add something to a function that added a new data value to an object. Unbidden it wrote three new tests, good tests. I wrote my own test by cutting and pasting a test it wrote with a modification, it pointed out that I didn’t edit the comment so I told it to do so. It also screwed up the imports of my tests pretty bad, some imports that worked before got changed for no good reason. It replaced the JetBrains NotNull annotation with a totally different annotation. It was able to figure out how to update a DAO object when I added a new field. It got the type of the field wrong when updating the object corresponding to a row in that database column even though it wrote the liquibase migration and should have known the type —- we had chatted plenty about that migration. It got many things right but I had to fix a lot of mistakes. It is not clear that it really saves time.
- imp0cat 2y ago
  Let's be clear here, Codeium kinda sucks. Yeah, it's free and it works, somewhat. But I wouldn't trust it much.
- MarcelOlsz 2y ago
  Try using Cursor with the latest claude-3-5-sonnet-20241022.
  - PaulHoule 2y ago
    Unfortunately I “think different” and use Windows. I use Microsoft Copilot and would say it is qualitatively similar to codeium in quality, a real quantitative eval would be a lot of work.
    - MarcelOlsz 2y ago
      Cursor (cursor.com) is just a vscode wrapper, should work fine with Windows. If you're already in the AI coding space I seriously urge you to at least give it a go.
      - PaulHoule 2y ago
        I'll look into it. I'll add that my experience with the Codium plugin for IntelliJ is night and day different from the Windsurf editor from Codium. The first one "just doesn't work" and struggles to see files that are in my project, the second basically works.
        MarcelOlsz 2y ago
        You can also look into https://www.greptile.com/ https://www.greptile.com/ to ask codebase questions. There's so many AI coding tools out there now. I've heard good things about https://codebuddy.ca/ https://codebuddy.ca/ as well (for IntelliJ) and https://www.continue.dev/ https://www.continue.dev/ (also for IntelliJ). >The first one "just doesn't work" Haha. You're on a roll.
  - lubujackson 2y ago
    Seconding Cursor. I have a friend who used Copilot 6 mo. ago and found it vaguely helpful... but turned him on to Cursor and it's a whole new ballgame. Cross between actually useful autocomplete, personalized StackOverflow and error diagnosis (just paste and error message in chat). I know I am just scratching the usefulness and I pretty much never do changes across multiple files, but I definitely see firm net positives at this point.
MarcelOlsz 2y ago
It's containerized and I have a script that takes care of everything from the ground up :) I've tested this on multiple OS' and friends computers. I'm thankful to old me for writing a readme for current me lol. >Please tell me this is satire. No. I started doing TDD. It's fun to think about a piece of functionality, write out some tests, and then slowly make it pass. Removes a lot of cognitive load for me and serves as a micro todo. It's also nice that when you're working and think of something to add, you can just write out a quick test for it and add it to kanban later. I can't tell you how many times I've worked on projects that are gigantic in complexity and don't do any testing, or use typescript, or both. You're always like 25% paranoid about everything you do and it's the worst.
- sarchertech 2y ago
  >It's a huge motivator actually to write a piece of code with the reward being the ability to send it to the LLM to create some tests and then seeing a nice stream of green checkmarks. Yeah that’s not TDD.
  - MarcelOlsz 2y ago
    Don't you have a book to get to writing instead of leaving useless comments? Haha.
    - sarchertech 2y ago
      More importantly I have a French cleat wall to finish, a Christmas present to make for my wife, and a toddler and infant to keep from killing themselves. But I also have a day job and I can’t even begin to imagine how much extra work someone doing “TDD” by writing a function and then fixing it in place with a whole suite of generated tests would cause me. I’m fine with TDD. I do it myself fairly often. I also go back in and delete the tests that I used to build it that aren’t actually going to be useful a year from now.
      - MarcelOlsz 2y ago
dang 2y ago
"Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- sarchertech 2y ago
  We continued this discussion and the OP actually meant that they write tests first and then use those tests to help them write code. However what they said was they write code then send that to an LLM to generate tests. There’s no other way to interpret what they wrote. There’s no way to get “I write tests using plain English and an LLM, then use those tests to help me write code from “I write code, I send that code to an LLM to generate tests, those tests give me a satisfying series of green checks”. This is a case of the OP writing something that is close to the opposite of what they meant and you can’t reasonably iron man that to get to a stronger position for them. I get that my wording was too harsh, but I estimated a 50% chance it was actually satire.

highfrequency 2y ago

> I now have a test suite that can rebuild my entire app from the ground up What does this mean?

MarcelOlsz 2y ago
Sorry, should have been more clear. Firebase is (or was) a PITA when I started the app I'm working on a few years ago. I have a lot of records in my db that I need to validate after normalizing the data. I used to have an admin page that spit out a bunch of json data with some basic filtering and self-rolled testing that I could verify at a glance. After a few years off from this project, I refactored it all, and part of that refactoring was building a test suite that I can run. When ran, it will rebuild, normalize, and verify all the data in my app (scraped data). When I deploy, it will also run these tests and then email if something breaks, but skip the seeding portion. I had plans to do this before but the firebase emulator still had a lot of issues a few years ago, and refactoring this project gave me the freedom to finally build a proper testing environment and make my entire app make full use of my local firebase emulator without issue. I like giving it my test cases in plain english. It still gets them wrong sometimes but 90% of the time they are good to go.

rr808 2y ago

I struggle to get github copilot to create any unit tests that provide any value. How to you get it to create really useful tests?

BillyTheKing 2y ago
Would recommend to try out anthropic sonnet 3.5 for this one - usually generates decent unit tests for reasonably sized functions
MarcelOlsz 2y ago
I use claude-3-5-sonnet-20241022 with a very explicit .cursorrules file with the cursor editor.
- ponector 2y ago
  Can you share your .cursorrules? For me cursor is not much better than autocomplete, but I'm writing mostly e2e tests.
  - MarcelOlsz 2y ago
    You can find a bunch on https://cursor.directory/ https://cursor.directory/.