5 ms·

Love the idea & spitballing ways to generalize to coding.. Thought experiment: as you write code, an LLM generates tests for it & the IDE runs those tests as y

Love the idea & spitballing ways to generalize to coding.. Thought experiment: as you write code, an LLM generates tests for it & the IDE runs those tests as you type, showing which ones are passing & failing, updating in real time. Imagine 10-100 tests that take <1ms to run, being rerun with every keystroke, and the result being shown in a non-intrusive way. The tests could appear in a separated panel next to your code, and pass/fail status in the gutter of that panel. As simple as red and green dots for tests that passed or failed in the last run. The presence or absence and content of certain tests, plus their pass/fail state, tells you what the code you’re writing does from an outside perspective. Not seeing the LLM write a test you think you’ll need? Either your test generator prompt is wrong, or the code you’re writing doesn’t do the things you think they do! Making it realtime helps you shape the code. Or if you want to do traditional TDD, the tooling could be reversed so you write the tests and the LLM makes them pass as soon as you stop typing by writing the code.

hnthrowaway121 1y ago
Yes the reverse makes much more sense to me. AI help to spec out the software & then the code has an accepted definition of correctness. People focus on this way less than they should I think
callc 1y ago
Humans writing the test first and LLM writing the code is much better than the reverse. And that is because tests are simply the “truth” and “intention” of the code as a contract. When you give up the work of deciding what the expected inputs and outputs of the code/program is you are no longer in the drivers seat.
- JimDabell 1y ago
  > When you give up the work of deciding what the expected inputs and outputs of the code/program is you are no longer in the drivers seat. You don’t need to write tests for that, you need to write acceptance criteria.
  - ThunderSizzle 1y ago
    As in, a developer would write something in e.g. gherkin, and AI would automatically create the matching unit tests and the production code? That would be interesting. Of course, gherkin tends to just be transpiled into generated code that is customized for the particular test, so I'm not sure how AI can really abstract it away too much.
    - kamaal 1y ago
      All of this at the end reduces to a simple fact at the end of the discussion. You need some of way of precisely telling AI what to do. As it turns out there is only that much you can do with text. Come to think of it, you can write a whole book about a scenery, and yet 100 people will imagine it quite differently. And still that actual photograph would be totally different compared to the imagination of all those 100 people. As it turns out if you wish to describe something accurately enough, you have to write mathematical statements, in other words statements that reduce to true/false answers. We could skip to the end of the discussion here, and say you are better of either writing code directly or test cases. This is just people revisiting logic programming all over again.
      - motorest 1y ago
        > You need some of way of precisely telling AI what to do. I think this is the detail you are not getting quite right. The truth of the matter is that you don't need precision to get acceptable results, at least in 100% of the cases. As everything in software engineering, there is indeed "good enough". Also worth noting, LLMs allow anyone to improve upon "good enough". > As it turns out if you wish to describe something accurately enough, you have to write mathematical statements, in other words statements that reduce to true/false answers. Not really. Nothing prevents you to refer to high-level sets of requirements. For example, if you tell a LLM "enforce Google's style guide", you don't have to concern yourself with how many spaces are in a tab. LLMs have been migrating towards instruction files and prompt files for a while, too.
        1y ago
- kamaal 1y ago
  >>Humans writing the test first and LLM writing the code is much better than the reverse. Isn't that logic programming/Prolog? You basically write the sequence of conditions(i.e tests in our lingo) that have to be true, and the compiler(now AI) generates code for your. Perhaps there has to be a relook on how Logic programming can be done in the modern era to make this more seamless.
- willtemperley 1y ago
  Yes this is fundamental to actually designing software. Still, it would be perfectly reasonable to ask "please write a test which gives y output for x input".
- quantumHazer 1y ago
  I disagree. You can simply code in a way that all test passes and you have more problem than before reviewing the code that is being generated.
cjonas 1y ago
Then do you need tests to validate your tests are correct, otherwise the LLM might just generate passing code even if the test is bad? Or write code that games the system because it's easier to hardcode an output value then to do the actual work. There probably is a setup where this works well, but the LLM and humans need to be able to move across the respective boundaries fluidly... Writing clear requirements and letting the AI take care of the bulk of both sides seems more streamlined and productive.
- cellis 1y ago
  The harder part is “test invalidation”. For instance if a feature no longer makes sense, the human / test validator must painstakingly go through and delete obsolete specs. An idea I’d like to try is to “separate” the concerns; only QA agents can delete specs, engineer agents must conform to the suite, and make a strong case to the qa agent for deletion.
andsoitis 1y ago
> Imagine 10-100 tests that take <1ms to run, being rerun with every keystroke, and the result being shown in a non-intrusive way. Doesn’t seem like high ROI to run full suite of tests on each keystroke. Most keystrokes yield an incomplete program, so you want to be smarter about when you run the tests to get a reasonably good trade off.
- vidarh 1y ago
  You could prune this drastically by just tokenizing the file with a lexer suitable for the language, turn them into a canonical state (e.g. replace the contents of any comment tokens with identical text), and check if the token state has changed. If you have a restartable lexer, you can even re-tokenize only from the current line until the state converges again or you encounter a syntax error.
- deleted 1y ago
  [deleted]
scottgg 1y ago
WallabyJS does something along these lines, although I don’t think it is contextually understanding which tests to highlight https://wallabyjs.com/ https://wallabyjs.com/
William_BB 1y ago
There's no way this would work for any serious C++ codebase. Compile times alone make this impossible I'm also not sure how LLM could guess what the tests should be without having written all of the code, e.g. imagine writing code for a new data structure
- motorest 1y ago
  > There's no way this would work for any serious C++ codebase. Compile times alone make this impossible There's nothing in C++ that prevents this. If build times are your bogeyman, you'd be pleased to know that all mainstream build systems support incremental builds.
  - William_BB 1y ago
    The original example was (paraphrasing) "rerunning 10-100 tests that take 1ms after each keystroke". Even with incremental builds, that surely does not sound plausible? I only mentioned C++ because that's my main working language, but this wouldn't sound reasonable for Rust either, no?
    - motorest 1y ago
      > The original example was (paraphrasing) "rerunning 10-100 tests that take 1ms after each keystroke". Yeah, OP's point is completely unrealistic and doesn't reflect real-world experience. This sort of test watchers is mundane in any project involving JavaScript, and not even those tests re-run at each keystroke. Watch mode triggers tests when they detect changes, and waits for test executions to finish to re-run tests. This feature consists of running a small command line app that is designed to run a command whenever specific files within a project tree are touched. There is zero requirement to only watch for JavaScript files or only trigger npm build when a file changes. To be very clear, this means that right now anyone at all, including you and me, can install a watcher, configure it to run make test/cutest/etc when any file in your project is touched, and call it a day. This is a 5 minute job. By the way, nowadays even Microsoft's dotnet tool supports watch mode, which means there's out-of-the-box support to "rerunning 10-100 tests that take 1ms after each keystroke".
motorest 1y ago
> Thought experiment: as you write code, an LLM generates tests for it & the IDE runs those tests as you type, showing which ones are passing & failing, updating in real time. Imagine 10-100 tests that take <1ms to run, being rerun with every keystroke, and the result being shown in a non-intrusive way. I think this is a bad approach. Tests enforce invariants, and they are exactly the type of code we don't want LLMs to touch willy-nilly. You want your tests to only change if you explicitly want them to, and even then only the tests should change. Once you adopt that constraint, you'll quickly realize ever single detail of your thought experiment is already a mundane workflow in any developer's day-to-day activities. Consider the fact that watch mode is a staple of any JavaScript testing framework, and those even found their way into .NET a couple of years ago. So, your thought experiment is something professional software developers have been doing for what? A decade now?
- cadamsdotcom 1y ago
  I think tests should be rewritten as much as needed. But to counter the invariant part, maybe let the user zoom back and forth through past revisions and pull in whatever they want to the current version, in case something important is deleted? And then allow “pinning” of some stuff so it can’t be changed? Would that solve for your concerns?
  - motorest 1y ago
    > I think tests should be rewritten as much as needed. Yes, I agree. The nuance is that they need to be rewritten independently and without touching the code. You can't change both and expect to get a working system. I'm speaking based on personal experience, by the way. Today's LLMs don't enforce correctness out of the box and agent mode has only one goal: getting things to work. I had agent mode flip invariants in tests when trying to fix unit tests it broke, and I'm talking about egregious changes such as flipping requirements in line with "normal users should not have access to the admin panel" to "normal users should have access to the admin panel". The worst part is that if agent mode is left unsupervised, it will even adjust the CSS to make sure normal users have a seamless experience going through the admin panel.
federiconafria 1y ago
That's already part of most IDE's and they know which tests to re-run, because of coverage, so it's really fast. It also updates the coverage on the fly, you don't even have to look at the test output to know that you've broken something since the tests are not reaching your lines. https://gavindraper.com/2020/05/27/VS-Code-Continious-Testing/ https://gavindraper.com/2020/05/27/VS-Code-Continious-Testin...
Cthulhu_ 1y ago
Besides generating the tests, automatically running tests on edit and showing the results inline is already a thing. I think it'd be better to do it the other way around, start with the tests and let the LLM implement it until all tests are green. Test driven development.
squigz 1y ago
> Imagine 10-100 tests that take <1ms to run, being rerun with every keystroke, and the result being shown in a non-intrusive way. Even if this were possible, this seems like an absolutely colossal waste of energy - both the computer's, and my own. Why would I want incomplete tests generated after every keystroke? Why would I test an incomplete if statement or some such?

hnthrowaway121 1y ago

Yes the reverse makes much more sense to me. AI help to spec out the software & then the code has an accepted definition of correctness. People focus on this way less than they should I think

callc 1y ago

Humans writing the test first and LLM writing the code is much better than the reverse. And that is because tests are simply the “truth” and “intention” of the code as a contract. When you give up the work of deciding what the expected inputs and outputs of the code/program is you are no longer in the drivers seat.

JimDabell 1y ago
> When you give up the work of deciding what the expected inputs and outputs of the code/program is you are no longer in the drivers seat. You don’t need to write tests for that, you need to write acceptance criteria.
- ThunderSizzle 1y ago
  As in, a developer would write something in e.g. gherkin, and AI would automatically create the matching unit tests and the production code? That would be interesting. Of course, gherkin tends to just be transpiled into generated code that is customized for the particular test, so I'm not sure how AI can really abstract it away too much.
  - kamaal 1y ago
    All of this at the end reduces to a simple fact at the end of the discussion. You need some of way of precisely telling AI what to do. As it turns out there is only that much you can do with text. Come to think of it, you can write a whole book about a scenery, and yet 100 people will imagine it quite differently. And still that actual photograph would be totally different compared to the imagination of all those 100 people. As it turns out if you wish to describe something accurately enough, you have to write mathematical statements, in other words statements that reduce to true/false answers. We could skip to the end of the discussion here, and say you are better of either writing code directly or test cases. This is just people revisiting logic programming all over again.
    - motorest 1y ago
      > You need some of way of precisely telling AI what to do. I think this is the detail you are not getting quite right. The truth of the matter is that you don't need precision to get acceptable results, at least in 100% of the cases. As everything in software engineering, there is indeed "good enough". Also worth noting, LLMs allow anyone to improve upon "good enough". > As it turns out if you wish to describe something accurately enough, you have to write mathematical statements, in other words statements that reduce to true/false answers. Not really. Nothing prevents you to refer to high-level sets of requirements. For example, if you tell a LLM "enforce Google's style guide", you don't have to concern yourself with how many spaces are in a tab. LLMs have been migrating towards instruction files and prompt files for a while, too.
      - 1y ago
kamaal 1y ago
>>Humans writing the test first and LLM writing the code is much better than the reverse. Isn't that logic programming/Prolog? You basically write the sequence of conditions(i.e tests in our lingo) that have to be true, and the compiler(now AI) generates code for your. Perhaps there has to be a relook on how Logic programming can be done in the modern era to make this more seamless.
willtemperley 1y ago
Yes this is fundamental to actually designing software. Still, it would be perfectly reasonable to ask "please write a test which gives y output for x input".
quantumHazer 1y ago
I disagree. You can simply code in a way that all test passes and you have more problem than before reviewing the code that is being generated.

cjonas 1y ago

Then do you need tests to validate your tests are correct, otherwise the LLM might just generate passing code even if the test is bad? Or write code that games the system because it's easier to hardcode an output value then to do the actual work. There probably is a setup where this works well, but the LLM and humans need to be able to move across the respective boundaries fluidly... Writing clear requirements and letting the AI take care of the bulk of both sides seems more streamlined and productive.

cellis 1y ago
The harder part is “test invalidation”. For instance if a feature no longer makes sense, the human / test validator must painstakingly go through and delete obsolete specs. An idea I’d like to try is to “separate” the concerns; only QA agents can delete specs, engineer agents must conform to the suite, and make a strong case to the qa agent for deletion.

andsoitis 1y ago

> Imagine 10-100 tests that take <1ms to run, being rerun with every keystroke, and the result being shown in a non-intrusive way. Doesn’t seem like high ROI to run full suite of tests on each keystroke. Most keystrokes yield an incomplete program, so you want to be smarter about when you run the tests to get a reasonably good trade off.

vidarh 1y ago
You could prune this drastically by just tokenizing the file with a lexer suitable for the language, turn them into a canonical state (e.g. replace the contents of any comment tokens with identical text), and check if the token state has changed. If you have a restartable lexer, you can even re-tokenize only from the current line until the state converges again or you encounter a syntax error.
deleted 1y ago
[deleted]

scottgg 1y ago

WallabyJS does something along these lines, although I don’t think it is contextually understanding which tests to highlight https://wallabyjs.com/ https://wallabyjs.com/

William_BB 1y ago

There's no way this would work for any serious C++ codebase. Compile times alone make this impossible I'm also not sure how LLM could guess what the tests should be without having written all of the code, e.g. imagine writing code for a new data structure

motorest 1y ago
> There's no way this would work for any serious C++ codebase. Compile times alone make this impossible There's nothing in C++ that prevents this. If build times are your bogeyman, you'd be pleased to know that all mainstream build systems support incremental builds.
- William_BB 1y ago
  The original example was (paraphrasing) "rerunning 10-100 tests that take 1ms after each keystroke". Even with incremental builds, that surely does not sound plausible? I only mentioned C++ because that's my main working language, but this wouldn't sound reasonable for Rust either, no?
  - motorest 1y ago
    > The original example was (paraphrasing) "rerunning 10-100 tests that take 1ms after each keystroke". Yeah, OP's point is completely unrealistic and doesn't reflect real-world experience. This sort of test watchers is mundane in any project involving JavaScript, and not even those tests re-run at each keystroke. Watch mode triggers tests when they detect changes, and waits for test executions to finish to re-run tests. This feature consists of running a small command line app that is designed to run a command whenever specific files within a project tree are touched. There is zero requirement to only watch for JavaScript files or only trigger npm build when a file changes. To be very clear, this means that right now anyone at all, including you and me, can install a watcher, configure it to run make test/cutest/etc when any file in your project is touched, and call it a day. This is a 5 minute job. By the way, nowadays even Microsoft's dotnet tool supports watch mode, which means there's out-of-the-box support to "rerunning 10-100 tests that take 1ms after each keystroke".

motorest 1y ago

> Thought experiment: as you write code, an LLM generates tests for it & the IDE runs those tests as you type, showing which ones are passing & failing, updating in real time. Imagine 10-100 tests that take <1ms to run, being rerun with every keystroke, and the result being shown in a non-intrusive way. I think this is a bad approach. Tests enforce invariants, and they are exactly the type of code we don't want LLMs to touch willy-nilly. You want your tests to only change if you explicitly want them to, and even then only the tests should change. Once you adopt that constraint, you'll quickly realize ever single detail of your thought experiment is already a mundane workflow in any developer's day-to-day activities. Consider the fact that watch mode is a staple of any JavaScript testing framework, and those even found their way into .NET a couple of years ago. So, your thought experiment is something professional software developers have been doing for what? A decade now?

cadamsdotcom 1y ago
I think tests should be rewritten as much as needed. But to counter the invariant part, maybe let the user zoom back and forth through past revisions and pull in whatever they want to the current version, in case something important is deleted? And then allow “pinning” of some stuff so it can’t be changed? Would that solve for your concerns?
- motorest 1y ago
  > I think tests should be rewritten as much as needed. Yes, I agree. The nuance is that they need to be rewritten independently and without touching the code. You can't change both and expect to get a working system. I'm speaking based on personal experience, by the way. Today's LLMs don't enforce correctness out of the box and agent mode has only one goal: getting things to work. I had agent mode flip invariants in tests when trying to fix unit tests it broke, and I'm talking about egregious changes such as flipping requirements in line with "normal users should not have access to the admin panel" to "normal users should have access to the admin panel". The worst part is that if agent mode is left unsupervised, it will even adjust the CSS to make sure normal users have a seamless experience going through the admin panel.

federiconafria 1y ago

That's already part of most IDE's and they know which tests to re-run, because of coverage, so it's really fast. It also updates the coverage on the fly, you don't even have to look at the test output to know that you've broken something since the tests are not reaching your lines. https://gavindraper.com/2020/05/27/VS-Code-Continious-Testing/ https://gavindraper.com/2020/05/27/VS-Code-Continious-Testin...

Cthulhu_ 1y ago

Besides generating the tests, automatically running tests on edit and showing the results inline is already a thing. I think it'd be better to do it the other way around, start with the tests and let the LLM implement it until all tests are green. Test driven development.

squigz 1y ago

> Imagine 10-100 tests that take <1ms to run, being rerun with every keystroke, and the result being shown in a non-intrusive way. Even if this were possible, this seems like an absolutely colossal waste of energy - both the computer's, and my own. Why would I want incomplete tests generated after every keystroke? Why would I test an incomplete if statement or some such?