3 ms·

REPL-driven development seems like a natural fit with current agent concepts nb: Bruce also did figwheel, a hot/live reloading tool for Clojurescript https://

by tosh 1y ago

REPL-driven development seems like a natural fit with current agent concepts nb: Bruce also did figwheel, a hot/live reloading tool for Clojurescript https://figwheel.org https://figwheel.org

pona-a 1y ago
I'm no sure, if you mean having the LLM interact with the REPL. REPL requires a lot of discipline from the developer to keep track of its state. LLMs seem to be far worse at this sort of long-term state tracking than most humans. It would likely keep forgetting things, calling undefined functions, or mistaking its own interfaces.
- tosh 1y ago
  I agree, it’s tricky, but agents are also great at inspecting stuff right now the advice in the readme makes a lot of sense: use git (or similar) for checkpointing a bit like climbing with ropes and anchors
- diggan 1y ago
  > REPL requires a lot of discipline from the developer to keep track of its state More discipline than what is required when trying to keep track of all of that in your head? Personally I use a REPL because I'm lazy and dumb, and the REPL helps me keep track and makes sure I don't do stupid mistakes. If I wanna know what value a thing has, I just ask the REPL, instead of trying to retrace some logic by reading the code again.
  - dkarl 1y ago
    What I don't like about building systems at the REPL is that when I investigate a possibility and it works out, I have to identify the code that's actually needed, clean it up, and turn my ad-hoc tests in the REPL into unit tests. There's a trade-off between quick experimentation and the mess I create in search of immediate feedback. I think an LLM could help with that. Ideally, I could say, "Please add the widget-muckle-frobnicate function from the current REPL session to the codebase. Fix or flag any shortcuts I took in its implementation, such as hardcoding configuration and naming functions 'foo' or 'bar', and add unit tests that include the examples I tried in the REPL as well as property-based tests to exercise edge cases."
    - diggan 1y ago
      > Please add the widget-muckle-frobnicate function from the current REPL session to the codebase. I think this is your problem, those two things should be the same, not two different things. When I'm working with a Clojure REPL, I write down code in a .clj file, then evaluate that code via the REPL but without leaving my editor. "Moving" something from a REPL session to the codebase is a matter of saving the current file you're editing. I wrote more about the exact process I typically use when going from "prototyping with a repl, to having a test case" (https://news.ycombinator.com/item?id=44106445 https://news.ycombinator.com/item?id=44106445) and it's essentially changing "comment" to "deftest" then saving the file again, now it's a unit test. I do find value from LLMs in general, but I don't see much value in taking a process that takes a minute or less and trying to make LLMs do it.
      - sokoloff 1y ago
        Exactly (on the first two paragraphs). I (almost) never type anything into the REPL window itself. I might type a "5" and evaluate it just to see if the REPL is busy or not, but otherwise, everything the REPL evaluates comes from an emacs source code buffer, sent via CIDER to the REPL. At most, it might be in a "Rich comment" and thus not evaluated next time I load that file, but those tend to be small, interactive tests and not defns. Unless emacs crashes with the buffer unsaved, there's nothing the REPL has that the source file doesn't.
      - dkarl 1y ago
        I usually have a buffer that is kind of like a Jupyter notebook, and it accumulates a lot of stuff that may or may not turn out to be useful in the end. If I'm doing the work to keep things tidy as I go, so it's close to production ready at the end, then it isn't much different from traditional programming. It ends up just being TDD in one file instead of two.
- pydry 1y ago
  REPL requires less discipline if you combine it with TDD. If there's an "abandon ship, tear down and set up the right state from scratch" red button it helps.
worldsayshi 1y ago
I would like to see how to easily get regression tests from REPL driven development. In general I wonder how to quickly set up mocks and such with chatbot integrations.
- diggan 1y ago
  > how to easily get regression tests from REPL driven development Usually it's a matter of renaming "comment" to "deftest", and move some things around. Not sure how/why it would be different when using a LLM compared to doing it manually.
  - rads 1y ago
    There is often a misunderstanding that “REPL driven development” means typing things into the REPL prompt interactively and accumulating state that’s not easy to reproduce. Writing code down in comment blocks as you go is the solution to that, as you mentioned. With that technique I have no fear of restarting my REPL and getting back to where I was.
    - diggan 1y ago
      That's true. To be fair, if you come from a language like Ruby, JavaScript or Python, a REPL is basically a standalone experience, so I understand why people initially think it's like that, the name is more or less the same :) "Interactive Development" is maybe a better overall term that can lead to people thinking "Oh, what is that?" rather than "I've tried that already, what's the point?"
      - worldsayshi 1y ago
        I like that. So you're saying that the switch between REPL and writing a test is much more seamless in a clojure system. That makes a lot of sense to me.
        dandelion9 1y ago

by tosh 1y ago

REPL-driven development seems like a natural fit with current agent concepts nb: Bruce also did figwheel, a hot/live reloading tool for Clojurescript https://figwheel.org https://figwheel.org

pona-a 1y ago

I'm no sure, if you mean having the LLM interact with the REPL. REPL requires a lot of discipline from the developer to keep track of its state. LLMs seem to be far worse at this sort of long-term state tracking than most humans. It would likely keep forgetting things, calling undefined functions, or mistaking its own interfaces.

tosh 1y ago
I agree, it’s tricky, but agents are also great at inspecting stuff right now the advice in the readme makes a lot of sense: use git (or similar) for checkpointing a bit like climbing with ropes and anchors
diggan 1y ago
> REPL requires a lot of discipline from the developer to keep track of its state More discipline than what is required when trying to keep track of all of that in your head? Personally I use a REPL because I'm lazy and dumb, and the REPL helps me keep track and makes sure I don't do stupid mistakes. If I wanna know what value a thing has, I just ask the REPL, instead of trying to retrace some logic by reading the code again.
- dkarl 1y ago
  What I don't like about building systems at the REPL is that when I investigate a possibility and it works out, I have to identify the code that's actually needed, clean it up, and turn my ad-hoc tests in the REPL into unit tests. There's a trade-off between quick experimentation and the mess I create in search of immediate feedback. I think an LLM could help with that. Ideally, I could say, "Please add the widget-muckle-frobnicate function from the current REPL session to the codebase. Fix or flag any shortcuts I took in its implementation, such as hardcoding configuration and naming functions 'foo' or 'bar', and add unit tests that include the examples I tried in the REPL as well as property-based tests to exercise edge cases."
  - diggan 1y ago
    > Please add the widget-muckle-frobnicate function from the current REPL session to the codebase. I think this is your problem, those two things should be the same, not two different things. When I'm working with a Clojure REPL, I write down code in a .clj file, then evaluate that code via the REPL but without leaving my editor. "Moving" something from a REPL session to the codebase is a matter of saving the current file you're editing. I wrote more about the exact process I typically use when going from "prototyping with a repl, to having a test case" (https://news.ycombinator.com/item?id=44106445 https://news.ycombinator.com/item?id=44106445) and it's essentially changing "comment" to "deftest" then saving the file again, now it's a unit test. I do find value from LLMs in general, but I don't see much value in taking a process that takes a minute or less and trying to make LLMs do it.
    - sokoloff 1y ago
      Exactly (on the first two paragraphs). I (almost) never type anything into the REPL window itself. I might type a "5" and evaluate it just to see if the REPL is busy or not, but otherwise, everything the REPL evaluates comes from an emacs source code buffer, sent via CIDER to the REPL. At most, it might be in a "Rich comment" and thus not evaluated next time I load that file, but those tend to be small, interactive tests and not defns. Unless emacs crashes with the buffer unsaved, there's nothing the REPL has that the source file doesn't.
    - dkarl 1y ago
      I usually have a buffer that is kind of like a Jupyter notebook, and it accumulates a lot of stuff that may or may not turn out to be useful in the end. If I'm doing the work to keep things tidy as I go, so it's close to production ready at the end, then it isn't much different from traditional programming. It ends up just being TDD in one file instead of two.
pydry 1y ago
REPL requires less discipline if you combine it with TDD. If there's an "abandon ship, tear down and set up the right state from scratch" red button it helps.

worldsayshi 1y ago

I would like to see how to easily get regression tests from REPL driven development. In general I wonder how to quickly set up mocks and such with chatbot integrations.

diggan 1y ago
> how to easily get regression tests from REPL driven development Usually it's a matter of renaming "comment" to "deftest", and move some things around. Not sure how/why it would be different when using a LLM compared to doing it manually.
- rads 1y ago
  There is often a misunderstanding that “REPL driven development” means typing things into the REPL prompt interactively and accumulating state that’s not easy to reproduce. Writing code down in comment blocks as you go is the solution to that, as you mentioned. With that technique I have no fear of restarting my REPL and getting back to where I was.
  - diggan 1y ago
    That's true. To be fair, if you come from a language like Ruby, JavaScript or Python, a REPL is basically a standalone experience, so I understand why people initially think it's like that, the name is more or less the same :) "Interactive Development" is maybe a better overall term that can lead to people thinking "Oh, what is that?" rather than "I've tried that already, what's the point?"
    - worldsayshi 1y ago
      I like that. So you're saying that the switch between REPL and writing a test is much more seamless in a clojure system. That makes a lot of sense to me.
      - dandelion9 1y ago