6 ms·

The thing I still don't understand about REPL driven development is how you manage state, and how you manage threads. If I have a task running and it depends on

by gaze 4y ago

The thing I still don't understand about REPL driven development is how you manage state, and how you manage threads. If I have a task running and it depends on a collection of global variables, those global variables now need some machinery around them so that they can be edited from the REPL thread. If you replace a function, there now needs to be decisions made about when you now begin usage of the new function. To me, the understanding that some set of things may be replaced right under you adds a great deal of additional engineering complexity. I've gotten various answers about how one deals with this. One being "yes add the machinery" or "just yolo it, yes it's a race condition but it's rarely an issue," neither of which I find particularly satisfying. Concerning state, I've occasionally found myself developing a long-lived lisp image, and then I need to restart the VM for one reason or another, and then found that nothing works. The state of the in-memory image had gotten totally out of sync with the codebase. Perhaps this is a manner of discipline in lisp, but I greatly appreciate the replacement of discipline (be it memory management or the aforementioned situation) with machinery of the language itself.

mark_l_watson 4y ago
On LispWorks REPL, I use the Threads and Processes Viewer.
Joker_vD 4y ago
> how you manage state It's already done for you by the runtime: it holds the state in the process memory. Now, with edit-compile-run-print loop if you want your state to persist between runs, you gotta implement some scheme of data persistence. > it depends on a collection of global variables, those global variables now need some machinery around them so that they can be edited from the REPL thread. In Erlang, you create a public named ETS table (basically, a thread-safe dictionary) and put "global settings" in there instead (and read them from there, too). I'd imagine LISP has something similar. And mind you, I personally prefer the ECRPL instead of REPL, even when working in Python.
- solarmist 4y ago
  What is ECRPL?
  - Joker_vD 4y ago
    "Edit-compile-run-print loop", a non-standard (but fairly clear) term I've used in the beginning of my comment.
    - solarmist 4y ago
      Sure. I just didn’t know it.
Jtsummers 4y ago
The "discipline" is to put the source code into a source file, the same thing you do with every other language. Don't do this and expect positive results: SOME-PACKAGE> (defun a-critical-function (...) ...) The discipline to instead do this: ;; some-package.lisp or whatever (defun a-critical-function (...) ...) Is table stakes for other languages, and Lispers are no worse programmers than programmers in other languages so why would they be incapable of this basic discipline? Only fools would write that in the REPL, never commit it to a source file, and be surprised when they couldn't reproduce the system state later on. Same with loading data. It comes from a database, file, or other source. Preserve the method of loading the data in source like all other code.
- sillysaurusx 4y ago
  > Only fools would write that in the REPL, never commit it to a source file, and be surprised when they couldn't reproduce the system state later on. This happened a few times while I was developing https://laarc.io https://laarc.io. It’s so addictive to just paste new functions into a repl and see the changes instantly that it’s easy to go a few hours without committing and realize your changes rely on a now-deleted function. :) Wouldn’t trade that workflow for the world though. C++ compile/run separation makes it horribly obvious how much productivity you lose. It feels like walking into a pit of molasses. But even Python isn’t much better — I’m constantly tabbing to the repl, typing ctrl-R “reload” <enter>, then pressing the up arrow twice to evaluate the previous function example I was working on. I wish there was an auto reload feature that would just reload everything every time I evaluate a repl expression. (I’ve been writing Bel lisp in Python, and I put in some code to do just that for bel.interact()’s repl. It’s so much nicer not having to reload manually all the time.)
  - tonyarkles 4y ago
    > It’s so addictive to just paste new functions into a repl and see the changes instantly that it’s easy to go a few hours without committing and realize your changes rely on a now-deleted function. :) I think Slime is a pretty happy middle ground. Instead of just pasting, the function gets written in a source file and then C-c C-c to evaluate it in the repl. I did once or twice run into similar issues though from depending on older or newer versions of functions that changed but weren’t completely reflected until a clean restart. But that’s been pretty rare and definitely wasted way less time than “make clean all” before committing C++ code :D
sillysaurusx 4y ago
You’re right. I think the other commenters aren’t being straightforward with you. It is a race condition to load a file one function at a time, because any other thread can preempt you. Arc has a particularly elegant solution to this. Any code you want to happen atomically, you wrap in (atomic …) So (atomic x y z) Will do x, y, then z. During this time, no other threads are allowed to run. Therefore your load-file function might look like (pseudo code because phone): (def load (filename) (atomic (each form (read-file filename) (eval form)))) Now any time you call (load “code.arc”), it’s guaranteed that none of the other threads will run “mid-update”. If your file contains definitions that overwrite all functions in your program, then your entire program is guaranteed to update atomically. Under the hood, (atomic …) is implemented with a recursive lock (cf. Python’s RLock). That way, if you call a function that calls atomic, which calls another function that calls atomic, you won’t deadlock — it’s the same thread, and the same thread can always acquire the lock recursively. And that’s it. The lock is literally a single instance of a recursive mutex, stored globally, created at program startup. Astute readers will notice one pitfall: suppose there are 10 threads running, and then you load file, which replaces all of the functions those threads were running. What happens? Each thread is paused in the middle of some existing function. That function will continue to exist until nothing refers to it. Since those threads refer to those functions (because we’re paused at some spot in the function), the currently-executing functions wont vanish until all the threads wake up and return. … which is particularly problematic if your thread is a while true: “do this forever” loop! There’s no way to update it anymore. You’d have to kill the thread and restart. Which is why the solution is “don’t do that, do this.” Get rid of the while loop, and call yourself recursively. Now whenever the new function loads, calling that function by name means you’ll jump into the new function, abandoning the old one. In languages without tail recursion (lookin at you Python, bastard), you can still achieve this by making sure your thread runner is a while-true loop that just calls some other function, and nothing else.
- tmtvl 4y ago
  But if your tail recursion gets compiled down into the standard "label whatever, code blablabla, set address register to label whatever, jump" type-code, how do you change the address that the register gets set to? Unless you do a lookup in the environment every time you recur to find where you have to jump to, but that would kill your performance stone dead.
jgalt212 4y ago
> The state of the in-memory image had gotten totally out of sync with the codebase. I have not used it, but I heard that Clojure Clerk is supposed to help with this. https://github.com/nextjournal/clerk https://github.com/nextjournal/clerk
septic_balls 4y ago
*lisp image
lilactown 4y ago
IMO this is one of the main benefits of Clojure from a lisp-er perspective: it actually takes multithreading seriously in its design. It basically chose the "yes add the machinery" option, but built the machinery into the design of the language. For instance, vars (i.e. global bindings) are mutable storage containers that can be bound on a per-thread basis. `(def foo "bar")` creates a new var and interns it in the current namespace, binding a root value "bar". Using it in code looks up the current value of it, either in thread-local storage or in the root binding. Any thread looking up the root binding of the var will see a newly bound value immediately. However! Data in Clojure is immutable, which means that even if a running program started by looking up the current value of a var (or any of the other concurrency-safe containers in Clojure), once it has that value it is guaranteed not to change underneath you. e.g. if you have a process that looks up the current value of a var and does some work with it for 5s, an in-flight process will not be altered if the var is redefined in the middle of the work. Other processes could even be kicked off that would alter the var, and it will have no affect on each other unless you explicitly synchronize, since the data inside the var is immutable.

by gaze 4y ago

mark_l_watson 4y ago

On LispWorks REPL, I use the Threads and Processes Viewer.

Joker_vD 4y ago

> how you manage state It's already done for you by the runtime: it holds the state in the process memory. Now, with edit-compile-run-print loop if you want your state to persist between runs, you gotta implement some scheme of data persistence. > it depends on a collection of global variables, those global variables now need some machinery around them so that they can be edited from the REPL thread. In Erlang, you create a public named ETS table (basically, a thread-safe dictionary) and put "global settings" in there instead (and read them from there, too). I'd imagine LISP has something similar. And mind you, I personally prefer the ECRPL instead of REPL, even when working in Python.

solarmist 4y ago
What is ECRPL?
- Joker_vD 4y ago
  "Edit-compile-run-print loop", a non-standard (but fairly clear) term I've used in the beginning of my comment.
  - solarmist 4y ago
    Sure. I just didn’t know it.

Jtsummers 4y ago

The "discipline" is to put the source code into a source file, the same thing you do with every other language. Don't do this and expect positive results: SOME-PACKAGE> (defun a-critical-function (...) ...) The discipline to instead do this: ;; some-package.lisp or whatever (defun a-critical-function (...) ...) Is table stakes for other languages, and Lispers are no worse programmers than programmers in other languages so why would they be incapable of this basic discipline? Only fools would write that in the REPL, never commit it to a source file, and be surprised when they couldn't reproduce the system state later on. Same with loading data. It comes from a database, file, or other source. Preserve the method of loading the data in source like all other code.

sillysaurusx 4y ago
> Only fools would write that in the REPL, never commit it to a source file, and be surprised when they couldn't reproduce the system state later on. This happened a few times while I was developing https://laarc.io https://laarc.io. It’s so addictive to just paste new functions into a repl and see the changes instantly that it’s easy to go a few hours without committing and realize your changes rely on a now-deleted function. :) Wouldn’t trade that workflow for the world though. C++ compile/run separation makes it horribly obvious how much productivity you lose. It feels like walking into a pit of molasses. But even Python isn’t much better — I’m constantly tabbing to the repl, typing ctrl-R “reload” <enter>, then pressing the up arrow twice to evaluate the previous function example I was working on. I wish there was an auto reload feature that would just reload everything every time I evaluate a repl expression. (I’ve been writing Bel lisp in Python, and I put in some code to do just that for bel.interact()’s repl. It’s so much nicer not having to reload manually all the time.)
- tonyarkles 4y ago
  > It’s so addictive to just paste new functions into a repl and see the changes instantly that it’s easy to go a few hours without committing and realize your changes rely on a now-deleted function. :) I think Slime is a pretty happy middle ground. Instead of just pasting, the function gets written in a source file and then C-c C-c to evaluate it in the repl. I did once or twice run into similar issues though from depending on older or newer versions of functions that changed but weren’t completely reflected until a clean restart. But that’s been pretty rare and definitely wasted way less time than “make clean all” before committing C++ code :D

sillysaurusx 4y ago

You’re right. I think the other commenters aren’t being straightforward with you. It is a race condition to load a file one function at a time, because any other thread can preempt you. Arc has a particularly elegant solution to this. Any code you want to happen atomically, you wrap in (atomic …) So (atomic x y z) Will do x, y, then z. During this time, no other threads are allowed to run. Therefore your load-file function might look like (pseudo code because phone): (def load (filename) (atomic (each form (read-file filename) (eval form)))) Now any time you call (load “code.arc”), it’s guaranteed that none of the other threads will run “mid-update”. If your file contains definitions that overwrite all functions in your program, then your entire program is guaranteed to update atomically. Under the hood, (atomic …) is implemented with a recursive lock (cf. Python’s RLock). That way, if you call a function that calls atomic, which calls another function that calls atomic, you won’t deadlock — it’s the same thread, and the same thread can always acquire the lock recursively. And that’s it. The lock is literally a single instance of a recursive mutex, stored globally, created at program startup. Astute readers will notice one pitfall: suppose there are 10 threads running, and then you load file, which replaces all of the functions those threads were running. What happens? Each thread is paused in the middle of some existing function. That function will continue to exist until nothing refers to it. Since those threads refer to those functions (because we’re paused at some spot in the function), the currently-executing functions wont vanish until all the threads wake up and return. … which is particularly problematic if your thread is a while true: “do this forever” loop! There’s no way to update it anymore. You’d have to kill the thread and restart. Which is why the solution is “don’t do that, do this.” Get rid of the while loop, and call yourself recursively. Now whenever the new function loads, calling that function by name means you’ll jump into the new function, abandoning the old one. In languages without tail recursion (lookin at you Python, bastard), you can still achieve this by making sure your thread runner is a while-true loop that just calls some other function, and nothing else.

tmtvl 4y ago
But if your tail recursion gets compiled down into the standard "label whatever, code blablabla, set address register to label whatever, jump" type-code, how do you change the address that the register gets set to? Unless you do a lookup in the environment every time you recur to find where you have to jump to, but that would kill your performance stone dead.

jgalt212 4y ago

> The state of the in-memory image had gotten totally out of sync with the codebase. I have not used it, but I heard that Clojure Clerk is supposed to help with this. https://github.com/nextjournal/clerk https://github.com/nextjournal/clerk

septic_balls 4y ago

*lisp image

lilactown 4y ago

IMO this is one of the main benefits of Clojure from a lisp-er perspective: it actually takes multithreading seriously in its design. It basically chose the "yes add the machinery" option, but built the machinery into the design of the language. For instance, vars (i.e. global bindings) are mutable storage containers that can be bound on a per-thread basis. `(def foo "bar")` creates a new var and interns it in the current namespace, binding a root value "bar". Using it in code looks up the current value of it, either in thread-local storage or in the root binding. Any thread looking up the root binding of the var will see a newly bound value immediately. However! Data in Clojure is immutable, which means that even if a running program started by looking up the current value of a var (or any of the other concurrency-safe containers in Clojure), once it has that value it is guaranteed not to change underneath you. e.g. if you have a process that looks up the current value of a var and does some work with it for 5s, an in-flight process will not be altered if the var is redefined in the middle of the work. Other processes could even be kicked off that would alter the var, and it will have no affect on each other unless you explicitly synchronize, since the data inside the var is immutable.