4 ms·

I have worked on massive LISP projects that were developed using REPL-driven development, and your fears are critically valid. Even worse, this basically hamstr

I have worked on massive LISP projects that were developed using REPL-driven development, and your fears are critically valid. Even worse, this basically hamstrings it as a standalone application. The workflow to _use_ the application becomes "start the REPL under emacs and launch it", which is, in my personal opinion, a completely unacceptable distribution model. And if you don't use emacs, good luck to you! Once set up as REPL-based development, Smalltalk and LISP both have to essentially ship their "whole environment" to deploy the application. This only works out well for small projects that do not need to be distributed, or whose distribution you, the developer, have a lot of control over. (There's an aside to be said about building binaries from a LISP compiler, but most binaries I have seen produced by such compilers still need access to the required libraries, which sort of hoses it as a distribution solution. This could be a result of the build system used on the projects I work on.) And your documentation concern is valid, but I think even worse is the design concern: when designing an API, you have to stop and design it and consider the various concerns. If you develop it as you need it / as you go, it is possible to arrive at sub-optimal designs. Hacking the "next piece" out at each step is likely to lead to slanted designs with poor APIs, requiring future refactoring. You see this with programmers at every level: if you ask them to write a program to do a single thing, then add more (related) capabilities, and continue ad nauseum, at some point the program will need to be refactored. This suggests that, without prior planning, REPL-based design is set up to incur technical debt. Unfortunately, my real-life experience confirms this. Moreover, refactoring something in a REPL is... a chore, to say the least.

mikelevins 6y ago
Your observations are valid, and need to be kept in mind when developing in an exploratory, interactive way. Your alarm seems a little overblown. Maybe it reflects a particular bad experience? I've delivered a bunch of products built in the way I prefer to work, and it's been a long time since I've had the problems you're describing. Maybe that's because I ran into such such problems early in my career and learned ways of handling them. And yeah, if you just iterate and explore, you can wind up in lacunae. You do need to develop some actual goal and a vision of how to get there. Interactive development can be really helpful for the exploring part, but it can't pick your goals for you. In short, interactive development is a way of working that's good for some cases and some people (me, for instance). It's not a cure-all or an objectively superior methodology for all people or all cases. I don't want it to take over the world. I just want it to continue to be an option because it's the way I prefer to work.
- O_H_E 6y ago
  Oh man, I appreciate your pragmatism and open-mindness. It is refreshing to see someone advocate for something without trying to "take over the world" and proving others wrong. Thanks for sharing your experience amd positivity with the world.
- Folcon 6y ago
  Echoing this, I'd really like it to be an option in other languages, because it's just a pleasant way to work and it's nice to have the option.
- discardable_dan 6y ago
  Thanks for taking the time to respond. I wrote up my comment just to point out some of the weaknesses I have identified in REPL-based development, and how to watch out for those pitfalls as a program scales. For what it's worth, I agree with your sentiment, and believe REPL-based development should always be an option. I have a lot of success using REPL-based "prototyping", even in python, where I will quickly try/test something at a REPL before scaling it up, and having that utility to try/test/experiment quickly is often invaluable (especially if the standard build time is on the scale of minutes).
fouric 6y ago
> Even worse, this basically hamstrings it as a standalone application. The workflow to _use_ the application becomes "start the REPL under emacs and launch it" This seems like an incidental property (symptom of poor engineering discipline, which manifests itself in other ways in other development paradigms e.g. lack of documentation from "agile" teams) of the particular applications you've worked with and not an essential property of the (architecture decisions resulting from the) REPL-driven development style. What makes you think that this is actually due to REPL-driven development? I develop several tools from the REPL and was able to easily convert each one to a standalone tool (when I attempted to do so).
- discardable_dan 6y ago
  The experience I have had on large, REPL-driven development is that continued development is assumed to also work under REPL-driven environments, so debugging or supporting them begins with "try typing X into your REPL." Many of the distributables I have seen also maintains REPL interactivity as part of the live deployment solution, with similar debugging solutions. While this may not be unilateral, "ship the deployment environment" seems to be a running theme. I'm interested in your experience in avoiding this, and the process you used.
  - fouric 6y ago
    The batch-processing tools I wrote (e.g. a simulator) had CLI shims that called the same interesting functions that you would at the REPL. The interactive tools I wrote all had functions that acted as a zero-effort entry point anyway, so the conversion to standalone tool just consisted of building an image that called that function. Additionally, my development environment (SLIME) talked with my application over a network socket, so there's no linkage between it and the application - at least, any more so than the usual problem of "Common Lisp binaries are hard to make small" but that's not specific to REPL-driven development. The simplicity of my "solution" makes me think that we might be talking at different levels here, but I can't think of what the disconnect might be.
    - 6y ago

mikelevins 6y ago

Your observations are valid, and need to be kept in mind when developing in an exploratory, interactive way. Your alarm seems a little overblown. Maybe it reflects a particular bad experience? I've delivered a bunch of products built in the way I prefer to work, and it's been a long time since I've had the problems you're describing. Maybe that's because I ran into such such problems early in my career and learned ways of handling them. And yeah, if you just iterate and explore, you can wind up in lacunae. You do need to develop some actual goal and a vision of how to get there. Interactive development can be really helpful for the exploring part, but it can't pick your goals for you. In short, interactive development is a way of working that's good for some cases and some people (me, for instance). It's not a cure-all or an objectively superior methodology for all people or all cases. I don't want it to take over the world. I just want it to continue to be an option because it's the way I prefer to work.

O_H_E 6y ago
Oh man, I appreciate your pragmatism and open-mindness. It is refreshing to see someone advocate for something without trying to "take over the world" and proving others wrong. Thanks for sharing your experience amd positivity with the world.
Folcon 6y ago
Echoing this, I'd really like it to be an option in other languages, because it's just a pleasant way to work and it's nice to have the option.
discardable_dan 6y ago
Thanks for taking the time to respond. I wrote up my comment just to point out some of the weaknesses I have identified in REPL-based development, and how to watch out for those pitfalls as a program scales. For what it's worth, I agree with your sentiment, and believe REPL-based development should always be an option. I have a lot of success using REPL-based "prototyping", even in python, where I will quickly try/test something at a REPL before scaling it up, and having that utility to try/test/experiment quickly is often invaluable (especially if the standard build time is on the scale of minutes).

fouric 6y ago

> Even worse, this basically hamstrings it as a standalone application. The workflow to _use_ the application becomes "start the REPL under emacs and launch it" This seems like an incidental property (symptom of poor engineering discipline, which manifests itself in other ways in other development paradigms e.g. lack of documentation from "agile" teams) of the particular applications you've worked with and not an essential property of the (architecture decisions resulting from the) REPL-driven development style. What makes you think that this is actually due to REPL-driven development? I develop several tools from the REPL and was able to easily convert each one to a standalone tool (when I attempted to do so).

discardable_dan 6y ago
The experience I have had on large, REPL-driven development is that continued development is assumed to also work under REPL-driven environments, so debugging or supporting them begins with "try typing X into your REPL." Many of the distributables I have seen also maintains REPL interactivity as part of the live deployment solution, with similar debugging solutions. While this may not be unilateral, "ship the deployment environment" seems to be a running theme. I'm interested in your experience in avoiding this, and the process you used.
- fouric 6y ago
  The batch-processing tools I wrote (e.g. a simulator) had CLI shims that called the same interesting functions that you would at the REPL. The interactive tools I wrote all had functions that acted as a zero-effort entry point anyway, so the conversion to standalone tool just consisted of building an image that called that function. Additionally, my development environment (SLIME) talked with my application over a network socket, so there's no linkage between it and the application - at least, any more so than the usual problem of "Common Lisp binaries are hard to make small" but that's not specific to REPL-driven development. The simplicity of my "solution" makes me think that we might be talking at different levels here, but I can't think of what the disconnect might be.
  - 6y ago