4 ms·

Explorative Programming

70 points by signa11 3y ago

dusted 3y ago
That's what I made hedonjs for, a kind of repl for doing more than the node repl, it allows you to slowly build out your program without restarting it, adding and editing functions (and callbacks, with a slight bit of trickery) and variable values while the program runs. Of course, this is nothing new, it's an amazing strength of lisp in such environments as slime for emacs or alive for vscode.
ElevenLathe 3y ago
This all smells very Smalltalk-y, but I'm always excited to see more takes on environments like this.
brianhama 3y ago
I have to say, this is definitely not how I code. Maybe I’m in the minority, but I don’t think so
- bayindirh 3y ago
  Same here. I can visualize what I'm trying to make pretty easily, and see the end result I want to have before even touching the keyboard in most cases. The only time I poke and play with the code is when I'm poking hardware to understand what it's doing, IOW when I'm playing with a relatively black box.
  - Calavar 3y ago
    My personal experience is that it depends on the context. Writing new functionality is usually easy if it's self contained. If I know what inputs I will have and what outputs I want, then I generally have a pretty solid idea of how to derive one from the other before the first keystroke. But if I find that the new functionality is turning into an internal API that will be consumed in multiple places for multiple use cases, things aren't so obvious anymore. You have a combinatorial explosion of design parameters: not just inputs, outputs, and algorithm, but also issues like ownership of data, ownership of the stack, which inputs are supplied at which point in the process, when is it legal to consume outputs and in what ways, push vs. pull interface, two functions for similar functionality vs. one with a flag. When the design parameter space gets this large, it's hard to reason about it ahead of time, and sometimes the most straightforward API doesn't reveal itself until you write code that uses it and realize that there's needless boilerplate or inconsistency that opens up room for logic errors. Then you go back and refactor to eliminate that. Casey Muratori calls this iterative process semantic compression [1]. I don't agree with all his software development takes, but I think he hits this one pretty squarely on the head. The key insight is that an experienced programmer can write good code on the first pass, but it usually takes a second pass to write great code. [1] https://caseymuratori.com/blog_0015 https://caseymuratori.com/blog_0015
  - MarceColl 3y ago
    This is not for your typical glue code, or standard algorithms, this is for discovery work, novel programs, library api design, and exploring new systems and ideas. If you are able to take novel greenfield work, map it all out in your head and get to writing, good for you. But I somehow doubt it
    - bayindirh 3y ago
      I do not write glue code 99% of the time, but write libraries, small tools, etc. The last time I did this, it was a completely novel framework for material simulations using boundary element method which happened to be my Ph.D. thesis. It did not completely turned out the way I wanted, but I was able to get the theoretical performance I targeted for (i.e. literally drowning the system in computation), nailed the data formats workflow we wanted to have, and it turned out to be pretty good. It can scale from laptops to high end servers, and on a typical laptop, it's 30x faster than a well tuned MATLAB implementation, while being at least as accurate. The formulae and algorithms we used for calculations were novel. It was mostly designed on paper, with some whiteboard work, and implemented with zero memory leaks along the way, plus we hit the performance numbers we were hoping in second beta IIRC. I'll shortly start on its second iteration, borrowing the high performance architecture, data types, but not the overall workflow. We'll try to make it even more user friendly. So yeah, I do this during implementing APIs, libraries and novel programs.
      - MarceColl 3y ago
        As I said, if you can do this, good for you. I'm just explaining how I need to do it. However, your whiteboard and paper is my exploratory programming. I don't see it as it being that different.
        bayindirh 3y ago
        Honestly, you said "I doubt you do", so this is why I wrote this long answer. I just visualize the problem on paper. You have this, you want this, and to go from A to B, we need to do this, etc. Exploratory programming is nothing bad, or what I do is nothing good or exceptional. Humans' brains are wired differently, and as long as we arrive to the point we want, it's all good in my eyes. There's no superiority in any of the methods. We're just cooking the same food differently. That's all.
- liampulles 3y ago
  For me it took some practice, but I got there in the end. https://liampulles.com/jira-tickets.html https://liampulles.com/jira-tickets.html
  - swah 3y ago
    I like your post a lot - but I wouldn't do a side project like this - seems to take a bit of the fun from coding, I guess.
    - liampulles 3y ago
      Totally fair
- kkfx 3y ago
  It's essentially the classic Smalltalk workstation model. End user programming, where the user live in a live easy to bend environment and just have to write few bits, some listings, to bend the desktop to it's wish. You are in a MUA composition windows and want to solve an equation? You have some bits that do so in your system, no need to switch "application", just feed the input to the relevant function and get locally the result. You just want to change something? Easy to do, it's just few bits of code. This is the most effective and powerful use of a computer we have ever invented. It was dropped because no one who works in sw business want power users and easy to bend code, it's nature is collaborative, not commercial. It's nature allow people to understand and those who became "alphabets" do not like Alphabet's or Oracle's o Meta-information business and they can perfectly goes on without them.
- MarceColl 3y ago
  That's fine, I was just explaining how I code and the problems with the current tools. How do you approach new problems?
deckiedan 3y ago
My day job is Django - and the ipdb based `shell_plus` from the `django_extensions` package is invaluable.[1] I have a shell open almost all the time, and use it constantly for testing functions, writing queries etc. It drives me crazy watching other Devs change a query, wait for the server to reload and refresh the browser repeatedly, when editing in the REPL is so much faster. Being able to `dir` and tab complete experimentally is so fast. (I also have this in the ipdb debugger... Also Django extensions has a debugger inside templates which is really nice) Being able to `%edit 1-20` and have the repl session instantly in my editor to copy paste is fantastic too. I also use a lot of TDD for tasks that fit that style, mostly where I'm writing business logic or data processing functions... but for exploring data / APIs /queries / data shaping, interactive just feels instantly productive. Once I've got the API or whatever it is to correctly produce the happy path, it's then easy to copy that into the codebase knowing what shape to expect, I can then turn that into unit tests and throw all the weird external data and edge cases at it with reasonable confidence about what should be happening underneath. It also has a mode to print out all SQL statements it runs underneath your code - which is great for checking that you've not missed a join or something and got n+1 sneaking in [1] https://django-extensions.readthedocs.io/en/latest/ https://django-extensions.readthedocs.io/en/latest/
- karmakaze 3y ago
  I knew someone would mention REPL-based development. We really should have better tooling in this area where we could set a breakpoint and when hit, change both code and/or data and continue. The REPL is so incredibly productive with immediate feedback that the long ORM expression actually runs the SQL query you were intending (or notice it does something you didn't intend).
- agumonkey 3y ago
  I had to fight a lot to stop colleagues doing the long change a variable, reload, renavigate, trigger some dom event and wait. Obviously I come from a lisp heavy culture, but I'm still stumped how many people stay in those high latency exploration mindset.
  - 3y ago
kkfx 3y ago
In few years you'll discover classic Smalltalk workstations, modern Pharo, with a probably nicer language than Smalltalk itself. This model is essentially the original desktop model with it's end-user programming concept.
- MarceColl 3y ago
  I've already tried pharo, my problem there is with the next step, making that production ready. I need tooling that allows explorative programming and that can be transitioned into production. If not it's just another source of rigidness, just later in the process
  - kkfx 3y ago
    That's not much a Pharo or CL issue but a social issue: actual IT development, and for the last decades, have gone in a regressive path for commercial reasons and it's from enough from these systems and paradigms to have a nearly impossible to fill gap in between. Personally I live digitally in Emacs, so I have some tooling for the present word, mostly hidden under Emacs or *nix system, but yes, it's still limited, less then Pharo since I can read my mails, org-attach my files and so on, but not integrate much more without all the modern integration issues of a an IT built to be isolated not integrated to sell programs before, services thereafter. However I see a very slow and long trend: the more current IT evolve the more it tend back to the IT pioneer era, in the "modern past" all was widget-based GUIs, not even Microsoft recommend CLIs for certain works, most apps have WebUIs witch are limited DocUIs but with the same doc concept, NotebookUI are more and more widespread, here and there someone recommend LaTeX instead of WYSIWYG word processor, someone else recommend R instead of spreadsheets and so on. Perhaps in 50 years we will have Smalltalk, Lisp and so on again, at least at concept/paradigm level.
  - ElevenLathe 3y ago
    Smalltalk and Excel are both good to draw on. Another programming tradition that is actually very popular but seems to have limited influence on "professional" programming languages is Wolfram/Mathematica. The Wolfram Cloud product has some very interesting features in terms of exposing your Mathematica code as a service on the internet. Here's an interesting demo: https://www.youtube.com/watch?v=3VqlCm_T2CA https://www.youtube.com/watch?v=3VqlCm_T2CA. Mathematica also popularized the "notebook" style of dev which is also very nice for exploration.
TheAlchemist 3y ago
Well, that makes me think about Excel ! I think what the author describes is why Excel is so heavily used. It's often not easy to clearly define the specifications - but it's easy to play a bit with the data the user knows, and discover some flaws in the logic, corner cases etc. And also very easy to prototype processes. I almost miss it.
- postepowanieadm 3y ago
  I once made a pdf text extraction tool in sql. By accident. Having snapshots of data on every step was really nice.
cjs_ac 3y ago
I have this problem with planning. My planning mostly ends up being sequencing of tasks rather than working how to do them. Software is made out of ideas, and code is just a symbolic representation of those ideas. Writing the code is the easy part; working out what ideas to represent is the hard part of programming. If your planning involves working out what ideas to represent, if you do that in greater and greater detail, eventually you stop planning the work and cross over into doing the work. > File based organization, right from the start you need to start deciding what goes where. Of course I think most of us start with a single file, full of random functions and tests. But at some point this file becomes unbearable, impossible to find what you are looking for. At that point you have to make decisions and start splitting the file at logical points. Deal with what is private and what is not. It may not seem like much, but it's very annoying. In a project I'm working on at home, I started off with an extremely naive implementation, the single file of functions described above, and when the messiness became unbearable, I started building object-oriented abstractions to make sense of it all. But the structure of those abstractions was discovered while making the mess - if I'd tried to build that at the start, I'd have built the wrong thing. I'm very fortunate in that in both my day job and my recreational programming, I'm the only person working on that particular codebase at the time. (At work, I develop what are basically plugins for the framework-type-thing that's our main product.) Consequently, I have the freedom to make a mess and clean it up once I've muddled about in the solution space to discover what's going to work best, and so most of the solutions in the blog post are rather unnecessary for me. In the end, two months of work on my home project was thrown away and rewritten, which I could never have got away with at my job, but the project is all the better for it.
- cmiller1 3y ago
  > But the structure of those abstractions was discovered while making the mess For me at least it's more like bang out the naive implementation mess then "discover" the proper structure of the abstractions in bed that night or in the shower the next morning.
- deergomoo 3y ago
  This comment very well captures what I was trying (and failing) to explain to a colleague recently, when talking about why TDD doesn’t fit the way I solve problems. It’s almost like tending to something as it grows, discovering what the solution most wants to be with respect to the problem and the environment (e.g. programming language, server vs browser etc.)
MarceColl 3y ago
I'm the author, was not expecting to see this here today, somebody posted it yesterday as well but got no traction. It's just a small essay about how I approach programming, particularly when understanding problems and exploring new ideas. It's a little bit of explanation of a project I'm doing as well as something I'd have liked to read years ago when I was sure I was just doing it wrong.
- agumonkey 3y ago
  Do you have buddies or web acquaintances talking about similar ideas / goals ? I've always been searching for this mindset but I rarely read / hear about it.
- norman784 3y ago
  This sounds much like how unison[0] works. I would say that if it were to use a C like syntax, the succeed chances would go up. Would be cool if it targets WASM, this something I'm super interested right now, there are a lot of advantages on targeting WASM one of them is the interoperability with other components written in different languages, this will be when the component model[1] get's finished and implemented. [0]http://unison-lang.org http://unison-lang.org [1]https://component-model.bytecodealliance.org https://component-model.bytecodealliance.org
  - MarceColl 3y ago
    Unison was very much an inspiration for my thoughts, but statement based programming (C-syntax) is pretty much out of the question since they rely on hashing functions and on some specific ways of specifying computation so it can work as it does with the whole distribution of computation and universal database of code. It is actually one of the problems I have with common lisp, its impossible to do a semantic hash on arbitrary common lisp functions so I create more history that I need, since a reordering of independent let forms cannot be easily detected to preserve functionality
    - brabel 3y ago
      Your ideas sounded very much like a mixup of Common Lisp with SLIME, Smalltalk interactivity and Unison-like storage of code in a database instead of files. I've tried all of them, I think the closest thing I've seen to what you describe, which I also find very attractive, is the GT Smalltalk environment: https://gtoolkit.com/ https://gtoolkit.com/ Have you tried that? They call this idea "moldable development" as you can "mold" your environment to your needs. Even though I loved it, I ended up not using it much, mostly because it's a bit too heavy to keep handy for exploration all the time when needed (it takes like 1GB of RAM even when idle!)... as I already can do most of that with emacs, which is much lighter, I just stick with it.
xixixao 3y ago
I agree. It’s a tall order to replace all the pieces. You need: - programming language not oriented around files - PL compiler designed first and foremost for incremental complication / interpretation - New source control that’s syntax-aware and watch not commit based - New IDE that allows navigating the tree and space And any of these are a reason for people to not use it: - CommonLisp? Most people don’t like it. Could you adapt TS? Rust? - If it’s not Git, can’t use GitHub - Emacs? Could it be a VS Code plugin? A fork? Where is a billionaire techie who could foot the bill to build this? It’s a mammoth effort.
- MarceColl 3y ago
  Thats exactly why I started with common lisp, I have a working prototype of all of this (not a complex version control system yet) in about 60 lines of code. The IDE was a hacked 50 line emacs major mode, it's really not that much work to get started. It has of course a lot of rough edges. About the reasons not to use it, thats fine for me, for now this is for me and to serve as inspiration, I want to work with CL. If anyone else wants to tacke it in another language, all the best for me :) Git compatibility and output to files is something that the system should have, indeed
creesch 3y ago
> I think I'm not alone in this. Hopefully? No, it works in a similar way for me. I can do abstractions to a certain degree in my head and plan ahead but also do need to work on a problem and trial things. It's interesting that the article mentions Javascript as not having an immediate feedback cycle. Which is true enough for a lot of the modern Javascript stack. Although live reloading is also a thing depending on what aspect you are working on. > Imagine a service that never stops, you diff the existing functions with the new functions, only updating what needs to be updated. If there is a change in the rate of errors then you automatically rollback the deployed functions. I am pretty sure that this is already a concept in use by companies. Although the exact naming for it, I can't remember. It also highly depends on the sort of domain you are active in. The more critical your data is, the less likely it is that you want to risk using this type of development and testing model. Not to mention that it seems lean in the logical fallacy of "only errors indicate faults" while the most devastating bugs often are those that don't cause clear errors. For example, when dealing with data your particular system might have no increased errors but due to a data change a system down the line might. Another potential issue is that because you are developing this in an explorative manner, it also doesn't leave much room for a review process. Or any of the myriad of other processes and methods that normally should be in place to evaluate the thing under development. In addition to the above, the way the article does describes the envisioned process leaves very little room for proper documentation. More specifically, it feels like a process where documentation will be more of an afterthought than it already is within many teams. Off-topic for the subject, but using monospaced fonts for blog posts sucks for readability, imho.
- MarceColl 3y ago
  Javascript has decent feedback cycle, but pretty bad program state persistence as you reload things, there are some attemps and sometimes Vite is able to do it, but it's a pretty sad state of affairs in general. The end remarks are more to let the imagination run a little bit free than anything particularly structured, the part about deployment doesnt really exist as it is in most companies, there is a concept of canary deployments, blue green, etc, but not on a running process ala Erlang or Elixir. Still, Im not sure if it is even a good idea since starting from a clean slate is usually good. I need to do more testing around this. There is no interest in documentation while the process of discovering is going on, because the documentation would be stale as soon as you move on in the discovery process. Its not even a fault of the system, it just doesn't make sense. Once you find a good enough design, then you transition into settling it, documenting it, writing more thorough tests. About the review process, this is mostly targeted at individuals and teams where there is a certain trust involved. But for knowledge spread that's why I also tackle doing this collaboratively, as I think it could even increase the effectiveness of this approach. The design was pretty fast done with a template. I'll see if I have time today to test other fonts and see how it improves. Thanks for taking the time to read
laserbeam 3y ago
I remember some of the earlier talks from Jonathan Blow about how he was designing Jai (the programming language) to alleviate some of these problems and support explorative programmings. One of the ideas was to write code such that if you copy paste things into a different context, the syntax changes are minimal. It should be trivial to copy paste a block of code to turn it into a function. It should be trivial to turn a lambda into a full blown function. Type signatures should look the same whether they are function arguments or variable definitions etc. Not advocating for Jai per se... But most languages are pretty bad at supporting this style, some are better.
- ReleaseCandidat 3y ago
  > But most languages are pretty bad at supporting this style, some are better. The first and most important step is having a language without statements and only expressions. But, as Jai needs a `return` in functions, it already failed the trivial test. https://github.com/BSVino/JaiPrimer/blob/master/JaiPrimer.md https://github.com/BSVino/JaiPrimer/blob/master/JaiPrimer.md
tpoacher 3y ago
> I am very bad at planning and visualizing what a program should look before I start writing it This is what top-down programming is for (though, arguably, a lot easier for 'smaller' projects)
anonzzzies 3y ago
This is what Lisp is good at; emacs/slime and you can experiment/plan your project together. Python tries this with it's only repl (see the Django comment also; that solution works ok, but nothing compared to slime) or with notebooks and stuff. It's far behind though and I would much rather forever just use emacs/slime for everything. I do for my own projects, but for others I have to cry in my soup.
crq-yml 3y ago
I believe it's the combination of needing exploration but also wanting "safe checkpoints" that leads me now towards writing in terms of code generation first and making big iterations through the generator, and then subsequently testing and exploring interactively. I've been working in a blend of Forth, Lua, and Python this way. All three have REPLs. Python is the "fattest" of the three, and does more of the dirty work with the OS and file processing since those things are built in. Lua is the one I actually want to use for the primary task of generating source because of its small size, familiarity and flexibility. Forth is the tip of the spear and solves concrete lower-level stuff without needing the help of another layer. While I haven't yet made this process very sophisticated - it's mostly string substitution, pretty prints and state machines - it has a natural way of tightening the screws on any specification, because each step presents challenges about which layer I want to use, and whether I want to break with locally idiomatic usage and take the approach of inlining something from above instead.
buescher 3y ago
As an aside, is there anything like the Smalltalk/Pharo object browser and IDE for javascript in a web browser?
- brabel 3y ago
  SqueakJS: https://squeak.js.org/run/#highdpi&zip=https://files.squeak.org/6.1alpha/Squeak6.1alpha-22998-32bit/Squeak6.1alpha-22998-32bit.zip https://squeak.js.org/run/#highdpi&zip=https://files.squeak.... EDIT: oh sorry, didn't notice you asked specifically for JavaScript (not just Smalltalk in the browser).
- xkriva11 3y ago
  There are several distinct versions of Lively Kernel https://lively-kernel.org https://lively-kernel.org https://lively-kernel.org/lively4/lively4-jens/start.html https://lively-kernel.org/lively4/lively4-jens/start.html
  - buescher 3y ago
    Lively is very impressive in its scope and beyond what I imagined anyone might have done already. Thanks! It seems a little brittle - the examples threw errors for me. It’s also a bit less coupled to the browser that what I was picturing - I was thinking something more like a lot of bells and whistles added to say Chrome devtools.
JanisErdmanis 3y ago
I wholeheartedly agree that the rigidity of most current development environments prevents us from understanding the problem. I would even say it shapes the problems we try to explore the same way a spoken language does. Many points were the reasons why I am now using Julia to develop a large-scope full-stack application. Revise (package) allows to evolve the program interactively while preserving the system state. For instance, I can have an instance of a struct in the REPL; I change one of the modules, and the available behaviour of the instance can be immediately tried out without recreating the state. Or, when interactively developing a webserver, I find an error on one of the served pages. I changed the piece of code in the module, and it is immediately reflected when reloaded while keeping the existing state. System introspection is easy as everything is available from the REPL. Default printing is available for all user-defined types, which can be specialised on a wish. Thus, printing an object or even getting to the precise point in the module call stack with an Infiltrator is always accessible. One pain point regarding file-based organisation is that, in most languages, every file is within its namespace. In Julia, these things are orthogonal. You can have a single namespace organised over multiple files or multiple namespaces within a single file. This generally makes it easier to navigate a large namespace while not being bothered with making explicit imports, etc., when one is still in the exploratory phase. I can’t really relate to tree-based exploration. Very few times do I find that the direction I have found is wrong, so I do a git reset to a previous commit. Also, there are a few sunk costs migrating from one approach to another when most of the functions are written to be pure. Another essential piece for explorative development is types. Although some planning is required, it is generally worth it. In Julia, those are checked at runtime, which nevertheless significantly helps to locate issues quickly when doing test-driven development.

70 points by signa11 3y ago

That's what I made hedonjs for, a kind of repl for doing more than the node repl, it allows you to slowly build out your program without restarting it, adding and editing functions (and callbacks, with a slight bit of trickery) and variable values while the program runs. Of course, this is nothing new, it's an amazing strength of lisp in such environments as slime for emacs or alive for vscode.

ElevenLathe 3y ago

This all smells very Smalltalk-y, but I'm always excited to see more takes on environments like this.

brianhama 3y ago

I have to say, this is definitely not how I code. Maybe I’m in the minority, but I don’t think so

bayindirh 3y ago
Same here. I can visualize what I'm trying to make pretty easily, and see the end result I want to have before even touching the keyboard in most cases. The only time I poke and play with the code is when I'm poking hardware to understand what it's doing, IOW when I'm playing with a relatively black box.
- Calavar 3y ago
  My personal experience is that it depends on the context. Writing new functionality is usually easy if it's self contained. If I know what inputs I will have and what outputs I want, then I generally have a pretty solid idea of how to derive one from the other before the first keystroke. But if I find that the new functionality is turning into an internal API that will be consumed in multiple places for multiple use cases, things aren't so obvious anymore. You have a combinatorial explosion of design parameters: not just inputs, outputs, and algorithm, but also issues like ownership of data, ownership of the stack, which inputs are supplied at which point in the process, when is it legal to consume outputs and in what ways, push vs. pull interface, two functions for similar functionality vs. one with a flag. When the design parameter space gets this large, it's hard to reason about it ahead of time, and sometimes the most straightforward API doesn't reveal itself until you write code that uses it and realize that there's needless boilerplate or inconsistency that opens up room for logic errors. Then you go back and refactor to eliminate that. Casey Muratori calls this iterative process semantic compression [1]. I don't agree with all his software development takes, but I think he hits this one pretty squarely on the head. The key insight is that an experienced programmer can write good code on the first pass, but it usually takes a second pass to write great code. [1] https://caseymuratori.com/blog_0015 https://caseymuratori.com/blog_0015
- MarceColl 3y ago
  This is not for your typical glue code, or standard algorithms, this is for discovery work, novel programs, library api design, and exploring new systems and ideas. If you are able to take novel greenfield work, map it all out in your head and get to writing, good for you. But I somehow doubt it
  - bayindirh 3y ago
    I do not write glue code 99% of the time, but write libraries, small tools, etc. The last time I did this, it was a completely novel framework for material simulations using boundary element method which happened to be my Ph.D. thesis. It did not completely turned out the way I wanted, but I was able to get the theoretical performance I targeted for (i.e. literally drowning the system in computation), nailed the data formats workflow we wanted to have, and it turned out to be pretty good. It can scale from laptops to high end servers, and on a typical laptop, it's 30x faster than a well tuned MATLAB implementation, while being at least as accurate. The formulae and algorithms we used for calculations were novel. It was mostly designed on paper, with some whiteboard work, and implemented with zero memory leaks along the way, plus we hit the performance numbers we were hoping in second beta IIRC. I'll shortly start on its second iteration, borrowing the high performance architecture, data types, but not the overall workflow. We'll try to make it even more user friendly. So yeah, I do this during implementing APIs, libraries and novel programs.
    - MarceColl 3y ago
      As I said, if you can do this, good for you. I'm just explaining how I need to do it. However, your whiteboard and paper is my exploratory programming. I don't see it as it being that different.
      - bayindirh 3y ago
        Honestly, you said "I doubt you do", so this is why I wrote this long answer. I just visualize the problem on paper. You have this, you want this, and to go from A to B, we need to do this, etc. Exploratory programming is nothing bad, or what I do is nothing good or exceptional. Humans' brains are wired differently, and as long as we arrive to the point we want, it's all good in my eyes. There's no superiority in any of the methods. We're just cooking the same food differently. That's all.
liampulles 3y ago
For me it took some practice, but I got there in the end. https://liampulles.com/jira-tickets.html https://liampulles.com/jira-tickets.html
- swah 3y ago
  I like your post a lot - but I wouldn't do a side project like this - seems to take a bit of the fun from coding, I guess.
  - liampulles 3y ago
    Totally fair
kkfx 3y ago
It's essentially the classic Smalltalk workstation model. End user programming, where the user live in a live easy to bend environment and just have to write few bits, some listings, to bend the desktop to it's wish. You are in a MUA composition windows and want to solve an equation? You have some bits that do so in your system, no need to switch "application", just feed the input to the relevant function and get locally the result. You just want to change something? Easy to do, it's just few bits of code. This is the most effective and powerful use of a computer we have ever invented. It was dropped because no one who works in sw business want power users and easy to bend code, it's nature is collaborative, not commercial. It's nature allow people to understand and those who became "alphabets" do not like Alphabet's or Oracle's o Meta-information business and they can perfectly goes on without them.
MarceColl 3y ago
That's fine, I was just explaining how I code and the problems with the current tools. How do you approach new problems?

deckiedan 3y ago

My day job is Django - and the ipdb based `shell_plus` from the `django_extensions` package is invaluable.[1] I have a shell open almost all the time, and use it constantly for testing functions, writing queries etc. It drives me crazy watching other Devs change a query, wait for the server to reload and refresh the browser repeatedly, when editing in the REPL is so much faster. Being able to `dir` and tab complete experimentally is so fast. (I also have this in the ipdb debugger... Also Django extensions has a debugger inside templates which is really nice) Being able to `%edit 1-20` and have the repl session instantly in my editor to copy paste is fantastic too. I also use a lot of TDD for tasks that fit that style, mostly where I'm writing business logic or data processing functions... but for exploring data / APIs /queries / data shaping, interactive just feels instantly productive. Once I've got the API or whatever it is to correctly produce the happy path, it's then easy to copy that into the codebase knowing what shape to expect, I can then turn that into unit tests and throw all the weird external data and edge cases at it with reasonable confidence about what should be happening underneath. It also has a mode to print out all SQL statements it runs underneath your code - which is great for checking that you've not missed a join or something and got n+1 sneaking in [1] https://django-extensions.readthedocs.io/en/latest/ https://django-extensions.readthedocs.io/en/latest/

karmakaze 3y ago
I knew someone would mention REPL-based development. We really should have better tooling in this area where we could set a breakpoint and when hit, change both code and/or data and continue. The REPL is so incredibly productive with immediate feedback that the long ORM expression actually runs the SQL query you were intending (or notice it does something you didn't intend).
agumonkey 3y ago
I had to fight a lot to stop colleagues doing the long change a variable, reload, renavigate, trigger some dom event and wait. Obviously I come from a lisp heavy culture, but I'm still stumped how many people stay in those high latency exploration mindset.
- 3y ago

kkfx 3y ago

In few years you'll discover classic Smalltalk workstations, modern Pharo, with a probably nicer language than Smalltalk itself. This model is essentially the original desktop model with it's end-user programming concept.

MarceColl 3y ago
I've already tried pharo, my problem there is with the next step, making that production ready. I need tooling that allows explorative programming and that can be transitioned into production. If not it's just another source of rigidness, just later in the process
- kkfx 3y ago
  That's not much a Pharo or CL issue but a social issue: actual IT development, and for the last decades, have gone in a regressive path for commercial reasons and it's from enough from these systems and paradigms to have a nearly impossible to fill gap in between. Personally I live digitally in Emacs, so I have some tooling for the present word, mostly hidden under Emacs or *nix system, but yes, it's still limited, less then Pharo since I can read my mails, org-attach my files and so on, but not integrate much more without all the modern integration issues of a an IT built to be isolated not integrated to sell programs before, services thereafter. However I see a very slow and long trend: the more current IT evolve the more it tend back to the IT pioneer era, in the "modern past" all was widget-based GUIs, not even Microsoft recommend CLIs for certain works, most apps have WebUIs witch are limited DocUIs but with the same doc concept, NotebookUI are more and more widespread, here and there someone recommend LaTeX instead of WYSIWYG word processor, someone else recommend R instead of spreadsheets and so on. Perhaps in 50 years we will have Smalltalk, Lisp and so on again, at least at concept/paradigm level.
- ElevenLathe 3y ago
  Smalltalk and Excel are both good to draw on. Another programming tradition that is actually very popular but seems to have limited influence on "professional" programming languages is Wolfram/Mathematica. The Wolfram Cloud product has some very interesting features in terms of exposing your Mathematica code as a service on the internet. Here's an interesting demo: https://www.youtube.com/watch?v=3VqlCm_T2CA https://www.youtube.com/watch?v=3VqlCm_T2CA. Mathematica also popularized the "notebook" style of dev which is also very nice for exploration.

TheAlchemist 3y ago

Well, that makes me think about Excel ! I think what the author describes is why Excel is so heavily used. It's often not easy to clearly define the specifications - but it's easy to play a bit with the data the user knows, and discover some flaws in the logic, corner cases etc. And also very easy to prototype processes. I almost miss it.

postepowanieadm 3y ago
I once made a pdf text extraction tool in sql. By accident. Having snapshots of data on every step was really nice.

cjs_ac 3y ago

I have this problem with planning. My planning mostly ends up being sequencing of tasks rather than working how to do them. Software is made out of ideas, and code is just a symbolic representation of those ideas. Writing the code is the easy part; working out what ideas to represent is the hard part of programming. If your planning involves working out what ideas to represent, if you do that in greater and greater detail, eventually you stop planning the work and cross over into doing the work. > File based organization, right from the start you need to start deciding what goes where. Of course I think most of us start with a single file, full of random functions and tests. But at some point this file becomes unbearable, impossible to find what you are looking for. At that point you have to make decisions and start splitting the file at logical points. Deal with what is private and what is not. It may not seem like much, but it's very annoying. In a project I'm working on at home, I started off with an extremely naive implementation, the single file of functions described above, and when the messiness became unbearable, I started building object-oriented abstractions to make sense of it all. But the structure of those abstractions was discovered while making the mess - if I'd tried to build that at the start, I'd have built the wrong thing. I'm very fortunate in that in both my day job and my recreational programming, I'm the only person working on that particular codebase at the time. (At work, I develop what are basically plugins for the framework-type-thing that's our main product.) Consequently, I have the freedom to make a mess and clean it up once I've muddled about in the solution space to discover what's going to work best, and so most of the solutions in the blog post are rather unnecessary for me. In the end, two months of work on my home project was thrown away and rewritten, which I could never have got away with at my job, but the project is all the better for it.

cmiller1 3y ago
> But the structure of those abstractions was discovered while making the mess For me at least it's more like bang out the naive implementation mess then "discover" the proper structure of the abstractions in bed that night or in the shower the next morning.
deergomoo 3y ago
This comment very well captures what I was trying (and failing) to explain to a colleague recently, when talking about why TDD doesn’t fit the way I solve problems. It’s almost like tending to something as it grows, discovering what the solution most wants to be with respect to the problem and the environment (e.g. programming language, server vs browser etc.)

MarceColl 3y ago

I'm the author, was not expecting to see this here today, somebody posted it yesterday as well but got no traction. It's just a small essay about how I approach programming, particularly when understanding problems and exploring new ideas. It's a little bit of explanation of a project I'm doing as well as something I'd have liked to read years ago when I was sure I was just doing it wrong.

agumonkey 3y ago
Do you have buddies or web acquaintances talking about similar ideas / goals ? I've always been searching for this mindset but I rarely read / hear about it.
norman784 3y ago
This sounds much like how unison[0] works. I would say that if it were to use a C like syntax, the succeed chances would go up. Would be cool if it targets WASM, this something I'm super interested right now, there are a lot of advantages on targeting WASM one of them is the interoperability with other components written in different languages, this will be when the component model[1] get's finished and implemented. [0]http://unison-lang.org http://unison-lang.org [1]https://component-model.bytecodealliance.org https://component-model.bytecodealliance.org
- MarceColl 3y ago
  Unison was very much an inspiration for my thoughts, but statement based programming (C-syntax) is pretty much out of the question since they rely on hashing functions and on some specific ways of specifying computation so it can work as it does with the whole distribution of computation and universal database of code. It is actually one of the problems I have with common lisp, its impossible to do a semantic hash on arbitrary common lisp functions so I create more history that I need, since a reordering of independent let forms cannot be easily detected to preserve functionality
  - brabel 3y ago
    Your ideas sounded very much like a mixup of Common Lisp with SLIME, Smalltalk interactivity and Unison-like storage of code in a database instead of files. I've tried all of them, I think the closest thing I've seen to what you describe, which I also find very attractive, is the GT Smalltalk environment: https://gtoolkit.com/ https://gtoolkit.com/ Have you tried that? They call this idea "moldable development" as you can "mold" your environment to your needs. Even though I loved it, I ended up not using it much, mostly because it's a bit too heavy to keep handy for exploration all the time when needed (it takes like 1GB of RAM even when idle!)... as I already can do most of that with emacs, which is much lighter, I just stick with it.

xixixao 3y ago

I agree. It’s a tall order to replace all the pieces. You need: - programming language not oriented around files - PL compiler designed first and foremost for incremental complication / interpretation - New source control that’s syntax-aware and watch not commit based - New IDE that allows navigating the tree and space And any of these are a reason for people to not use it: - CommonLisp? Most people don’t like it. Could you adapt TS? Rust? - If it’s not Git, can’t use GitHub - Emacs? Could it be a VS Code plugin? A fork? Where is a billionaire techie who could foot the bill to build this? It’s a mammoth effort.

MarceColl 3y ago
Thats exactly why I started with common lisp, I have a working prototype of all of this (not a complex version control system yet) in about 60 lines of code. The IDE was a hacked 50 line emacs major mode, it's really not that much work to get started. It has of course a lot of rough edges. About the reasons not to use it, thats fine for me, for now this is for me and to serve as inspiration, I want to work with CL. If anyone else wants to tacke it in another language, all the best for me :) Git compatibility and output to files is something that the system should have, indeed

creesch 3y ago

> I think I'm not alone in this. Hopefully? No, it works in a similar way for me. I can do abstractions to a certain degree in my head and plan ahead but also do need to work on a problem and trial things. It's interesting that the article mentions Javascript as not having an immediate feedback cycle. Which is true enough for a lot of the modern Javascript stack. Although live reloading is also a thing depending on what aspect you are working on. > Imagine a service that never stops, you diff the existing functions with the new functions, only updating what needs to be updated. If there is a change in the rate of errors then you automatically rollback the deployed functions. I am pretty sure that this is already a concept in use by companies. Although the exact naming for it, I can't remember. It also highly depends on the sort of domain you are active in. The more critical your data is, the less likely it is that you want to risk using this type of development and testing model. Not to mention that it seems lean in the logical fallacy of "only errors indicate faults" while the most devastating bugs often are those that don't cause clear errors. For example, when dealing with data your particular system might have no increased errors but due to a data change a system down the line might. Another potential issue is that because you are developing this in an explorative manner, it also doesn't leave much room for a review process. Or any of the myriad of other processes and methods that normally should be in place to evaluate the thing under development. In addition to the above, the way the article does describes the envisioned process leaves very little room for proper documentation. More specifically, it feels like a process where documentation will be more of an afterthought than it already is within many teams. Off-topic for the subject, but using monospaced fonts for blog posts sucks for readability, imho.

MarceColl 3y ago
Javascript has decent feedback cycle, but pretty bad program state persistence as you reload things, there are some attemps and sometimes Vite is able to do it, but it's a pretty sad state of affairs in general. The end remarks are more to let the imagination run a little bit free than anything particularly structured, the part about deployment doesnt really exist as it is in most companies, there is a concept of canary deployments, blue green, etc, but not on a running process ala Erlang or Elixir. Still, Im not sure if it is even a good idea since starting from a clean slate is usually good. I need to do more testing around this. There is no interest in documentation while the process of discovering is going on, because the documentation would be stale as soon as you move on in the discovery process. Its not even a fault of the system, it just doesn't make sense. Once you find a good enough design, then you transition into settling it, documenting it, writing more thorough tests. About the review process, this is mostly targeted at individuals and teams where there is a certain trust involved. But for knowledge spread that's why I also tackle doing this collaboratively, as I think it could even increase the effectiveness of this approach. The design was pretty fast done with a template. I'll see if I have time today to test other fonts and see how it improves. Thanks for taking the time to read

laserbeam 3y ago

I remember some of the earlier talks from Jonathan Blow about how he was designing Jai (the programming language) to alleviate some of these problems and support explorative programmings. One of the ideas was to write code such that if you copy paste things into a different context, the syntax changes are minimal. It should be trivial to copy paste a block of code to turn it into a function. It should be trivial to turn a lambda into a full blown function. Type signatures should look the same whether they are function arguments or variable definitions etc. Not advocating for Jai per se... But most languages are pretty bad at supporting this style, some are better.

ReleaseCandidat 3y ago
> But most languages are pretty bad at supporting this style, some are better. The first and most important step is having a language without statements and only expressions. But, as Jai needs a `return` in functions, it already failed the trivial test. https://github.com/BSVino/JaiPrimer/blob/master/JaiPrimer.md https://github.com/BSVino/JaiPrimer/blob/master/JaiPrimer.md

tpoacher 3y ago

> I am very bad at planning and visualizing what a program should look before I start writing it This is what top-down programming is for (though, arguably, a lot easier for 'smaller' projects)

anonzzzies 3y ago

This is what Lisp is good at; emacs/slime and you can experiment/plan your project together. Python tries this with it's only repl (see the Django comment also; that solution works ok, but nothing compared to slime) or with notebooks and stuff. It's far behind though and I would much rather forever just use emacs/slime for everything. I do for my own projects, but for others I have to cry in my soup.

crq-yml 3y ago

I believe it's the combination of needing exploration but also wanting "safe checkpoints" that leads me now towards writing in terms of code generation first and making big iterations through the generator, and then subsequently testing and exploring interactively. I've been working in a blend of Forth, Lua, and Python this way. All three have REPLs. Python is the "fattest" of the three, and does more of the dirty work with the OS and file processing since those things are built in. Lua is the one I actually want to use for the primary task of generating source because of its small size, familiarity and flexibility. Forth is the tip of the spear and solves concrete lower-level stuff without needing the help of another layer. While I haven't yet made this process very sophisticated - it's mostly string substitution, pretty prints and state machines - it has a natural way of tightening the screws on any specification, because each step presents challenges about which layer I want to use, and whether I want to break with locally idiomatic usage and take the approach of inlining something from above instead.

buescher 3y ago

As an aside, is there anything like the Smalltalk/Pharo object browser and IDE for javascript in a web browser?

brabel 3y ago
SqueakJS: https://squeak.js.org/run/#highdpi&zip=https://files.squeak.org/6.1alpha/Squeak6.1alpha-22998-32bit/Squeak6.1alpha-22998-32bit.zip https://squeak.js.org/run/#highdpi&zip=https://files.squeak.... EDIT: oh sorry, didn't notice you asked specifically for JavaScript (not just Smalltalk in the browser).
xkriva11 3y ago
There are several distinct versions of Lively Kernel https://lively-kernel.org https://lively-kernel.org https://lively-kernel.org/lively4/lively4-jens/start.html https://lively-kernel.org/lively4/lively4-jens/start.html
- buescher 3y ago
  Lively is very impressive in its scope and beyond what I imagined anyone might have done already. Thanks! It seems a little brittle - the examples threw errors for me. It’s also a bit less coupled to the browser that what I was picturing - I was thinking something more like a lot of bells and whistles added to say Chrome devtools.

JanisErdmanis 3y ago

I wholeheartedly agree that the rigidity of most current development environments prevents us from understanding the problem. I would even say it shapes the problems we try to explore the same way a spoken language does. Many points were the reasons why I am now using Julia to develop a large-scope full-stack application. Revise (package) allows to evolve the program interactively while preserving the system state. For instance, I can have an instance of a struct in the REPL; I change one of the modules, and the available behaviour of the instance can be immediately tried out without recreating the state. Or, when interactively developing a webserver, I find an error on one of the served pages. I changed the piece of code in the module, and it is immediately reflected when reloaded while keeping the existing state. System introspection is easy as everything is available from the REPL. Default printing is available for all user-defined types, which can be specialised on a wish. Thus, printing an object or even getting to the precise point in the module call stack with an Infiltrator is always accessible. One pain point regarding file-based organisation is that, in most languages, every file is within its namespace. In Julia, these things are orthogonal. You can have a single namespace organised over multiple files or multiple namespaces within a single file. This generally makes it easier to navigate a large namespace while not being bothered with making explicit imports, etc., when one is still in the exploratory phase. I can’t really relate to tree-based exploration. Very few times do I find that the direction I have found is wrong, so I do a git reset to a previous commit. Also, there are a few sunk costs migrating from one approach to another when most of the functions are written to be pure. Another essential piece for explorative development is types. Although some planning is required, it is generally worth it. In Julia, those are checked at runtime, which nevertheless significantly helps to locate issues quickly when doing test-driven development.