8 ms·

The Build is Always Broken

95 points by mhd 7y ago

dbmikus 7y ago
This is what Bazel, Pants, etc (and I'm sure a bunch of other build tools) do for you. You define a dependency graph, then per commit you can see what targets have been modified and then you can run commands on only targets dependent on the modified ones. I'm not sure why this blog post is written like these things don't exist. EDIT: on a re read, I'm a bit off the mark. The author seems to want the compiler or interpreter that reads the code to automatically process the dependency graph of what changed (at the finest grain possible) and take appropriate action. The company I work at has tools for automatically updating our dependency graph relationships, and then commits/diffs execute commands against the dependencies of modified targets as part of the code review and CD process. This is pretty close to what the author is suggesting. I still think this blog post is written a bit too dramatically.
- hibbelig 7y ago
  > This is what Bazel, Pants, etc (and I'm sure a bunch of other build tools) do for you. You define a dependency graph, then per commit you can see what targets have been modified and then you can run commands on only targets dependent on the modified ones. This is way less live than the author hints at. Let's take Emacs as an example -- it can be extended using Emacs Lisp. If I write such an extension in Emacs Lisp, and I find that one of the functions isn't right, I go to the function and hit Ctrl-Meta-x and then the function is redefined in the running Emacs, and so I can try again to see if it works now. Or take the Fish shell as an example. It provides a method to define a function that is automatically saved. So you write a shell function "foo" and you try it out (in your interactive shell) and you see it's not right. You go "funcedit foo" which pops up an editor with the function in it. You make your change and save and exit the editor. You run the function again to see if it now does what you wanted. A similar experience can be had, I guess, using a Java IDE when working on a web application, if your stack supports hot reload (with JRebel?). You run the application in the debugger, you find it's incorrect, you make a change to the affected method, you save and the IDE hot-reloads the new method into the existing web application and you just try again. But for all of the above examples, it is still the case that there are two processes -- one redefines things while developing, and the other process kicks in when you turn off your computer and then turn it on again. I think in Smalltalk, you make code changes using something like the "hot reload" thing, but the new code is automatically saved to "the image". And when you want to run the system, you open "the image".
whateveracct 7y ago
Sounds like Nix to me.
mcintyre1994 7y ago
Is rebuilding everything from scratch on every build actually that common? If use use docker’s —cache-from flag on build, then assuming you’ve already pushed an image if you don’t make any changes on the part of the code that image is for you’re not going to actually rebuild that. That part of your build is just docker pull, build using cache, no changes so no work.
- bathtub365 7y ago
  Incremental compilation isn’t the only point of the article
- alkonaut 7y ago
  I'd say for compiled languages it's probably the norm. (i.e. where builds take one huge source repository and produce one set of compiled binaries e.g. a desktop app). Depending on how clever the build system is, I have been burned by "build hygiene" in the past. Some would go so far as even nuking the source repository and cloning it again rather than updating or running "git clean" etc, to be absolutely sure that no a single byte was left from a previous build (Git is pretty good at this, but other version control systems not so much).
  - iainmerrick 7y ago
    The norm is: - Use an incremental build during development - If you hit weird errors, try a clean build - Always use a clean build for releases Make is vulnerable to weird errors, as are newer systems like Gradle. Better build systems work hard to be 100% correct. Bazel is the best example I've seen. I'd say working with an extremely reliable build system is just as eye-opening as working with a good live coding system. If you never have to do a clean build, and the incremental build is fast, that's most of the benefits of live coding right there.
    - alkonaut 7y ago
      Incremental where you need to stop and restart running is still much slower than hot-edits (as in editing a dynamic/script language file) or hot-reloading statically compiled as C#). So clean build, incremental build+restart, incremental build while running I think is the 3 levels. I wonder if what the article is on about wrt. smalltalk is the last one.
jnwatson 7y ago
gcc can generate make dependencies since forever ago. That with ccache and distcc enables an environment where most of the time, compiling just a file or two plus linking needs to be done. For C at least, this problem has been solved for a while.
- AceJohnny2 7y ago
  Not really. Indeed, the building blocks for fast iteration are there, but as always in the 40-years-old ecosystem of C, no coherent solution has emerged and dominated the landscape.
jakear 7y ago
This is one area where Electron really shines. For example, VS Code is a reasonably large project, yet one can make a modification and play with it within a couple seconds (~200ms to recompile the changes, a second or two to reload the window). I’d be curious if anyone here knows how InteliJ or Eclipse compares?
- jdc 7y ago
  Aren't there native GUI toolkits for JavaScript too though?
  - jakear 7y ago
    There are, and those may have fast incremental builds too, but I wouldn’t know.
graton 7y ago
I'm used to using a Continuous Integration (CI) system which doesn't allow code to be merged unless it builds and tests pass. This does not allow for super fast merging as builds take time and testing takes time. But it prevents builds from breaking. The system I have been using is: https://zuul-ci.org/ https://zuul-ci.org/ Not exactly what the main focus of the article. But when the article talks about "the build is broken" then using a CI system which prevents that is good. It doesn't prevent an individual developer from breaking their local build of course.
benschulz 7y ago
I believe many posts here miss the core point. The article is talking about _liveness_, i.e. modifying the program while it is running. One example of this would be Squeak[1]. Another perhaps more relatable one is modifying your HTML5 app from the browser console. [1]: https://squeak.org/ https://squeak.org/
- mhd 7y ago
  Yes, it seems most of the posters are mostly familiar with the JS build … situation, where the last addendum of the post regarding multiple languages would apply anyway (as you're often building JS, HTML and CSS in one fell swoop). Modfiying web apps from the browser seems to be a rare case, interestingly. I think there were a forays[1] made in that direction, but the need for external tools to shoe-horn a half-shod module system onto JS was the final straw for that. Never mind the convenience of some nigh-essential language features (SCSS, ES6). [1]: https://www.amber-lang.net/ https://www.amber-lang.net/
- dbmikus 7y ago
  In Smalltalk, does anyone actually modify the production application live? I would think you'd want to modify the live application in devel, and then copy it over. At this point, you can either copy over the difference or just replace the whole production application. I'm not sure which would be faster. I don't think liveness is important for a production system. I prefer production to evolve in discrete chunks. Liveness makes sense for development. I suppose one other benefit of a live system is that you could deploy an update without without restarting the application. But at some level, you would be restarting part of the application, and you are just shifting the update logic from the networking layer to the function call layer or object layer or whatever minimal layer of swappable component you can deploy.
  - Kinrany 7y ago
    I barely touched Smalltalk, but I think people value liveness as a feature for the user, not the developer. This is basically ultimate customization: you can change the source code of your local copy.
skrebbel 7y ago
People love to complain about Webpack, but it pretty much does all of this out of box, all the way down to hot-reloading UI components. In that sense it's way ahead of all C++, C#, Java, etc build systems I know. Doing an `npm run dev` type of command in a reasonably set up JS project is very much this staged execution model the author talks about. Everything gets cached, on disk and in memory, only changes are recompiled, and reloads are fast, partial, and often hot-swapped. It's quite close to the maximum level of "liveness" I can imagine. It's kind of amazing how JS took over older and more established languages in this sense - and we're still not content (judging by how popular it is to complain about webpack). This is great.
- Insanity 7y ago
  I'm not a web dev but used webpack some years ago. Is the issue not that webpack works fine once you finally get it working, and the complaints are about all the issues before it does?
  - skrebbel 7y ago
    Yes it is, but this article highlights what an achievement it is that webpack is possible at all. I think Bracha would gladly struggle through some config if that was all that stood between him and his holy grail of liveness. Plus webpack got a lot better even in that regard, and there's alternatives like Parcel pushing the curve. Web devs sometimes don't realize how good they have it :-)
  - baddox 7y ago
    Perhaps, but you’d get similarly unsatisfactory reviews from any technology before you have learned how to use it. Criticism of how difficult it is to learn a technology is valid, but it’s not a valid criticism to say that a technology is not working well before you have learned how to use it.
- zelly 7y ago
  Arguably Skaffold and other container tools are approaching Webpack in its nonstop build-edit-rebuild workflow. But it's still not part of the code's build process. I'd like to see something like Webpack for native code. The hypothetical build system can, whenever files change, run the build again (in a container), execute tests (complaining when you break one), or let you keep the debugger open while you're editing (best attempt made to fast forward). This is probably better suited for modern languages with proper modules and incremental builds, like Rust, C++20, Java9.
discreteevent 7y ago
"what kind of debugger does your build tool provide?" That's what drives me mad about all these configuration heavy (declarative?) systems. Just give me a debugger and an imperative language and I'll step through it myself when there's a problem. It's what is happening anyway so it's a really leaky abstraction.
boffinism 7y ago
What? Sometimes live or pseudo-live is helpful when programming or debugging, and therefore the whole concept of building software from scratch is invalid?
- numpad0 7y ago
  "building software from scratch" as in `make clean && make` rather than F5 or Shift+F12, not as in "cat > a.out"
  - juped 7y ago
    The solution to that being time-consuming is just to not make clean
jupp0r 7y ago
OP should take a look at bazel and reevaluate his opinions. Things like executing only tests when necessary, caching/remote executing everything and stellar cross language support make it easier than ever to rebuild the world every time.
- kevingadd 7y ago
  (Can't tell if this is the joke) OP worked at Google as one of the creators of Dart so I assume he had a chance to encounter Bazel. Reading the post I got the feeling that these takes are partly informed by his encounters with die-hard Build System Believers.
  - iainmerrick 7y ago
    Here's a question for die-hard live coding believers: what gets checked into source control? I enjoy live coding, but the idea of checking in an "image" of the running system sounds horrible. You want to clearly distinguish between the key stuff and all the incidental state. The key stuff is what gets checked in, and that's your build system right there. (I don't ask just for the sake of arguing, I'm interested in the answer! And I don't see it addressed anywhere in the article.)
  - jupp0r 7y ago
    No joke intended. From reading the article, where cmake and scons are criticized, my impression was that many of OPs concerns could be addressed by using something more fundamentally sound like bazel.
hyperion2010 7y ago
Another point that seems related is the difficulty of call/cc semantics when dealing with changes in state that are outside the semantics of the virtual machine. It seems to me that there is no easy way to avoid cases like described in the article without forcing explicit accounting of a massive amount of implicit state. Depending on your use case the tradeoff might not be worth it. On the other hand, I would argue that the idea of a build is an entirely valid and useful construct, it merely represents a point in state space against which test cases (and production) must run. It is impossible to get away from that state. Some tools can make managing it easier, and some make it virtually impossible to manage. Imagine that we discovered a magical halting oracle and that we could compile everything instantaneously, we would still have to figure out what combination of states were valid, and we would call that the 'build'.
- iainmerrick 7y ago
  Yes, that's a good way of expressing it! I was just thinking that a running application is a little kernel of Code (the stuff that gets checked into source control) and big wrapper of State (what the user is currently doing with it). The build system view is that the Code is sacrosanct, and what we need to focus on is checking the right stuff into source control and trying to ensure the latest commit is always correct and self-consistent. We should always be able to throw away all the state and rebuild everything from scratch. The live coding view is that the user is more important than the Code, so we should focus on letting them get stuff done as effectively as possible. And programmers are users too! Therefore we should be able to modify the code without losing any of the user's state. Both views are correct and what's actually needed is a good balance between the two.
moomin 7y ago
I have two main issues here 1) liveness is great, but you run into fundamental problems when you upgrade a data structure that’s actually in use in memory. This problem is seen during deployment in a compile-time typed language and in production in a runtime typed language. Also, the more we incrementally patch a live environment the harder it becomes to specify what, exactly, is running in production.
alkonaut 7y ago
Not sure I understand the concrete suggestion here. When I develop I have "live" edits. (JS reloads, C# edit-and-continue, jvm hot reloading etc). But that's stile prone to being out of date, which is why there is a server that builds everything from scratch ensuring that we can reproduce it (as the local developers' edit loop sells those guarantees for speed). Obviously even a compiler on a server can be caching and clever and not rebuild everything (if you use a source package manager like cargo then you may run into this, but if you have a binary package manager like nuget then you don't - each compilation unit is either required to build or it isn't, and external dependencies are always just fetched).
- nunb 7y ago
  Gilad Bracha is one of the Smalltalk luminaries (his dialect is called Newspeak) and so he's really alluding to the ST way of doing things, where textual code is instantly compiled to machine-code or JIT-ed ... he's basically against the idea of a compiler as a separate binary that spits out an executable. I believe that's why he goes on about 'live-ness' which is a very smalltalk concept of a "live-image" of the running program. I doubt what he means is directly translatable to the Javascript world, although perhaps a running browser with JS in it can considered to be the live-image (like, say, the Clojurescript/React/Figwheel way of working). But as you note, in the JS-dev world, very often you do need to reboot and start again (say if an external resource like a CSS has changed).
dwenzek 7y ago
Unison (https://www.unisonweb.org/ https://www.unisonweb.org/) is a new programming language designed around a really neat idea that removes the necessity of builds. Unison’s core idea is that code is immutable and identified by its content. A function name can be changed with absolutely no recompilation nor risk to break another piece of code.
desc 7y ago
It's useful for development environments to cache things which have not changed in order to shorten the feedback loop. Many do. The problem is guaranteeing correctness. Correct cache invalidation is Hard(TM). Therefore, the CI system builds from scratch in order to guarantee a clean, correct and reproducible output from a given snapshot of the source. As for 'live programming' against a running image, this write-up starts with the implicit assumption that this is always a good thing. While it can be useful on occasion, the reason for restarting the application after changing its source code is the same reason we favour functional, immutable styles: mutable state is a pain and it's best to push it out of the thing you're working with if at all possible. Modifying the running image risks ending up with internal states which no single snapshot of the source could ever have yielded.
kstenerud 7y ago
The biggest problem I see with live systems is this: The longer you run without a restart, the more dependent you become on the current state. And the more dependent you become, the more likely it is that restarts will be catastrophic due to lost implicit state.
- specialist 7y ago
  Keen point. There was (is?) a live code editing capable browser which updates the source code. My failing memory seems to remember creating a project and mounting directories. Chrome? Great idea, but IIRC, also very brittle. And because it had it's own notions of "project", there was some impedance mismatch with the IDE.
kohlerm 7y ago
I do not really see the point of this article. If you want modular software you need to somehow declare dependencies to existing (maybe open source) software components. If you only want to do this via import statements then you would have to use some long URLs pointing to the exact version. You would have to include some GIT commit hash in case the component is hosted on GIT. That doesn't look like a practical approach to me. If you don't need modularity then as others point out Smalltalk solved this long time ago. Also as others pointed out, gradle, bazel etc solve most of the issues mentioned (incremental builds, distributed (!) caching of build resources).
MrBuddyCasino 7y ago
The Eclipse incremental compiler and the ability to swap method bodies during runtime via debugger connection gives a taste of this for the JVM. If only class redefinitions were supported...
bjornsing 7y ago
Kids today are so spoiled. :P I remember back in my Ericsson days when an incremental build would take 40 minutes, and then another 20 or so to load the resulting 100 MB monolithic executable onto hardware and boot it up. That kind of turnaround time teaches you to think through your code thoroughly, and use the type system to catch bugs early.
iainmerrick 7y ago
As others have noted, the title is confusing; this is really talking about live code reloading. "The build" => offline compilation in general, "is always broken" => is not as good as live reloading. I've used live reloading in HyperCard, where your application (the "stack") is always running and you can interactively edit the code for each UI element. I think this can work really well, but only under certain constraints: - You have a very clean separation between code and data (in HyperCard, the data is the set of cards and backgrounds, and the contents of all the fields; the code is the event handlers attached to each object). - The data format is stable. - There are clear checkpoints where your data is stable and no code is running. Those are sometimes the case in some applications. Arguably, all of them are nice properties and therefore good targets to aim for. But there are plenty of normal scenarios that don't meet those conditions: - You have an expensive long-running task. Trying to change the code half-way through is a fool's errand. You need to re-run it from the start, and/or find a way to split it up into smaller steps. Unit tests and a good incremental build system are your friends here. - You're working on the core data structures. Again, trying to update the code while all the data is still in memory is a bad idea. Just use a good build system and try to make the compilation and startup time as fast as possible. - You're changing the navigation system, so the user's current position in the app won't make sense any more. You'll need to reset back to the front page (or whatever). For that last one, "apply changes" in C# or Android Studio will generally do the right thing, even though those are mostly traditional build systems. There isn't really a hard distinction between a build system and live coding, it's a spectrum. I'd argue further that a build system (in the traditional "make" fashion) is more fundamental, because you can achieve anything with a build system (even if it's tedious and inefficient) whereas some tasks are just intractable with live coding. A fast, reliable and correct build system (like Bazel, as several people have mentioned already) is the best foundation for everything else.
machawinka 7y ago
This is something that emacs users do naturally every time they update a elisp function at runtime.

95 points by mhd 7y ago

This is what Bazel, Pants, etc (and I'm sure a bunch of other build tools) do for you. You define a dependency graph, then per commit you can see what targets have been modified and then you can run commands on only targets dependent on the modified ones. I'm not sure why this blog post is written like these things don't exist. EDIT: on a re read, I'm a bit off the mark. The author seems to want the compiler or interpreter that reads the code to automatically process the dependency graph of what changed (at the finest grain possible) and take appropriate action. The company I work at has tools for automatically updating our dependency graph relationships, and then commits/diffs execute commands against the dependencies of modified targets as part of the code review and CD process. This is pretty close to what the author is suggesting. I still think this blog post is written a bit too dramatically.

hibbelig 7y ago
> This is what Bazel, Pants, etc (and I'm sure a bunch of other build tools) do for you. You define a dependency graph, then per commit you can see what targets have been modified and then you can run commands on only targets dependent on the modified ones. This is way less live than the author hints at. Let's take Emacs as an example -- it can be extended using Emacs Lisp. If I write such an extension in Emacs Lisp, and I find that one of the functions isn't right, I go to the function and hit Ctrl-Meta-x and then the function is redefined in the running Emacs, and so I can try again to see if it works now. Or take the Fish shell as an example. It provides a method to define a function that is automatically saved. So you write a shell function "foo" and you try it out (in your interactive shell) and you see it's not right. You go "funcedit foo" which pops up an editor with the function in it. You make your change and save and exit the editor. You run the function again to see if it now does what you wanted. A similar experience can be had, I guess, using a Java IDE when working on a web application, if your stack supports hot reload (with JRebel?). You run the application in the debugger, you find it's incorrect, you make a change to the affected method, you save and the IDE hot-reloads the new method into the existing web application and you just try again. But for all of the above examples, it is still the case that there are two processes -- one redefines things while developing, and the other process kicks in when you turn off your computer and then turn it on again. I think in Smalltalk, you make code changes using something like the "hot reload" thing, but the new code is automatically saved to "the image". And when you want to run the system, you open "the image".

whateveracct 7y ago

Sounds like Nix to me.

mcintyre1994 7y ago

Is rebuilding everything from scratch on every build actually that common? If use use docker’s —cache-from flag on build, then assuming you’ve already pushed an image if you don’t make any changes on the part of the code that image is for you’re not going to actually rebuild that. That part of your build is just docker pull, build using cache, no changes so no work.

bathtub365 7y ago
Incremental compilation isn’t the only point of the article
alkonaut 7y ago
I'd say for compiled languages it's probably the norm. (i.e. where builds take one huge source repository and produce one set of compiled binaries e.g. a desktop app). Depending on how clever the build system is, I have been burned by "build hygiene" in the past. Some would go so far as even nuking the source repository and cloning it again rather than updating or running "git clean" etc, to be absolutely sure that no a single byte was left from a previous build (Git is pretty good at this, but other version control systems not so much).
- iainmerrick 7y ago
  The norm is: - Use an incremental build during development - If you hit weird errors, try a clean build - Always use a clean build for releases Make is vulnerable to weird errors, as are newer systems like Gradle. Better build systems work hard to be 100% correct. Bazel is the best example I've seen. I'd say working with an extremely reliable build system is just as eye-opening as working with a good live coding system. If you never have to do a clean build, and the incremental build is fast, that's most of the benefits of live coding right there.
  - alkonaut 7y ago
    Incremental where you need to stop and restart running is still much slower than hot-edits (as in editing a dynamic/script language file) or hot-reloading statically compiled as C#). So clean build, incremental build+restart, incremental build while running I think is the 3 levels. I wonder if what the article is on about wrt. smalltalk is the last one.

jnwatson 7y ago

gcc can generate make dependencies since forever ago. That with ccache and distcc enables an environment where most of the time, compiling just a file or two plus linking needs to be done. For C at least, this problem has been solved for a while.

AceJohnny2 7y ago
Not really. Indeed, the building blocks for fast iteration are there, but as always in the 40-years-old ecosystem of C, no coherent solution has emerged and dominated the landscape.

jakear 7y ago

This is one area where Electron really shines. For example, VS Code is a reasonably large project, yet one can make a modification and play with it within a couple seconds (~200ms to recompile the changes, a second or two to reload the window). I’d be curious if anyone here knows how InteliJ or Eclipse compares?

jdc 7y ago
Aren't there native GUI toolkits for JavaScript too though?
- jakear 7y ago
  There are, and those may have fast incremental builds too, but I wouldn’t know.

graton 7y ago

I'm used to using a Continuous Integration (CI) system which doesn't allow code to be merged unless it builds and tests pass. This does not allow for super fast merging as builds take time and testing takes time. But it prevents builds from breaking. The system I have been using is: https://zuul-ci.org/ https://zuul-ci.org/ Not exactly what the main focus of the article. But when the article talks about "the build is broken" then using a CI system which prevents that is good. It doesn't prevent an individual developer from breaking their local build of course.

benschulz 7y ago

I believe many posts here miss the core point. The article is talking about _liveness_, i.e. modifying the program while it is running. One example of this would be Squeak[1]. Another perhaps more relatable one is modifying your HTML5 app from the browser console. [1]: https://squeak.org/ https://squeak.org/

mhd 7y ago
Yes, it seems most of the posters are mostly familiar with the JS build … situation, where the last addendum of the post regarding multiple languages would apply anyway (as you're often building JS, HTML and CSS in one fell swoop). Modfiying web apps from the browser seems to be a rare case, interestingly. I think there were a forays[1] made in that direction, but the need for external tools to shoe-horn a half-shod module system onto JS was the final straw for that. Never mind the convenience of some nigh-essential language features (SCSS, ES6). [1]: https://www.amber-lang.net/ https://www.amber-lang.net/
dbmikus 7y ago
In Smalltalk, does anyone actually modify the production application live? I would think you'd want to modify the live application in devel, and then copy it over. At this point, you can either copy over the difference or just replace the whole production application. I'm not sure which would be faster. I don't think liveness is important for a production system. I prefer production to evolve in discrete chunks. Liveness makes sense for development. I suppose one other benefit of a live system is that you could deploy an update without without restarting the application. But at some level, you would be restarting part of the application, and you are just shifting the update logic from the networking layer to the function call layer or object layer or whatever minimal layer of swappable component you can deploy.
- Kinrany 7y ago
  I barely touched Smalltalk, but I think people value liveness as a feature for the user, not the developer. This is basically ultimate customization: you can change the source code of your local copy.

skrebbel 7y ago

People love to complain about Webpack, but it pretty much does all of this out of box, all the way down to hot-reloading UI components. In that sense it's way ahead of all C++, C#, Java, etc build systems I know. Doing an `npm run dev` type of command in a reasonably set up JS project is very much this staged execution model the author talks about. Everything gets cached, on disk and in memory, only changes are recompiled, and reloads are fast, partial, and often hot-swapped. It's quite close to the maximum level of "liveness" I can imagine. It's kind of amazing how JS took over older and more established languages in this sense - and we're still not content (judging by how popular it is to complain about webpack). This is great.

Insanity 7y ago
I'm not a web dev but used webpack some years ago. Is the issue not that webpack works fine once you finally get it working, and the complaints are about all the issues before it does?
- skrebbel 7y ago
  Yes it is, but this article highlights what an achievement it is that webpack is possible at all. I think Bracha would gladly struggle through some config if that was all that stood between him and his holy grail of liveness. Plus webpack got a lot better even in that regard, and there's alternatives like Parcel pushing the curve. Web devs sometimes don't realize how good they have it :-)
- baddox 7y ago
  Perhaps, but you’d get similarly unsatisfactory reviews from any technology before you have learned how to use it. Criticism of how difficult it is to learn a technology is valid, but it’s not a valid criticism to say that a technology is not working well before you have learned how to use it.
zelly 7y ago
Arguably Skaffold and other container tools are approaching Webpack in its nonstop build-edit-rebuild workflow. But it's still not part of the code's build process. I'd like to see something like Webpack for native code. The hypothetical build system can, whenever files change, run the build again (in a container), execute tests (complaining when you break one), or let you keep the debugger open while you're editing (best attempt made to fast forward). This is probably better suited for modern languages with proper modules and incremental builds, like Rust, C++20, Java9.

discreteevent 7y ago

"what kind of debugger does your build tool provide?" That's what drives me mad about all these configuration heavy (declarative?) systems. Just give me a debugger and an imperative language and I'll step through it myself when there's a problem. It's what is happening anyway so it's a really leaky abstraction.

boffinism 7y ago

What? Sometimes live or pseudo-live is helpful when programming or debugging, and therefore the whole concept of building software from scratch is invalid?

numpad0 7y ago
"building software from scratch" as in `make clean && make` rather than F5 or Shift+F12, not as in "cat > a.out"
- juped 7y ago
  The solution to that being time-consuming is just to not make clean

jupp0r 7y ago

OP should take a look at bazel and reevaluate his opinions. Things like executing only tests when necessary, caching/remote executing everything and stellar cross language support make it easier than ever to rebuild the world every time.

kevingadd 7y ago
(Can't tell if this is the joke) OP worked at Google as one of the creators of Dart so I assume he had a chance to encounter Bazel. Reading the post I got the feeling that these takes are partly informed by his encounters with die-hard Build System Believers.
- iainmerrick 7y ago
  Here's a question for die-hard live coding believers: what gets checked into source control? I enjoy live coding, but the idea of checking in an "image" of the running system sounds horrible. You want to clearly distinguish between the key stuff and all the incidental state. The key stuff is what gets checked in, and that's your build system right there. (I don't ask just for the sake of arguing, I'm interested in the answer! And I don't see it addressed anywhere in the article.)
- jupp0r 7y ago
  No joke intended. From reading the article, where cmake and scons are criticized, my impression was that many of OPs concerns could be addressed by using something more fundamentally sound like bazel.

hyperion2010 7y ago

Another point that seems related is the difficulty of call/cc semantics when dealing with changes in state that are outside the semantics of the virtual machine. It seems to me that there is no easy way to avoid cases like described in the article without forcing explicit accounting of a massive amount of implicit state. Depending on your use case the tradeoff might not be worth it. On the other hand, I would argue that the idea of a build is an entirely valid and useful construct, it merely represents a point in state space against which test cases (and production) must run. It is impossible to get away from that state. Some tools can make managing it easier, and some make it virtually impossible to manage. Imagine that we discovered a magical halting oracle and that we could compile everything instantaneously, we would still have to figure out what combination of states were valid, and we would call that the 'build'.

iainmerrick 7y ago
Yes, that's a good way of expressing it! I was just thinking that a running application is a little kernel of Code (the stuff that gets checked into source control) and big wrapper of State (what the user is currently doing with it). The build system view is that the Code is sacrosanct, and what we need to focus on is checking the right stuff into source control and trying to ensure the latest commit is always correct and self-consistent. We should always be able to throw away all the state and rebuild everything from scratch. The live coding view is that the user is more important than the Code, so we should focus on letting them get stuff done as effectively as possible. And programmers are users too! Therefore we should be able to modify the code without losing any of the user's state. Both views are correct and what's actually needed is a good balance between the two.

moomin 7y ago

I have two main issues here 1) liveness is great, but you run into fundamental problems when you upgrade a data structure that’s actually in use in memory. This problem is seen during deployment in a compile-time typed language and in production in a runtime typed language. Also, the more we incrementally patch a live environment the harder it becomes to specify what, exactly, is running in production.

alkonaut 7y ago

Not sure I understand the concrete suggestion here. When I develop I have "live" edits. (JS reloads, C# edit-and-continue, jvm hot reloading etc). But that's stile prone to being out of date, which is why there is a server that builds everything from scratch ensuring that we can reproduce it (as the local developers' edit loop sells those guarantees for speed). Obviously even a compiler on a server can be caching and clever and not rebuild everything (if you use a source package manager like cargo then you may run into this, but if you have a binary package manager like nuget then you don't - each compilation unit is either required to build or it isn't, and external dependencies are always just fetched).

nunb 7y ago
Gilad Bracha is one of the Smalltalk luminaries (his dialect is called Newspeak) and so he's really alluding to the ST way of doing things, where textual code is instantly compiled to machine-code or JIT-ed ... he's basically against the idea of a compiler as a separate binary that spits out an executable. I believe that's why he goes on about 'live-ness' which is a very smalltalk concept of a "live-image" of the running program. I doubt what he means is directly translatable to the Javascript world, although perhaps a running browser with JS in it can considered to be the live-image (like, say, the Clojurescript/React/Figwheel way of working). But as you note, in the JS-dev world, very often you do need to reboot and start again (say if an external resource like a CSS has changed).

dwenzek 7y ago

Unison (https://www.unisonweb.org/ https://www.unisonweb.org/) is a new programming language designed around a really neat idea that removes the necessity of builds. Unison’s core idea is that code is immutable and identified by its content. A function name can be changed with absolutely no recompilation nor risk to break another piece of code.

desc 7y ago

It's useful for development environments to cache things which have not changed in order to shorten the feedback loop. Many do. The problem is guaranteeing correctness. Correct cache invalidation is Hard(TM). Therefore, the CI system builds from scratch in order to guarantee a clean, correct and reproducible output from a given snapshot of the source. As for 'live programming' against a running image, this write-up starts with the implicit assumption that this is always a good thing. While it can be useful on occasion, the reason for restarting the application after changing its source code is the same reason we favour functional, immutable styles: mutable state is a pain and it's best to push it out of the thing you're working with if at all possible. Modifying the running image risks ending up with internal states which no single snapshot of the source could ever have yielded.

kstenerud 7y ago

The biggest problem I see with live systems is this: The longer you run without a restart, the more dependent you become on the current state. And the more dependent you become, the more likely it is that restarts will be catastrophic due to lost implicit state.

specialist 7y ago
Keen point. There was (is?) a live code editing capable browser which updates the source code. My failing memory seems to remember creating a project and mounting directories. Chrome? Great idea, but IIRC, also very brittle. And because it had it's own notions of "project", there was some impedance mismatch with the IDE.

kohlerm 7y ago

I do not really see the point of this article. If you want modular software you need to somehow declare dependencies to existing (maybe open source) software components. If you only want to do this via import statements then you would have to use some long URLs pointing to the exact version. You would have to include some GIT commit hash in case the component is hosted on GIT. That doesn't look like a practical approach to me. If you don't need modularity then as others point out Smalltalk solved this long time ago. Also as others pointed out, gradle, bazel etc solve most of the issues mentioned (incremental builds, distributed (!) caching of build resources).

MrBuddyCasino 7y ago

The Eclipse incremental compiler and the ability to swap method bodies during runtime via debugger connection gives a taste of this for the JVM. If only class redefinitions were supported...

bjornsing 7y ago

Kids today are so spoiled. :P I remember back in my Ericsson days when an incremental build would take 40 minutes, and then another 20 or so to load the resulting 100 MB monolithic executable onto hardware and boot it up. That kind of turnaround time teaches you to think through your code thoroughly, and use the type system to catch bugs early.

iainmerrick 7y ago

As others have noted, the title is confusing; this is really talking about live code reloading. "The build" => offline compilation in general, "is always broken" => is not as good as live reloading. I've used live reloading in HyperCard, where your application (the "stack") is always running and you can interactively edit the code for each UI element. I think this can work really well, but only under certain constraints: - You have a very clean separation between code and data (in HyperCard, the data is the set of cards and backgrounds, and the contents of all the fields; the code is the event handlers attached to each object). - The data format is stable. - There are clear checkpoints where your data is stable and no code is running. Those are sometimes the case in some applications. Arguably, all of them are nice properties and therefore good targets to aim for. But there are plenty of normal scenarios that don't meet those conditions: - You have an expensive long-running task. Trying to change the code half-way through is a fool's errand. You need to re-run it from the start, and/or find a way to split it up into smaller steps. Unit tests and a good incremental build system are your friends here. - You're working on the core data structures. Again, trying to update the code while all the data is still in memory is a bad idea. Just use a good build system and try to make the compilation and startup time as fast as possible. - You're changing the navigation system, so the user's current position in the app won't make sense any more. You'll need to reset back to the front page (or whatever). For that last one, "apply changes" in C# or Android Studio will generally do the right thing, even though those are mostly traditional build systems. There isn't really a hard distinction between a build system and live coding, it's a spectrum. I'd argue further that a build system (in the traditional "make" fashion) is more fundamental, because you can achieve anything with a build system (even if it's tedious and inefficient) whereas some tasks are just intractable with live coding. A fast, reliable and correct build system (like Bazel, as several people have mentioned already) is the best foundation for everything else.

machawinka 7y ago

This is something that emacs users do naturally every time they update a elisp function at runtime.