6 ms·

> I wouldn't have known how to reproduce that "working state". So what? You don't need to reproduce that working state: The work has already been done! Think

by geocar 10y ago

> I wouldn't have known how to reproduce that "working state". So what? You don't need to reproduce that working state: The work has already been done! Think about how the interactive approach Lisp and Smalltalk programmers use is superior to the compile/run/rerun approach used by C++ and Java programmers -- by exploring the problem in a nonlinear way, you can find your way to a solution more quickly. Docker package management is like that: You make some changes, you commit often, fork your instance and try lots of different things. Eventually you find something that works and you tag it. You share your tags with other people. This is an extremely productive way to get work done.

oconnore 10y ago
Working with Lisp images: it's horrible when you lose track of what runtime change allows your code to continue working. You typically keep a working compilation between code and your intended environment, and use the run time to experiment in between changes.
- DigitalJack 10y ago
  I think this all the time. I have to refresh my clojure repl because I forget what I've eval'd or not and the cognitive load of mentally tracking that state is not something I want to bother with.
  - geocar 10y ago
    Clojure is unusually leaky compared to Unix/Linux or (for example) Erlang. If we design systems out of a collection of processes, that we can replace at will, then what we actually have is a collection of checkpoints and branches that we can try simultaneously. Replacing individual branches allows us to try out changes there, without having to rebuild (and wait) the entire system. Another way to think about this, is that Clojure is more like subversion than like git: It might be better than RCS, but you're still stuck with a linear revision history, and new changes can't completely conflict with old objects and old changes that are created. And yet another way, is to imagine all of those old class definitions that old objects hold onto are actually global variables, because they have global (time) extent, and yet are unreachable, so it's difficult to find out whether your new code is being affected by those old settings. Smalltalk, and even other (esp. Commercial) Lisps don't have this problem. It has nothing to do with interactivity, but with hidden state.
- jxy 10y ago
  This reminds me of one of the good things APL has. You can interactively change anything in your workspace, and all the things (array, function, operators) can be serialized, programmatically within the language.
- yaowenjiaozi 10y ago
  In Smalltalk implementations like Pharo, Squeak etc changes are captured in a changes file. If the image crashes before being saved, it is possible to cherry pick and replay unsaved changes from the changes file.
  - geocar 10y ago
    You can do this with Docker as well: Mount two commits and diff between them, pull bits into a third. It's also a surprisingly convenient way to build tests.
adekok 10y ago
> You don't need to reproduce that working state: The work has already been done! My background is physics, and the philosophy there is "If you can't reproduce it, it doesn't exist." In my experience with software, sharing opaque binary blobs is a Bad Thing. It doesn't matter if the blob is a proprietary firmware image, or a "open source" docker image. It's a magic unreproducible image. And (to me, at least) therefore suspect. > Docker package management is like that: You make some changes, you commit often, fork your instance and try lots of different things. Eventually you find something that works and you tag it. Again, the physics philosophy is "If you don't know what you did, you didn't do it". For software, programming by randomly trying things is one of the worst methods available. You should understand what you're doing, and understand the system you're building. > You share your tags with other people. This is an extremely productive way to get work done. Sure. It's a way to quickly create, test, roll back, and share opaque binary blobs. It's "productive" in that you get things done. But you have no real idea what you got done. You just have a magic image that "works". I would not hire a programmer with that kind of attitude. I've been burnt by that method, and those kind of programmers, too many times.
- Keats 10y ago
  We wrote an article about the same thing https://blog.wearewizards.io/why-docker-is-not-the-answer-to-reproducible-research-and-why-nix-may-be https://blog.wearewizards.io/why-docker-is-not-the-answer-to... (Tom has a background in physics as well)
  - geocar 10y ago
    You are assuming "that reproducibility is good". I believe this assumption needs to be challenged.
    - coldtea 10y ago
      Any significant contribution you might have for software engineering in regards to that? Aside from merely believing that. I can't even tell what you mean. If I want to rebuild my machine same as before, or deploy build 100s of servers at different points in time with the same setup, reproducibility is necessary. There's not even one argument against that. If you have any, please oblige us. Should deployments be a surprise?
falcolas 10y ago
How do you recommend applying security updates to the image, short of running (for example) `apt-get dist-upgrade`? What if that breaks? Do you just continue with the out-of-date image?
- bjt 10y ago
  This is the biggie for me. When the next heartbleed comes out, it would REALLY suck to be running a bunch of opaque images that you have no idea how to patch.
  - rdtsc 10y ago
    That's the thing, I surprised how many people were "ok" with that binary imagine that someone built on their laptop. Few saw an issue with it, and instead praised it as a great win. I mean, it was a win, not having to untangle the hairball of rpm/deb dependencies. I like that part too. But not understanding the downside was what shocked me. It was like everyone drank the magic kool-aid potion.
    - geocar 10y ago
      > I surprised how many people were "ok" with that binary imagine that someone built Really? This surprises you? The gross majority of the world runs software this way. The real issues are quality control and release process.
  - geocar 10y ago
    That's an unrelated problem. Developing your systems interactively, instead of reimplementing them over and over again produces the same results, just faster. If your servers aren't ephemeral, or you only have "one machine", or if you don't know about heartbleed patching, then you're going to be fucked anyway.
- cyphar 10y ago
  There's tools like zypper-docker that make this possible.
- 10y ago
sciurus 10y ago
It sure sounds like you are advocating Docker as a way to produce and distribute a http://martinfowler.com/bliki/SnowflakeServer.html http://martinfowler.com/bliki/SnowflakeServer.html We've been down this road before. Hacking something until it works, then imaging it and passing it on to your coworkers is a maintenance nightmare.
- geocar 10y ago
  I appreciate how difficult it is to understand someone when you begin the conversation by assuming you understand them. For example, did you read that article? You know Docker has none of those problems > The first problem with a snowflake server is that it's difficult to reproduce. Should your hardware start having problems, this means that it's difficult to fire up another server to support the same functions. docker tag/push/pull > If you need to run a cluster, you get difficulties keeping all of the instances of the cluster in sync. docker start tag > You can't easily mirror your production environment for testing. docker commit/export/import > When you get production faults, you can't investigate them by reproducing the transaction execution in a development environment. docker push/pull > The true fragility of snowflakes, however, comes when you need to change them. Snowflakes soon become hard to understand and modify. Upgrades of one bit software cause unpredictable knock-on effects. docker history/diff The issue that Martin is describing is one where people work on the server - update it in place, and do not have any change management process besides "do stuff" and any quality checking process besides "check stuff". He spent over 500 words to say that's probably not a good idea. No kidding. Now. What I'm actually advocating is interactive development: The exact same way we develop our APL workspaces and our Smalltalk images, can be used to design and build server infrastructure. Instead of a development cycle that goes: * Edit nixfile/dockerfile/vagrantfile * Wait (sometimes hours) * Test things * Repeat You get a very short: * Do stuff * Get happy * Tag the results Then when you want to run "in production", you publish your tags/image, review the history with your team, not dissimilar to how Smalltalk people will clean up their changes file, and then introduce the new version into the cluster. Once you're satisfied the world hasn't ended, you can decommission the old images. You can even make scripts that do these many of these steps automatically (divert x% of traffic to new instances, collect results, and so on). If you have a well-designed infrastructure, you can have junior developers use those scripts to make changes on their first day. Now. I get it, you have realised that if you start over often enough you get good at it. I get that you've seen people who treat systems development as a completely linear descent into unmaintainable bat barf that can only be recovered by a bad day reinstalling everything and "starting over", I also get that you think amortising that cost by taking those hours to rebuild your entire system when glibc changes is necessary to avoid that bad day. I also get you like static external configuration like Dockerfiles or Vagrantfiles, and view them as instrumental in the audit of that process. However while you understood that I disagree with that, you didn't understand why: If you had a decade or so of Smalltalk or Lisp experience instead of just ruby experience, you might have known exactly what I meant, but instead of asking, you just ignored the part you didn't understand, making a straw man argument that you did understand. After all, what's more likely, someone knows something important that you don't? Or someone knows something unimportant and just isn't as smart as you?

by geocar 10y ago

oconnore 10y ago

Working with Lisp images: it's horrible when you lose track of what runtime change allows your code to continue working. You typically keep a working compilation between code and your intended environment, and use the run time to experiment in between changes.

DigitalJack 10y ago
I think this all the time. I have to refresh my clojure repl because I forget what I've eval'd or not and the cognitive load of mentally tracking that state is not something I want to bother with.
- geocar 10y ago
  Clojure is unusually leaky compared to Unix/Linux or (for example) Erlang. If we design systems out of a collection of processes, that we can replace at will, then what we actually have is a collection of checkpoints and branches that we can try simultaneously. Replacing individual branches allows us to try out changes there, without having to rebuild (and wait) the entire system. Another way to think about this, is that Clojure is more like subversion than like git: It might be better than RCS, but you're still stuck with a linear revision history, and new changes can't completely conflict with old objects and old changes that are created. And yet another way, is to imagine all of those old class definitions that old objects hold onto are actually global variables, because they have global (time) extent, and yet are unreachable, so it's difficult to find out whether your new code is being affected by those old settings. Smalltalk, and even other (esp. Commercial) Lisps don't have this problem. It has nothing to do with interactivity, but with hidden state.
jxy 10y ago
This reminds me of one of the good things APL has. You can interactively change anything in your workspace, and all the things (array, function, operators) can be serialized, programmatically within the language.
yaowenjiaozi 10y ago
In Smalltalk implementations like Pharo, Squeak etc changes are captured in a changes file. If the image crashes before being saved, it is possible to cherry pick and replay unsaved changes from the changes file.
- geocar 10y ago
  You can do this with Docker as well: Mount two commits and diff between them, pull bits into a third. It's also a surprisingly convenient way to build tests.

adekok 10y ago

> You don't need to reproduce that working state: The work has already been done! My background is physics, and the philosophy there is "If you can't reproduce it, it doesn't exist." In my experience with software, sharing opaque binary blobs is a Bad Thing. It doesn't matter if the blob is a proprietary firmware image, or a "open source" docker image. It's a magic unreproducible image. And (to me, at least) therefore suspect. > Docker package management is like that: You make some changes, you commit often, fork your instance and try lots of different things. Eventually you find something that works and you tag it. Again, the physics philosophy is "If you don't know what you did, you didn't do it". For software, programming by randomly trying things is one of the worst methods available. You should understand what you're doing, and understand the system you're building. > You share your tags with other people. This is an extremely productive way to get work done. Sure. It's a way to quickly create, test, roll back, and share opaque binary blobs. It's "productive" in that you get things done. But you have no real idea what you got done. You just have a magic image that "works". I would not hire a programmer with that kind of attitude. I've been burnt by that method, and those kind of programmers, too many times.

Keats 10y ago
We wrote an article about the same thing https://blog.wearewizards.io/why-docker-is-not-the-answer-to-reproducible-research-and-why-nix-may-be https://blog.wearewizards.io/why-docker-is-not-the-answer-to... (Tom has a background in physics as well)
- geocar 10y ago
  You are assuming "that reproducibility is good". I believe this assumption needs to be challenged.
  - coldtea 10y ago
    Any significant contribution you might have for software engineering in regards to that? Aside from merely believing that. I can't even tell what you mean. If I want to rebuild my machine same as before, or deploy build 100s of servers at different points in time with the same setup, reproducibility is necessary. There's not even one argument against that. If you have any, please oblige us. Should deployments be a surprise?

falcolas 10y ago

How do you recommend applying security updates to the image, short of running (for example) `apt-get dist-upgrade`? What if that breaks? Do you just continue with the out-of-date image?

bjt 10y ago
This is the biggie for me. When the next heartbleed comes out, it would REALLY suck to be running a bunch of opaque images that you have no idea how to patch.
- rdtsc 10y ago
  That's the thing, I surprised how many people were "ok" with that binary imagine that someone built on their laptop. Few saw an issue with it, and instead praised it as a great win. I mean, it was a win, not having to untangle the hairball of rpm/deb dependencies. I like that part too. But not understanding the downside was what shocked me. It was like everyone drank the magic kool-aid potion.
  - geocar 10y ago
    > I surprised how many people were "ok" with that binary imagine that someone built Really? This surprises you? The gross majority of the world runs software this way. The real issues are quality control and release process.
- geocar 10y ago
  That's an unrelated problem. Developing your systems interactively, instead of reimplementing them over and over again produces the same results, just faster. If your servers aren't ephemeral, or you only have "one machine", or if you don't know about heartbleed patching, then you're going to be fucked anyway.
cyphar 10y ago
There's tools like zypper-docker that make this possible.
10y ago

sciurus 10y ago

It sure sounds like you are advocating Docker as a way to produce and distribute a http://martinfowler.com/bliki/SnowflakeServer.html http://martinfowler.com/bliki/SnowflakeServer.html We've been down this road before. Hacking something until it works, then imaging it and passing it on to your coworkers is a maintenance nightmare.

geocar 10y ago
I appreciate how difficult it is to understand someone when you begin the conversation by assuming you understand them. For example, did you read that article? You know Docker has none of those problems > The first problem with a snowflake server is that it's difficult to reproduce. Should your hardware start having problems, this means that it's difficult to fire up another server to support the same functions. docker tag/push/pull > If you need to run a cluster, you get difficulties keeping all of the instances of the cluster in sync. docker start tag > You can't easily mirror your production environment for testing. docker commit/export/import > When you get production faults, you can't investigate them by reproducing the transaction execution in a development environment. docker push/pull > The true fragility of snowflakes, however, comes when you need to change them. Snowflakes soon become hard to understand and modify. Upgrades of one bit software cause unpredictable knock-on effects. docker history/diff The issue that Martin is describing is one where people work on the server - update it in place, and do not have any change management process besides "do stuff" and any quality checking process besides "check stuff". He spent over 500 words to say that's probably not a good idea. No kidding. Now. What I'm actually advocating is interactive development: The exact same way we develop our APL workspaces and our Smalltalk images, can be used to design and build server infrastructure. Instead of a development cycle that goes: * Edit nixfile/dockerfile/vagrantfile * Wait (sometimes hours) * Test things * Repeat You get a very short: * Do stuff * Get happy * Tag the results Then when you want to run "in production", you publish your tags/image, review the history with your team, not dissimilar to how Smalltalk people will clean up their changes file, and then introduce the new version into the cluster. Once you're satisfied the world hasn't ended, you can decommission the old images. You can even make scripts that do these many of these steps automatically (divert x% of traffic to new instances, collect results, and so on). If you have a well-designed infrastructure, you can have junior developers use those scripts to make changes on their first day. Now. I get it, you have realised that if you start over often enough you get good at it. I get that you've seen people who treat systems development as a completely linear descent into unmaintainable bat barf that can only be recovered by a bad day reinstalling everything and "starting over", I also get that you think amortising that cost by taking those hours to rebuild your entire system when glibc changes is necessary to avoid that bad day. I also get you like static external configuration like Dockerfiles or Vagrantfiles, and view them as instrumental in the audit of that process. However while you understood that I disagree with that, you didn't understand why: If you had a decade or so of Smalltalk or Lisp experience instead of just ruby experience, you might have known exactly what I meant, but instead of asking, you just ignored the part you didn't understand, making a straw man argument that you did understand. After all, what's more likely, someone knows something important that you don't? Or someone knows something unimportant and just isn't as smart as you?