8 ms·

Stop Filing Bugs, File a Container

171 points by tolmasky 10y ago

tolmasky 10y ago
Hi I'm part of the RunKit team and am happy to answer any questions!
eridius 10y ago
This looks really cool. I wish I could do this with other languages.
- Myztiq 10y ago
  You could do something similar with http://code.runnable.com/ http://code.runnable.com/ [Disclaimer - I work at Runnable]
Animats 10y ago
So now someone can reproduce the bug, but can't test a fix to it. They can take apart the container and attempt to reproduce in their own environment, but that's a lot of work.
- tolmasky 10y ago
  We provide a download link on the left which will download the file with all the dependencies shrink-wrapped, so that you can certainly start hacking on it locally too. This is a great start and a lot easier than manually trying to npm install the right combination of sub-dependencies, etc. If you have any other ideas we're happy to implement!
  - contingencies 10y ago
    The point is that complexity is most effectively managed by discrete (modular) consideration, not another layer of abstraction, which tends to hide it. Build process is a discrete development process problem space with its own group of established solutions. It should not be conflated with bug reporting. All problems in computer science can be solved by another level of indirection, except of course for the problem of too many indirections. - David Wheeler
    - sametmax 10y ago
      If you are in the type of bug runkit targets, getting the exact same env right will take you days. With runkit, you do have a little setup to do, but the days can be used to debug. This is a great concept : remove the work to reproduce an environment, and use the freed resource to understand said env. Major advantages are: - the bug reporter may not know his/her case is special and won't provide you with the information you need to see it. Providing a running container with the bug just obliviate that. - the running container can come with a docker file, which gives you everything: libs, versions, settings. It's basically a summary of the stuff you need for it to go wrong. - you can try locally the stuff, even if your setup is completly different, without messing with your setup. Because complex bugs rarely are reproducible with a bunch or pip install, apt get and other yum incantations. It's not good for all bugs. I would say it's actually bad for most bugs. But for the bugs that needs it, it seems fantastic.
      - contingencies 10y ago
        getting the exact same env right will take you days If your codebase has significant undocumented environmental dependencies and these are hard to script in place, there's something far more fundamentally wrong with your development process than bug reporting.
        sametmax 10y ago
        Lol you react like most of us have software deployed in controlled environments. I create Python libs that are used in so many various configurations by so many people I don't know. They come back and say "hey, I got this stack trace" (in the best case). Now I have to play the guessing game. Is it me, is it you ? Wrong path ? Permission problem ? Bad conf ? Network is having trouble ? Server is this particular linux version and and there is something important about the SEL setup here ? Oh but upstart/systemd don't behave the same way. Output is redirected here look. Stuff is not the expected encoding. Na it was not a bug but you're file is corrupted. What the heck is this data format ? Etc, etc. You thing you though about every single thing ? Your error handling is perfect for all IO ? You deal with all encoding, all user inputs perfectly ? You know all the little OS peculiarities that will make your subprocess run in the exact way you think ? Of course you don't, nobody is perfect, we don't have infinite resources. But there are infinite ways to fail. And for many things, you can figure it out with with just your code base and the error because it's a simple cause. But from time to time arrive this terrible bug that is a mix between a strange LOCAL, this particular version of the VM you use but only in one time zone with this env variable set. And for that, yes, a good container with a reproducible bug in the proper env is an interesting idea. Apparently, you don't ship software that is wildly used enough or you would not be that arrogant. Deploying on your own 100 servers is hard. Try seing your code deployed on 1000 servers that you don't own, nor configure.
- rhizome 10y ago
  Filing a container-disassembler container RSN.
amelius 10y ago
Or send a coredump :)
TheAceOfHearts 10y ago
This looks really awesome. I really like that you allow users to try it out without having to sign up. I think probably the biggest hurdle with something like this gaining traction is having the users remember that it's available. Is it possible to fork from an existing container? If that's possible, you can probably get maintainers to include a link to a working container with the lib already setup in the project's new issue template.
- tolmasky 10y ago
  You can link to an existing notebook and add "/clone" to the end of the URL to basically get this behavior. That's how the 'Try It Now' button at the bottom of the page works, we should definitely make this more obvious. Additionally, maintainers can embed RunKit on their pages with our embed API: https://runkit.com/embed https://runkit.com/embed. For example, Lodash uses embed on their documentation: https://lodash.com/docs/ https://lodash.com/docs/.
chriswarbo 10y ago
I find Nix invaluable for this kind of thing; especially how it rebuilds anything who's dependencies have changed, and caches everything else. I tried to play with docker, but haven't seen much use for it in for my current workflows.
- yagni3 10y ago
  +1 for nix. _nix-shell_ is a great way to get stuff up and running without having to wrestle with the container boundry.
- swift 10y ago
  I don't work on projects where Docker is useful in production, but I've found it tremendously useful in continuous integration. It guarantees a reproducible environment, which a VM could do, but just as importantly it documents how to produce that environment from scratch. I've found that hugely valuable. I've always liked the idea of Nix but I've never used it. How easy is it to extract and understand the details of the environment you've constructed to build your project?
  - baq 10y ago
    the problem is docker builds aren't really reproducible; e.g. if you're using ubuntu, you have to do apt-get update or your apt-get installs will fail with 404s. i also haven't used nix, but i hear it truly solves the problem.
  - chriswarbo 10y ago
    > Docker... documents how to produce that environment from scratch. I've found that hugely valuable. > How easy is it to extract and understand the details of the [Nix] environment you've constructed to build your project? The "extract" part is easy: Nix uses a declarative language to describe the environment, so in that sense it's like docker, but more fine-grained: each package is built in isolation, and installed to a read-only filesystem. As far as "understanding the details", it depends on what you want to know. You can use "nix-shell" to enter a build environment in a shell, e.g. nix-shell -E 'some Nix expression' will enter the build environment of 'some Nix expression', nix-shell -p 'some Nix expression' will create an environment with 'some Nix expression' installed, etc. For example, I'm currently tracking down a space leak in a Haskell program by running the following command after each edit of the source code: nix-shell -p 'with import <nixpkgs> {}; profiledHaskellPackages.callPackage (runCabal2nix { url = ../myTester; }) { myDependency = profiledHaskellPackages.callPackage (runCabal2nix { url = ./.; }) {}; }' --run 'tester +RTS -M100M -xc' The command 'tester +RTS -M100M -xc' is a Haskell program which dies if it uses more than 100MB of heap, exposing the space leak and dumping a stack trace. This will be run in an environment containing Haskell packages built from the source code in "./." (the current directory, for "myDependency") and "../myTester" (for the "tester" command). These packages and all of their dependencies will be built with profiling enabled. Running this command over and over will just re-use the previously-built environment. If I edit the source in ./. or in ../myTester, Nix will notice that the hashes have changed, rebuild the packages and their dependents and make a new environment using those. This makes it easy for me to test potential fixes.
feld 10y ago
I hate to be negative, but this feels like an invitation for very poor bug quality reports where now you have to debug an application AND an OS (the container).
- deleted 10y ago
  [deleted]
- Me1000 10y ago
  Hi there, I'm on the RunKit team. I think we've all been in a situation where we have a bug reproducing in front of us but when we ask for help by describe or even give code to someone else they say "hmm, it works on my machine". Since software rarely runs in a vacuum, having better system level access can sometimes be the only way to debug it. Our hope here is that RunKit notebooks make it easier to see what's wrong, even when the problem is less obvious.
  - monochromatic 10y ago
    Probably makes more sense to use the container as a backup for cases like that. If the bug does reproduce on other machines, then a code snippet is going to be easier to diagnose than messing around with a whole container.
    - tolmasky 10y ago
      I think there may be a bit of a misunderstanding here. RunKit does not send you a container for you to interact with manually, in fact there is nothing in the RunKit UI to "mess around" with the container itself. RunKit is a coding notebook that is backed by containers (and automatic shrink-wrapping of dependencies) to guarantee determinism over multiple runs. In other words, it is a code snippet, but with a run button attached. Nothing prevents you from 1) copy pasting the code to your own computer to run it exactly the same way as you would have previously done, or even better 2) hitting the "download" link on the left which downloads the code + shrink-wrap file so you are using the same dependencies as the user. On top of that, we have also made stack traces and other elements of the UI a lot friendlier. From a reproducibility perspective, the notebook represents an undeniable instance that the bug did happen, along with all the background information you are usually asked (what version of node? what version of package? any other dependencies?). The goal is to make that all apparent in the code itself, vs extraneous other files like package.json or such. Here is an example: https://runkit.com/tolmasky/my-bug/1.0.0 https://runkit.com/tolmasky/my-bug/1.0.0 If you are interested in the underlying technology, we've documented it here: http://blog.runkit.com/2015/09/10/time-traveling-in-node.js-notebooks.html http://blog.runkit.com/2015/09/10/time-traveling-in-node.js-...
saosebastiao 10y ago
I wish we had better reproducibility tools. It should be possible to write a failing test case, and everything that is not of value to that test case gets stripped out, containing just the parts of the program/environment that can affect the test case (including sources of non-determinism like clocks and RNGs). Kinda like C-Reduce, but on steroids. And we should be able to look at the problem probabilistically too: given the form of the test case, the value pipeline through the debugger, and the structure of the program, we should be able to narrow down places to look as much as possible using some form of bayesian search theory.
- mrkgnao 10y ago
  Minimal test cases sounds like QuickCheck.
mattwhilden 10y ago
We have something "similar" for the .NET Native compiler. You pass a special flag to the compiler and it will generate a file with more or less everything we need in house to diagnose your issue. Compared to cut down repros, the packages are huge (200+MB) and debugging requires more gumption but we save a huge amount of time going back and forth with customers. It's not ideal but we'd rather have a "bad" repro than never hear about it. We've been taking customer issues this way for almost 2 years and it's been a good thing to have in the tool belt.
dullroar 10y ago
Am I the only one put off by the bad "s-t" ligatures as being too twee? It may or may not be a good approach, but I stopped reading literally ("Archer" literally) got too annoyed by the distraction of that font face. Edit: I want to downvote myself-my wife's W10 box show no annoying ligatures. My Linux Mint 18/Firefox setup is otherwise. Sorry for the distraction!
deavmi 10y ago
Very clever approach to filing a bug.
voiper1 10y ago
Wow. Not sure why all the hate. This is amazing. It's like the live javascript/html/css editors, but for systems! https://runkit.com/npm/lodash https://runkit.com/npm/lodash and poof, you can run queries against lodash and share and they can edit it. (which you could do in a plunkr too, if you loaded it). I didn't even need to sign up... Looks like http://code.runnable.com/ http://code.runnable.com/ does similar for more stacks, but you have to sign up for that.
corndoge 10y ago
Down in microcontroller land we generally just send an email describing the problem. But not knowing much about web development, I suppose problems in that area are much more complicated.
shade23 10y ago
I think this would make a lot of sense on platforms that undergo massive fragmentation(all fingers pointing at android here). Browsers are also fragmented , but websites like Can I Use It[1] should be able to reduce the problem to a manageable extent. I love the idea of reproducible bugs. Because often the hardest part of solving any bug is first being able to reproduce it followed by finding the source. Since I presume these containers would also help in reliably reproducing those one-off bugs which happen only on first install/ first open / on some other event. [1]:http://caniuse.com/ http://caniuse.com/

171 points by tolmasky 10y ago

tolmasky 10y ago

Hi I'm part of the RunKit team and am happy to answer any questions!

eridius 10y ago

This looks really cool. I wish I could do this with other languages.

Myztiq 10y ago
You could do something similar with http://code.runnable.com/ http://code.runnable.com/ [Disclaimer - I work at Runnable]

Animats 10y ago

So now someone can reproduce the bug, but can't test a fix to it. They can take apart the container and attempt to reproduce in their own environment, but that's a lot of work.

tolmasky 10y ago
We provide a download link on the left which will download the file with all the dependencies shrink-wrapped, so that you can certainly start hacking on it locally too. This is a great start and a lot easier than manually trying to npm install the right combination of sub-dependencies, etc. If you have any other ideas we're happy to implement!
- contingencies 10y ago
  The point is that complexity is most effectively managed by discrete (modular) consideration, not another layer of abstraction, which tends to hide it. Build process is a discrete development process problem space with its own group of established solutions. It should not be conflated with bug reporting. All problems in computer science can be solved by another level of indirection, except of course for the problem of too many indirections. - David Wheeler
  - sametmax 10y ago
    If you are in the type of bug runkit targets, getting the exact same env right will take you days. With runkit, you do have a little setup to do, but the days can be used to debug. This is a great concept : remove the work to reproduce an environment, and use the freed resource to understand said env. Major advantages are: - the bug reporter may not know his/her case is special and won't provide you with the information you need to see it. Providing a running container with the bug just obliviate that. - the running container can come with a docker file, which gives you everything: libs, versions, settings. It's basically a summary of the stuff you need for it to go wrong. - you can try locally the stuff, even if your setup is completly different, without messing with your setup. Because complex bugs rarely are reproducible with a bunch or pip install, apt get and other yum incantations. It's not good for all bugs. I would say it's actually bad for most bugs. But for the bugs that needs it, it seems fantastic.
    - contingencies 10y ago
      getting the exact same env right will take you days If your codebase has significant undocumented environmental dependencies and these are hard to script in place, there's something far more fundamentally wrong with your development process than bug reporting.
      - sametmax 10y ago
        Lol you react like most of us have software deployed in controlled environments. I create Python libs that are used in so many various configurations by so many people I don't know. They come back and say "hey, I got this stack trace" (in the best case). Now I have to play the guessing game. Is it me, is it you ? Wrong path ? Permission problem ? Bad conf ? Network is having trouble ? Server is this particular linux version and and there is something important about the SEL setup here ? Oh but upstart/systemd don't behave the same way. Output is redirected here look. Stuff is not the expected encoding. Na it was not a bug but you're file is corrupted. What the heck is this data format ? Etc, etc. You thing you though about every single thing ? Your error handling is perfect for all IO ? You deal with all encoding, all user inputs perfectly ? You know all the little OS peculiarities that will make your subprocess run in the exact way you think ? Of course you don't, nobody is perfect, we don't have infinite resources. But there are infinite ways to fail. And for many things, you can figure it out with with just your code base and the error because it's a simple cause. But from time to time arrive this terrible bug that is a mix between a strange LOCAL, this particular version of the VM you use but only in one time zone with this env variable set. And for that, yes, a good container with a reproducible bug in the proper env is an interesting idea. Apparently, you don't ship software that is wildly used enough or you would not be that arrogant. Deploying on your own 100 servers is hard. Try seing your code deployed on 1000 servers that you don't own, nor configure.
rhizome 10y ago
Filing a container-disassembler container RSN.

amelius 10y ago

Or send a coredump :)

TheAceOfHearts 10y ago

This looks really awesome. I really like that you allow users to try it out without having to sign up. I think probably the biggest hurdle with something like this gaining traction is having the users remember that it's available. Is it possible to fork from an existing container? If that's possible, you can probably get maintainers to include a link to a working container with the lib already setup in the project's new issue template.

tolmasky 10y ago
You can link to an existing notebook and add "/clone" to the end of the URL to basically get this behavior. That's how the 'Try It Now' button at the bottom of the page works, we should definitely make this more obvious. Additionally, maintainers can embed RunKit on their pages with our embed API: https://runkit.com/embed https://runkit.com/embed. For example, Lodash uses embed on their documentation: https://lodash.com/docs/ https://lodash.com/docs/.

chriswarbo 10y ago

I find Nix invaluable for this kind of thing; especially how it rebuilds anything who's dependencies have changed, and caches everything else. I tried to play with docker, but haven't seen much use for it in for my current workflows.

yagni3 10y ago
+1 for nix. _nix-shell_ is a great way to get stuff up and running without having to wrestle with the container boundry.
swift 10y ago
I don't work on projects where Docker is useful in production, but I've found it tremendously useful in continuous integration. It guarantees a reproducible environment, which a VM could do, but just as importantly it documents how to produce that environment from scratch. I've found that hugely valuable. I've always liked the idea of Nix but I've never used it. How easy is it to extract and understand the details of the environment you've constructed to build your project?
- baq 10y ago
  the problem is docker builds aren't really reproducible; e.g. if you're using ubuntu, you have to do apt-get update or your apt-get installs will fail with 404s. i also haven't used nix, but i hear it truly solves the problem.
- chriswarbo 10y ago
  > Docker... documents how to produce that environment from scratch. I've found that hugely valuable. > How easy is it to extract and understand the details of the [Nix] environment you've constructed to build your project? The "extract" part is easy: Nix uses a declarative language to describe the environment, so in that sense it's like docker, but more fine-grained: each package is built in isolation, and installed to a read-only filesystem. As far as "understanding the details", it depends on what you want to know. You can use "nix-shell" to enter a build environment in a shell, e.g. nix-shell -E 'some Nix expression' will enter the build environment of 'some Nix expression', nix-shell -p 'some Nix expression' will create an environment with 'some Nix expression' installed, etc. For example, I'm currently tracking down a space leak in a Haskell program by running the following command after each edit of the source code: nix-shell -p 'with import <nixpkgs> {}; profiledHaskellPackages.callPackage (runCabal2nix { url = ../myTester; }) { myDependency = profiledHaskellPackages.callPackage (runCabal2nix { url = ./.; }) {}; }' --run 'tester +RTS -M100M -xc' The command 'tester +RTS -M100M -xc' is a Haskell program which dies if it uses more than 100MB of heap, exposing the space leak and dumping a stack trace. This will be run in an environment containing Haskell packages built from the source code in "./." (the current directory, for "myDependency") and "../myTester" (for the "tester" command). These packages and all of their dependencies will be built with profiling enabled. Running this command over and over will just re-use the previously-built environment. If I edit the source in ./. or in ../myTester, Nix will notice that the hashes have changed, rebuild the packages and their dependents and make a new environment using those. This makes it easy for me to test potential fixes.

feld 10y ago

I hate to be negative, but this feels like an invitation for very poor bug quality reports where now you have to debug an application AND an OS (the container).

deleted 10y ago
[deleted]
Me1000 10y ago
Hi there, I'm on the RunKit team. I think we've all been in a situation where we have a bug reproducing in front of us but when we ask for help by describe or even give code to someone else they say "hmm, it works on my machine". Since software rarely runs in a vacuum, having better system level access can sometimes be the only way to debug it. Our hope here is that RunKit notebooks make it easier to see what's wrong, even when the problem is less obvious.
- monochromatic 10y ago
  Probably makes more sense to use the container as a backup for cases like that. If the bug does reproduce on other machines, then a code snippet is going to be easier to diagnose than messing around with a whole container.
  - tolmasky 10y ago
    I think there may be a bit of a misunderstanding here. RunKit does not send you a container for you to interact with manually, in fact there is nothing in the RunKit UI to "mess around" with the container itself. RunKit is a coding notebook that is backed by containers (and automatic shrink-wrapping of dependencies) to guarantee determinism over multiple runs. In other words, it is a code snippet, but with a run button attached. Nothing prevents you from 1) copy pasting the code to your own computer to run it exactly the same way as you would have previously done, or even better 2) hitting the "download" link on the left which downloads the code + shrink-wrap file so you are using the same dependencies as the user. On top of that, we have also made stack traces and other elements of the UI a lot friendlier. From a reproducibility perspective, the notebook represents an undeniable instance that the bug did happen, along with all the background information you are usually asked (what version of node? what version of package? any other dependencies?). The goal is to make that all apparent in the code itself, vs extraneous other files like package.json or such. Here is an example: https://runkit.com/tolmasky/my-bug/1.0.0 https://runkit.com/tolmasky/my-bug/1.0.0 If you are interested in the underlying technology, we've documented it here: http://blog.runkit.com/2015/09/10/time-traveling-in-node.js-notebooks.html http://blog.runkit.com/2015/09/10/time-traveling-in-node.js-...

saosebastiao 10y ago

I wish we had better reproducibility tools. It should be possible to write a failing test case, and everything that is not of value to that test case gets stripped out, containing just the parts of the program/environment that can affect the test case (including sources of non-determinism like clocks and RNGs). Kinda like C-Reduce, but on steroids. And we should be able to look at the problem probabilistically too: given the form of the test case, the value pipeline through the debugger, and the structure of the program, we should be able to narrow down places to look as much as possible using some form of bayesian search theory.

mrkgnao 10y ago
Minimal test cases sounds like QuickCheck.

mattwhilden 10y ago

We have something "similar" for the .NET Native compiler. You pass a special flag to the compiler and it will generate a file with more or less everything we need in house to diagnose your issue. Compared to cut down repros, the packages are huge (200+MB) and debugging requires more gumption but we save a huge amount of time going back and forth with customers. It's not ideal but we'd rather have a "bad" repro than never hear about it. We've been taking customer issues this way for almost 2 years and it's been a good thing to have in the tool belt.

dullroar 10y ago

Am I the only one put off by the bad "s-t" ligatures as being too twee? It may or may not be a good approach, but I stopped reading literally ("Archer" literally) got too annoyed by the distraction of that font face. Edit: I want to downvote myself-my wife's W10 box show no annoying ligatures. My Linux Mint 18/Firefox setup is otherwise. Sorry for the distraction!

deavmi 10y ago

Very clever approach to filing a bug.

voiper1 10y ago

Wow. Not sure why all the hate. This is amazing. It's like the live javascript/html/css editors, but for systems! https://runkit.com/npm/lodash https://runkit.com/npm/lodash and poof, you can run queries against lodash and share and they can edit it. (which you could do in a plunkr too, if you loaded it). I didn't even need to sign up... Looks like http://code.runnable.com/ http://code.runnable.com/ does similar for more stacks, but you have to sign up for that.

corndoge 10y ago

Down in microcontroller land we generally just send an email describing the problem. But not knowing much about web development, I suppose problems in that area are much more complicated.

shade23 10y ago

I think this would make a lot of sense on platforms that undergo massive fragmentation(all fingers pointing at android here). Browsers are also fragmented , but websites like Can I Use It[1] should be able to reduce the problem to a manageable extent. I love the idea of reproducible bugs. Because often the hardest part of solving any bug is first being able to reproduce it followed by finding the source. Since I presume these containers would also help in reliably reproducing those one-off bugs which happen only on first install/ first open / on some other event. [1]:http://caniuse.com/ http://caniuse.com/