4 ms·

That's one interesting project. As someone who relies heavily on collaboration with people using Jupyter Notebook. The most annoying points about reproducing th

That's one interesting project. As someone who relies heavily on collaboration with people using Jupyter Notebook. The most annoying points about reproducing their work are the environment and the hidden state of Jupyter Notebooks. This does to address directly the second problem. It does however by sacrificing flexibility. I might need to change a cell just to test a new thing (without affecting the other cells) but thats a trade off if you focus on reproducibility. I know that requirements.txt is the standard solution to the other problem. But generating and using it is annoying. The command pio freeze will list all the packages in bloated way (there is better ways) but I always hoped to find a notebook system that will integrate this information natively and have a way to embed that into a notebook in a form that I can share with other people. Unfortunately I can't see support for something in any of the available solutions (at least up to my knowledge).

akshayka 3y ago
Yes, the second half of reproducibility is for sure packages. A solution for reproducible environments is on our roadmap (https://marimo-team.notion.site/The-marimo-roadmap-e5460b9f203e4adeae633523fce87a35 https://marimo-team.notion.site/The-marimo-roadmap-e5460b9f2...), but we haven't quite figured it out yet. It's a bit challenging because Python has so many different solutions for package management. If you have any ideas we'd love to hear them.
- bluish29 3y ago
  The link redirect does not specify which point in the list you are referring to but I guess it is "Install missing packages from...". If so, then I really wonder if you mean supporting something like '!pip install numpy' like Jupyter or something else? I don't think this is really a solution, not to mention that this raise the question. Does it support running shell commands using '!' like Jupyter Notebook?
  - akshayka 3y ago
    Oh, sorry for not being more clear. That's not the one. It's "Package management: make notebooks reproducible down to the packages they use": https://marimo-team.notion.site/840c475fd7ca4e3a8c6f20c86fce51a0?v=0a96d9f02aa2459388dbadf36a9b6167&p=a67d9dd024f94defb995cd2bc6f6c4b6&pm=s https://marimo-team.notion.site/840c475fd7ca4e3a8c6f20c86fce... Does that align with what you're talking about? That page has some scrawled brainstormed notes. But we haven't spent time designing a solution yet.
    - bluish29 3y ago
      Thanks. That is precisely what I was talking about in my comment. It would solve the problem if we have some like that integrated natively. I understand that between pip, conda, mamba and all the others it would be hard problem to solve. But at least auto generating requirements.txt would be easier. But to be honest the hard part is identify packages and where they are from not what to do with information. Good luck with the development.
- aidos 3y ago
  People always complain about pip and python packaging but it’s never been an issue for me. I create a requirements.base.txt that has the versions of things I want installed. I then: pip freeze -r requirements.base.txt > requirements.txt Install is then simply: pip install -r requirements.txt Updating / installing something new is a matter of adding to the base file and then refreezing.
  - bluish29 3y ago
    There are several problems with this approach, notably you don't get information about specific platform stuff. You don't get information on how these package are installed (conda, mamba..etc). And it does not account for dependincies version conflicts which life very hard.
    - aidos 3y ago
      I don’t understand the platform thing, is that something to do with running on Windows? Why wouldn’t you just pip install? Why bring conda etc into the mix? If you have conflicts then you have to reconcile those at point of initial install - pip deals with that for you. I’ve never had a situation in 15 years of Python packages where there wasn’t a working combination of versions. These are genuine questions btw. I see these common complaints and wonder how I’ve not ever had issues with it.
      - bluish29 3y ago
        I will try to summarize the complaints (mine at least) in obvious simple points 1- pip freeze will miss packages not installed by pip (i.e. Conda). 2- It does include all packages, even not used in the project. 3- It just dumps all packages, their dependencies and sub-dependencies. Even without conflicts, if you happen to change a package, then it is very hard to keep track of dependencies and sub-dependencies that need to be removed. At some point, your file will be a hot mess. 4. If you install specific platform package version then this information will not be tracked
        3y ago
- tsenit 3y ago
  [dead]
- gcarvalho 3y ago
  The third half is data which only exists on your machine :P And even if it’s on some shared storage, it may have been generated by another unreproducible notebook or worse, manually.
331c8c71 3y ago
Nix is the only solution for reproducible environments that I would call rock-solid. It comes with costs and the gpu-related stuff is especially tricky e.g. https://www.canva.dev/blog/engineering/supporting-gpu-accelerated-machine-learning-with-kubernetes-and-nix/ https://www.canva.dev/blog/engineering/supporting-gpu-accele...
BerislavLopac 3y ago
You should try pip-tools.

akshayka 3y ago

Yes, the second half of reproducibility is for sure packages. A solution for reproducible environments is on our roadmap (https://marimo-team.notion.site/The-marimo-roadmap-e5460b9f203e4adeae633523fce87a35 https://marimo-team.notion.site/The-marimo-roadmap-e5460b9f2...), but we haven't quite figured it out yet. It's a bit challenging because Python has so many different solutions for package management. If you have any ideas we'd love to hear them.

bluish29 3y ago
The link redirect does not specify which point in the list you are referring to but I guess it is "Install missing packages from...". If so, then I really wonder if you mean supporting something like '!pip install numpy' like Jupyter or something else? I don't think this is really a solution, not to mention that this raise the question. Does it support running shell commands using '!' like Jupyter Notebook?
- akshayka 3y ago
  Oh, sorry for not being more clear. That's not the one. It's "Package management: make notebooks reproducible down to the packages they use": https://marimo-team.notion.site/840c475fd7ca4e3a8c6f20c86fce51a0?v=0a96d9f02aa2459388dbadf36a9b6167&p=a67d9dd024f94defb995cd2bc6f6c4b6&pm=s https://marimo-team.notion.site/840c475fd7ca4e3a8c6f20c86fce... Does that align with what you're talking about? That page has some scrawled brainstormed notes. But we haven't spent time designing a solution yet.
  - bluish29 3y ago
    Thanks. That is precisely what I was talking about in my comment. It would solve the problem if we have some like that integrated natively. I understand that between pip, conda, mamba and all the others it would be hard problem to solve. But at least auto generating requirements.txt would be easier. But to be honest the hard part is identify packages and where they are from not what to do with information. Good luck with the development.
aidos 3y ago
People always complain about pip and python packaging but it’s never been an issue for me. I create a requirements.base.txt that has the versions of things I want installed. I then: pip freeze -r requirements.base.txt > requirements.txt Install is then simply: pip install -r requirements.txt Updating / installing something new is a matter of adding to the base file and then refreezing.
- bluish29 3y ago
  There are several problems with this approach, notably you don't get information about specific platform stuff. You don't get information on how these package are installed (conda, mamba..etc). And it does not account for dependincies version conflicts which life very hard.
  - aidos 3y ago
    I don’t understand the platform thing, is that something to do with running on Windows? Why wouldn’t you just pip install? Why bring conda etc into the mix? If you have conflicts then you have to reconcile those at point of initial install - pip deals with that for you. I’ve never had a situation in 15 years of Python packages where there wasn’t a working combination of versions. These are genuine questions btw. I see these common complaints and wonder how I’ve not ever had issues with it.
    - bluish29 3y ago
      I will try to summarize the complaints (mine at least) in obvious simple points 1- pip freeze will miss packages not installed by pip (i.e. Conda). 2- It does include all packages, even not used in the project. 3- It just dumps all packages, their dependencies and sub-dependencies. Even without conflicts, if you happen to change a package, then it is very hard to keep track of dependencies and sub-dependencies that need to be removed. At some point, your file will be a hot mess. 4. If you install specific platform package version then this information will not be tracked
      - 3y ago
tsenit 3y ago
[dead]
gcarvalho 3y ago
The third half is data which only exists on your machine :P And even if it’s on some shared storage, it may have been generated by another unreproducible notebook or worse, manually.

331c8c71 3y ago

Nix is the only solution for reproducible environments that I would call rock-solid. It comes with costs and the gpu-related stuff is especially tricky e.g. https://www.canva.dev/blog/engineering/supporting-gpu-accelerated-machine-learning-with-kubernetes-and-nix/ https://www.canva.dev/blog/engineering/supporting-gpu-accele...

BerislavLopac 3y ago

You should try pip-tools.