5 ms·
I just find reproducible notebooks at the internet. It is really rare to find them from coworkers. If they aren't trained as developers, it is almost impossible
by neves 7y ago
I just find reproducible notebooks at the internet. It is really rare to find them from coworkers. If they aren't trained as developers, it is almost impossible. Their solution for this problem looks really efficient and is really simple and brilliant:
> Writing Polynote’s code interpretation from scratch allowed us to do away with this global, mutable state. By keeping track of the variables defined in each cell, Polynote constructs the input state for a given cell based on the cells that have run above it. Making the position of a cell important in its execution semantics enforces the principle of least surprise, allowing users to read the notebook from top to bottom. It ensures reproducibility by making it far more likely that running the notebook sequentially will work.
- type_enthusiast 7y agoThanks for the kind feedback. It's a young project to be honest, but I'm pretty proud of what we've done with only two contributors so far. With community participation I think we could support many more languages pretty quickly!
- yomritoyj 7y agoI really have always wished for reproducibility. Thanks for taking up this feature. How do you handle aliasing and references inside objects? Suppose I have #Cell 1 a = [1,2,3] b = (a,True) #Cell 2 b[0][0] = 5 #Cell 3 print(sum(a)) Now if I change Cell 2 to # Cell 2' b[0][0] = 4 and execute, Cell 3's result becomes stale. Do you track such dependencies? Would really love to read more about the underlying implementation.
- type_enthusiast 7y agoIf you mutate an object itself, we can't really track that. There's no magic going on; you can break the state if you use mutable objects. It's less of an issue in Scala where immutable data structures are the norm, but I can imagine it would be disappointing in Python. Currently it takes a shallow copy of the state output by each cell, meaning every value is going to be a primitive value or a reference. If it's a reference to mutable state, you're kind of on your own with respect to keeping reproducibility. I felt like this was a good compromise between strictly enforced reproducibility and practicality; if it turns out to be confusing we could consider deep copying the state, or having an option to do that (I could imagine it being pretty bad for efficiency in a lot of ML use cases, though).
- friggeri 7y agoSome related links if you’re interested in this kind of approach: https://nbviewer.jupyter.org/github/friggeri/notebooks/blob/master/tracking_inconsistencies_in_notebooks.ipynb https://nbviewer.jupyter.org/github/friggeri/notebooks/blob/... https://github.com/jupytercalpoly/reactivepy https://github.com/jupytercalpoly/reactivepy https://dataflownb.github.io/ https://dataflownb.github.io/ https://github.com/stitchfix/nodebook https://github.com/stitchfix/nodebook
- datagy 7y agoYou could also try Voilà to make reproducible standalone app from notebook https://blog.jupyter.org/and-voilà-f6a2c08a4a93 https://blog.jupyter.org/and-voilà-f6a2c08a4a93