3 ms·
What most people don't realize is that iPython (sans this notebook stuff) scales horizontally (across boxes) too. Get a bunch of UNIX surplus cheapo pizza boxes
by wantreprenr007 11y ago
What most people don't realize is that iPython (sans this notebook stuff) scales horizontally (across boxes) too. Get a bunch of UNIX surplus cheapo pizza boxes with decent CPUs and start crunching cheaper than buying new SuperMicro boxes... let some die and it's still cheaper than buying new gear.
If you want a Linux cluster and don't want to set that shit up manually, Rocks Clusters FTW.
I haven't had good luck with this new notebook system, it requires installing too much and doesn't just work like Python packages and it's too much of a pain, but maybe it will get less painful... Maybe they should just provide desktop apps which pull down/update and vendor their own per-language runtimes and give a way to interact with those isolated (vendored) runtimes, rather than a bunch of clunky instructions that go stale quickly. For server-side, something similar that doesnt interfere with native packages.
For somewhat older Python notebook apps, ReInteract is nice.
Source: a former Stanford HPC sysadmin
- detaro 11y ago> this new notebook system Jupyter? Or are you thinking of something else?
- gh02t 11y agoOnly drawback is power consumption. Running a basement/dungeon cluster on surplus (or even brand new) hardware, you're going to end up paying through the nose on power bills. With regard to a smoother distribution workflow, Enthought and Anaconda (the latter being my preference) go a long way to ease this. They keep curated package sets of scientific software that are known to work together smoothly, plus include their own package management systems that work well when installed standalone e.g. in a user's home direcctory. You can do this with pip and/or virtualenvs, but it's not as smooth and that is really optimized for a different use case I think.