5 ms·

I can see this being useful for teaching environments, but I'm not so sure how useful this would be for actual scientific research. Most universities tend to h

by mileswu 14y ago

I can see this being useful for teaching environments, but I'm not so sure how useful this would be for actual scientific research. Most universities tend to have their own computing cluster, which will make storing the (potentially large) datasets easier, and furthermore use of the CPU time is free! Javascript ploting for the next set of graphs I need to produce for my research though is something I'm actually pondering about. It has the ability to be interactive which can be really helpful.

draven 14y ago
Useful for heavy computation on smallish datasets, but not for moderate computation on big datasets. Here at work the datasets are several gigabytes so this would not be a solution. After reading the blog post this looks like an enhanced IPython notebook with some additional packages. It could be a good platform to test-drive their technology though. (I hope this comment does not come across as being too negative --- the tech is cool --- but I really wonder what the use case for this tool is.)
- pwang 14y ago
  > Useful for heavy computation on smallish datasets, but not for moderate computation on big datasets. Here at work the datasets are several gigabytes so this would not be a solution. This is just a closed beta for the very first initial version of the product. We absolutely intend to make it a powerful platform for computation on big datasets. Many people are storing data in S3, and many datasets are sourced from locations on the web or remote data providers, anyway. With Wakari, you can have extremely efficient access to those datasets via IOPro (http://continuum.io/iopro http://continuum.io/iopro), which is adding indexing on S3 data in the next version. We will also be adding sharing, collaboration, and publishing features in the next version. I completely understand that it might not fit every situation (or your particular situation right now), but I hope that eventually you'll find these other features fairly compelling. > After reading the blog post this looks like an enhanced IPython notebook with some additional packages. Not the case at all. We provide a fully sandboxed Linux environment, accessible via the browser, with a complete installation of Anaconda Pro. The IPython notebook is an additional feature of Wakari, and not vice versa. One thing we haven't showcased very much is the fact that you have concurrent access to many different Python environments. Python 2.6 + Numpy 1.5? No problem. Want to try out Numpy 1.7 on Python 3.3? Just change the drop-down, and you get another shell with that. Want to play around with Travis Oliphant's Numba compiler for Python (https://store.continuum.io/cshop/numbapro https://store.continuum.io/cshop/numbapro), but don't want to go through the hassle of installing LLVM, LLVM-Py, and Numba on your own system? You're just one button click away. Like I said, I can understand that Wakari isn't going to solve everyone's problems right now, but it's fairly cool even in this initial version, and it'll only improve over time. :-)
  - dbaupp 14y ago
    Will you (or do you) offer PyPy support too? It is looking to be the fastest Python implementation around, and even more so as its support for Numpy matures.
    - pwang 14y ago
      Possibly; it is not on our immediate roadmap, but is something we can add if there is enough demand. If you are interested in accelerating Numpy, have you seen Numba? One decorator can net you a several hundred times speedup over pure python, and in many cases, a significant speedup over Numpy: http://www.slideshare.net/teoliphant/numba-lightning http://www.slideshare.net/teoliphant/numba-lightning http://continuum.io/blog/the-python-and-the-complied-python http://continuum.io/blog/the-python-and-the-complied-python https://vimeo.com/continuumanalytics/review/53105906/9f5dcbb86f https://vimeo.com/continuumanalytics/review/53105906/9f5dcbb...
  - mileswu 14y ago
    > We absolutely intend to make it a powerful platform for computation on big datasets. The datasets I run on tend to be several terabytes and all the clusters that I use utilize various distributed filesystems (eg. Lustre [1] or Dcache [2] or Xrootd [3]) to store these files across local storage on all the nodes of the cluster. I think for big datasets some kind of support for these distributed filesystems will be necessary as most private (scientific) computing clusters use them. The complication is of course that to use these filesystems, one needs to build support for each access protocol into a python module (off the top of my head I'm not sure if they exist yet; perhaps they do). While most of these filesystems do offer standard posix 'mounting', this is usually not recommended for performance reasons. Perhaps I'm outside of the intended use case anyway. To a certain extent this problem is solved for my field of research (of course nothing ever works all the time heh), as all LHC/OSG Computing Grid centers run the same Scientific Linux distribution with the theory being that any code that works on one should work on another. But this sounds great for teaching classes and labs! I look forward to using it. [1] http://wiki.lustre.org/index.php/Main_Page http://wiki.lustre.org/index.php/Main_Page [2] http://www.dcache.org/ http://www.dcache.org/ [3] http://xrootd.slac.stanford.edu/ http://xrootd.slac.stanford.edu/
paddy_m 14y ago
We see it being helpful for research in a couple of ways. You have all of the great python ecosystem at your fingertips for analysis. The really exciting part for scientists is the way you can seamlessly share code. Instead of having to install someone else's complete coding environment with all of the compilation steps, you will be able to simply see their code run.
jh73 14y ago
It's certainly true that a lot of principle investigators think this, but computing time is never free. A lot of the clusters have to win their own grants or fight for funding from the university. Nevertheless it looks like they are planning to include private cloud integration. I'm really looking forward to using this.
- paddy_m 14y ago
  We will be coming out with a private cloud version. That's an interesting use case, managing your own cluster in a university. We anticipate demand for private cloud from private industry where security criteria are different.
pwang 14y ago
> Most universities tend to have their own computing cluster, which will make storing the (potentially large) datasets easier, and furthermore use of the CPU time is free! We plan on making the entire Wakari stack available to deploy on internal clusters. (Obviously there are many businesses that don't want to upload their data to a third-party service.) >Javascript ploting for the next set of graphs I need to produce for my research though is something I'm actually pondering about. It has the ability to be interactive which can be really helpful. Those plotting features are very alpha at this point, but they are all components that will be in Bokeh (https://github.com/ContinuumIO/bokeh https://github.com/ContinuumIO/bokeh), which will be available as a standalone plotting system for Python.
rajbot 14y ago
> Javascript ploting for the next set of graphs I need to produce for my research though is something I'm actually pondering about. It has the ability to be interactive which can be really helpful. If you haven't already, you should check out the wonderful ipython HTML Notebook: http://ipython.org/ipython-doc/dev/interactive/htmlnotebook.html http://ipython.org/ipython-doc/dev/interactive/htmlnotebook....

by mileswu 14y ago

draven 14y ago

Useful for heavy computation on smallish datasets, but not for moderate computation on big datasets. Here at work the datasets are several gigabytes so this would not be a solution. After reading the blog post this looks like an enhanced IPython notebook with some additional packages. It could be a good platform to test-drive their technology though. (I hope this comment does not come across as being too negative --- the tech is cool --- but I really wonder what the use case for this tool is.)

pwang 14y ago
> Useful for heavy computation on smallish datasets, but not for moderate computation on big datasets. Here at work the datasets are several gigabytes so this would not be a solution. This is just a closed beta for the very first initial version of the product. We absolutely intend to make it a powerful platform for computation on big datasets. Many people are storing data in S3, and many datasets are sourced from locations on the web or remote data providers, anyway. With Wakari, you can have extremely efficient access to those datasets via IOPro (http://continuum.io/iopro http://continuum.io/iopro), which is adding indexing on S3 data in the next version. We will also be adding sharing, collaboration, and publishing features in the next version. I completely understand that it might not fit every situation (or your particular situation right now), but I hope that eventually you'll find these other features fairly compelling. > After reading the blog post this looks like an enhanced IPython notebook with some additional packages. Not the case at all. We provide a fully sandboxed Linux environment, accessible via the browser, with a complete installation of Anaconda Pro. The IPython notebook is an additional feature of Wakari, and not vice versa. One thing we haven't showcased very much is the fact that you have concurrent access to many different Python environments. Python 2.6 + Numpy 1.5? No problem. Want to try out Numpy 1.7 on Python 3.3? Just change the drop-down, and you get another shell with that. Want to play around with Travis Oliphant's Numba compiler for Python (https://store.continuum.io/cshop/numbapro https://store.continuum.io/cshop/numbapro), but don't want to go through the hassle of installing LLVM, LLVM-Py, and Numba on your own system? You're just one button click away. Like I said, I can understand that Wakari isn't going to solve everyone's problems right now, but it's fairly cool even in this initial version, and it'll only improve over time. :-)
- dbaupp 14y ago
  Will you (or do you) offer PyPy support too? It is looking to be the fastest Python implementation around, and even more so as its support for Numpy matures.
  - pwang 14y ago
    Possibly; it is not on our immediate roadmap, but is something we can add if there is enough demand. If you are interested in accelerating Numpy, have you seen Numba? One decorator can net you a several hundred times speedup over pure python, and in many cases, a significant speedup over Numpy: http://www.slideshare.net/teoliphant/numba-lightning http://www.slideshare.net/teoliphant/numba-lightning http://continuum.io/blog/the-python-and-the-complied-python http://continuum.io/blog/the-python-and-the-complied-python https://vimeo.com/continuumanalytics/review/53105906/9f5dcbb86f https://vimeo.com/continuumanalytics/review/53105906/9f5dcbb...
- mileswu 14y ago
  > We absolutely intend to make it a powerful platform for computation on big datasets. The datasets I run on tend to be several terabytes and all the clusters that I use utilize various distributed filesystems (eg. Lustre [1] or Dcache [2] or Xrootd [3]) to store these files across local storage on all the nodes of the cluster. I think for big datasets some kind of support for these distributed filesystems will be necessary as most private (scientific) computing clusters use them. The complication is of course that to use these filesystems, one needs to build support for each access protocol into a python module (off the top of my head I'm not sure if they exist yet; perhaps they do). While most of these filesystems do offer standard posix 'mounting', this is usually not recommended for performance reasons. Perhaps I'm outside of the intended use case anyway. To a certain extent this problem is solved for my field of research (of course nothing ever works all the time heh), as all LHC/OSG Computing Grid centers run the same Scientific Linux distribution with the theory being that any code that works on one should work on another. But this sounds great for teaching classes and labs! I look forward to using it. [1] http://wiki.lustre.org/index.php/Main_Page http://wiki.lustre.org/index.php/Main_Page [2] http://www.dcache.org/ http://www.dcache.org/ [3] http://xrootd.slac.stanford.edu/ http://xrootd.slac.stanford.edu/

paddy_m 14y ago

We see it being helpful for research in a couple of ways. You have all of the great python ecosystem at your fingertips for analysis. The really exciting part for scientists is the way you can seamlessly share code. Instead of having to install someone else's complete coding environment with all of the compilation steps, you will be able to simply see their code run.

jh73 14y ago

It's certainly true that a lot of principle investigators think this, but computing time is never free. A lot of the clusters have to win their own grants or fight for funding from the university. Nevertheless it looks like they are planning to include private cloud integration. I'm really looking forward to using this.

paddy_m 14y ago
We will be coming out with a private cloud version. That's an interesting use case, managing your own cluster in a university. We anticipate demand for private cloud from private industry where security criteria are different.

pwang 14y ago

> Most universities tend to have their own computing cluster, which will make storing the (potentially large) datasets easier, and furthermore use of the CPU time is free! We plan on making the entire Wakari stack available to deploy on internal clusters. (Obviously there are many businesses that don't want to upload their data to a third-party service.) >Javascript ploting for the next set of graphs I need to produce for my research though is something I'm actually pondering about. It has the ability to be interactive which can be really helpful. Those plotting features are very alpha at this point, but they are all components that will be in Bokeh (https://github.com/ContinuumIO/bokeh https://github.com/ContinuumIO/bokeh), which will be available as a standalone plotting system for Python.

rajbot 14y ago

> Javascript ploting for the next set of graphs I need to produce for my research though is something I'm actually pondering about. It has the ability to be interactive which can be really helpful. If you haven't already, you should check out the wonderful ipython HTML Notebook: http://ipython.org/ipython-doc/dev/interactive/htmlnotebook.html http://ipython.org/ipython-doc/dev/interactive/htmlnotebook....