4 ms·

> We absolutely intend to make it a powerful platform for computation on big datasets. The datasets I run on tend to be several terabytes and all the clusters

by mileswu 14y ago

> We absolutely intend to make it a powerful platform for computation on big datasets. The datasets I run on tend to be several terabytes and all the clusters that I use utilize various distributed filesystems (eg. Lustre [1] or Dcache [2] or Xrootd [3]) to store these files across local storage on all the nodes of the cluster. I think for big datasets some kind of support for these distributed filesystems will be necessary as most private (scientific) computing clusters use them. The complication is of course that to use these filesystems, one needs to build support for each access protocol into a python module (off the top of my head I'm not sure if they exist yet; perhaps they do). While most of these filesystems do offer standard posix 'mounting', this is usually not recommended for performance reasons. Perhaps I'm outside of the intended use case anyway. To a certain extent this problem is solved for my field of research (of course nothing ever works all the time heh), as all LHC/OSG Computing Grid centers run the same Scientific Linux distribution with the theory being that any code that works on one should work on another. But this sounds great for teaching classes and labs! I look forward to using it. [1] http://wiki.lustre.org/index.php/Main_Page http://wiki.lustre.org/index.php/Main_Page [2] http://www.dcache.org/ http://www.dcache.org/ [3] http://xrootd.slac.stanford.edu/ http://xrootd.slac.stanford.edu/

pwang 14y ago
Thanks for the feedback! You're right that for scientific use cases, the data access story defines mode and methods of computation. Your use case of having a large dataset stored across many files on a distributed FS is one of the cornerstone motivations for building Blaze, next-generation distributed Numpy: https://speakerdeck.com/sdiehl/blaze-next-generation-numpy https://speakerdeck.com/sdiehl/blaze-next-generation-numpy http://vimeopro.com/continuumanalytics/pydata-nyc-2012 http://vimeopro.com/continuumanalytics/pydata-nyc-2012 Rather than traditional "load all the data into memory" sort of approaches, Blaze is inherently out-of-core, and allows the user to define mappings of an index space onto local or remote files, and then manipulate that structure at the Python prompt as easily as if it were a small matrix stored in memory. There are existing PGAS approaches that are similar in spirit, but they tend to invoke heavyweight MPI machinery or make assumptions about the regularity or structure of the dataset that is being distributed across the cluster. Our guidance in designing and implementing Blaze is to make simple distributed things easy, and hard things possible; we are not trying to solve decades of distributed computing and linear algebra problems as a start-up. :-) So, hopefully you can see that with Blaze as your data access mechanism, and Wakari as a web-based front-end, you should be able to do large scale compute from within a web browser.

by mileswu 14y ago

pwang 14y ago

Thanks for the feedback! You're right that for scientific use cases, the data access story defines mode and methods of computation. Your use case of having a large dataset stored across many files on a distributed FS is one of the cornerstone motivations for building Blaze, next-generation distributed Numpy: https://speakerdeck.com/sdiehl/blaze-next-generation-numpy https://speakerdeck.com/sdiehl/blaze-next-generation-numpy http://vimeopro.com/continuumanalytics/pydata-nyc-2012 http://vimeopro.com/continuumanalytics/pydata-nyc-2012 Rather than traditional "load all the data into memory" sort of approaches, Blaze is inherently out-of-core, and allows the user to define mappings of an index space onto local or remote files, and then manipulate that structure at the Python prompt as easily as if it were a small matrix stored in memory. There are existing PGAS approaches that are similar in spirit, but they tend to invoke heavyweight MPI machinery or make assumptions about the regularity or structure of the dataset that is being distributed across the cluster. Our guidance in designing and implementing Blaze is to make simple distributed things easy, and hard things possible; we are not trying to solve decades of distributed computing and linear algebra problems as a start-up. :-) So, hopefully you can see that with Blaze as your data access mechanism, and Wakari as a web-based front-end, you should be able to do large scale compute from within a web browser.