6 ms·

DARPA gives $3M to Continuum Analytics to improve Python for big data

216 points by astaire 14y ago

pjscott 14y ago
More info from Continuum Analytics: http://continuum.io/blog/continuum-wins-darpa-xdata-funding http://continuum.io/blog/continuum-wins-darpa-xdata-funding The money is going to develop Blaze and Bokeh: http://blaze.pydata.org/ http://blaze.pydata.org/ https://github.com/ContinuumIO/Bokeh https://github.com/ContinuumIO/Bokeh
- gojomo 14y ago
  I see that Blaze is BSD-licensed... but Bokeh has no clear licensing information. Anyone know the intended Bokeh license? (The Dr. Dobbs article doesn't mention licensing at all; the Continuum release implies the XDATA program has a focus on open source but doesn't specify more about Bokeh.)
  - jsrn 14y ago
    "Bokeh will be BSD/MIT." https://github.com/ContinuumIO/Bokeh/issues/7 https://github.com/ContinuumIO/Bokeh/issues/7
- gjreda 14y ago
  I think the lack of a ggplot2 equivalent is one of the main things holding people back from fully converting from R to Python - I know it is for me. Once you really understand the ggplot2 syntax and style, making your graphics seems to come naturally. It's fantastic. I'm extremely excited to see Bokeh develop more.
  - alokv28 14y ago
    I've found matplotlib [1] to be an intuitive and customizable way to make beautiful plots with python [1]: http://matplotlib.org/ http://matplotlib.org/
- primelens 14y ago
  Better D3 integration from within Python would be awesome.
  - coldtea 14y ago
    Why, is there ANY D3 integration with Python?
    - pwang 14y ago
      https://github.com/mikedewar/d3py https://github.com/mikedewar/d3py
  - paddy_m 14y ago
    Bokeh has three components. BokehJS runs in the browser, the Bokeh server and the Bokeh Python client library. BokehJS draws plots in the browser with canvas. BokehJS maintains a data model of plot models (plots, renederers of different types, tools) with Backbone JS. It has a a connection to the Bokeh server via web sockets. The Bokeh server keeps track of plot objects and other models with redis. It keeps these persistently in documents. The python client connects to the server and either create new plots or updates existing plots. You can create a plot with python and get a url to embed that plot in any web page. This means that your python code can just be a script that doesn't have to worry about serving up a web page. We intend to write client libraries in other languages like java, R, and ruby to name a few.
    - siddboots 14y ago
      So... the server would maintain a set of grammar-of-graphics-like objects, which would be replicated as backbone models? In theory we could use whichever graphics libraries we want on top the backbone layer, right? Bokeh is starting to sound like everything I've ever wanted.
wilfra 14y ago
Article calls it both a 'donation' and an 'investment' - which is it?
- paddy_m 14y ago
  We are working to build open source software for them that meets their needs. I think they are making an investment in the sense that any customer is invested in the outcome of a project. It wasn't a VC investment.
- gojomo 14y ago
  I'd guess 'grant' would be a better term than either for what's happening here. AFAIK, DARPA doesn't take equity, but does make grants based on intended deliverables. ('Investment' was used in the article in the general sense of "an investment in our children's future" or "investing in shared infrastructure", rather than expecting a specific liquid return like cash dividends or an appreciated exit event.)
- ryguytilidie 14y ago
  Since DARPA can't take equity in companies, it doesn't really matter. Its a donation in that the department will get nothing in return. Its an investment in that it will ideally make America a better place. Think of everything DARPA does as both a donation and an investment.
dekhn 14y ago
Blaze has the potential to be great. Numpy/Scipy (and previously, numarry and NumPy) are, at least to scientists, some of the most useful libraries in Python. Being able to express computation in a high level language, and having it run in native code, yet still be able to access the data in python, is often a major productivity enhancer. I'm still a bit skeptical Blaze will catch on. It's ambitious, and the python numeric community has had to chase each new implementation (with serious issues with backwards compatibility and cost of adoption) over the years. I would actually far prefer numpy and its limited functionality, over a much more complex system that didn't have any adopters.
- siddboots 14y ago
  Having a bit of a look at the docs^1, I don't really see it as "yet another arrays package". The API exposes generalized table/array operations, and a subset of those features can be implemented using NumPy's ndarrays. If the interface is clean, then I don't think adoption will be an issue: there simply isn't anything else that does this. You are absolutely right that both of these projects are extremely ambitious, but, given the $3 mil grant, I can't help but be optimistic. 1 http://blaze.pydata.org/docs/overview.html#blaze-is-a-generalization-of-numpy http://blaze.pydata.org/docs/overview.html#blaze-is-a-genera...
- shazzdeeds 14y ago
  DARPA is also working closely with the Blaze team on several projects to enhance its functionality and performance, so it will continue to improve. I'm not aware of any money being directly invested, but the organization has a way of encouraging projects it finds promising.
eliben 14y ago
This is great! Continuum Analytics are doing cool stuff for numeric / big data Python and it's great to see the language getting more and more traction in this domain.
- chm 14y ago
  Will this mean better scientific computing with Python?
  - deleted 14y ago
    [deleted]
  - hazov 14y ago
    Depends on what you call scientific computing, Continuum Analytics appears to have a data/stats focus (I now work as a statistician and people talks good about their products, I never used them), there I believe Python has the possibility to displace R and Octave/SciLab/MatLab as the language for light academic stuff, for CFD (Computational Fluid Dynamics) and CEM (Computational Electrodynamics) guys generally run on C, C++ and Fortran, or whatever runs Numerical Linear Algebra faster, I doubt Python will take this post anytime soon.
    - dchichkov 14y ago
      I doubt, that you will find it easy to beat well written numeric Python code with naive C, C++ or Fortran approach. Python is not inherently slower than C, C++ or Fortran. It is just a language after all. If you have a fast implementation of computational engine, like Numpy, you can reach speeds of C++ in computational tasks. If you have an optimizer that simplifies your math expressions, before running them - you can get better than naive C++ approach. An optimizer and computational engine that optimally offloads work from CPU to GPU can give an order of magnitude advantage over naive C++/blas code. And to give you a concrete example - there is a nice Python library called Theano, that is doing just than.
      - hazov 14y ago
ahaefner 14y ago
I saw a presentation about Blaze at PyData. The CEO of Continuum was the author of Numpy and said they are doing everything he would have done differently in Blaze. Also, Blaze is attempting to abstract the concept of having huge arrays distributed across multiple systems, clusters, clouds, whatever. So imagine having large distributed arrays that you can easily perform computations on. Could be very powerful if it works as planned.
qompiler 14y ago
All I see in these libraries are standard data types which are already available in Python. Are these libraries even being developed by mathematicians? It looks like someone hired a bunch of first grade computer science students to chunk out wrappers.
- mmcnickle 14y ago
  There are many new data types introduced by these libraries, distinct from the standard library types. The N-dimentional array is the most notable one.
toolslive 14y ago
"If they can learn an easy language, they won't have to rely on an external software development group to complete their analysis." Although scientists earn more credit here than business people, allow me to make a skeptical remark: This is exactly the same reason they had in the past regarding with SQL. "Business people can't wait for programmers to run their queries; if they can learn an easy language, they won't have to rely on programmers to do that for them..." In the end, the business people did not do SQL; they just force developers to do it for them, using SQL. History repeats itself, perpetuating fallacies
- yummyfajitas 14y ago
  Many business people do write SQL. Typically in a poorly administered access database, but they do write SQL. Scientists also already use several easy languages (r, MATLAB, SAS, increasingly python) rather than relying on outside developers.
- pwang 14y ago
  > In the end, the business people did not do SQL False. There are literally hundreds of thousands of people using SQL every day to query and analyze their data, that would not be able to write a for-loop or tell you what a pointer is. The same goes for R. Just because the needs of business analytics have grown to a point where developers are needed to build extremely sophisticated things with SQL, does not mean that simple SQL does not serve its original purpose beautifully. The same is true for Python. You can do incredibly sophisticated things with Python, but there are many, many scientists and data analysts that use it to do simple things that would not quite fit into a SQL mold, but also don't require a deep knowledge of CS.
indeyets 14y ago
Would be really cool, if part of this money went for funding PyPy's NumPy effort (they still need $15K to be fully funded)
ateev23 14y ago
This will sure make me to learn python over ruby

216 points by astaire 14y ago

More info from Continuum Analytics: http://continuum.io/blog/continuum-wins-darpa-xdata-funding http://continuum.io/blog/continuum-wins-darpa-xdata-funding The money is going to develop Blaze and Bokeh: http://blaze.pydata.org/ http://blaze.pydata.org/ https://github.com/ContinuumIO/Bokeh https://github.com/ContinuumIO/Bokeh

gojomo 14y ago
I see that Blaze is BSD-licensed... but Bokeh has no clear licensing information. Anyone know the intended Bokeh license? (The Dr. Dobbs article doesn't mention licensing at all; the Continuum release implies the XDATA program has a focus on open source but doesn't specify more about Bokeh.)
- jsrn 14y ago
  "Bokeh will be BSD/MIT." https://github.com/ContinuumIO/Bokeh/issues/7 https://github.com/ContinuumIO/Bokeh/issues/7
gjreda 14y ago
I think the lack of a ggplot2 equivalent is one of the main things holding people back from fully converting from R to Python - I know it is for me. Once you really understand the ggplot2 syntax and style, making your graphics seems to come naturally. It's fantastic. I'm extremely excited to see Bokeh develop more.
- alokv28 14y ago
  I've found matplotlib [1] to be an intuitive and customizable way to make beautiful plots with python [1]: http://matplotlib.org/ http://matplotlib.org/
primelens 14y ago
Better D3 integration from within Python would be awesome.
- coldtea 14y ago
  Why, is there ANY D3 integration with Python?
  - pwang 14y ago
    https://github.com/mikedewar/d3py https://github.com/mikedewar/d3py
- paddy_m 14y ago
  Bokeh has three components. BokehJS runs in the browser, the Bokeh server and the Bokeh Python client library. BokehJS draws plots in the browser with canvas. BokehJS maintains a data model of plot models (plots, renederers of different types, tools) with Backbone JS. It has a a connection to the Bokeh server via web sockets. The Bokeh server keeps track of plot objects and other models with redis. It keeps these persistently in documents. The python client connects to the server and either create new plots or updates existing plots. You can create a plot with python and get a url to embed that plot in any web page. This means that your python code can just be a script that doesn't have to worry about serving up a web page. We intend to write client libraries in other languages like java, R, and ruby to name a few.
  - siddboots 14y ago
    So... the server would maintain a set of grammar-of-graphics-like objects, which would be replicated as backbone models? In theory we could use whichever graphics libraries we want on top the backbone layer, right? Bokeh is starting to sound like everything I've ever wanted.

wilfra 14y ago

Article calls it both a 'donation' and an 'investment' - which is it?

paddy_m 14y ago
We are working to build open source software for them that meets their needs. I think they are making an investment in the sense that any customer is invested in the outcome of a project. It wasn't a VC investment.
gojomo 14y ago
I'd guess 'grant' would be a better term than either for what's happening here. AFAIK, DARPA doesn't take equity, but does make grants based on intended deliverables. ('Investment' was used in the article in the general sense of "an investment in our children's future" or "investing in shared infrastructure", rather than expecting a specific liquid return like cash dividends or an appreciated exit event.)
ryguytilidie 14y ago
Since DARPA can't take equity in companies, it doesn't really matter. Its a donation in that the department will get nothing in return. Its an investment in that it will ideally make America a better place. Think of everything DARPA does as both a donation and an investment.

dekhn 14y ago

Blaze has the potential to be great. Numpy/Scipy (and previously, numarry and NumPy) are, at least to scientists, some of the most useful libraries in Python. Being able to express computation in a high level language, and having it run in native code, yet still be able to access the data in python, is often a major productivity enhancer. I'm still a bit skeptical Blaze will catch on. It's ambitious, and the python numeric community has had to chase each new implementation (with serious issues with backwards compatibility and cost of adoption) over the years. I would actually far prefer numpy and its limited functionality, over a much more complex system that didn't have any adopters.

siddboots 14y ago
Having a bit of a look at the docs^1, I don't really see it as "yet another arrays package". The API exposes generalized table/array operations, and a subset of those features can be implemented using NumPy's ndarrays. If the interface is clean, then I don't think adoption will be an issue: there simply isn't anything else that does this. You are absolutely right that both of these projects are extremely ambitious, but, given the $3 mil grant, I can't help but be optimistic. 1 http://blaze.pydata.org/docs/overview.html#blaze-is-a-generalization-of-numpy http://blaze.pydata.org/docs/overview.html#blaze-is-a-genera...
shazzdeeds 14y ago
DARPA is also working closely with the Blaze team on several projects to enhance its functionality and performance, so it will continue to improve. I'm not aware of any money being directly invested, but the organization has a way of encouraging projects it finds promising.

eliben 14y ago

This is great! Continuum Analytics are doing cool stuff for numeric / big data Python and it's great to see the language getting more and more traction in this domain.

chm 14y ago
Will this mean better scientific computing with Python?
- deleted 14y ago
  [deleted]
- hazov 14y ago
  Depends on what you call scientific computing, Continuum Analytics appears to have a data/stats focus (I now work as a statistician and people talks good about their products, I never used them), there I believe Python has the possibility to displace R and Octave/SciLab/MatLab as the language for light academic stuff, for CFD (Computational Fluid Dynamics) and CEM (Computational Electrodynamics) guys generally run on C, C++ and Fortran, or whatever runs Numerical Linear Algebra faster, I doubt Python will take this post anytime soon.
  - dchichkov 14y ago
    I doubt, that you will find it easy to beat well written numeric Python code with naive C, C++ or Fortran approach. Python is not inherently slower than C, C++ or Fortran. It is just a language after all. If you have a fast implementation of computational engine, like Numpy, you can reach speeds of C++ in computational tasks. If you have an optimizer that simplifies your math expressions, before running them - you can get better than naive C++ approach. An optimizer and computational engine that optimally offloads work from CPU to GPU can give an order of magnitude advantage over naive C++/blas code. And to give you a concrete example - there is a nice Python library called Theano, that is doing just than.
    - hazov 14y ago

ahaefner 14y ago

I saw a presentation about Blaze at PyData. The CEO of Continuum was the author of Numpy and said they are doing everything he would have done differently in Blaze. Also, Blaze is attempting to abstract the concept of having huge arrays distributed across multiple systems, clusters, clouds, whatever. So imagine having large distributed arrays that you can easily perform computations on. Could be very powerful if it works as planned.

qompiler 14y ago

All I see in these libraries are standard data types which are already available in Python. Are these libraries even being developed by mathematicians? It looks like someone hired a bunch of first grade computer science students to chunk out wrappers.

mmcnickle 14y ago
There are many new data types introduced by these libraries, distinct from the standard library types. The N-dimentional array is the most notable one.

toolslive 14y ago

"If they can learn an easy language, they won't have to rely on an external software development group to complete their analysis." Although scientists earn more credit here than business people, allow me to make a skeptical remark: This is exactly the same reason they had in the past regarding with SQL. "Business people can't wait for programmers to run their queries; if they can learn an easy language, they won't have to rely on programmers to do that for them..." In the end, the business people did not do SQL; they just force developers to do it for them, using SQL. History repeats itself, perpetuating fallacies

yummyfajitas 14y ago
Many business people do write SQL. Typically in a poorly administered access database, but they do write SQL. Scientists also already use several easy languages (r, MATLAB, SAS, increasingly python) rather than relying on outside developers.
pwang 14y ago
> In the end, the business people did not do SQL False. There are literally hundreds of thousands of people using SQL every day to query and analyze their data, that would not be able to write a for-loop or tell you what a pointer is. The same goes for R. Just because the needs of business analytics have grown to a point where developers are needed to build extremely sophisticated things with SQL, does not mean that simple SQL does not serve its original purpose beautifully. The same is true for Python. You can do incredibly sophisticated things with Python, but there are many, many scientists and data analysts that use it to do simple things that would not quite fit into a SQL mold, but also don't require a deep knowledge of CS.

indeyets 14y ago

Would be really cool, if part of this money went for funding PyPy's NumPy effort (they still need $15K to be fully funded)

ateev23 14y ago

This will sure make me to learn python over ruby