5 ms·

The State of Statistics in Julia

69 points by wallflower 14y ago

StefanKarpinski 14y ago
It's rather amazing how far this has come in a very short time. Excellent post, John!
andrewvc 14y ago
I tried julia last week looking for an R replacement. I was pretty excited but the web REPL is broken on the current OSX build. Try as I might I could not get the web repl to run when building it myself. It still seems quite immature, but I'm very excited about it's future.
- StefanKarpinski 14y ago
  We're phasing out the web repl since it's not actively developed anymore. At some point I'm going to take a crack at providing a Julia backend to iPython Notebook [1]. [1] http://ipython.org/ipython-doc/dev/interactive/htmlnotebook.html http://ipython.org/ipython-doc/dev/interactive/htmlnotebook....
  - timClicks 14y ago
    That will actually be an interesting test of the iPython architecture. Hopefully the team got it right with client/server, zmq and websockets.
  - andrewvc 14y ago
    That's unfortunate as I was trying to get an r-studio like experience.
    - StefanKarpinski 14y ago
      No reason that can't happen too, but the current web repl is more of a proof-of-concept.
- paulgb 14y ago
  I tried Julia recently and am very excited about it as well, especially with the addition of DataFrames. The only thing that worries me is the disorganized nature of the Base and Core packages. I fear that 5 years from now Julia will have a mess of functions in the global namespace like PHP, rather than a clean standard library like Python. Is this fear misplaced?
  - StefanKarpinski 14y ago
    The namespace mechanism was only added a few months ago and we're still working out some of the kinks with how it works. The plan going forward is to slice Base into an organized hierarchy but still have everything available by default without too many imports. The "using" keyword works rather similarly to the same keyword in C++ and now in Ruby 2.0 [1], and aims to give the best of both the "single huge namespace" (C, PHP, Matlab) and many small namespaces approaches (Python, Java). [1] http://blog.headius.com/2012/11/refining-ruby.html http://blog.headius.com/2012/11/refining-ruby.html
    - jordigh 14y ago
      Make sure you get namespaces right, right now. This is one of the biggest problems in Matlab, and 20+ years later, they still can't fix them, and the namespace problem is still huge.
  - JeffBezanson 14y ago
    Nothing more will be added to Core (though a few things might be removed), and very little more will be added to Base. Most new stuff in the future will be in separate packages and modules.
deleted 14y ago
[deleted]
pseut 14y ago
This might be slightly off-topic, but are DataFrames intended to be similar to R's data.frame, where the contents of the entire table are loaded into memory and operated on all at once, or more like SQL with the possibility of aggregate functions calculated incrementally? Since a lot of statistics can be calculated incrementally (OLS, obviously, but even something like MLE based on nonlinear optimization, if you allow for multiple passes through the dataset) R's approach really bugs me... even though providing the right tools for an aggregate function approach would be (I imagine) quite a bit more difficult.
- johnmyleswhite 14y ago
  DataFrames are in-memory. It sounds like you're describing what we're calling DataStreams, which are still a work in progress, but do already exist. And we're also building SGD descent for doing things like OLS incrementally.

69 points by wallflower 14y ago

It's rather amazing how far this has come in a very short time. Excellent post, John!

I tried julia last week looking for an R replacement. I was pretty excited but the web REPL is broken on the current OSX build. Try as I might I could not get the web repl to run when building it myself. It still seems quite immature, but I'm very excited about it's future.

StefanKarpinski 14y ago
We're phasing out the web repl since it's not actively developed anymore. At some point I'm going to take a crack at providing a Julia backend to iPython Notebook [1]. [1] http://ipython.org/ipython-doc/dev/interactive/htmlnotebook.html http://ipython.org/ipython-doc/dev/interactive/htmlnotebook....
- timClicks 14y ago
  That will actually be an interesting test of the iPython architecture. Hopefully the team got it right with client/server, zmq and websockets.
- andrewvc 14y ago
  That's unfortunate as I was trying to get an r-studio like experience.
  - StefanKarpinski 14y ago
    No reason that can't happen too, but the current web repl is more of a proof-of-concept.
paulgb 14y ago
I tried Julia recently and am very excited about it as well, especially with the addition of DataFrames. The only thing that worries me is the disorganized nature of the Base and Core packages. I fear that 5 years from now Julia will have a mess of functions in the global namespace like PHP, rather than a clean standard library like Python. Is this fear misplaced?
- StefanKarpinski 14y ago
  The namespace mechanism was only added a few months ago and we're still working out some of the kinks with how it works. The plan going forward is to slice Base into an organized hierarchy but still have everything available by default without too many imports. The "using" keyword works rather similarly to the same keyword in C++ and now in Ruby 2.0 [1], and aims to give the best of both the "single huge namespace" (C, PHP, Matlab) and many small namespaces approaches (Python, Java). [1] http://blog.headius.com/2012/11/refining-ruby.html http://blog.headius.com/2012/11/refining-ruby.html
  - jordigh 14y ago
    Make sure you get namespaces right, right now. This is one of the biggest problems in Matlab, and 20+ years later, they still can't fix them, and the namespace problem is still huge.
- JeffBezanson 14y ago
  Nothing more will be added to Core (though a few things might be removed), and very little more will be added to Base. Most new stuff in the future will be in separate packages and modules.

deleted 14y ago

[deleted]

pseut 14y ago

This might be slightly off-topic, but are DataFrames intended to be similar to R's data.frame, where the contents of the entire table are loaded into memory and operated on all at once, or more like SQL with the possibility of aggregate functions calculated incrementally? Since a lot of statistics can be calculated incrementally (OLS, obviously, but even something like MLE based on nonlinear optimization, if you allow for multiple passes through the dataset) R's approach really bugs me... even though providing the right tools for an aggregate function approach would be (I imagine) quite a bit more difficult.

johnmyleswhite 14y ago
DataFrames are in-memory. It sounds like you're describing what we're calling DataStreams, which are still a work in progress, but do already exist. And we're also building SGD descent for doing things like OLS incrementally.