5 ms·

Julia Package Ecosystem Pulse

63 points by idunning 12y ago

idunning 12y ago
I should point out the main Julia package listing is at http://pkg.julialang.org/ http://pkg.julialang.org/, this is the "executive summary" page. At the bottom of this "Pulse" page are test results - we run the tests for every package every night on both Julia stable (0.3) and unstable (0.4) to make sure everything is still working nicely together. Although, breakages on 0.4 just mean cool stuff is happening on master.
- jjoonathan 12y ago
  You aren't kidding: Gadfly still takes 30 seconds to import on 0.3, but I hear there's some sort of binary caching in 0.4 that should fix it.
  - ViralBShah 12y ago
    We should have static compilation in 0.4.
    - tlmr 12y ago
      Of modules or entire programs?
      - one-more-minute 12y ago
        I don't know exactly the current status or roadmap (if there is one), but the idea is to have both eventually. Packages and files will be cached automatically so that they load much more quickly, and it will be possible (eventually) to output standalone executables.
phkahler 12y ago
The stars seem to indicate popularity but not quality. They should have people rate it 1-5 and give a histogram like other places do, it's much more informative.
jordigh 12y ago
I was thinking the other day, before 2004, did we ever say "ecosystem" to refer to a bunch of software packages in a particular domain? I have a feeling that this book is what convinced us all to refer to our software collections as an ecosystem, and we have forgotten that it was this book's idea: http://www.amazon.ca/Software-Ecosystem-Understanding-Indispensable-Technology/dp/0262633310 http://www.amazon.ca/Software-Ecosystem-Understanding-Indisp... The word is ubiquitous now, but if you think about it, it's a bit of a funny way to refer to software, with allusions to predators, prey, producers, consumers, and population dynamics.
- johnmyleswhite 12y ago
  Looks like the growth curve starts before then: https://books.google.com/ngrams/graph?content=software+ecosystem&year_start=1990&year_end=2014&corpus=15&smoothing=3&share=&direct_url=t1%3B%2Csoftware%20ecosystem%3B%2Cc0 https://books.google.com/ngrams/graph?content=software+ecosy...
  - jordigh 12y ago
    Interesting, although the smoothing can be a bit misleading. As I expected, "open source" does not exist before 1998, when Eric Raymond, Bruce Perens, et al coined the term: https://books.google.com/ngrams/graph?content=open+source&case_insensitive=on&year_start=1990&year_end=2008&corpus=15&smoothing=0&share=&direct_url=t4%3B%2Copen%20source%3B%2Cc0%3B%2Cs0%3B%3BOpen%20Source%3B%2Cc0%3B%3Bopen%20source%3B%2Cc0%3B%3BOpen%20source%3B%2Cc0 https://books.google.com/ngrams/graph?content=open+source&ca... However, with data smoothing, it makes it appear is if the term existed before 1998: https://books.google.com/ngrams/graph?content=open+source&case_insensitive=on&year_start=1990&year_end=2008&corpus=15&smoothing=3&share=&direct_url=t4%3B%2Copen%20source%3B%2Cc0%3B%2Cs0%3B%3BOpen%20Source%3B%2Cc0%3B%3Bopen%20source%3B%2Cc0%3B%3BOpen%20source%3B%2Cc0 https://books.google.com/ngrams/graph?content=open+source&ca...
- glomph 12y ago
  The metaphor is widely used in lots of other areas though. Education, Science, Research, Business and so on.
peatmoss 12y ago
As primarily an R / stats user, the progress Julia has been making in terms of libraries has been staggering. I have just enough historic experience with Python to be similarly impressed if I place myself in that mindset. I recently attended a Software Carpentry instructor training session and witnessed a breakout between social scientists using R and natural sciences people using Python. Julia will have "made it" when orgs like SWC can start reasonably offering Julia sessions that draw both communities.
- timClicks 12y ago
  That's an interesting statement, because Software Carpentry intentionally positions itself as a follower rather than a leader in technology. That is, Python & R workshops are given because of the wide adoption of Python & R. Julia workshops would need to follow demand.
  - peatmoss 12y ago
    Agreed, that was my point if it wasn't clear. At the point where there are enough people from natural sciences and social sciences looking to learn Julia at an introductory SWC level, Julia will have made it.
transfire 12y ago
I've written a few small Julia programs and have found it to be a very productive language, in most respects. Unfortunately its module system is horrid. If you look at almost all Julia programs, they are essentially one big file. Any secondary files just get included into a main file wholeclothe. It's like working with archaic PHP all over again.
- cschmidt 12y ago
  Don't people used Module in practice? http://julia.readthedocs.org/en/latest/manual/modules/?highlight=module http://julia.readthedocs.org/en/latest/manual/modules/?highl... I haven't read up on Julia yet, so I'm curious.
  - sgt101 12y ago
    I've only played, but a few hours into it I wanted to partition up my project into separate places in order to separate concerns . I found modules to be a good way to do that once I realised that it's Pascal/modular 2 / ada again and not objects. I think that the problem may be that people are tripped up by the separation of type and modularity when they are used to that being coupled in c++ and java.
- Fede_V 12y ago
  I agree with this. I think in large part this is because the developers decided to try as much as possible to 'decouple' the concept of module from the filesystem. I think this is clever in the abstract, but in practice, it's a lot of headache for very, very little gain and a lot of added confusion. The main Julia team is a bunch of incredibly smart and dedicated guys who know a lot more about designing language than I do, but I think this was a case of trying to be too clever for your own good.
  - CyberDildonics 12y ago
    So what are the module systems you like and what do you like about them?
msravi 12y ago
I've been using Julia 0.3 for algorithmic trading in production for about 3 months now, and the language is truly, astoundingly productive. The only thing lacking for me at this point is true multithreading support (so I can fork off a python thread from within julia) - I currently get around this by using a socket interface to pass messages to python. Edit: For anyone who's interested, an indicative problem is this: http://stackoverflow.com/questions/21113030/async-thread-deadlock-in-julia-when-used-with-pycall http://stackoverflow.com/questions/21113030/async-thread-dea...
- tlmr 12y ago
  How would you compare this to using python with an easy JIT like numba?
  - msravi 12y ago
    I haven't used numba or benchmarked it, so I'm not really qualified to comment. I do however vastly prefer the MATLAB-like syntax of julia to that of python.
  - Fede_V 12y ago
    I actually both in terms of speed, and in terms of writing code they are remarkably similar approaches. numba forces you into certain ways of writing code to achieve significant speed ups, but usually that's not too restrictive. If you are used to writing numerical python code, you'll feel write at home in Julia - especially with the amazing PyCall module.
  - curiouslearn 12y ago
    Whenever I have tried to use Numba, I find it is slower than standard python. OTOH, Julia just works. Gives me the same performance as C.
- ViralBShah 12y ago
  There is a threads branch and we expect to have support in 0.4.
x43b 12y ago
Any recommendations for books for a Matlab user looking to use Julia? Thanks
- one-more-minute 12y ago
  There's not much in the way of books, but a few will be coming out soon, I believe. "Learning Julia" by Leah Hanson, for example. See also [1] for more learning resources. [1]: http://julialang.org/learning/ http://julialang.org/learning/
- srparish 12y ago
  Not a book, but scipy had a two-part introduction to Julia talk that's worth watching: part 1: https://www.youtube.com/watch?v=vWkgEddb4-A https://www.youtube.com/watch?v=vWkgEddb4-A part 2: https://www.youtube.com/watch?v=I3JH5Bg46yU https://www.youtube.com/watch?v=I3JH5Bg46yU
Aleman360 12y ago
I've been giving Julia a shot lately because I really like MATLAB syntax and IJulia is awesome for analysis reports. There are a few warts though: 1. I really don't like how the entire standard library is included in the default namespace. I know this is how MATLAB does it, but my function names are constantly colliding with built-in ones. 2. The documentation is hard to navigate. I'm used to MSDN. Compared to that, providing huge pages of functions with a sentence or two of prose and no examples is not that helpful. 3. Plotting tools need work. Gadfly produces beautiful output but interactivity is weak (you can only zoom in on Y-axis) and it chokes when you tell it to plot a lot of data. It should downsample based on display resolution... I should be able to tell it to graph 100,000 points without crashing my browser.
- thorwwaay 12y ago
  > but my function names are constantly colliding with built-in ones I'm sure you know this, but the casual drive-by reader may not. Functions with the same name aren't a collision. The parameter types of your function and a builtin of the same name all have to match in order for there to be a collision. In julia "module" introduces a namespace: http://julia.readthedocs.org/en/latest/manual/modules/ http://julia.readthedocs.org/en/latest/manual/modules/
- StefanKarpinski 12y ago
  Glad you're enjoying Julia :-) > I really don't like how the entire standard library is included in the default namespace. I know this is how MATLAB does it, but my function names are constantly colliding with built-in ones. This may not be obvious, but if you don't use a name from the standard library, then you are free to use it yourself since bindings imported by `using` are done lazily. For example: julia> e = "some message" "some message" julia> Base.e e = 2.7182818284590... julia> println(e) some message julia> Base.e e = 2.7182818284590... On the other hand, if you don't assign to `e` then you can use the built-in definition of `e` like so (in a new session): julia> e e = 2.7182818284590... If you want to use both your own `e` and `Base.e` then you do need to qualify `Base.e` – but this is exactly the same in any other language – if you use the same name from two different namespaces, you have to qualify them. Making the standard library namespace smaller or more granular wouldn't help in any way. > The documentation is hard to navigate. I'm used to MSDN. Compared to that, providing huge pages of functions with a sentence or two of prose and no examples is not that helpful. I definitely agree that our documentation isn't up to the standards of Microsoft. Pull requests are always welcomed and are a lovely way to give back. Over time the documentation will get better – I suspect we're already doing fairly well given the age of the language. > Plotting tools need work. Gadfly produces beautiful output but interactivity is weak (you can only zoom in on Y-axis) and it chokes when you tell it to plot a lot of data. It should downsample based on display resolution... I should be able to tell it to graph 100,000 points without crashing my browser. I'm curious what plotting system does actually let you plot 100,000 points without choking. Automatic downsampling would certainly be a nice advanced feature, but the mature plotting systems I'm familiar with do not do this. For example, neither R nor matplotlib do. Is there some Microsoft plotting library that does?
antman 12y ago
Calling julia functions from python would be nice
- ihnorton 12y ago
  All of the basic pieces are available here: https://github.com/JuliaLang/pyjulia https://github.com/JuliaLang/pyjulia

63 points by idunning 12y ago

I should point out the main Julia package listing is at http://pkg.julialang.org/ http://pkg.julialang.org/, this is the "executive summary" page. At the bottom of this "Pulse" page are test results - we run the tests for every package every night on both Julia stable (0.3) and unstable (0.4) to make sure everything is still working nicely together. Although, breakages on 0.4 just mean cool stuff is happening on master.

jjoonathan 12y ago
You aren't kidding: Gadfly still takes 30 seconds to import on 0.3, but I hear there's some sort of binary caching in 0.4 that should fix it.
- ViralBShah 12y ago
  We should have static compilation in 0.4.
  - tlmr 12y ago
    Of modules or entire programs?
    - one-more-minute 12y ago
      I don't know exactly the current status or roadmap (if there is one), but the idea is to have both eventually. Packages and files will be cached automatically so that they load much more quickly, and it will be possible (eventually) to output standalone executables.

phkahler 12y ago

The stars seem to indicate popularity but not quality. They should have people rate it 1-5 and give a histogram like other places do, it's much more informative.

jordigh 12y ago

I was thinking the other day, before 2004, did we ever say "ecosystem" to refer to a bunch of software packages in a particular domain? I have a feeling that this book is what convinced us all to refer to our software collections as an ecosystem, and we have forgotten that it was this book's idea: http://www.amazon.ca/Software-Ecosystem-Understanding-Indispensable-Technology/dp/0262633310 http://www.amazon.ca/Software-Ecosystem-Understanding-Indisp... The word is ubiquitous now, but if you think about it, it's a bit of a funny way to refer to software, with allusions to predators, prey, producers, consumers, and population dynamics.

johnmyleswhite 12y ago
Looks like the growth curve starts before then: https://books.google.com/ngrams/graph?content=software+ecosystem&year_start=1990&year_end=2014&corpus=15&smoothing=3&share=&direct_url=t1%3B%2Csoftware%20ecosystem%3B%2Cc0 https://books.google.com/ngrams/graph?content=software+ecosy...
- jordigh 12y ago
  Interesting, although the smoothing can be a bit misleading. As I expected, "open source" does not exist before 1998, when Eric Raymond, Bruce Perens, et al coined the term: https://books.google.com/ngrams/graph?content=open+source&case_insensitive=on&year_start=1990&year_end=2008&corpus=15&smoothing=0&share=&direct_url=t4%3B%2Copen%20source%3B%2Cc0%3B%2Cs0%3B%3BOpen%20Source%3B%2Cc0%3B%3Bopen%20source%3B%2Cc0%3B%3BOpen%20source%3B%2Cc0 https://books.google.com/ngrams/graph?content=open+source&ca... However, with data smoothing, it makes it appear is if the term existed before 1998: https://books.google.com/ngrams/graph?content=open+source&case_insensitive=on&year_start=1990&year_end=2008&corpus=15&smoothing=3&share=&direct_url=t4%3B%2Copen%20source%3B%2Cc0%3B%2Cs0%3B%3BOpen%20Source%3B%2Cc0%3B%3Bopen%20source%3B%2Cc0%3B%3BOpen%20source%3B%2Cc0 https://books.google.com/ngrams/graph?content=open+source&ca...
glomph 12y ago
The metaphor is widely used in lots of other areas though. Education, Science, Research, Business and so on.

peatmoss 12y ago

As primarily an R / stats user, the progress Julia has been making in terms of libraries has been staggering. I have just enough historic experience with Python to be similarly impressed if I place myself in that mindset. I recently attended a Software Carpentry instructor training session and witnessed a breakout between social scientists using R and natural sciences people using Python. Julia will have "made it" when orgs like SWC can start reasonably offering Julia sessions that draw both communities.

timClicks 12y ago
That's an interesting statement, because Software Carpentry intentionally positions itself as a follower rather than a leader in technology. That is, Python & R workshops are given because of the wide adoption of Python & R. Julia workshops would need to follow demand.
- peatmoss 12y ago
  Agreed, that was my point if it wasn't clear. At the point where there are enough people from natural sciences and social sciences looking to learn Julia at an introductory SWC level, Julia will have made it.

transfire 12y ago

I've written a few small Julia programs and have found it to be a very productive language, in most respects. Unfortunately its module system is horrid. If you look at almost all Julia programs, they are essentially one big file. Any secondary files just get included into a main file wholeclothe. It's like working with archaic PHP all over again.

cschmidt 12y ago
Don't people used Module in practice? http://julia.readthedocs.org/en/latest/manual/modules/?highlight=module http://julia.readthedocs.org/en/latest/manual/modules/?highl... I haven't read up on Julia yet, so I'm curious.
- sgt101 12y ago
  I've only played, but a few hours into it I wanted to partition up my project into separate places in order to separate concerns . I found modules to be a good way to do that once I realised that it's Pascal/modular 2 / ada again and not objects. I think that the problem may be that people are tripped up by the separation of type and modularity when they are used to that being coupled in c++ and java.
Fede_V 12y ago
I agree with this. I think in large part this is because the developers decided to try as much as possible to 'decouple' the concept of module from the filesystem. I think this is clever in the abstract, but in practice, it's a lot of headache for very, very little gain and a lot of added confusion. The main Julia team is a bunch of incredibly smart and dedicated guys who know a lot more about designing language than I do, but I think this was a case of trying to be too clever for your own good.
- CyberDildonics 12y ago
  So what are the module systems you like and what do you like about them?

msravi 12y ago

I've been using Julia 0.3 for algorithmic trading in production for about 3 months now, and the language is truly, astoundingly productive. The only thing lacking for me at this point is true multithreading support (so I can fork off a python thread from within julia) - I currently get around this by using a socket interface to pass messages to python. Edit: For anyone who's interested, an indicative problem is this: http://stackoverflow.com/questions/21113030/async-thread-deadlock-in-julia-when-used-with-pycall http://stackoverflow.com/questions/21113030/async-thread-dea...

tlmr 12y ago
How would you compare this to using python with an easy JIT like numba?
- msravi 12y ago
  I haven't used numba or benchmarked it, so I'm not really qualified to comment. I do however vastly prefer the MATLAB-like syntax of julia to that of python.
- Fede_V 12y ago
  I actually both in terms of speed, and in terms of writing code they are remarkably similar approaches. numba forces you into certain ways of writing code to achieve significant speed ups, but usually that's not too restrictive. If you are used to writing numerical python code, you'll feel write at home in Julia - especially with the amazing PyCall module.
- curiouslearn 12y ago
  Whenever I have tried to use Numba, I find it is slower than standard python. OTOH, Julia just works. Gives me the same performance as C.
ViralBShah 12y ago
There is a threads branch and we expect to have support in 0.4.

x43b 12y ago

Any recommendations for books for a Matlab user looking to use Julia? Thanks

one-more-minute 12y ago
There's not much in the way of books, but a few will be coming out soon, I believe. "Learning Julia" by Leah Hanson, for example. See also [1] for more learning resources. [1]: http://julialang.org/learning/ http://julialang.org/learning/
srparish 12y ago
Not a book, but scipy had a two-part introduction to Julia talk that's worth watching: part 1: https://www.youtube.com/watch?v=vWkgEddb4-A https://www.youtube.com/watch?v=vWkgEddb4-A part 2: https://www.youtube.com/watch?v=I3JH5Bg46yU https://www.youtube.com/watch?v=I3JH5Bg46yU

Aleman360 12y ago

I've been giving Julia a shot lately because I really like MATLAB syntax and IJulia is awesome for analysis reports. There are a few warts though: 1. I really don't like how the entire standard library is included in the default namespace. I know this is how MATLAB does it, but my function names are constantly colliding with built-in ones. 2. The documentation is hard to navigate. I'm used to MSDN. Compared to that, providing huge pages of functions with a sentence or two of prose and no examples is not that helpful. 3. Plotting tools need work. Gadfly produces beautiful output but interactivity is weak (you can only zoom in on Y-axis) and it chokes when you tell it to plot a lot of data. It should downsample based on display resolution... I should be able to tell it to graph 100,000 points without crashing my browser.

thorwwaay 12y ago
> but my function names are constantly colliding with built-in ones I'm sure you know this, but the casual drive-by reader may not. Functions with the same name aren't a collision. The parameter types of your function and a builtin of the same name all have to match in order for there to be a collision. In julia "module" introduces a namespace: http://julia.readthedocs.org/en/latest/manual/modules/ http://julia.readthedocs.org/en/latest/manual/modules/
StefanKarpinski 12y ago
Glad you're enjoying Julia :-) > I really don't like how the entire standard library is included in the default namespace. I know this is how MATLAB does it, but my function names are constantly colliding with built-in ones. This may not be obvious, but if you don't use a name from the standard library, then you are free to use it yourself since bindings imported by `using` are done lazily. For example: julia> e = "some message" "some message" julia> Base.e e = 2.7182818284590... julia> println(e) some message julia> Base.e e = 2.7182818284590... On the other hand, if you don't assign to `e` then you can use the built-in definition of `e` like so (in a new session): julia> e e = 2.7182818284590... If you want to use both your own `e` and `Base.e` then you do need to qualify `Base.e` – but this is exactly the same in any other language – if you use the same name from two different namespaces, you have to qualify them. Making the standard library namespace smaller or more granular wouldn't help in any way. > The documentation is hard to navigate. I'm used to MSDN. Compared to that, providing huge pages of functions with a sentence or two of prose and no examples is not that helpful. I definitely agree that our documentation isn't up to the standards of Microsoft. Pull requests are always welcomed and are a lovely way to give back. Over time the documentation will get better – I suspect we're already doing fairly well given the age of the language. > Plotting tools need work. Gadfly produces beautiful output but interactivity is weak (you can only zoom in on Y-axis) and it chokes when you tell it to plot a lot of data. It should downsample based on display resolution... I should be able to tell it to graph 100,000 points without crashing my browser. I'm curious what plotting system does actually let you plot 100,000 points without choking. Automatic downsampling would certainly be a nice advanced feature, but the mature plotting systems I'm familiar with do not do this. For example, neither R nor matplotlib do. Is there some Microsoft plotting library that does?

antman 12y ago

Calling julia functions from python would be nice

ihnorton 12y ago
All of the basic pieces are available here: https://github.com/JuliaLang/pyjulia https://github.com/JuliaLang/pyjulia