6 ms·

If you were trying to motivate someone embedded in Python / scipy / pandas / etc to move to Julia at this time , what would you say?

eigenspace 6y ago
I think this post: https://julialang.org/blog/2012/02/why-we-created-julia/ https://julialang.org/blog/2012/02/why-we-created-julia/ presents the aims of the Julia project at a high level quite well. That post was partly aspirational at the time, and partly already achieved. Now, 8 years later, I'd say essentially everything there has been achieved. The most important difference between Julia now and Julia then is the gigantic, vibrant and interacting ecosystem that's sprung up (though share are other important technical differences too!) _______________________________________________ Here's the pitch I'd give: Julia's value proposition is mostly geared towards people trying to do things that are just too difficult, awkward or slow in other languages. It's not just about speed, dynamism and friendly syntax. It's also about solving the expression problem and providing unprecedented composability. So while it may not necessarily have great appeal to 'end users' (who are people just calling numpy/scipy/pandas/whatever functions the way they were meant to be called by the library writers), it does appeal to people who make the sort of things end users want so the community is currently swelling with very talented people making state of the art research libraries. The language is designed for 'composability', so that it's very easy to combine packages together in ways the authors didn't necessarily intend or forsee. This approach is really starting to bear fruit. Our package ecosystem has some real, state of the art stuff not easily available in other languages, and it's often built in less time and with less resources. For 'end users', whether or not Julia makes sense for them really just depends on if the things they need are already available. Another huge advantage is that even if you're not some super researcher developer, Julia substantially lowers the bar for mere mortals such as myself to do package development. The composability I mentioned above means that I can stand on the shoulders of giants as I implement my own stuff, plus the fact that the majority of libraries are written in pure julia means that I can easily "peek under the hood" to figure out how they work and learn from them. This is really special and can't be replicated in python, because 'peeking under the hood' invariably means reading very specialized C code, rather than Python code. I think Julia will someday eat Python for scientific computing, but it may be a while before that becomes obvious to Python users. In the meantime, we have an excellent package for calling and running Python code from withing Julia, called PyCall.jl [1], so that can make the burden of switching even easier. [1] https://github.com/JuliaPy/PyCall.jl https://github.com/JuliaPy/PyCall.jl
- usgroup 6y ago
  Thank you for the detailed answer. I guess I should just give it a proper try by converting an old project into Julia. So here is an example and my worries ... My project uses GDAL, PIL, SentinelSat libraries to collect Sentinel 1 imagery and process it, pandas, numpy, scipy extensively. Would I find a bunch of those modules missing? I assume so ... I guess then I might be better of to trying a new project in Julia. Something not too easy but not too hard.
  - eigenspace 6y ago
    I don't know much about geospatial stuff, but I do know there are people working on that in the community. I'd be rather surprised if you couldn't find Julia packages that fill those needs for you. Worst case scenario, you might need to do the GDAL, PIL and SentinelSat stuff with PyCall and then the Pandas, Numpy and Scipy stuff should all be quite well supported from the Julia side. I can ask around to see if there's anyone who can answer this better though.
  - amkkma 6y ago
    > pandas, numpy, scipy Julia more than covers all the functionality of those packages. >PIL https://github.com/JuliaImages/Images.jl https://github.com/JuliaImages/Images.jl > GDAL https://github.com/yeesian/ArchGDAL.jl https://github.com/yeesian/ArchGDAL.jl https://github.com/JuliaGeo https://github.com/JuliaGeo >SentinelSat I'd use PyCall.jl for that. It's very easy and does automatic conversion of types etc
    - ViralBShah 6y ago
      ArchGDAL.jl is great. We just added it to Circuitscape.jl too - a conservation ecology package.
- marmaduke 6y ago
  This is a good pitch and I think composability is the major selling point for programming in the large. Time will tell though if the ecosystem doesn’t fragment as different approches or competing packages appear, for political if not technical reasons. The Torch vs TensorFlow choice could have been in Julia. > I think Julia will someday eat Python for scientific computing, but it may be a while before that becomes obvious to Python users. I can’t say I understand this rather pervasive attitude. If you have well tested code in a language which you can easily interface with, why bother rewriting it except for reasons of performance or compositionality? Or perhaps you mean that we’ll all be writing Julia even if some algorithms still sit in Python?
sgillen 6y ago
The biggest value prop for me is when you need something done fast that’s not already in scipy etc. In python you will need to write a C extension or fumble with numba / cython / whatever, in Julia you just write it. I also think writing out math is much more concise in Julia, no more np.whatever(np.sin(x.detach()) + cos(np.sqrt(y)).mean().
- usgroup 6y ago
  It seems about the right time to give it a go. How is the package ecosystem in your experience? Can I expect fairly decent support for numerical optimisation stuff , graphing? Munging xml/Json/etc? Will I find that I have to write my own oauth2 client library to call API? Etc
  - amkkma 6y ago
    >numerical optimisation stuff Yes, far exceeding python actually. - https://github.com/jump-dev/JuMP.jl https://github.com/jump-dev/JuMP.jl - https://github.com/JuliaNLSolvers https://github.com/JuliaNLSolvers Are just some. > graphing Excellent support. - https://github.com/JuliaPlots https://github.com/JuliaPlots - https://github.com/JuliaPlots/Makie.jl https://github.com/JuliaPlots/Makie.jl >Munging xml/Json etc - https://github.com/quinnj/JSON3.jl https://github.com/quinnj/JSON3.jl - https://github.com/JuliaData https://github.com/JuliaData - https://github.com/Algocircle/Cascadia.jl https://github.com/Algocircle/Cascadia.jl - https://github.com/queryverse/Query.jl https://github.com/queryverse/Query.jl >oauth2 etc - https://github.com/JuliaCloud https://github.com/JuliaCloud - https://github.com/JuliaWeb/HTTP.jl https://github.com/JuliaWeb/HTTP.jl for anything not available yet, you can call it from python using PyCall.jjl
marmaduke 6y ago
I’m also looking at this now after more than a decade in Python. I don’t think “move” is the right word but rather “coexist”. You can call back and forth between Python and Julia, so why pretend you have to choose? For example, matplotlib is one of the more feature complete plotting backends for Plots.jl. The question I am currently researching is not what Julia is good at but what are the pitfalls and drawbacks of particular technical choices. I know them by heart now for Python, and my team relies on knowing what doesn’t work well. For Julia it’s an unknown unknown. Understandably all the blogs are about why Julia is great so it makes seems to go slowly and coexist instead of moving.
- sgillen 6y ago
  >> I know them by heart now for Python... Mind elaborating?
lambdatronics 6y ago
I got fed up with Python when I found out that xarray couldn't support unitful quantities. In Julia, it just works. In Python, it turns into a massive project due to duck-typing & single-dispatch. It's been 5 years and xarray still doesn't have units, although it's getting close. Julia's equivalent to xarray: https://github.com/JuliaArrays/AxisArrays.jl https://github.com/JuliaArrays/AxisArrays.jl Original issue explaining the problem: https://github.com/pydata/xarray/issues/525 https://github.com/pydata/xarray/issues/525 Latest issue detailing the work: https://github.com/pydata/xarray/issues/3594 https://github.com/pydata/xarray/issues/3594

eigenspace 6y ago

I think this post: https://julialang.org/blog/2012/02/why-we-created-julia/ https://julialang.org/blog/2012/02/why-we-created-julia/ presents the aims of the Julia project at a high level quite well. That post was partly aspirational at the time, and partly already achieved. Now, 8 years later, I'd say essentially everything there has been achieved. The most important difference between Julia now and Julia then is the gigantic, vibrant and interacting ecosystem that's sprung up (though share are other important technical differences too!) _______________________________________________ Here's the pitch I'd give: Julia's value proposition is mostly geared towards people trying to do things that are just too difficult, awkward or slow in other languages. It's not just about speed, dynamism and friendly syntax. It's also about solving the expression problem and providing unprecedented composability. So while it may not necessarily have great appeal to 'end users' (who are people just calling numpy/scipy/pandas/whatever functions the way they were meant to be called by the library writers), it does appeal to people who make the sort of things end users want so the community is currently swelling with very talented people making state of the art research libraries. The language is designed for 'composability', so that it's very easy to combine packages together in ways the authors didn't necessarily intend or forsee. This approach is really starting to bear fruit. Our package ecosystem has some real, state of the art stuff not easily available in other languages, and it's often built in less time and with less resources. For 'end users', whether or not Julia makes sense for them really just depends on if the things they need are already available. Another huge advantage is that even if you're not some super researcher developer, Julia substantially lowers the bar for mere mortals such as myself to do package development. The composability I mentioned above means that I can stand on the shoulders of giants as I implement my own stuff, plus the fact that the majority of libraries are written in pure julia means that I can easily "peek under the hood" to figure out how they work and learn from them. This is really special and can't be replicated in python, because 'peeking under the hood' invariably means reading very specialized C code, rather than Python code. I think Julia will someday eat Python for scientific computing, but it may be a while before that becomes obvious to Python users. In the meantime, we have an excellent package for calling and running Python code from withing Julia, called PyCall.jl [1], so that can make the burden of switching even easier. [1] https://github.com/JuliaPy/PyCall.jl https://github.com/JuliaPy/PyCall.jl

usgroup 6y ago
Thank you for the detailed answer. I guess I should just give it a proper try by converting an old project into Julia. So here is an example and my worries ... My project uses GDAL, PIL, SentinelSat libraries to collect Sentinel 1 imagery and process it, pandas, numpy, scipy extensively. Would I find a bunch of those modules missing? I assume so ... I guess then I might be better of to trying a new project in Julia. Something not too easy but not too hard.
- eigenspace 6y ago
  I don't know much about geospatial stuff, but I do know there are people working on that in the community. I'd be rather surprised if you couldn't find Julia packages that fill those needs for you. Worst case scenario, you might need to do the GDAL, PIL and SentinelSat stuff with PyCall and then the Pandas, Numpy and Scipy stuff should all be quite well supported from the Julia side. I can ask around to see if there's anyone who can answer this better though.
- amkkma 6y ago
  > pandas, numpy, scipy Julia more than covers all the functionality of those packages. >PIL https://github.com/JuliaImages/Images.jl https://github.com/JuliaImages/Images.jl > GDAL https://github.com/yeesian/ArchGDAL.jl https://github.com/yeesian/ArchGDAL.jl https://github.com/JuliaGeo https://github.com/JuliaGeo >SentinelSat I'd use PyCall.jl for that. It's very easy and does automatic conversion of types etc
  - ViralBShah 6y ago
    ArchGDAL.jl is great. We just added it to Circuitscape.jl too - a conservation ecology package.
marmaduke 6y ago
This is a good pitch and I think composability is the major selling point for programming in the large. Time will tell though if the ecosystem doesn’t fragment as different approches or competing packages appear, for political if not technical reasons. The Torch vs TensorFlow choice could have been in Julia. > I think Julia will someday eat Python for scientific computing, but it may be a while before that becomes obvious to Python users. I can’t say I understand this rather pervasive attitude. If you have well tested code in a language which you can easily interface with, why bother rewriting it except for reasons of performance or compositionality? Or perhaps you mean that we’ll all be writing Julia even if some algorithms still sit in Python?

sgillen 6y ago

The biggest value prop for me is when you need something done fast that’s not already in scipy etc. In python you will need to write a C extension or fumble with numba / cython / whatever, in Julia you just write it. I also think writing out math is much more concise in Julia, no more np.whatever(np.sin(x.detach()) + cos(np.sqrt(y)).mean().

usgroup 6y ago
It seems about the right time to give it a go. How is the package ecosystem in your experience? Can I expect fairly decent support for numerical optimisation stuff , graphing? Munging xml/Json/etc? Will I find that I have to write my own oauth2 client library to call API? Etc
- amkkma 6y ago
  >numerical optimisation stuff Yes, far exceeding python actually. - https://github.com/jump-dev/JuMP.jl https://github.com/jump-dev/JuMP.jl - https://github.com/JuliaNLSolvers https://github.com/JuliaNLSolvers Are just some. > graphing Excellent support. - https://github.com/JuliaPlots https://github.com/JuliaPlots - https://github.com/JuliaPlots/Makie.jl https://github.com/JuliaPlots/Makie.jl >Munging xml/Json etc - https://github.com/quinnj/JSON3.jl https://github.com/quinnj/JSON3.jl - https://github.com/JuliaData https://github.com/JuliaData - https://github.com/Algocircle/Cascadia.jl https://github.com/Algocircle/Cascadia.jl - https://github.com/queryverse/Query.jl https://github.com/queryverse/Query.jl >oauth2 etc - https://github.com/JuliaCloud https://github.com/JuliaCloud - https://github.com/JuliaWeb/HTTP.jl https://github.com/JuliaWeb/HTTP.jl for anything not available yet, you can call it from python using PyCall.jjl

marmaduke 6y ago

I’m also looking at this now after more than a decade in Python. I don’t think “move” is the right word but rather “coexist”. You can call back and forth between Python and Julia, so why pretend you have to choose? For example, matplotlib is one of the more feature complete plotting backends for Plots.jl. The question I am currently researching is not what Julia is good at but what are the pitfalls and drawbacks of particular technical choices. I know them by heart now for Python, and my team relies on knowing what doesn’t work well. For Julia it’s an unknown unknown. Understandably all the blogs are about why Julia is great so it makes seems to go slowly and coexist instead of moving.

sgillen 6y ago
>> I know them by heart now for Python... Mind elaborating?

lambdatronics 6y ago

I got fed up with Python when I found out that xarray couldn't support unitful quantities. In Julia, it just works. In Python, it turns into a massive project due to duck-typing & single-dispatch. It's been 5 years and xarray still doesn't have units, although it's getting close. Julia's equivalent to xarray: https://github.com/JuliaArrays/AxisArrays.jl https://github.com/JuliaArrays/AxisArrays.jl Original issue explaining the problem: https://github.com/pydata/xarray/issues/525 https://github.com/pydata/xarray/issues/525 Latest issue detailing the work: https://github.com/pydata/xarray/issues/3594 https://github.com/pydata/xarray/issues/3594