6 ms·

Is there also something about timing? Given that R and Python are so mature and active, this really leaves not much space for Julia. Yes, Julia is faster than

Is there also something about timing? Given that R and Python are so mature and active, this really leaves not much space for Julia. Yes, Julia is faster than R and Python (just the language, not packages), but R and Python can easily integrate with C/C++ and many R and Python packages are in fact internally C/C++. This really leaves not much advantage for Julia against R and Python. If Julia had appeared earlier, it may have had better position in the data analytics programing ecosystem, and gain more attention and opportunity to grow. Now, the whole data industry is booming, but people would rather contribute to the matured R and Python for real work, instead of waiting for Julia to grow.

oscardssmith 5y ago
I think Julia has a pretty good path forward since it is often faster than C++ (especially once you consider language interop overhead), and it is much more pleasant to write than c++. Since Julia and most of the packages are written in Julia (for the most part), Julia makes it really easy for users to become developers. Python and R have much bigger barriers to entry since users will usually hit a binary blob, which makes it much harder to interact with library implimentations.
jakobnissen 5y ago
I don't think that's so convincing, actually. When I learned programming, Python was _just_ replacing Perl as the "common language" in bioinformatics. Until then, there were lots of people who claimed Perl would essentially never be dethroned because it had become the standard. The issue is that while _some_ users only experience the surface level of packages like PyTorch, Numpy, SciPy, Pandas, and so on, people who actually have to develop these packages must work with the C++ and C underneath. To the developers, the problem of Python's performance is front and center, and a real issue they face every day. So, the developer folks will make the switch first, as they realize it's much nicer to develop Julia packages in Julia than it is to develop Python packages in C. Over time, this will mean the development power switches to Julia - and as soon as these developers have build tools as good or better than existing tools, the majority of users will slowly migrate, too.
- blackbear_ 5y ago
  > So, the developer folks will make the switch first, as they realize it's much nicer to develop Julia packages in Julia than it is to develop Python packages in C. Not so fast. You should also consider that writing the core in C/C++ allows some of those packages to offer bindings for several languages other than python (for example, both tensorflow and torch have bindings for Go, .NET, Rust, Haskell, R, Java, JavaScript). This significantly expands the use-cases that these frameworks can cover without requiring a complete rewrite for every platform. For example, it is relatively easy to have data scientists train models with Python and deploy such models to mobile/edge devices and web apps.
  - jakobnissen 5y ago
    That's true, but for most packages, that's not really an issue. I.e. there might be a tensorflow library, but there is no libnumpy or libpandas, as far as I know. As I also mention in the blog post, if you set out to create fundamental library software like FFTW, you should indeed use a static language, and Julia is no good.
    - awaythrowact 5y ago
      > there is no libnumpy or libpandas I think this is changing with the rise of Apache Arrow. See eg here, slide 38 and on: https://www.slideshare.net/wesm/pycon-colombia-2020-python-for-data-analysis-past-present-and-future https://www.slideshare.net/wesm/pycon-colombia-2020-python-f...
      - Kalanos 5y ago
        Arrow is just an intermediary
        awaythrowact 5y ago
        What is your point? (I honestly do not understand.) Blackbear's comment was that writing libraries in C allows those libraries to be deployed broadly in many compute environments. Jakob's reply (as I understood it) was that outside of the big Deep Learning libraries, this has not really happened. There is no C implementation of Pandas that allows for redeployment in other non-python compute contexts. My point was that, with Arrow, this type of cross platform compatibility is coming to python dataframe libraries. You can prototype Dask code that runs on your laptop, then deploy it to a production Spark cluster, knowing the same Arrow engine is underpinning both. Or at least that's the vision. Obviously Arrow is still relatively young. But the point is, it's far from certain that the long-term global optimum for the ecosystem isn't sticking with "all libraries are written in C".
dnautics 5y ago
I dunno. I have written FFIs in many languages and by far julia was the easiest to call a c abi in. I would say if anything what holds people back in julia is a drive to RIIJ instead of calling the c abi (despite that julia's linalg basically does exactly that with BLAS).
- ska 5y ago
  Lua is pretty smooth in this respect.
ska 5y ago
> This really leaves not much advantage for Julia against R and Python. It's a bit of a chicken and egg problem. Really the argument is the problems with dual-language model will never go away, so Python & R face an ongoing cost disadvantage in the long run. However, they are also much more mature. So assuming previous assertion is correct, you need to get enough people interested in Julia to grow to the tipping point. I'm not really sure it will work, but can see the argument. After all technically superior platforms for this existed when python got started (heck, early python was next to useless for data work beyond file parsing) but that didn't dissuade it's use. The non-technical arguments made it a good-enough language and once that had momentum it's compelling. On the third hand, what julia really needs to attract right now is mostly library/package writers - that's a tiny fraction of the users of python, but basically steers "what's next". However, for library writers, how many people might use it can be a deciding factor on investing the time...
- awaythrowact 5y ago
  > for library writers, how many people might use it can be a deciding factor on investing the time Yes. Especially given that a high percentage of library development these days is being done by industry, not just academia. (At least in machine learning.) They're not writing packages for fun or for academic citations, they want users.
  - BadInformatics 5y ago
    Julia does appear to have a higher than usual proportion of package authors to total users, FWIW. I think it's both a blessing and a curse, because feedback around gaps in the ecosystem is sometimes brushed away with "write your own package". That is, because a large part of the community writes packages to scratch their own itch (I've yet to see one written for citations), they expect others to do the same (which may not be fair).
thebooktocome 5y ago
It should go without saying that Julia also has had a C FFI for years. And a Python FFI. And an R FFI. The C++ one was dodgy last I looked (which was several years ago; I'm sure it's better now). In practice I typically wrote explicit C wrappers, restoring genericity on either side of the wrapper. Faintly tedious, but nothing a couple macros couldn't solve.
- hobofan 5y ago
  Yes, Julia has them, but if you use them, it basically renders one of the promised advantages of Julia ("it does away with the two-language problem") moot, as you now have two languages anyway. If you have to languages anyway, why not use one of the more established languages? ---- I personally don't subscribe to the idea that the "two-language problem" is a problem that needs fixing. In the end your stack will have multiple languages anyway, unless you rewrite everything is Julia, which is utopian, so having one more isn't really a problem (and it might also be one that you are already using).
  - thebooktocome 5y ago
    > Yes, Julia is faster than R and Python (just the language, not packages), but R and Python can easily integrate with C/C++ and many R and Python packages are in fact internally C/C++. This is what I was responding to. The grandparent argued that Julia had less library support than R and Python because, so they implied, those languages could interop with C/C++ in a way Julia can't. The two language problem refers to code in user space, anyway. It's not like Rust, where there are so many wheel-reinventing projects in progress. If you peruse the main Julia package registry, and you'll see plenty of library-wrapping modules.
ummonk 5y ago
Also the slowness of Python is overstated - for many of the niches Julia is targeting, Python is plenty fast if you use NumPy / CuPy / SciPi. Yes, it's easier to write the equivalent for Julia using native Julia code, but it's not clear to me that the average user will care.
- ska 5y ago
  > Python is plenty fast if you use NumPy / CuPy / SciPi. I think this misses the point. You are basically saying that python is fast enough in cases that someone has written already written a fast package in not-python (plus bindings), whereas the value proposition for julia is basically: writing such packages in the future will be lots better. Obviously the average user doesn't care, but if enough whiz-bang stuff starts to come out in Julia but not python, they'll move.
  - ummonk 5y ago
    I think it will be interesting if enough clever new whiz-bang stuff comes out particularly based on Julia's metaprogramming powers. However, when talking about the "two language problem", the main issue traditionally was prototyping computational code in Python / Matlab / whatever and then having to drop into Fortran / C / Cuda for optimizing it. That's pretty much a solved problem in Python now - thanks to the well developed ecosystem, you don't really have to leave Python to write fast numerical code.
    - klaff 5y ago
      The first problems I came to Julia to solve were ODE simulations in which it's just as critical that the stuff I write and rewrite over and over again (the actual differential equations of my model and associated simulated controller callbacks) runs fast as it is that the pre-written stuff of the library (the solvers or integrators) runs fast. To the best of my knowledge I can't do that in python. This example (DifferentialEquations.jl in particular and SciML.ai in general) is one of the places where Julia leads the pack. When I first used Julia it felt rather strange in a lot of ways (I had used python for about 20 years), but after 2 years when I go back to python (and of course there are places where it makes more sense to use python) it feels primitive as there are a lot of very nice features of Julia that I suddenly miss.

oscardssmith 5y ago

I think Julia has a pretty good path forward since it is often faster than C++ (especially once you consider language interop overhead), and it is much more pleasant to write than c++. Since Julia and most of the packages are written in Julia (for the most part), Julia makes it really easy for users to become developers. Python and R have much bigger barriers to entry since users will usually hit a binary blob, which makes it much harder to interact with library implimentations.

jakobnissen 5y ago

I don't think that's so convincing, actually. When I learned programming, Python was _just_ replacing Perl as the "common language" in bioinformatics. Until then, there were lots of people who claimed Perl would essentially never be dethroned because it had become the standard. The issue is that while _some_ users only experience the surface level of packages like PyTorch, Numpy, SciPy, Pandas, and so on, people who actually have to develop these packages must work with the C++ and C underneath. To the developers, the problem of Python's performance is front and center, and a real issue they face every day. So, the developer folks will make the switch first, as they realize it's much nicer to develop Julia packages in Julia than it is to develop Python packages in C. Over time, this will mean the development power switches to Julia - and as soon as these developers have build tools as good or better than existing tools, the majority of users will slowly migrate, too.

blackbear_ 5y ago
> So, the developer folks will make the switch first, as they realize it's much nicer to develop Julia packages in Julia than it is to develop Python packages in C. Not so fast. You should also consider that writing the core in C/C++ allows some of those packages to offer bindings for several languages other than python (for example, both tensorflow and torch have bindings for Go, .NET, Rust, Haskell, R, Java, JavaScript). This significantly expands the use-cases that these frameworks can cover without requiring a complete rewrite for every platform. For example, it is relatively easy to have data scientists train models with Python and deploy such models to mobile/edge devices and web apps.
- jakobnissen 5y ago
  That's true, but for most packages, that's not really an issue. I.e. there might be a tensorflow library, but there is no libnumpy or libpandas, as far as I know. As I also mention in the blog post, if you set out to create fundamental library software like FFTW, you should indeed use a static language, and Julia is no good.
  - awaythrowact 5y ago
    > there is no libnumpy or libpandas I think this is changing with the rise of Apache Arrow. See eg here, slide 38 and on: https://www.slideshare.net/wesm/pycon-colombia-2020-python-for-data-analysis-past-present-and-future https://www.slideshare.net/wesm/pycon-colombia-2020-python-f...
    - Kalanos 5y ago
      Arrow is just an intermediary
      - awaythrowact 5y ago
        What is your point? (I honestly do not understand.) Blackbear's comment was that writing libraries in C allows those libraries to be deployed broadly in many compute environments. Jakob's reply (as I understood it) was that outside of the big Deep Learning libraries, this has not really happened. There is no C implementation of Pandas that allows for redeployment in other non-python compute contexts. My point was that, with Arrow, this type of cross platform compatibility is coming to python dataframe libraries. You can prototype Dask code that runs on your laptop, then deploy it to a production Spark cluster, knowing the same Arrow engine is underpinning both. Or at least that's the vision. Obviously Arrow is still relatively young. But the point is, it's far from certain that the long-term global optimum for the ecosystem isn't sticking with "all libraries are written in C".

dnautics 5y ago

I dunno. I have written FFIs in many languages and by far julia was the easiest to call a c abi in. I would say if anything what holds people back in julia is a drive to RIIJ instead of calling the c abi (despite that julia's linalg basically does exactly that with BLAS).

ska 5y ago
Lua is pretty smooth in this respect.

ska 5y ago

> This really leaves not much advantage for Julia against R and Python. It's a bit of a chicken and egg problem. Really the argument is the problems with dual-language model will never go away, so Python & R face an ongoing cost disadvantage in the long run. However, they are also much more mature. So assuming previous assertion is correct, you need to get enough people interested in Julia to grow to the tipping point. I'm not really sure it will work, but can see the argument. After all technically superior platforms for this existed when python got started (heck, early python was next to useless for data work beyond file parsing) but that didn't dissuade it's use. The non-technical arguments made it a good-enough language and once that had momentum it's compelling. On the third hand, what julia really needs to attract right now is mostly library/package writers - that's a tiny fraction of the users of python, but basically steers "what's next". However, for library writers, how many people might use it can be a deciding factor on investing the time...

awaythrowact 5y ago
> for library writers, how many people might use it can be a deciding factor on investing the time Yes. Especially given that a high percentage of library development these days is being done by industry, not just academia. (At least in machine learning.) They're not writing packages for fun or for academic citations, they want users.
- BadInformatics 5y ago
  Julia does appear to have a higher than usual proportion of package authors to total users, FWIW. I think it's both a blessing and a curse, because feedback around gaps in the ecosystem is sometimes brushed away with "write your own package". That is, because a large part of the community writes packages to scratch their own itch (I've yet to see one written for citations), they expect others to do the same (which may not be fair).

thebooktocome 5y ago

It should go without saying that Julia also has had a C FFI for years. And a Python FFI. And an R FFI. The C++ one was dodgy last I looked (which was several years ago; I'm sure it's better now). In practice I typically wrote explicit C wrappers, restoring genericity on either side of the wrapper. Faintly tedious, but nothing a couple macros couldn't solve.

hobofan 5y ago
Yes, Julia has them, but if you use them, it basically renders one of the promised advantages of Julia ("it does away with the two-language problem") moot, as you now have two languages anyway. If you have to languages anyway, why not use one of the more established languages? ---- I personally don't subscribe to the idea that the "two-language problem" is a problem that needs fixing. In the end your stack will have multiple languages anyway, unless you rewrite everything is Julia, which is utopian, so having one more isn't really a problem (and it might also be one that you are already using).
- thebooktocome 5y ago
  > Yes, Julia is faster than R and Python (just the language, not packages), but R and Python can easily integrate with C/C++ and many R and Python packages are in fact internally C/C++. This is what I was responding to. The grandparent argued that Julia had less library support than R and Python because, so they implied, those languages could interop with C/C++ in a way Julia can't. The two language problem refers to code in user space, anyway. It's not like Rust, where there are so many wheel-reinventing projects in progress. If you peruse the main Julia package registry, and you'll see plenty of library-wrapping modules.

ummonk 5y ago

Also the slowness of Python is overstated - for many of the niches Julia is targeting, Python is plenty fast if you use NumPy / CuPy / SciPi. Yes, it's easier to write the equivalent for Julia using native Julia code, but it's not clear to me that the average user will care.

ska 5y ago
> Python is plenty fast if you use NumPy / CuPy / SciPi. I think this misses the point. You are basically saying that python is fast enough in cases that someone has written already written a fast package in not-python (plus bindings), whereas the value proposition for julia is basically: writing such packages in the future will be lots better. Obviously the average user doesn't care, but if enough whiz-bang stuff starts to come out in Julia but not python, they'll move.
- ummonk 5y ago
  I think it will be interesting if enough clever new whiz-bang stuff comes out particularly based on Julia's metaprogramming powers. However, when talking about the "two language problem", the main issue traditionally was prototyping computational code in Python / Matlab / whatever and then having to drop into Fortran / C / Cuda for optimizing it. That's pretty much a solved problem in Python now - thanks to the well developed ecosystem, you don't really have to leave Python to write fast numerical code.
  - klaff 5y ago
    The first problems I came to Julia to solve were ODE simulations in which it's just as critical that the stuff I write and rewrite over and over again (the actual differential equations of my model and associated simulated controller callbacks) runs fast as it is that the pre-written stuff of the library (the solvers or integrators) runs fast. To the best of my knowledge I can't do that in python. This example (DifferentialEquations.jl in particular and SciML.ai in general) is one of the places where Julia leads the pack. When I first used Julia it felt rather strange in a lot of ways (I had used python for about 20 years), but after 2 years when I go back to python (and of course there are places where it makes more sense to use python) it feels primitive as there are a lot of very nice features of Julia that I suddenly miss.