3 ms·

I don't think that's so convincing, actually. When I learned programming, Python was _just_ replacing Perl as the "common language" in bioinformatics. Until the

I don't think that's so convincing, actually. When I learned programming, Python was _just_ replacing Perl as the "common language" in bioinformatics. Until then, there were lots of people who claimed Perl would essentially never be dethroned because it had become the standard. The issue is that while _some_ users only experience the surface level of packages like PyTorch, Numpy, SciPy, Pandas, and so on, people who actually have to develop these packages must work with the C++ and C underneath. To the developers, the problem of Python's performance is front and center, and a real issue they face every day. So, the developer folks will make the switch first, as they realize it's much nicer to develop Julia packages in Julia than it is to develop Python packages in C. Over time, this will mean the development power switches to Julia - and as soon as these developers have build tools as good or better than existing tools, the majority of users will slowly migrate, too.

blackbear_ 5y ago
> So, the developer folks will make the switch first, as they realize it's much nicer to develop Julia packages in Julia than it is to develop Python packages in C. Not so fast. You should also consider that writing the core in C/C++ allows some of those packages to offer bindings for several languages other than python (for example, both tensorflow and torch have bindings for Go, .NET, Rust, Haskell, R, Java, JavaScript). This significantly expands the use-cases that these frameworks can cover without requiring a complete rewrite for every platform. For example, it is relatively easy to have data scientists train models with Python and deploy such models to mobile/edge devices and web apps.
- jakobnissen 5y ago
  That's true, but for most packages, that's not really an issue. I.e. there might be a tensorflow library, but there is no libnumpy or libpandas, as far as I know. As I also mention in the blog post, if you set out to create fundamental library software like FFTW, you should indeed use a static language, and Julia is no good.
  - awaythrowact 5y ago
    > there is no libnumpy or libpandas I think this is changing with the rise of Apache Arrow. See eg here, slide 38 and on: https://www.slideshare.net/wesm/pycon-colombia-2020-python-for-data-analysis-past-present-and-future https://www.slideshare.net/wesm/pycon-colombia-2020-python-f...
    - Kalanos 5y ago
      Arrow is just an intermediary
      - awaythrowact 5y ago
        What is your point? (I honestly do not understand.) Blackbear's comment was that writing libraries in C allows those libraries to be deployed broadly in many compute environments. Jakob's reply (as I understood it) was that outside of the big Deep Learning libraries, this has not really happened. There is no C implementation of Pandas that allows for redeployment in other non-python compute contexts. My point was that, with Arrow, this type of cross platform compatibility is coming to python dataframe libraries. You can prototype Dask code that runs on your laptop, then deploy it to a production Spark cluster, knowing the same Arrow engine is underpinning both. Or at least that's the vision. Obviously Arrow is still relatively young. But the point is, it's far from certain that the long-term global optimum for the ecosystem isn't sticking with "all libraries are written in C".
      - Kalanos 5y ago
        In response to "rise of," I too was excited about Arrow until I played with it and realized it didn't even provide a shape attribute. Anyways, people shouldn't be dependent on a low level lang like C to write fast code.
        awaythrowact 5y ago
        Fair. I agree Arrow is still more of a vision than anything else. > it didn't even provide a shape attribute I suspect this has to do with the project's focus. I think they aspire to be a back-end to DataFrame libraries, which are generally 2d. I think they (correctly) are ceding the "n-dimensional tensor computation" space to the current incumbents.
- oscardssmith 5y ago
  This is doable in Julia too. DifferentialEquations.jl has python and R bindings. I expect more to follow as Julia starts getting more best in class packages.
dochtman 5y ago
Writing Python packages in Rust with PyO3 is pretty awesome, though.
- awaythrowact 5y ago
  What are your favorite open source Python packages written in Rust?
  - amgoodman 5y ago
    tokenizers
  - ZeroCool2u 5y ago
    tokenizers is a great example. Some other, perhaps simpler ones, are orjson and tantivy.
  - dochtman 5y ago
    I mostly live in the Rust world, not the Python world, but I worked on two Python packages wrapping over my own Rust libraries and it's been pretty great. https://github.com/InstantDomain/instant-distance https://github.com/InstantDomain/instant-distance https://github.com/InstantDomain/instant-segment https://github.com/InstantDomain/instant-segment

blackbear_ 5y ago

> So, the developer folks will make the switch first, as they realize it's much nicer to develop Julia packages in Julia than it is to develop Python packages in C. Not so fast. You should also consider that writing the core in C/C++ allows some of those packages to offer bindings for several languages other than python (for example, both tensorflow and torch have bindings for Go, .NET, Rust, Haskell, R, Java, JavaScript). This significantly expands the use-cases that these frameworks can cover without requiring a complete rewrite for every platform. For example, it is relatively easy to have data scientists train models with Python and deploy such models to mobile/edge devices and web apps.

jakobnissen 5y ago
That's true, but for most packages, that's not really an issue. I.e. there might be a tensorflow library, but there is no libnumpy or libpandas, as far as I know. As I also mention in the blog post, if you set out to create fundamental library software like FFTW, you should indeed use a static language, and Julia is no good.
- awaythrowact 5y ago
  > there is no libnumpy or libpandas I think this is changing with the rise of Apache Arrow. See eg here, slide 38 and on: https://www.slideshare.net/wesm/pycon-colombia-2020-python-for-data-analysis-past-present-and-future https://www.slideshare.net/wesm/pycon-colombia-2020-python-f...
  - Kalanos 5y ago
    Arrow is just an intermediary
    - awaythrowact 5y ago
      What is your point? (I honestly do not understand.) Blackbear's comment was that writing libraries in C allows those libraries to be deployed broadly in many compute environments. Jakob's reply (as I understood it) was that outside of the big Deep Learning libraries, this has not really happened. There is no C implementation of Pandas that allows for redeployment in other non-python compute contexts. My point was that, with Arrow, this type of cross platform compatibility is coming to python dataframe libraries. You can prototype Dask code that runs on your laptop, then deploy it to a production Spark cluster, knowing the same Arrow engine is underpinning both. Or at least that's the vision. Obviously Arrow is still relatively young. But the point is, it's far from certain that the long-term global optimum for the ecosystem isn't sticking with "all libraries are written in C".
    - Kalanos 5y ago
      In response to "rise of," I too was excited about Arrow until I played with it and realized it didn't even provide a shape attribute. Anyways, people shouldn't be dependent on a low level lang like C to write fast code.
      - awaythrowact 5y ago
        Fair. I agree Arrow is still more of a vision than anything else. > it didn't even provide a shape attribute I suspect this has to do with the project's focus. I think they aspire to be a back-end to DataFrame libraries, which are generally 2d. I think they (correctly) are ceding the "n-dimensional tensor computation" space to the current incumbents.
oscardssmith 5y ago
This is doable in Julia too. DifferentialEquations.jl has python and R bindings. I expect more to follow as Julia starts getting more best in class packages.

dochtman 5y ago

Writing Python packages in Rust with PyO3 is pretty awesome, though.

awaythrowact 5y ago
What are your favorite open source Python packages written in Rust?
- amgoodman 5y ago
  tokenizers
- ZeroCool2u 5y ago
  tokenizers is a great example. Some other, perhaps simpler ones, are orjson and tantivy.
- dochtman 5y ago
  I mostly live in the Rust world, not the Python world, but I worked on two Python packages wrapping over my own Rust libraries and it's been pretty great. https://github.com/InstantDomain/instant-distance https://github.com/InstantDomain/instant-distance https://github.com/InstantDomain/instant-segment https://github.com/InstantDomain/instant-segment