5 ms·

yep, it is the current state of things. However, it would be better to have all this huge ecosystem written in a single performant language. People trade percei

yep, it is the current state of things. However, it would be better to have all this huge ecosystem written in a single performant language. People trade perceivable simplicity for things that are more beneficial long term.

fnord123 7y ago
>However, it would be better to have all this huge ecosystem written in a single performant language. It's not the language, it's the linkage model. Python glues C components together and that's why experienced professionals don't blink when going all in on Python: the performance can be easily cranked up to C level.
- jacques_chester 7y ago
  Within the scope of a linked C library, this is true. But when your Python code is shuttling chunks of data between libraries it's less so. Pure Julia has the advantage that it can perform global optimisations that Python-wrapping-C can't.
  - mlthoughts2018 7y ago
    I disagree. In practice the things I can do between Cython, llvmlite and numba are very low effort, wide reach optimizations. Practically, I have never found a case when a feature of Julia would have made this much easier for me. As a result, the tradeoff of Julia continues to be 100% focused on the switching costs, infrastructure maturity, drop in equivalents of well worn Python libraries, etc.
- socialdemocrat 7y ago
  The problem is that you need C/C++ expertise to build packages, which creates a barriers between users of packages and makers of packages. This is a big selling point for Julia. One is seeing much faster package development on Julia than on Python because you use the same language on both ends. Package users are often able to contribute to the packages they use. Also the Python-Linkage has major limitations. If you create anything that requires the user to pass a function defined in Python to the C/C++ layer you take a performance hit. Think about solvers e.g. For machine learning that is a big deal. In Julia you can write your own scientific models and do automatic differentiation and training on them. If you do the same in Python you get a huge performance penalty. C/C++ cannot really help you.
  - NeverFade 7y ago
    > The problem is that you need C/C++ expertise to build packages, which creates a barriers between users of packages and makers of packages. The vast majority of Python packages are in pure Python and require no C/++ skills at all.
    - socialdemocrat 7y ago
      Sure, but the large packages which are the big selling point for scientific computing are for the most part C/C++ - NumPy - Pandas - TensorFlow etc. In contrast the most popular scientific packages in Julia are almost all pure Julia.
      - NeverFade 7y ago
        Yes, but these large packages represent a very small portion of Python packages. Also, they still include very large amounts of Python code. So a typical user who knows only Python can still contribute even to these packages, and certainly to the other 99.9% of pure Python packages. The barriers only exist if you want to contribute to the C++ core of NumPy or TensorFlow, and these parts are so complex and performance critical that even in Julia I'm sure only experts would be touching them.
        socialdemocrat 7y ago
        I am not convinced. It does not reflect the frustrations I have heard from Python developers who ended up switching to Julia. I don't have personal experience interfacing C/C++ code with Python but I spent a lot of time with Lua and C/C++ and Lua is considered far easier to integrate than Python. Yet looking back and those experiences, while I thought it was super easy and cool back then, it was fraught with all sorts of problems with impedance mismatch. Especially shuffling large chunks of data back and forth between the interfaces of two languages is hard. You cannot take an arbitrary Python data structure and push it into say NumPy. I've seen in practice how much problems that can cause at one of my previous companies. We made a C++ application with a Python interface. We got major performance issues because we shuffled data back and forth as NumPy arrays. It also limited what we could do with the data. Rather than being rich objects with associated functionality, you are stuck with blobs of NumPy data, which is really just dumb data with generic operations associated with them. With Julia I can create e.g. an RBG buffer, where ever element is an RGB value and register functions working specifically for this type of data. In Python you would be stuck with pretending your NumPy array of floats or ints represents an RBG buffer. But don't take it from me. Talk to people who have switched from Python to Julia and let them describe the benefits. I meet a lot of these people and they talk about how much simpler code they can write and performance gains they get. Sure if your Python packages fit your niche great at the moment, no need to switch. But too many people twist themselves sideways to continue with the technology they have determined is all they ever need. Python developers should know that too. Lots of Java, C++, Perl etc guy at some point resisted going over to Python with lots of lame excuses.
- tastyminerals 7y ago
  I would argue that because it could have been Lua in the first place. Fastest C ffi. Faster, simpler scripting language with natural 1 - indexed arrays like in fortran, matlab. It got some traction, yes, from ML community thanks to Torch but Python had better batteries already at that time.
- sixbrx 7y ago
  Doesn't that "just drop down to C" get complicated when you need to pass callback functions though? Users will want to write them in the original language for convenience and to use closures safely and easily. Wouldn't that kill your performance if the callback function is just Python again but called many times (e.g. an integrand for a numeric integration function, which integrand depends on captured runtime params). This is no problem in Julia, what are people doing cases like this in Python land?
  - fnord123 7y ago
    Passing callbacks is so rare that this is definitely not a reason to switch languages and stacks and tool chains.

fnord123 7y ago

>However, it would be better to have all this huge ecosystem written in a single performant language. It's not the language, it's the linkage model. Python glues C components together and that's why experienced professionals don't blink when going all in on Python: the performance can be easily cranked up to C level.

jacques_chester 7y ago
Within the scope of a linked C library, this is true. But when your Python code is shuttling chunks of data between libraries it's less so. Pure Julia has the advantage that it can perform global optimisations that Python-wrapping-C can't.
- mlthoughts2018 7y ago
  I disagree. In practice the things I can do between Cython, llvmlite and numba are very low effort, wide reach optimizations. Practically, I have never found a case when a feature of Julia would have made this much easier for me. As a result, the tradeoff of Julia continues to be 100% focused on the switching costs, infrastructure maturity, drop in equivalents of well worn Python libraries, etc.
socialdemocrat 7y ago
The problem is that you need C/C++ expertise to build packages, which creates a barriers between users of packages and makers of packages. This is a big selling point for Julia. One is seeing much faster package development on Julia than on Python because you use the same language on both ends. Package users are often able to contribute to the packages they use. Also the Python-Linkage has major limitations. If you create anything that requires the user to pass a function defined in Python to the C/C++ layer you take a performance hit. Think about solvers e.g. For machine learning that is a big deal. In Julia you can write your own scientific models and do automatic differentiation and training on them. If you do the same in Python you get a huge performance penalty. C/C++ cannot really help you.
- NeverFade 7y ago
  > The problem is that you need C/C++ expertise to build packages, which creates a barriers between users of packages and makers of packages. The vast majority of Python packages are in pure Python and require no C/++ skills at all.
  - socialdemocrat 7y ago
    Sure, but the large packages which are the big selling point for scientific computing are for the most part C/C++ - NumPy - Pandas - TensorFlow etc. In contrast the most popular scientific packages in Julia are almost all pure Julia.
    - NeverFade 7y ago
      Yes, but these large packages represent a very small portion of Python packages. Also, they still include very large amounts of Python code. So a typical user who knows only Python can still contribute even to these packages, and certainly to the other 99.9% of pure Python packages. The barriers only exist if you want to contribute to the C++ core of NumPy or TensorFlow, and these parts are so complex and performance critical that even in Julia I'm sure only experts would be touching them.
      - socialdemocrat 7y ago
        I am not convinced. It does not reflect the frustrations I have heard from Python developers who ended up switching to Julia. I don't have personal experience interfacing C/C++ code with Python but I spent a lot of time with Lua and C/C++ and Lua is considered far easier to integrate than Python. Yet looking back and those experiences, while I thought it was super easy and cool back then, it was fraught with all sorts of problems with impedance mismatch. Especially shuffling large chunks of data back and forth between the interfaces of two languages is hard. You cannot take an arbitrary Python data structure and push it into say NumPy. I've seen in practice how much problems that can cause at one of my previous companies. We made a C++ application with a Python interface. We got major performance issues because we shuffled data back and forth as NumPy arrays. It also limited what we could do with the data. Rather than being rich objects with associated functionality, you are stuck with blobs of NumPy data, which is really just dumb data with generic operations associated with them. With Julia I can create e.g. an RBG buffer, where ever element is an RGB value and register functions working specifically for this type of data. In Python you would be stuck with pretending your NumPy array of floats or ints represents an RBG buffer. But don't take it from me. Talk to people who have switched from Python to Julia and let them describe the benefits. I meet a lot of these people and they talk about how much simpler code they can write and performance gains they get. Sure if your Python packages fit your niche great at the moment, no need to switch. But too many people twist themselves sideways to continue with the technology they have determined is all they ever need. Python developers should know that too. Lots of Java, C++, Perl etc guy at some point resisted going over to Python with lots of lame excuses.
tastyminerals 7y ago
I would argue that because it could have been Lua in the first place. Fastest C ffi. Faster, simpler scripting language with natural 1 - indexed arrays like in fortran, matlab. It got some traction, yes, from ML community thanks to Torch but Python had better batteries already at that time.
sixbrx 7y ago
Doesn't that "just drop down to C" get complicated when you need to pass callback functions though? Users will want to write them in the original language for convenience and to use closures safely and easily. Wouldn't that kill your performance if the callback function is just Python again but called many times (e.g. an integrand for a numeric integration function, which integrand depends on captured runtime params). This is no problem in Julia, what are people doing cases like this in Python land?
- fnord123 7y ago
  Passing callbacks is so rare that this is definitely not a reason to switch languages and stacks and tool chains.