9 ms·

Maybe I'm a cranky old man, but I think there's immense value in the fact that Python's (scientific and non-scientific) ecosystem has matured and has become sta

Maybe I'm a cranky old man, but I think there's immense value in the fact that Python's (scientific and non-scientific) ecosystem has matured and has become standard over the last 1-2 decades. Starting again from the ground up in a shiny new language (even if the language is perhaps slightly better) is a waste of so much effort. Transitioning from Python 2 to 3 was already a nightmare. I know there's some interoperability between Julia and Python, but a lot of stuff is not interoperable, and there are annoying differences in array indexing etc. Why can't people just settle on some tech, let it mature, and enjoy reaching higher and higher in capability? Why do we have to always tear things down and start from scratch? This causes a lot of artificial bit rot, there's a never ending treadmill of just porting well-developed concepts and mature code to slightly changed representations. It's the busywork of tech. Does the scientific community also want to follow web dev in their insane churn of frameworks and libraries where last week's best practice is laughable ancient stone-age primitiveness today?

Datenstrom 6y ago
There is an immense value in most of the ecosystem converging on Python over the last decade and especially with leaving MATLAB behind in the dust. I don't think decade long transitions from old tech to new tech is a bad time scale or even close to that of the churn of web technologies. Julia offers enough of an improvement over Python to warrant a switch over the next 5-10 years and leave Python behind in the same way. I believe Julia offers the same level of improvement over Python as Python does over MATLAB. I also believe that it has drawn too much from MATLAB only for the sake of being familiar to that group and not the computer science and software engineering communities though such as the one based indexing you mentioned. Unfortunately that ship has already sailed and we will likely have to wait another 10 years for the next one to hopefully fix those problems.
- ScottPJones 6y ago
  Back 5.5 years ago, I used to complain about the 1 based indexing and the column-major structure of matrices in Julia (both like Fortran), however, those issues have been solved by OffsetArrays and PermutedDimsArrays, giving far more flexibility that is possible in most other languages. It's silly to keep bringing up the issue of one based indexing, when you can use any integer as the base, just like Fortran 90 (so you can index by -5..5, for example). For some things, 0 based does make things easier, sure, but you can do that easily in Julia (and more!)
  - Datenstrom 6y ago
    Yes it is possible to make those changes and even if it wasn't it does not detract from the impressive and useful innovations Julia has made. That is why I usually refer to them as cosmetic but that is possibly not the correct word either because it does not convey the implications of having that default beyond simply being unattractive. Cultural may be the correct term. Julia claims that it wants to solve the two language problem. To do that it must win both the scientific compute, HPC, data science, machine learning, etc. group and the CS, software engineering, dev ops group. I've shown Julia to quite a few members of the latter and they always love the idea and technical capabilities of Julia but visibly cringe when they find out it uses one based indexing. They also almost universally find the syntax ugly. Unfortunately the available packages to change that also does not help because most people will never change the defaults and IMHO it would be bad to do so. When writing Python you follow the PEP-8 style guide, it is just best practice to follow the languages conventions so everyone is on the same page. This is the same problem with C++ trying to add all these new features and a solid package manager like Rust has, it will never fly, people just won't use it everywhere, it isn't going to be in the standard library, It isn't the default and it isn't part of the culture.
ddragon 6y ago
By that logic people should have kept using Fortran instead of Python (which was the new guy a couple decades ago). But Fortran will never be Python, and Python will never be Fortran, no matter how much maturity/improvements. Because maturity means getting close to it's potential (local maxima), without the ability to drastically move to a hypothetical global maxima (without breaking all it's legacy, like a more drastic Python 2 -> 3, or adding things indefinitely until you get effectively many languages/dialects in one like C++). New languages on the other side don't have as much baggage (and can start with increased knowledge from previous attempts) so they are able to freely potentially target better local maxima that old languages cannot anymore. Plus in this particular case, Python was not really designed to target the scientific domain (nor ML which did not even exist in the way we use nowadays), it was repeatedly retrofitted with possibly a lot more effort than would be necessary to create something better from scratch.
- bonoboTP 6y ago
  > By that logic people should have kept using Fortran instead of Python There were very real pain points with Fortran, C++, MATLAB etc. compared to Python. Now this is hard for me to argue in a few sentences as it's a distillation of having worked with many languages, but I feel that Python strikes a great balance for productivity. It feels like pouring your thoughts directly onto the screen, with no need to fight the language. The syntax is very clear, concise, "Pythonic" is a word of praise. As I said, this kind of debate is rarely productive and devolves to religious arguments, but that's my stance. The only issue with Python is speed, but if you use Numpy properly, even that isn't a big issue. > and can start with increased knowledge from previous attempts They often don't make use of that out of a sense of pride and not invented here.
  - krastanov 6y ago
    > The only issue with Python is speed, but if you use Numpy properly, even that isn't a big issue. Numpy leaves a couple of orders of magnitude of performance on the table, especially if you use small arrays and a lot of intermediary computation. It is terrible in terms of memory allocation overhead. Cython or @tensorflow.function fix much of that, but then you are not really using python anymore. > They often don't make use of that out of a sense of pride and not invented here. This is a lazy and rather hurtful accusation.
vikramkr 6y ago
Because people want to? I think a big part of it is that people like trying new things and trying to differentiate in some way. In something as competitive as web dev, it makes sense that the churn in frameworks is so high, because everyone is looking for just that little edge or unique special sauce that makes development a bit easier or makes the competitor a bit more obsolete, or attracts talent thats just a little bit better. For someone out there, a little edge in CSV parsing is going to make their lives just a bit easier and help them do just a bit more than the competing lab, and grad students/labor is cheap, so why not have a grad student play around with Julia and see if you can get anything out of it?
krastanov 6y ago
I love python and work mostly in it, and I have some issues with the current state of Julia, but I think you are underestimating how much the python science stack has changed recently in a very unstable manner. There is so much cutting edge science that simply can not be done with numpy/scipy. A very big part of this has been the acceptance of reverse design and automatic differentiation in many engineering fields, for which you simply have to use theano/jax/tensorflow/pytorch (I am not talking about machine learning here). Of these, I had to switch my simulations away from Theano when it was deprecated, I could not use pytorch because it does not support complex numbers, and I had to switch from Tensorflow 1 to Tensorflow 2, and I had to write by myself ridiculously basic things like ODE solvers. The existence of ad-hoc solutions like numba or cython also do not speak well of the limitations of the language. Most specialized science packages (qutip for instance, for quantum optics), reimplement tons of solver/integration/optimization logic and require plenty of cython (or C) code. Scipy does not play nice with most of the automatic differentiation libraries. In that context (a pretty universal context), Julia is looking pretty stable (relatively) and future proof in comparison. E.g. you can not really interoperate with scipy/tensorflow in a meaningful high-performance way. The value proposition of Julia is that this will be possible, today, and hopefully when future new paradigms appear. Also, on a tangent, Julia already has ODE solvers and Probabilistic Programming packages lightlyears ahead of what Python has.
- bonoboTP 6y ago
  We need more decades-stable stuff similar to LAPACK and BLAS, but for one step higher-level things. Just imagine if people had to rewrite all that numerical computation for every new language. Leaving some performance critical stuff in C is okay. We can expose them to a variety of languages then. > away from Theano [...] I had to switch from Tensorflow 1 to Tensorflow 2 I feel you. But I think this has more to do with Google's deprecation culture. TF1 was nice for its time, but I will never adopt TF2, as they will drop it any time as well. I have higher hopes for PyTorch though.
  - krastanov 6y ago
    One of the neighboring comments mentioned something about changing assumptions. This is what limits the creation of decades-stable stuff. If you leave the performance crucial stuff in C, that stuff will not operate with the novel code you wrote (autodiff, probabilistic programming, GPU/TPU hardware, etc). Julia is now experimenting with pure-Julia code that is faster than many LAPACK/BLAS implementations. But I actually care little about the fact that they are a bit faster. Same with the new ODE package in Julia, which is unsurpassed in terms of functionality. The actual reason these pure-Julia packages excite me is that they are generic enough to run relatively easily on different hardware (GPUs for instance), they are on top of modern compilers (LLVM), and most importantly, they are written in a language that permits much higher level of interoperability. To rephrase it much more simply: The fact that you can not use scipy special functions efficiently inside of any of the autodiff frameworks is exemplary of how incredibly limiting python is, when it comes to developing new numerical techniques. Julia does not have that problem as far as we know.
jjoonathan 6y ago
> Why can't people just settle on some tech Because underlying assumptions change. Python still has an enormous amount of baggage from its single-threaded and fully-interpreted past (and present, really -- but I'd expect fans to argue). Should we live with the kludgey, limited workarounds? Or switch to an ecosystem built on a more modern foundation? Only time will tell -- but I won't shed any tears for python if we decide on the second option.
throwaway894345 6y ago
> Starting again from the ground up in a shiny new language (even if the language is perhaps slightly better) is a waste of so much effort. Transitioning from Python 2 to 3 was already a nightmare. This is a sunk cost argument. Python’s stack got us a long way, but it still falls short in a lot of ways. First of all, it is eager so you can’t build up an entire query, have the system optimize it, and then execute it for greater performance. Secondly, you can’t easily parallelize operations on the same dataframe. Thirdly, it’s poorly suited to running in a web process. Fourthly, it’s all written in C/C++, so debugging, building/packaging/installation (esp on niche systems), etc is a pain and it prevents many of the optimizations Julia and others can do natively, etc. Fifthly, the APIs were horribly designed (especially matplotlib, good grief). Etc. I don’t know about Julia in general, but in most domains there’s a middle ground where people can have nice, thoughtfully designed tools (albeit with minor issues), and endless churn. It certainly feels like the scientific toolchain is one of those areas—fix a few of these enormous, glaring problems and then enjoy some stability.
- jpeloquin 6y ago
  > Fifthly, the APIs were horribly designed (especially matplotlib, good grief) I've seen comments like this fairly often, but am not really sure what about matplotlib's API makes it so bad. Hoping to learn why. The only equal-basis competitor to matplotlib that I'm familiar with is Matlab, which is definitely worse. Declarative plotting libraries like ggplot2 have a nicer API, but the grammar of graphics approach is an entirely different paradigm, which gets as thorny as anything else if you have to do something unexpected. For the kind of medium-level plotting where you're given prepared plot elements, but have to explicitly specify where to draw them in data or figure coords, what would be an example of a "good" API (better than matplotlib)?
  - throwaway894345 6y ago
    The API is state full and the args it accepts are arbitrary. It just tries to guess what you want it to do based on the types of the args you pass and their order. I’m not sure which plotting API is better, but there’s nothing unique about plotting in all of computing that requires an insane API.
leovailati 6y ago
Disruption is the price of progress. Like cars with electric motors vs combustion engines, sometimes you have to start from scratch. You can't just keep improving combustion engines forever, you eventually reach a technological limit. To your point though, I do see a lot of unnecessary disruption particularly in the web dev world. I think people like working on new stuff. It's exciting to take the first steps, see order of magnitude improvements every commit.
- eesmith 6y ago
  As an aside, I think that's a poor analogy. Electric cars have been around since the late 1800s, only a decade or two after the first internal combustion engine, and for a while set land speed records. The last 30 years of electric cars did not start from scratch. Electric motors reached a technological plateau. As https://en.wikipedia.org/wiki/Electric_car https://en.wikipedia.org/wiki/Electric_car points out, it required MOSFET power converters and lithium-ion batteries. Before then, it could appear that it reached a technological limit.
  - leovailati 6y ago
    Fair point! I guess I meant the entire powertrain, not just the motor per se.
ssivark 6y ago
> Why can't people just settle on some tech, let it mature, and enjoy reaching higher and higher in capability? Why do we have to always tear things down and start from scratch? At what level of language improvement does it warrant rebuilding things? Eg: What if there has actually been a change as revolutionary as multiple dispatch based on a good type system, which cannot be backported to Python, and provides huge advantages for a diverse set of applications? For many users, it would be crazy not to switch... especially given that it’s actually much easier to write performance and flexible libraries in the Julia ecosystem (writing Julia code) than in the Python ecosystem (writing C/C++/Cython code). You might be succumbing to a bit of the blub paradox here... Python might be good enough for all your needs, but people are justifiably enamored with Julia because it brings some really exciting capabilities to the table which allows them to solve serious impediments they experience, not just because it is a shiny new thing. Rhetorical question: Now that we have all this infrastructure for gasoline based cars, is that reason enough to hold back the “churn” to electric cars?
cbkeller 6y ago
I see what you mean, but from my perspective what this argument is missing is the context that a large and important subset of the scientific community has never moved to Python to begin with. In other words, there is an even crankier and older community for which Julia may be the first actual change in a very long time. This is particularly the case in HPC, where everyone still uses Fortran, C, or C++, and has never moved nor ever will move to Python because Python is fundamentally unsuited to these workloads. But in some cases, Julia is [1]. The best differential equation solvers (e.g., SUNDIALS [2] for a modern example) have been written in FORTRAN for the last 50 years. If Julia can challenge Fortran as the go-to language for this type of work (e.g. DifferentialEquations.jl [3] and SciML), that would hardly count as excessive churn. [1] https://github.com/jeff-regier/Celeste.jl https://github.com/jeff-regier/Celeste.jl [2] https://computing.llnl.gov/projects/sundials https://computing.llnl.gov/projects/sundials [3] https://github.com/SciML/DifferentialEquations.jl https://github.com/SciML/DifferentialEquations.jl
ram_rar 6y ago
I cannot resonate more with this argument. Everytime we had some issue with performance with Python, instead of rewriting from scratch, I wrote it in C and created Python Wrappers for it. Its much better to extend the existing ecosystem, than rewriting things from scratch over and over again.
- bonoboTP 6y ago
  Yes, there is a very long tail of libraries scattered across GitHub and the web in general. Snippets of obscure algorithms, APIs for various things, like ML datasets from 5 years ago etc. I heard there's some interop between Python and Julia, but I guess it comes with plenty of caveats, like it works for most cases, but will give you enough headache like "oh, of course this use case is not supported like that" etc. Once you're in Python you can do anything. Spin up a web server, render something in 3D, read obscure data formats... When I see academic code from 10 years ago written in Matlab I cringe because I know it will be a pain to work with that code. There was a big reason people moved away from Matlab as it cannot support large applications. In Python you can build huge code bases with modular structure, sane classes etc. as opposed to Matlab. And it's free and open source unlike Matlab. So the switch there was worth it. But I don't want all of today's great Python long tail ecosystem to be lost to obsolescence in 5 or 10 years. It's just sad to see so much effort go down the drain.
bleepblorp 6y ago
From my perspective, the fact that so much of computing has consolidated around a language that cannot support even the most rudimentary code completion or static analysis tools is deeply unfortunate. As a result of Python's mass adoption, uncountable thousands of hours of developer time have been wasted hunting down bugs that would have been caught by tooling in any statically-typed language. Unless Python can fix this, by adopting static typing, it's not entirely fit for purpose in the ecosystem niche it has come to dominate. Replacing Python with a better language would be a net productivity boon for developers and scientists almost everywhere.
- jakobnissen 6y ago
  I think you're hitting on the main pain point of Julia. My bet is that we'll see strong typing dominate in the coming decade, with assistance from linters/static analysis becoming default. Unfortunately, this is not too easy to create using Julia. Much easier than Python, sure, but not too easy. Tellingly, the Julia community apparently does not care for static analysis and have not begun to develop the capacity for it. As a consequence, much Julia code is written in a way that is un-analyzable, which digs it deeper into weak typing.

Datenstrom 6y ago

There is an immense value in most of the ecosystem converging on Python over the last decade and especially with leaving MATLAB behind in the dust. I don't think decade long transitions from old tech to new tech is a bad time scale or even close to that of the churn of web technologies. Julia offers enough of an improvement over Python to warrant a switch over the next 5-10 years and leave Python behind in the same way. I believe Julia offers the same level of improvement over Python as Python does over MATLAB. I also believe that it has drawn too much from MATLAB only for the sake of being familiar to that group and not the computer science and software engineering communities though such as the one based indexing you mentioned. Unfortunately that ship has already sailed and we will likely have to wait another 10 years for the next one to hopefully fix those problems.

ScottPJones 6y ago
Back 5.5 years ago, I used to complain about the 1 based indexing and the column-major structure of matrices in Julia (both like Fortran), however, those issues have been solved by OffsetArrays and PermutedDimsArrays, giving far more flexibility that is possible in most other languages. It's silly to keep bringing up the issue of one based indexing, when you can use any integer as the base, just like Fortran 90 (so you can index by -5..5, for example). For some things, 0 based does make things easier, sure, but you can do that easily in Julia (and more!)
- Datenstrom 6y ago
  Yes it is possible to make those changes and even if it wasn't it does not detract from the impressive and useful innovations Julia has made. That is why I usually refer to them as cosmetic but that is possibly not the correct word either because it does not convey the implications of having that default beyond simply being unattractive. Cultural may be the correct term. Julia claims that it wants to solve the two language problem. To do that it must win both the scientific compute, HPC, data science, machine learning, etc. group and the CS, software engineering, dev ops group. I've shown Julia to quite a few members of the latter and they always love the idea and technical capabilities of Julia but visibly cringe when they find out it uses one based indexing. They also almost universally find the syntax ugly. Unfortunately the available packages to change that also does not help because most people will never change the defaults and IMHO it would be bad to do so. When writing Python you follow the PEP-8 style guide, it is just best practice to follow the languages conventions so everyone is on the same page. This is the same problem with C++ trying to add all these new features and a solid package manager like Rust has, it will never fly, people just won't use it everywhere, it isn't going to be in the standard library, It isn't the default and it isn't part of the culture.

ddragon 6y ago

By that logic people should have kept using Fortran instead of Python (which was the new guy a couple decades ago). But Fortran will never be Python, and Python will never be Fortran, no matter how much maturity/improvements. Because maturity means getting close to it's potential (local maxima), without the ability to drastically move to a hypothetical global maxima (without breaking all it's legacy, like a more drastic Python 2 -> 3, or adding things indefinitely until you get effectively many languages/dialects in one like C++). New languages on the other side don't have as much baggage (and can start with increased knowledge from previous attempts) so they are able to freely potentially target better local maxima that old languages cannot anymore. Plus in this particular case, Python was not really designed to target the scientific domain (nor ML which did not even exist in the way we use nowadays), it was repeatedly retrofitted with possibly a lot more effort than would be necessary to create something better from scratch.

bonoboTP 6y ago
> By that logic people should have kept using Fortran instead of Python There were very real pain points with Fortran, C++, MATLAB etc. compared to Python. Now this is hard for me to argue in a few sentences as it's a distillation of having worked with many languages, but I feel that Python strikes a great balance for productivity. It feels like pouring your thoughts directly onto the screen, with no need to fight the language. The syntax is very clear, concise, "Pythonic" is a word of praise. As I said, this kind of debate is rarely productive and devolves to religious arguments, but that's my stance. The only issue with Python is speed, but if you use Numpy properly, even that isn't a big issue. > and can start with increased knowledge from previous attempts They often don't make use of that out of a sense of pride and not invented here.
- krastanov 6y ago
  > The only issue with Python is speed, but if you use Numpy properly, even that isn't a big issue. Numpy leaves a couple of orders of magnitude of performance on the table, especially if you use small arrays and a lot of intermediary computation. It is terrible in terms of memory allocation overhead. Cython or @tensorflow.function fix much of that, but then you are not really using python anymore. > They often don't make use of that out of a sense of pride and not invented here. This is a lazy and rather hurtful accusation.

vikramkr 6y ago

Because people want to? I think a big part of it is that people like trying new things and trying to differentiate in some way. In something as competitive as web dev, it makes sense that the churn in frameworks is so high, because everyone is looking for just that little edge or unique special sauce that makes development a bit easier or makes the competitor a bit more obsolete, or attracts talent thats just a little bit better. For someone out there, a little edge in CSV parsing is going to make their lives just a bit easier and help them do just a bit more than the competing lab, and grad students/labor is cheap, so why not have a grad student play around with Julia and see if you can get anything out of it?

krastanov 6y ago

I love python and work mostly in it, and I have some issues with the current state of Julia, but I think you are underestimating how much the python science stack has changed recently in a very unstable manner. There is so much cutting edge science that simply can not be done with numpy/scipy. A very big part of this has been the acceptance of reverse design and automatic differentiation in many engineering fields, for which you simply have to use theano/jax/tensorflow/pytorch (I am not talking about machine learning here). Of these, I had to switch my simulations away from Theano when it was deprecated, I could not use pytorch because it does not support complex numbers, and I had to switch from Tensorflow 1 to Tensorflow 2, and I had to write by myself ridiculously basic things like ODE solvers. The existence of ad-hoc solutions like numba or cython also do not speak well of the limitations of the language. Most specialized science packages (qutip for instance, for quantum optics), reimplement tons of solver/integration/optimization logic and require plenty of cython (or C) code. Scipy does not play nice with most of the automatic differentiation libraries. In that context (a pretty universal context), Julia is looking pretty stable (relatively) and future proof in comparison. E.g. you can not really interoperate with scipy/tensorflow in a meaningful high-performance way. The value proposition of Julia is that this will be possible, today, and hopefully when future new paradigms appear. Also, on a tangent, Julia already has ODE solvers and Probabilistic Programming packages lightlyears ahead of what Python has.

bonoboTP 6y ago
We need more decades-stable stuff similar to LAPACK and BLAS, but for one step higher-level things. Just imagine if people had to rewrite all that numerical computation for every new language. Leaving some performance critical stuff in C is okay. We can expose them to a variety of languages then. > away from Theano [...] I had to switch from Tensorflow 1 to Tensorflow 2 I feel you. But I think this has more to do with Google's deprecation culture. TF1 was nice for its time, but I will never adopt TF2, as they will drop it any time as well. I have higher hopes for PyTorch though.
- krastanov 6y ago
  One of the neighboring comments mentioned something about changing assumptions. This is what limits the creation of decades-stable stuff. If you leave the performance crucial stuff in C, that stuff will not operate with the novel code you wrote (autodiff, probabilistic programming, GPU/TPU hardware, etc). Julia is now experimenting with pure-Julia code that is faster than many LAPACK/BLAS implementations. But I actually care little about the fact that they are a bit faster. Same with the new ODE package in Julia, which is unsurpassed in terms of functionality. The actual reason these pure-Julia packages excite me is that they are generic enough to run relatively easily on different hardware (GPUs for instance), they are on top of modern compilers (LLVM), and most importantly, they are written in a language that permits much higher level of interoperability. To rephrase it much more simply: The fact that you can not use scipy special functions efficiently inside of any of the autodiff frameworks is exemplary of how incredibly limiting python is, when it comes to developing new numerical techniques. Julia does not have that problem as far as we know.

jjoonathan 6y ago

> Why can't people just settle on some tech Because underlying assumptions change. Python still has an enormous amount of baggage from its single-threaded and fully-interpreted past (and present, really -- but I'd expect fans to argue). Should we live with the kludgey, limited workarounds? Or switch to an ecosystem built on a more modern foundation? Only time will tell -- but I won't shed any tears for python if we decide on the second option.

throwaway894345 6y ago

> Starting again from the ground up in a shiny new language (even if the language is perhaps slightly better) is a waste of so much effort. Transitioning from Python 2 to 3 was already a nightmare. This is a sunk cost argument. Python’s stack got us a long way, but it still falls short in a lot of ways. First of all, it is eager so you can’t build up an entire query, have the system optimize it, and then execute it for greater performance. Secondly, you can’t easily parallelize operations on the same dataframe. Thirdly, it’s poorly suited to running in a web process. Fourthly, it’s all written in C/C++, so debugging, building/packaging/installation (esp on niche systems), etc is a pain and it prevents many of the optimizations Julia and others can do natively, etc. Fifthly, the APIs were horribly designed (especially matplotlib, good grief). Etc. I don’t know about Julia in general, but in most domains there’s a middle ground where people can have nice, thoughtfully designed tools (albeit with minor issues), and endless churn. It certainly feels like the scientific toolchain is one of those areas—fix a few of these enormous, glaring problems and then enjoy some stability.

jpeloquin 6y ago
> Fifthly, the APIs were horribly designed (especially matplotlib, good grief) I've seen comments like this fairly often, but am not really sure what about matplotlib's API makes it so bad. Hoping to learn why. The only equal-basis competitor to matplotlib that I'm familiar with is Matlab, which is definitely worse. Declarative plotting libraries like ggplot2 have a nicer API, but the grammar of graphics approach is an entirely different paradigm, which gets as thorny as anything else if you have to do something unexpected. For the kind of medium-level plotting where you're given prepared plot elements, but have to explicitly specify where to draw them in data or figure coords, what would be an example of a "good" API (better than matplotlib)?
- throwaway894345 6y ago
  The API is state full and the args it accepts are arbitrary. It just tries to guess what you want it to do based on the types of the args you pass and their order. I’m not sure which plotting API is better, but there’s nothing unique about plotting in all of computing that requires an insane API.

leovailati 6y ago

Disruption is the price of progress. Like cars with electric motors vs combustion engines, sometimes you have to start from scratch. You can't just keep improving combustion engines forever, you eventually reach a technological limit. To your point though, I do see a lot of unnecessary disruption particularly in the web dev world. I think people like working on new stuff. It's exciting to take the first steps, see order of magnitude improvements every commit.

eesmith 6y ago
As an aside, I think that's a poor analogy. Electric cars have been around since the late 1800s, only a decade or two after the first internal combustion engine, and for a while set land speed records. The last 30 years of electric cars did not start from scratch. Electric motors reached a technological plateau. As https://en.wikipedia.org/wiki/Electric_car https://en.wikipedia.org/wiki/Electric_car points out, it required MOSFET power converters and lithium-ion batteries. Before then, it could appear that it reached a technological limit.
- leovailati 6y ago
  Fair point! I guess I meant the entire powertrain, not just the motor per se.

ssivark 6y ago

> Why can't people just settle on some tech, let it mature, and enjoy reaching higher and higher in capability? Why do we have to always tear things down and start from scratch? At what level of language improvement does it warrant rebuilding things? Eg: What if there has actually been a change as revolutionary as multiple dispatch based on a good type system, which cannot be backported to Python, and provides huge advantages for a diverse set of applications? For many users, it would be crazy not to switch... especially given that it’s actually much easier to write performance and flexible libraries in the Julia ecosystem (writing Julia code) than in the Python ecosystem (writing C/C++/Cython code). You might be succumbing to a bit of the blub paradox here... Python might be good enough for all your needs, but people are justifiably enamored with Julia because it brings some really exciting capabilities to the table which allows them to solve serious impediments they experience, not just because it is a shiny new thing. Rhetorical question: Now that we have all this infrastructure for gasoline based cars, is that reason enough to hold back the “churn” to electric cars?

cbkeller 6y ago

I see what you mean, but from my perspective what this argument is missing is the context that a large and important subset of the scientific community has never moved to Python to begin with. In other words, there is an even crankier and older community for which Julia may be the first actual change in a very long time. This is particularly the case in HPC, where everyone still uses Fortran, C, or C++, and has never moved nor ever will move to Python because Python is fundamentally unsuited to these workloads. But in some cases, Julia is [1]. The best differential equation solvers (e.g., SUNDIALS [2] for a modern example) have been written in FORTRAN for the last 50 years. If Julia can challenge Fortran as the go-to language for this type of work (e.g. DifferentialEquations.jl [3] and SciML), that would hardly count as excessive churn. [1] https://github.com/jeff-regier/Celeste.jl https://github.com/jeff-regier/Celeste.jl [2] https://computing.llnl.gov/projects/sundials https://computing.llnl.gov/projects/sundials [3] https://github.com/SciML/DifferentialEquations.jl https://github.com/SciML/DifferentialEquations.jl

ram_rar 6y ago

I cannot resonate more with this argument. Everytime we had some issue with performance with Python, instead of rewriting from scratch, I wrote it in C and created Python Wrappers for it. Its much better to extend the existing ecosystem, than rewriting things from scratch over and over again.

bonoboTP 6y ago
Yes, there is a very long tail of libraries scattered across GitHub and the web in general. Snippets of obscure algorithms, APIs for various things, like ML datasets from 5 years ago etc. I heard there's some interop between Python and Julia, but I guess it comes with plenty of caveats, like it works for most cases, but will give you enough headache like "oh, of course this use case is not supported like that" etc. Once you're in Python you can do anything. Spin up a web server, render something in 3D, read obscure data formats... When I see academic code from 10 years ago written in Matlab I cringe because I know it will be a pain to work with that code. There was a big reason people moved away from Matlab as it cannot support large applications. In Python you can build huge code bases with modular structure, sane classes etc. as opposed to Matlab. And it's free and open source unlike Matlab. So the switch there was worth it. But I don't want all of today's great Python long tail ecosystem to be lost to obsolescence in 5 or 10 years. It's just sad to see so much effort go down the drain.

bleepblorp 6y ago

From my perspective, the fact that so much of computing has consolidated around a language that cannot support even the most rudimentary code completion or static analysis tools is deeply unfortunate. As a result of Python's mass adoption, uncountable thousands of hours of developer time have been wasted hunting down bugs that would have been caught by tooling in any statically-typed language. Unless Python can fix this, by adopting static typing, it's not entirely fit for purpose in the ecosystem niche it has come to dominate. Replacing Python with a better language would be a net productivity boon for developers and scientists almost everywhere.

jakobnissen 6y ago
I think you're hitting on the main pain point of Julia. My bet is that we'll see strong typing dominate in the coming decade, with assistance from linters/static analysis becoming default. Unfortunately, this is not too easy to create using Julia. Much easier than Python, sure, but not too easy. Tellingly, the Julia community apparently does not care for static analysis and have not begun to develop the capacity for it. As a consequence, much Julia code is written in a way that is un-analyzable, which digs it deeper into weak typing.