2 ms·

I’ve been a software engineer for many years. A lot of what I and my colleagues do is glue, sew and plumb things together. I’ve never had to write a crazy algo

I’ve been a software engineer for many years. A lot of what I and my colleagues do is glue, sew and plumb things together. I’ve never had to write a crazy algorithm. Nor have I had to do any graph traversal. Or anything that impacts performance. For data analysis, pandas and numpy are plenty powerful. At no time in my career have I said “ditch all this because it’s too slow”. Perhaps others have, but I’m perfectly fine with Python. It’s a beautiful language with ability to hack into basically any part of the language. Batteries, flint, tent and a campsite included.

quixoticelixer- 6y ago
I sort of agree, although there are certainly quite a few Julia users who do use it for performance. On the other hand I much prefer Julia to Python/Numpy/Pandas regardless of speed.
gameswithgo 6y ago
somebody has to make the stuff you guys glue together.
- systemvoltage 6y ago
  Totally. And they can use C/C++/Rust which is what most do when performance really matters. I don't see Julia as a general purpose langauge. I don't see high performance network stack written in Julia ever. I see Julia as an academic scripting language that happens to be fast * * Debatable.
  - dnautics 6y ago
    really? I would think the number of people who would need to use a high performance number crunching for their business logic is much higher than the number of people who need a high peroformance network stack. The network is usually pretty slow, and almost always the bottleneck is not any code that hasn't already been integrated into the OS kernel. For 99% of people, the OS network stack is fine.
  - quixoticelixer- 6y ago
    That's why Julia was created though. There is a lot of scientific code that needs to be fast and is written in C or Fortran, but those languages are mediocre in terms of scientific programming.
  - ddragon 6y ago
    "Ever" could be a very long time, and lots of things can happen. For example it might be possible to AoT compile parts of the code (as long as the types are fully defined) to export as a standalone library, and if that happens it would be beneficial to write the number crunching libraries for glue languages in Julia. I mean, there are already those (like DiffEq to Python and R), but if it's a binary that could have just as easily come from C it would be even better. And there are already works along this path, including for wasm deployment. The multithreading in Julia is very fast and easy to use, and I'm sure it will become robust and safer as well as the language gets older, which can very well allow for an Akka style framework that allow for large software with both high reliability (thanks to stuff like supervision trees) and structured code (since like Julia's ecosystem design favors small libraries composing with each other, the future of large Julia codebases could become the composition of many small high level but fast actors). There is also lots of interests in work-arounds on the garbage collector, like improved support to immutable structures (like 1.5 improvements on structs with references to mutable structures). It's possible that you might be able to more reliably write parts of the code that do not allocate (or allocate in predictable ways), so you'll be even handle those realtime code loops. Sure this is very speculative, but look at Python today compared to when it had Julia's age. It took 8 years since Julia became public to get to this point (2 years since it got the first stable release, 1 year since it got it's multithreading model), it will surely be different when it has triple that age.
snicker7 6y ago
CORRECTION: I am mistaken. See https://docs.python.org/3/library/ctypes.html https://docs.python.org/3/library/ctypes.html Python is a pretty poor glue language, in my experience. Calling C APIs from Python is not pleasant. This is how one might import the libm "error function" from Julia: const libm = "/lib64/libm.so.6" erf(x) = @ccall libm.erf(x::Float64)::Float64 Python requires writing/compiling a separate dll written in C (see [^1]). Python extensions are (reasonably) fast, sure. But they are not pleasant at all to write. [^1]: https://docs.python.org/3/extending/building.html https://docs.python.org/3/extending/building.html
- jhayward 6y ago
  Python has a perfectly useful CFFI. There's no need to write an extension module.
- dragonwriter 6y ago
  > Calling C APIs from Python is not pleasant. [...] Python requires writing/compiling a separate dll written in C (see [^1]). No, it doesn't, see [0] and [1]. [0]: https://docs.python.org/3/library/ctypes.html https://docs.python.org/3/library/ctypes.html [1]: https://cffi.readthedocs.io/en/latest/ https://cffi.readthedocs.io/en/latest/
- orbifold 6y ago
  Pybind11 is a dream to work with and there are python packages that implement hot reloading for C++ into a live python session aswell as autogeneration tools for it.
oivey 6y ago
I write crazy algorithms all the time, but I guess I have a more research oriented role. Ultimately someone has to write the crazy stuff you want to glue together, and the more productive those people are the sooner you can have those libraries available. As a user of this code I think there is a be benefit to you being able to read the source behind the algorithms without wading through esoteric FORTRAN, C, and/or CUDA.
ska 6y ago
This the problem with anecdotal data. In the past I have spent years mostly developing and implementing new algorithms, where appropriate using graph representations. I've also spent months at a stretch on low level performance tuning. I don't find python beautiful; it's a functional glue language and I don't hate it. It has an ok REPL and some quirky design choices. It has a decent C interop which makes it practical. What makes it useful for data analysis is the large number of packages and staggering number of person-years that have been put into them. The greater python community is great. The language is ok. Together that's pretty useful.

quixoticelixer- 6y ago

I sort of agree, although there are certainly quite a few Julia users who do use it for performance. On the other hand I much prefer Julia to Python/Numpy/Pandas regardless of speed.

gameswithgo 6y ago

somebody has to make the stuff you guys glue together.

systemvoltage 6y ago
Totally. And they can use C/C++/Rust which is what most do when performance really matters. I don't see Julia as a general purpose langauge. I don't see high performance network stack written in Julia ever. I see Julia as an academic scripting language that happens to be fast * * Debatable.
- dnautics 6y ago
  really? I would think the number of people who would need to use a high performance number crunching for their business logic is much higher than the number of people who need a high peroformance network stack. The network is usually pretty slow, and almost always the bottleneck is not any code that hasn't already been integrated into the OS kernel. For 99% of people, the OS network stack is fine.
- quixoticelixer- 6y ago
  That's why Julia was created though. There is a lot of scientific code that needs to be fast and is written in C or Fortran, but those languages are mediocre in terms of scientific programming.
- ddragon 6y ago
  "Ever" could be a very long time, and lots of things can happen. For example it might be possible to AoT compile parts of the code (as long as the types are fully defined) to export as a standalone library, and if that happens it would be beneficial to write the number crunching libraries for glue languages in Julia. I mean, there are already those (like DiffEq to Python and R), but if it's a binary that could have just as easily come from C it would be even better. And there are already works along this path, including for wasm deployment. The multithreading in Julia is very fast and easy to use, and I'm sure it will become robust and safer as well as the language gets older, which can very well allow for an Akka style framework that allow for large software with both high reliability (thanks to stuff like supervision trees) and structured code (since like Julia's ecosystem design favors small libraries composing with each other, the future of large Julia codebases could become the composition of many small high level but fast actors). There is also lots of interests in work-arounds on the garbage collector, like improved support to immutable structures (like 1.5 improvements on structs with references to mutable structures). It's possible that you might be able to more reliably write parts of the code that do not allocate (or allocate in predictable ways), so you'll be even handle those realtime code loops. Sure this is very speculative, but look at Python today compared to when it had Julia's age. It took 8 years since Julia became public to get to this point (2 years since it got the first stable release, 1 year since it got it's multithreading model), it will surely be different when it has triple that age.

snicker7 6y ago

CORRECTION: I am mistaken. See https://docs.python.org/3/library/ctypes.html https://docs.python.org/3/library/ctypes.html Python is a pretty poor glue language, in my experience. Calling C APIs from Python is not pleasant. This is how one might import the libm "error function" from Julia: const libm = "/lib64/libm.so.6" erf(x) = @ccall libm.erf(x::Float64)::Float64 Python requires writing/compiling a separate dll written in C (see [^1]). Python extensions are (reasonably) fast, sure. But they are not pleasant at all to write. [^1]: https://docs.python.org/3/extending/building.html https://docs.python.org/3/extending/building.html

jhayward 6y ago
Python has a perfectly useful CFFI. There's no need to write an extension module.
dragonwriter 6y ago
> Calling C APIs from Python is not pleasant. [...] Python requires writing/compiling a separate dll written in C (see [^1]). No, it doesn't, see [0] and [1]. [0]: https://docs.python.org/3/library/ctypes.html https://docs.python.org/3/library/ctypes.html [1]: https://cffi.readthedocs.io/en/latest/ https://cffi.readthedocs.io/en/latest/
orbifold 6y ago
Pybind11 is a dream to work with and there are python packages that implement hot reloading for C++ into a live python session aswell as autogeneration tools for it.

oivey 6y ago

I write crazy algorithms all the time, but I guess I have a more research oriented role. Ultimately someone has to write the crazy stuff you want to glue together, and the more productive those people are the sooner you can have those libraries available. As a user of this code I think there is a be benefit to you being able to read the source behind the algorithms without wading through esoteric FORTRAN, C, and/or CUDA.

ska 6y ago

This the problem with anecdotal data. In the past I have spent years mostly developing and implementing new algorithms, where appropriate using graph representations. I've also spent months at a stretch on low level performance tuning. I don't find python beautiful; it's a functional glue language and I don't hate it. It has an ok REPL and some quirky design choices. It has a decent C interop which makes it practical. What makes it useful for data analysis is the large number of packages and staggering number of person-years that have been put into them. The greater python community is great. The language is ok. Together that's pretty useful.