4 ms·

I already solve real problems in Python—why bother learning yet another language, and one our data engineers/backend folks don’t know? That larger context of im

I already solve real problems in Python—why bother learning yet another language, and one our data engineers/backend folks don’t know? That larger context of implementation / development / deployment is a huge deciding factor for real-world problem solving, not how “great” a language is from a design perspective.

snicker7 6y ago
It depends on the problem you have, "do excel-type business analysis" or "write a CRUD app" are things any language can do. Julia solves hard problems. Neuro-differential equations, energy grid optimization (Ivenia), database implementations (RelationalAI, see Jamie Brandon's talk on why Julia specifically is suitable for this), DSL implementations (Stan, &etc). Writing high-performance simulations (no, slow langs won't cut it, if you want tight loops you need low-level control such as cache-coherent memory layouts, vectorized instructions, &c).
- coldtea 6y ago
  >(no, slow langs won't cut it, if you want tight loops you need low-level control such as cache-coherent memory layouts, vectorized instructions, &c). Yawn. You can have all those wrapped under a high level API, with much nicer implementations, with more support, more documentation, more eyeballs (due to more adoption) in a "slow" language. You can even have them run faster than Julia's and in more stable and mature implementations (like there's LAPACK).
  - Recurecur 6y ago
    > You can have all those wrapped under a high level API, with much nicer implementations, with more support, more documentation, more eyeballs (due to more adoption) in a "slow" language. You 'can' have those things, but 'should' you? Julia is well designed, so 'nicer' is not a given. Julia is also quite a high level language with excellent metaprogramming facilities. Otherwise, everything you listed is based on adoption level. Why use a "slow" language if you don't have to? Especially when the faster language is actually better designed...
    - coldtea 6y ago
      >You 'can' have those things, but 'should' you? As opposed to what? Wait for Julia to catch up in 10 years? If so, yes, you absolutely should get those things from where they're already available. >Why use a "slow" language if you don't have to? Because that's where the action is, and who gets those niceties first. Julia hasn't even fixed their slow startup/load situation all these years...
      - Recurecur 6y ago
        Progress is sadly not instant. Julia seems to be gathering momentum rather than losing it. The future looks bright if trends continue! (As to the startup time problem, significant progress is being made... Have you looked at standalone binaries?)
  - snicker7 6y ago
    What's the point of the wrapper? import simulation from 'simulation_code_written_in_c' simulation.run() What value does the above code have?
    - coldtea 6y ago
      That particular strawman code little. Code that e.g. can be configured to load data in a particular way, and then run in a particular way, and is then run, hella fast in C, does. Numpy is not: import nympy numpy.run Neither is pandas and others... You might as well ask "what is the point of glue languages". Just ask anybody who ever embedded e.g. Python/Lua/Guile/etc in a C/C++ program (which has similar mechanics)...
      - snicker7 6y ago
        "What is the point of glue languages" indeed. Loading parameters into a program makes sense for a scripting language, especially for interacting with code you didn't write. But when 99% of your problem is solved with custom-made C/C++/whatever (as is the case with simulations), the value-add of a glue language is very little. Professionally, I've written trading simulations in C++ where Python was used for glue code. It added very little value and added more dependencies/headaches. Julia could've have replaced both.
  - jakobnissen 6y ago
    Right up until you have to implement, debug, or enchance these lower-level packages yourself. Then suddenly * 95% of your users can't contribute or even understand what's going on, because they don't understand the language it's written in * even if you do understand, you have to write in a much less expressive language (you came using Python for a reason, but end up doing all the hard work in C) * performance gets lots at the boundary between the languages (which is why Julia nearly always outperforms Numpy) * error messages and bugs often fall through the crack of the two languages The fundamental argument "you can just use Python, you just have to write all the hard stuff in not-Python" is very strange to me.
    - coldtea 6y ago
      >Right up until you have to implement, debug, or enchance these lower-level packages yourself That's the very benefit of a larger ecosystem: you don't. Whereas with a smaller ecosystem, you often do. >95% of your users can't contribute or even understand what's going on, because they don't understand the language it's written in That's even more of a problem with Julia, since the code written in Julia will be unfamiliar to both first level users (much much fewer use Julia than Python) and to library developers (fewer know Julian than C/C++, which such Python libs are written in).
FridgeSeal 6y ago
I’m personally of the opinion that one should learn new languages, so that even if you don’t put them into production, you’re exposed to the ideas and approaches they bring-which might be foreign to your current language of choice-and this can give you new ideas about how to best tackle problems in your day-to-day. I’ll probably never convince my .Net-wielding team members to write a web API in Haskell, and that’s ok, and I’m not about to rewrite my data pipelines in it (even though that might be fun), but the ideas about purity/mutation, cleanly-separating IO, composition, etc have benefitted my teammates and I immensely with the code at work though.
- michaericalribo 6y ago
  This is a good point! I like your way of approaching this space :) Though, what you are describing is very different from what GP was claiming, which is using Julia for production.
jakobnissen 6y ago
Julia didn't come about because a bunch of CS people wanted to make a pretty language. It came about because people experienced with scientific computing were frustrated with the current state of affairs and wanted to improve the situation. If Python works fine for your use case, then indeed, Julia probably seems like a waste of time. But if your situation - like mine was, three years ago - is that you program mostly in Python, but then constantly hit the performance wall and begin doing questionable gymnastics with Numpy, Cython or Numba to try to scale it, before eventually giving up and writing C code that is 15% business logic and 85% pointless boilerplate that segfaults if you look at it wrong... well, then the point of Julia is obvious. Right now, Julia mostly attracts the users for whom Python is obviously not good enough. Eventually, I hope the ecosystem matures enough that even a user who might otherwise use Python would use Julia simply because it's nicer.

snicker7 6y ago

It depends on the problem you have, "do excel-type business analysis" or "write a CRUD app" are things any language can do. Julia solves hard problems. Neuro-differential equations, energy grid optimization (Ivenia), database implementations (RelationalAI, see Jamie Brandon's talk on why Julia specifically is suitable for this), DSL implementations (Stan, &etc). Writing high-performance simulations (no, slow langs won't cut it, if you want tight loops you need low-level control such as cache-coherent memory layouts, vectorized instructions, &c).

coldtea 6y ago
>(no, slow langs won't cut it, if you want tight loops you need low-level control such as cache-coherent memory layouts, vectorized instructions, &c). Yawn. You can have all those wrapped under a high level API, with much nicer implementations, with more support, more documentation, more eyeballs (due to more adoption) in a "slow" language. You can even have them run faster than Julia's and in more stable and mature implementations (like there's LAPACK).
- Recurecur 6y ago
  > You can have all those wrapped under a high level API, with much nicer implementations, with more support, more documentation, more eyeballs (due to more adoption) in a "slow" language. You 'can' have those things, but 'should' you? Julia is well designed, so 'nicer' is not a given. Julia is also quite a high level language with excellent metaprogramming facilities. Otherwise, everything you listed is based on adoption level. Why use a "slow" language if you don't have to? Especially when the faster language is actually better designed...
  - coldtea 6y ago
    >You 'can' have those things, but 'should' you? As opposed to what? Wait for Julia to catch up in 10 years? If so, yes, you absolutely should get those things from where they're already available. >Why use a "slow" language if you don't have to? Because that's where the action is, and who gets those niceties first. Julia hasn't even fixed their slow startup/load situation all these years...
    - Recurecur 6y ago
      Progress is sadly not instant. Julia seems to be gathering momentum rather than losing it. The future looks bright if trends continue! (As to the startup time problem, significant progress is being made... Have you looked at standalone binaries?)
- snicker7 6y ago
  What's the point of the wrapper? import simulation from 'simulation_code_written_in_c' simulation.run() What value does the above code have?
  - coldtea 6y ago
    That particular strawman code little. Code that e.g. can be configured to load data in a particular way, and then run in a particular way, and is then run, hella fast in C, does. Numpy is not: import nympy numpy.run Neither is pandas and others... You might as well ask "what is the point of glue languages". Just ask anybody who ever embedded e.g. Python/Lua/Guile/etc in a C/C++ program (which has similar mechanics)...
    - snicker7 6y ago
      "What is the point of glue languages" indeed. Loading parameters into a program makes sense for a scripting language, especially for interacting with code you didn't write. But when 99% of your problem is solved with custom-made C/C++/whatever (as is the case with simulations), the value-add of a glue language is very little. Professionally, I've written trading simulations in C++ where Python was used for glue code. It added very little value and added more dependencies/headaches. Julia could've have replaced both.
- jakobnissen 6y ago
  Right up until you have to implement, debug, or enchance these lower-level packages yourself. Then suddenly * 95% of your users can't contribute or even understand what's going on, because they don't understand the language it's written in * even if you do understand, you have to write in a much less expressive language (you came using Python for a reason, but end up doing all the hard work in C) * performance gets lots at the boundary between the languages (which is why Julia nearly always outperforms Numpy) * error messages and bugs often fall through the crack of the two languages The fundamental argument "you can just use Python, you just have to write all the hard stuff in not-Python" is very strange to me.
  - coldtea 6y ago
    >Right up until you have to implement, debug, or enchance these lower-level packages yourself That's the very benefit of a larger ecosystem: you don't. Whereas with a smaller ecosystem, you often do. >95% of your users can't contribute or even understand what's going on, because they don't understand the language it's written in That's even more of a problem with Julia, since the code written in Julia will be unfamiliar to both first level users (much much fewer use Julia than Python) and to library developers (fewer know Julian than C/C++, which such Python libs are written in).

FridgeSeal 6y ago

I’m personally of the opinion that one should learn new languages, so that even if you don’t put them into production, you’re exposed to the ideas and approaches they bring-which might be foreign to your current language of choice-and this can give you new ideas about how to best tackle problems in your day-to-day. I’ll probably never convince my .Net-wielding team members to write a web API in Haskell, and that’s ok, and I’m not about to rewrite my data pipelines in it (even though that might be fun), but the ideas about purity/mutation, cleanly-separating IO, composition, etc have benefitted my teammates and I immensely with the code at work though.

michaericalribo 6y ago
This is a good point! I like your way of approaching this space :) Though, what you are describing is very different from what GP was claiming, which is using Julia for production.

jakobnissen 6y ago

Julia didn't come about because a bunch of CS people wanted to make a pretty language. It came about because people experienced with scientific computing were frustrated with the current state of affairs and wanted to improve the situation. If Python works fine for your use case, then indeed, Julia probably seems like a waste of time. But if your situation - like mine was, three years ago - is that you program mostly in Python, but then constantly hit the performance wall and begin doing questionable gymnastics with Numpy, Cython or Numba to try to scale it, before eventually giving up and writing C code that is 15% business logic and 85% pointless boilerplate that segfaults if you look at it wrong... well, then the point of Julia is obvious. Right now, Julia mostly attracts the users for whom Python is obviously not good enough. Eventually, I hope the ecosystem matures enough that even a user who might otherwise use Python would use Julia simply because it's nicer.