5 ms·

Language choices are less about language to me and more about ecosystem of libraries. Python has generally been very strong in the ml/data science realm. I know

Language choices are less about language to me and more about ecosystem of libraries. Python has generally been very strong in the ml/data science realm. I know Julia is catching up but am unsure just how much it covers. Example of python libraries I would consider needed as part of the data ecosystem, numpy, pytorch/tf, batch and stream processing frameworks like spark/flink/beam, workflow orchestration like kubeflow/airflow, data formats like pyarrow, etc. My company does most of our ml in google cloud so gcp libraries are also quite helpful. How much of that does Julia have equivalents to? How much coverage do those equivalents have? We’ve done some ml things outside python before and one general issue for most languages is leaving python there’s a high risk that something will be missing. Because of that if we use another language it’s preferred to keep scope of it’s project small. I think only big exception is on the data engineering side sometimes Java is better as a lot of batch/stream frameworks have best coverage in Java, although ml libraries is weaker there so our usage of Java is mainly data pipelines. Another issue is pytorch/tf in python are very dominant in research/projects. Often we clone relevant recent projects and try experimenting with them to see if they help. Swapping to Julia would hurt a ton in that area. edit: Also while I'm fond of python I'd be very open to seeing another language win. There are language design choices I dislike in python, but I like enough of the language and ecosystem as been too strong to leave most other languages worth pondering. If Julia grows enough that my coworkers start asking for Julia support I'd be happy to explore it. My pet preferred language is crystal (ruby like readability + types + good performance) but ecosystem wise it's tiny.

queuebert 5y ago
Python is the middle manager of languages. It sucks at everything, but always knows a guy.
- dgfitz 5y ago
  I saw a quote once that stuck with me: python is the second best language for everything, first at nothing.
  - d0mine 5y ago
    What are the best languages for "executable pseudocode" or "glue language" roles?
- antisthenes 5y ago
  And there's 1 guy he knows that's a 10x developer and always willing to take one for the team. I'll leave it to HN to figure out what that means :P
  - melony 5y ago
    Call native libraries?
    - isoprophlex 5y ago
      Write a metric shit ton of unmaintainable, underperforming pandas code and call it an effective solution
adgjlsfhk1 5y ago
One of the really nice things with Julia is some of the ecosystem needs disappear. Python needs a lot of ecosystem because none of the packages work together, and the language is slow so you have to make sure you are doing as much work as possible outside the language itself. To answer your question more specifically: Numpy -> Array + broadcasting (both in Julia Base) pytoch/tf -> Flux.jl (package) batch/stream processing -> you don't need it as much, but things like OnlineStats exist. Also Base has multithreaded and distributed computing. Spark in particular is one where it lets you use a cluster of 100 computers to be as fast as 1 computer running good code. pyarrow -> Arrow.jl (there's also really good packages for JSON, CSV, HD5 and a bunch of others) Let me know if you have any other questions. Always glad to answer!
- Mehdi2277 5y ago
  Good spark support would be a good answer for batch/stream processing. I am a little scared of definition of support. Apache beam supports like 5 runners (flink, spark, data flow, etc) but the quality of runner support is extremely inconsistent. I’ve also noticed even for python flink sometimes have very useful operations only in Java with no wrapper. Although honestly having data pipelines in one language and downstream users of the data in a different language works pretty well in my experience so mixing data pipeline languages is somewhat ok. What’s workflow orchestration choice? That’s main one you didn’t touch. My work area in on an ml training platform and a lot of my work can be described as wrapper work on kubeflow to allow dozens of other ml engineers to manage experiments/workflows. For python the main choices are kubeflow/airflow. Ray kind of but ray workflows are still quite new and missing a lot of useful features. I need some system to run hundreds of ml workflows (one workflow being like 5-10 tasks some short some long) per day and manage there tasks well. Broader area also includes libraries like weights and biases, bento ml, etc (experimentation management libraries). In theory you can have workflow manager in one language and workflow code in a different language. Main downside is it makes debugging locally workflows harder (breakpoints are a little sad across most language boundaries), but it is doable and we debated migrating to temporal (Java workflow system) before.
  - adgjlsfhk1 5y ago

queuebert 5y ago

Python is the middle manager of languages. It sucks at everything, but always knows a guy.

dgfitz 5y ago
I saw a quote once that stuck with me: python is the second best language for everything, first at nothing.
- d0mine 5y ago
  What are the best languages for "executable pseudocode" or "glue language" roles?
antisthenes 5y ago
And there's 1 guy he knows that's a 10x developer and always willing to take one for the team. I'll leave it to HN to figure out what that means :P
- melony 5y ago
  Call native libraries?
  - isoprophlex 5y ago
    Write a metric shit ton of unmaintainable, underperforming pandas code and call it an effective solution

adgjlsfhk1 5y ago

One of the really nice things with Julia is some of the ecosystem needs disappear. Python needs a lot of ecosystem because none of the packages work together, and the language is slow so you have to make sure you are doing as much work as possible outside the language itself. To answer your question more specifically: Numpy -> Array + broadcasting (both in Julia Base) pytoch/tf -> Flux.jl (package) batch/stream processing -> you don't need it as much, but things like OnlineStats exist. Also Base has multithreaded and distributed computing. Spark in particular is one where it lets you use a cluster of 100 computers to be as fast as 1 computer running good code. pyarrow -> Arrow.jl (there's also really good packages for JSON, CSV, HD5 and a bunch of others) Let me know if you have any other questions. Always glad to answer!

Mehdi2277 5y ago
Good spark support would be a good answer for batch/stream processing. I am a little scared of definition of support. Apache beam supports like 5 runners (flink, spark, data flow, etc) but the quality of runner support is extremely inconsistent. I’ve also noticed even for python flink sometimes have very useful operations only in Java with no wrapper. Although honestly having data pipelines in one language and downstream users of the data in a different language works pretty well in my experience so mixing data pipeline languages is somewhat ok. What’s workflow orchestration choice? That’s main one you didn’t touch. My work area in on an ml training platform and a lot of my work can be described as wrapper work on kubeflow to allow dozens of other ml engineers to manage experiments/workflows. For python the main choices are kubeflow/airflow. Ray kind of but ray workflows are still quite new and missing a lot of useful features. I need some system to run hundreds of ml workflows (one workflow being like 5-10 tasks some short some long) per day and manage there tasks well. Broader area also includes libraries like weights and biases, bento ml, etc (experimentation management libraries). In theory you can have workflow manager in one language and workflow code in a different language. Main downside is it makes debugging locally workflows harder (breakpoints are a little sad across most language boundaries), but it is doable and we debated migrating to temporal (Java workflow system) before.
- adgjlsfhk1 5y ago