5 ms·

> Python has exceptional libraries but, as a language, it's a bit dated on several fronts. I've been using Python for ML for the last 3 years and I've never fe

> Python has exceptional libraries but, as a language, it's a bit dated on several fronts. I've been using Python for ML for the last 3 years and I've never felt this way. It might be that I'm not all about the hip new languages, but I don't really see the benefit of making Python more ML/Haskell-ish. The ML use case for Python is roughly as follows: you design, train, and evaluate models. Then, if something is decent enough to use in production, you switch over your application to load and use that instead. I don't really see where Haskell or any language from the ML family can improve in that process. Sure, the code that you used to implement your model may improve slightly, but I don't see that code improving significantly. The fruit of your labor is usually a protobuf file (encoding the TensorFlow graph) or whatever your framework uses to encode the model you built. The code actually surrounding it is very minimal for most use cases. > In Julia, things are really small and composable. For example, you have a probabilistic programming library like Turing and a differentiable programming one like Flux, and it's trivial to implement some Bayesian neural networks. There's nothing stopping you from composing things in Python. But it's simply not a goal of 99% of ML libraries to be composable with other libraries. You're probably never gonna run TensorFlow and PyTorch in the same application (trust me, I've tried, it was a nightmare) and I don't see why you would compose a TensorFlow model with a PyTorch model without incurring tons of overhead in your application around gluing these two things.

amkkma 6y ago
>there's nothing stopping you from composing things in Python. In Julia you can compose a custom distribution, with a bayesian model with an ODE with a neural network with unit number types with custom Julia written CUDA kernels and multithreading. Edit: That are not designed specifically to work with each other Can python even hope to do a fraction of that, still be fast and differentiate through everything?
- swagonomixxx 6y ago
  I'm fairly certain everything you said is possible except for custom CUDA kernels in pure Python. You'd have to write the kernel in C++ and use it in your TensorFlow/PyTorch code. [0][1] [0]: https://www.tensorflow.org/guide/create_op https://www.tensorflow.org/guide/create_op [1]: https://pytorch.org/tutorials/advanced/cpp_extension.html https://pytorch.org/tutorials/advanced/cpp_extension.html
  - amkkma 6y ago
    It's definitely not possible. These have to be rewritten with the specific autodiff /ML framework in mind. Even then, you're not going to have fast custom types to be used on the GPU without dropping into C++
    - swagonomixxx 6y ago
      > It's definitely not possible. These have to be rewritten with the specific autodiff /ML framework in mind. I don't understand why it's not possible. You're asking if it's possible in the language. I don't see anything stopping you from, as you say, writing your own framework, or simply doing it in TF. Perhaps my ignorance of Julia is showing :)
      - chc 6y ago
        If you're rewriting everything to get the behavior rather than just hooking together preexisting pieces, that's not what's generally meant by composition.
      - socialdemocrat 6y ago
        Admittedly it is a bit hard to see why Julia is so different in this area if you have not spent some with it. It took me a bit time as well to see. But there really is a profound difference. The thing is that Python is really just a language to rearrange networks of nodes written in C++. A node written for one ML framework will be incompatible with a node written in another language. If you want them to work together you need to spend a lot of effort gluing them together, and most likely the performance benefits are lost. This is very different in Julia because you are using pure Julia code. What is specialized C++ nodes in Python, are for the most part just plain functions in Julia. There is nothing special about them. That is why they can be used anywhere. In Python frameworks each of these C++ nodes need to contain information about how auto-differentiation is done. That is why they are tailor made for one specific ML library. In Julia the whole automatic differentiation stuff is perform code transformations on regular Julia code. Are you familiar with LISP, and the idea of code as data? That is what we are talking about here. A regular syntax graph of regular Julia code gets manipulated to do autodiff. You are not organizing C++ nodes in a network.
- selectodude 6y ago
  The outcome is more important than the code. While that sounds very elegant, what does it actually add?
  - amkkma 6y ago
    https://www.reddit.com/r/Julia/comments/keuumi/bayesian_neural_ordinary_differential_equations/ https://www.reddit.com/r/Julia/comments/keuumi/bayesian_neur... See the paper. It allows for the ecosystem to have a combinatorial explosion of possibilities as each domain expert works on their own package. I can't foresee every application, but the emergence is the point. This is just a crazy example to show the bounds of what's possible.
  - socialdemocrat 6y ago
    The Julia approach gives quicker development. Some of these ML libraries such as Flux are so small that almost anyone can read the code and learn how it works and make modifications. It is not for anyone to jump into TensorFlow. The irony here is that tiny Julia libraries give the same power as much larger Python libraries which require highly specialized and trained developers to evolve and maintain. With Julia it is much easier to grow the eco system because you can make lots of relatively small packages which can then be combined in almost any way. The challenge for the Julia community today is really to make people new to the environment aware of this. I have encountered people who thought Flux couldn't do anything because it was so small. He was so used to the huge monolithic libraries in Python, that it did not occur to him that adding something like an new activation function is literally one line of code. He is used to that requiring adding various C++ nodes. Wrap those up and God knows what more steps you need when you extend something like TensorFlow. I don't think most people realize how powerful Julia is. They are not used to being able to combine libraries the way you can do in Julia. A big part of the effort for the Julia community will simply be to write more tutorial and introduction which better introduce beginners to these kinds of capabilities. Once you know Julia it is pretty obvious. But grab people's attention and make them realize Julia could be a solution to their problem, I think one needs a lot of sort of shallow and quick intros to these kinds of things.
    - 6y ago
socialdemocrat 6y ago
> There's nothing stopping you from composing things in Python. There is, and you pointed it out yourself: > I don't see why you would compose a TensorFlow model with a PyTorch model without incurring tons of overhead in your application around gluing these two things. This is where Julia is profoundly different. Reusing stuff from different libraries is trivial compared to Python. You could take an activation function made for one libraries use it without any modification or wrapping in another ML library. These is where Julia will win long term. If you look at the ML libraries in Julia they are tiny. The Python side of things require gargantuan effort because things are not composable. The wheel is reinvented over and over again.
Karrot_Kream 6y ago
> I've been using Python for ML for the last 3 years and I've never felt this way. It might be that I'm not all about the hip new languages, but I don't really see the benefit of making Python more ML/Haskell-ish. Julia is not ML-ish. In fact, at first blush, Julia reads very similarly to Python. You even get list comprehensions in Julia! > Sure, the code that you used to implement your model may improve slightly, but I don't see that code improving significantly. The fruit of your labor is usually a protobuf file (encoding the TensorFlow graph) or whatever your framework uses to encode the model you built. The code actually surrounding it is very minimal for most use cases. It's really about breaking down the compositional boundaries here. Upthread, someone talked about how they could play around with Bayesian Nets by simply mixing Flux.jl (the differential programming library) with Turing.jl (the Bayesian programming library). Mixing Tensorflow/PyTorch with PyStan, for example, can be a nightmare. That's also why there's so many implementations of probabilistic programming frameworks (Edward, PyMC3, PyStan, Pyro, etc); they all use different underlying libraries. In Julia they just all compose together. > I don't see why you would compose a TensorFlow model with a PyTorch model without incurring tons of overhead in your application around gluing these two things I find the change to be illuminating. Before I started spending more time with Julia, I would often do derivations on paper for more experimental work, and then implement it using TF/PyTorch in Python from scratch, reading other code where I could for some help. In Julia I can import a library and I'm ready to go. It feels just like working with math itself. Julia also lets you compose long trains of operators together. That also helps when I'm doing long sessions in the REPL exploring data. It lets me define a few functions in a file, import the file, and just pipe (Julia has a pipe operator which is just function application) output between functions to plot data or calculate errors. Moreover Julia is a lot more performant for REPL work than Python. In Python I'll usually work with a subset of a dataset to get an idea for it, then run a file with code to process the entire dataset. In Julia I can often prototype and run the algorithm in the REPL itself. I also want to stress that Julia is quite a bit faster for REPL development, both in terms of raw speed

amkkma 6y ago

>there's nothing stopping you from composing things in Python. In Julia you can compose a custom distribution, with a bayesian model with an ODE with a neural network with unit number types with custom Julia written CUDA kernels and multithreading. Edit: That are not designed specifically to work with each other Can python even hope to do a fraction of that, still be fast and differentiate through everything?

swagonomixxx 6y ago
I'm fairly certain everything you said is possible except for custom CUDA kernels in pure Python. You'd have to write the kernel in C++ and use it in your TensorFlow/PyTorch code. [0][1] [0]: https://www.tensorflow.org/guide/create_op https://www.tensorflow.org/guide/create_op [1]: https://pytorch.org/tutorials/advanced/cpp_extension.html https://pytorch.org/tutorials/advanced/cpp_extension.html
- amkkma 6y ago
  It's definitely not possible. These have to be rewritten with the specific autodiff /ML framework in mind. Even then, you're not going to have fast custom types to be used on the GPU without dropping into C++
  - swagonomixxx 6y ago
    > It's definitely not possible. These have to be rewritten with the specific autodiff /ML framework in mind. I don't understand why it's not possible. You're asking if it's possible in the language. I don't see anything stopping you from, as you say, writing your own framework, or simply doing it in TF. Perhaps my ignorance of Julia is showing :)
    - chc 6y ago
      If you're rewriting everything to get the behavior rather than just hooking together preexisting pieces, that's not what's generally meant by composition.
    - socialdemocrat 6y ago
      Admittedly it is a bit hard to see why Julia is so different in this area if you have not spent some with it. It took me a bit time as well to see. But there really is a profound difference. The thing is that Python is really just a language to rearrange networks of nodes written in C++. A node written for one ML framework will be incompatible with a node written in another language. If you want them to work together you need to spend a lot of effort gluing them together, and most likely the performance benefits are lost. This is very different in Julia because you are using pure Julia code. What is specialized C++ nodes in Python, are for the most part just plain functions in Julia. There is nothing special about them. That is why they can be used anywhere. In Python frameworks each of these C++ nodes need to contain information about how auto-differentiation is done. That is why they are tailor made for one specific ML library. In Julia the whole automatic differentiation stuff is perform code transformations on regular Julia code. Are you familiar with LISP, and the idea of code as data? That is what we are talking about here. A regular syntax graph of regular Julia code gets manipulated to do autodiff. You are not organizing C++ nodes in a network.
selectodude 6y ago
The outcome is more important than the code. While that sounds very elegant, what does it actually add?
- amkkma 6y ago
  https://www.reddit.com/r/Julia/comments/keuumi/bayesian_neural_ordinary_differential_equations/ https://www.reddit.com/r/Julia/comments/keuumi/bayesian_neur... See the paper. It allows for the ecosystem to have a combinatorial explosion of possibilities as each domain expert works on their own package. I can't foresee every application, but the emergence is the point. This is just a crazy example to show the bounds of what's possible.
- socialdemocrat 6y ago
  The Julia approach gives quicker development. Some of these ML libraries such as Flux are so small that almost anyone can read the code and learn how it works and make modifications. It is not for anyone to jump into TensorFlow. The irony here is that tiny Julia libraries give the same power as much larger Python libraries which require highly specialized and trained developers to evolve and maintain. With Julia it is much easier to grow the eco system because you can make lots of relatively small packages which can then be combined in almost any way. The challenge for the Julia community today is really to make people new to the environment aware of this. I have encountered people who thought Flux couldn't do anything because it was so small. He was so used to the huge monolithic libraries in Python, that it did not occur to him that adding something like an new activation function is literally one line of code. He is used to that requiring adding various C++ nodes. Wrap those up and God knows what more steps you need when you extend something like TensorFlow. I don't think most people realize how powerful Julia is. They are not used to being able to combine libraries the way you can do in Julia. A big part of the effort for the Julia community will simply be to write more tutorial and introduction which better introduce beginners to these kinds of capabilities. Once you know Julia it is pretty obvious. But grab people's attention and make them realize Julia could be a solution to their problem, I think one needs a lot of sort of shallow and quick intros to these kinds of things.
  - 6y ago

socialdemocrat 6y ago

> There's nothing stopping you from composing things in Python. There is, and you pointed it out yourself: > I don't see why you would compose a TensorFlow model with a PyTorch model without incurring tons of overhead in your application around gluing these two things. This is where Julia is profoundly different. Reusing stuff from different libraries is trivial compared to Python. You could take an activation function made for one libraries use it without any modification or wrapping in another ML library. These is where Julia will win long term. If you look at the ML libraries in Julia they are tiny. The Python side of things require gargantuan effort because things are not composable. The wheel is reinvented over and over again.

Karrot_Kream 6y ago

> I've been using Python for ML for the last 3 years and I've never felt this way. It might be that I'm not all about the hip new languages, but I don't really see the benefit of making Python more ML/Haskell-ish. Julia is not ML-ish. In fact, at first blush, Julia reads very similarly to Python. You even get list comprehensions in Julia! > Sure, the code that you used to implement your model may improve slightly, but I don't see that code improving significantly. The fruit of your labor is usually a protobuf file (encoding the TensorFlow graph) or whatever your framework uses to encode the model you built. The code actually surrounding it is very minimal for most use cases. It's really about breaking down the compositional boundaries here. Upthread, someone talked about how they could play around with Bayesian Nets by simply mixing Flux.jl (the differential programming library) with Turing.jl (the Bayesian programming library). Mixing Tensorflow/PyTorch with PyStan, for example, can be a nightmare. That's also why there's so many implementations of probabilistic programming frameworks (Edward, PyMC3, PyStan, Pyro, etc); they all use different underlying libraries. In Julia they just all compose together. > I don't see why you would compose a TensorFlow model with a PyTorch model without incurring tons of overhead in your application around gluing these two things I find the change to be illuminating. Before I started spending more time with Julia, I would often do derivations on paper for more experimental work, and then implement it using TF/PyTorch in Python from scratch, reading other code where I could for some help. In Julia I can import a library and I'm ready to go. It feels just like working with math itself. Julia also lets you compose long trains of operators together. That also helps when I'm doing long sessions in the REPL exploring data. It lets me define a few functions in a file, import the file, and just pipe (Julia has a pipe operator which is just function application) output between functions to plot data or calculate errors. Moreover Julia is a lot more performant for REPL work than Python. In Python I'll usually work with a subset of a dataset to get an idea for it, then run a file with code to process the entire dataset. In Julia I can often prototype and run the algorithm in the REPL itself. I also want to stress that Julia is quite a bit faster for REPL development, both in terms of raw speed