4 ms·

>there's nothing stopping you from composing things in Python. In Julia you can compose a custom distribution, with a bayesian model with an ODE with a neural

by amkkma 6y ago

>there's nothing stopping you from composing things in Python. In Julia you can compose a custom distribution, with a bayesian model with an ODE with a neural network with unit number types with custom Julia written CUDA kernels and multithreading. Edit: That are not designed specifically to work with each other Can python even hope to do a fraction of that, still be fast and differentiate through everything?

swagonomixxx 6y ago
I'm fairly certain everything you said is possible except for custom CUDA kernels in pure Python. You'd have to write the kernel in C++ and use it in your TensorFlow/PyTorch code. [0][1] [0]: https://www.tensorflow.org/guide/create_op https://www.tensorflow.org/guide/create_op [1]: https://pytorch.org/tutorials/advanced/cpp_extension.html https://pytorch.org/tutorials/advanced/cpp_extension.html
- amkkma 6y ago
  It's definitely not possible. These have to be rewritten with the specific autodiff /ML framework in mind. Even then, you're not going to have fast custom types to be used on the GPU without dropping into C++
  - swagonomixxx 6y ago
    > It's definitely not possible. These have to be rewritten with the specific autodiff /ML framework in mind. I don't understand why it's not possible. You're asking if it's possible in the language. I don't see anything stopping you from, as you say, writing your own framework, or simply doing it in TF. Perhaps my ignorance of Julia is showing :)
    - chc 6y ago
      If you're rewriting everything to get the behavior rather than just hooking together preexisting pieces, that's not what's generally meant by composition.
    - socialdemocrat 6y ago
      Admittedly it is a bit hard to see why Julia is so different in this area if you have not spent some with it. It took me a bit time as well to see. But there really is a profound difference. The thing is that Python is really just a language to rearrange networks of nodes written in C++. A node written for one ML framework will be incompatible with a node written in another language. If you want them to work together you need to spend a lot of effort gluing them together, and most likely the performance benefits are lost. This is very different in Julia because you are using pure Julia code. What is specialized C++ nodes in Python, are for the most part just plain functions in Julia. There is nothing special about them. That is why they can be used anywhere. In Python frameworks each of these C++ nodes need to contain information about how auto-differentiation is done. That is why they are tailor made for one specific ML library. In Julia the whole automatic differentiation stuff is perform code transformations on regular Julia code. Are you familiar with LISP, and the idea of code as data? That is what we are talking about here. A regular syntax graph of regular Julia code gets manipulated to do autodiff. You are not organizing C++ nodes in a network.
      - swagonomixxx 6y ago
        > A regular syntax graph of regular Julia code gets manipulated to do autodiff. You are not organizing C++ nodes in a network. That does sound very different than Python. Admittedly, I haven't used Julia, because I never had to for professional purposes. For personal projects, I use PyTorch, because I find it easy to use, intuitive, and I have a lot of experience with it. Maybe one day I will be forced to use Julia and will appreciate what it brings :)
      - mark_l_watson 6y ago
        That was a great explanation that makes me appreciate Julia more, thanks. I don’t much use Julia, but it is a beautiful language. I started a tiny side project for using Julia for non-numerical things like querying SPARQL endpoints, text processing, etc. Julia could be a universal, use for almost everything language.
    - dnautics 6y ago
      Let's say you write some code that handles numbers differently, say a library that implements quaternion math. For doing graphics manipulations. Then you try to use it with tensorflow maybe for some ai-driven optimization. Is it going to work? Probably not out of the box. You will have to do some munging of quaternion data type to shove them into tensorflow tensors. Do the same thing in Julia with Julia's flux, it probably will, and maybe even with the gpu That's the difference.
    - coldtea 6y ago
      >I don't understand why it's not possible. You're asking if it's possible in the language. I don't see anything stopping you from, as you say, writing your own framework, or simply doing it in TF. Perhaps my ignorance of Julia is showing :) It's not that it's not possible. Yeah, you could always "write your own framework". It's that it's not readily available by the language, ready-made to compose, without resorting to external libs on C/C++/Fortran, and so on. So, "not possible" not in the sense that "it's not possible to write a program that solves the halting problem". More like "it's not possible to write a web app in assembly". Which means, yeah, it's possible, technically, but you wouldn't like it, it would be full of holes, a pain to maintain, and not a good time investment.
    - mirekrusin 6y ago
      They key concept that allows for this high degree of composability is multiple dispatch; it’s like if you were able to define trait for any function on any parameter type or to overwrite any existing function to specialise/generalise on one or more parameter types. Ie instead of wrapping different libraries to glue them together to work on your data, you update existing functions to accept your data, ie you blend your thing into existing things instead of wrapping. And once blended your data works not only with that library but also with libraries that use this library without extra effort and with very terse final code.
  - jl2718 6y ago
    Maybe Numba with Enzyme will catch up to Julia’s features before Julia catches up to Python’s users. I think the reason why people use python is because it’s the only universally-accepted language for coding interviews.
    - pjmlp 6y ago
      Only interviews for Python related jobs. I am not asking anyone to write Python when it isn't part of the job.
      - jl2718 6y ago
        If you are only asking interviewees to demonstrate skills related to the job, it seems like you are deeply in the minority.
- anon_tor_12345 6y ago
  It is possible https://numba.pydata.org/numba-doc/latest/cuda-reference/kernel.html https://numba.pydata.org/numba-doc/latest/cuda-reference/ker... which basically calls NVPTX https://github.com/numba/numba/blob/ec6fa07c12c3703a52c2b4ac8a225a68539c3cc8/numba/cuda/compiler.py#L71 https://github.com/numba/numba/blob/ec6fa07c12c3703a52c2b4ac... which is exactly what CUDA does anyway; CUDA code is actually a frontend for an ISA https://llvm.org/docs/NVPTXUsage.html https://llvm.org/docs/NVPTXUsage.html
  - swagonomixxx 6y ago
    Thanks, I wasn't aware this existed!
selectodude 6y ago
The outcome is more important than the code. While that sounds very elegant, what does it actually add?
- amkkma 6y ago
  https://www.reddit.com/r/Julia/comments/keuumi/bayesian_neural_ordinary_differential_equations/ https://www.reddit.com/r/Julia/comments/keuumi/bayesian_neur... See the paper. It allows for the ecosystem to have a combinatorial explosion of possibilities as each domain expert works on their own package. I can't foresee every application, but the emergence is the point. This is just a crazy example to show the bounds of what's possible.
- socialdemocrat 6y ago
  The Julia approach gives quicker development. Some of these ML libraries such as Flux are so small that almost anyone can read the code and learn how it works and make modifications. It is not for anyone to jump into TensorFlow. The irony here is that tiny Julia libraries give the same power as much larger Python libraries which require highly specialized and trained developers to evolve and maintain. With Julia it is much easier to grow the eco system because you can make lots of relatively small packages which can then be combined in almost any way. The challenge for the Julia community today is really to make people new to the environment aware of this. I have encountered people who thought Flux couldn't do anything because it was so small. He was so used to the huge monolithic libraries in Python, that it did not occur to him that adding something like an new activation function is literally one line of code. He is used to that requiring adding various C++ nodes. Wrap those up and God knows what more steps you need when you extend something like TensorFlow. I don't think most people realize how powerful Julia is. They are not used to being able to combine libraries the way you can do in Julia. A big part of the effort for the Julia community will simply be to write more tutorial and introduction which better introduce beginners to these kinds of capabilities. Once you know Julia it is pretty obvious. But grab people's attention and make them realize Julia could be a solution to their problem, I think one needs a lot of sort of shallow and quick intros to these kinds of things.
  - 6y ago

by amkkma 6y ago

swagonomixxx 6y ago

I'm fairly certain everything you said is possible except for custom CUDA kernels in pure Python. You'd have to write the kernel in C++ and use it in your TensorFlow/PyTorch code. [0][1] [0]: https://www.tensorflow.org/guide/create_op https://www.tensorflow.org/guide/create_op [1]: https://pytorch.org/tutorials/advanced/cpp_extension.html https://pytorch.org/tutorials/advanced/cpp_extension.html

amkkma 6y ago
It's definitely not possible. These have to be rewritten with the specific autodiff /ML framework in mind. Even then, you're not going to have fast custom types to be used on the GPU without dropping into C++
- swagonomixxx 6y ago
  > It's definitely not possible. These have to be rewritten with the specific autodiff /ML framework in mind. I don't understand why it's not possible. You're asking if it's possible in the language. I don't see anything stopping you from, as you say, writing your own framework, or simply doing it in TF. Perhaps my ignorance of Julia is showing :)
  - chc 6y ago
    If you're rewriting everything to get the behavior rather than just hooking together preexisting pieces, that's not what's generally meant by composition.
  - socialdemocrat 6y ago
    Admittedly it is a bit hard to see why Julia is so different in this area if you have not spent some with it. It took me a bit time as well to see. But there really is a profound difference. The thing is that Python is really just a language to rearrange networks of nodes written in C++. A node written for one ML framework will be incompatible with a node written in another language. If you want them to work together you need to spend a lot of effort gluing them together, and most likely the performance benefits are lost. This is very different in Julia because you are using pure Julia code. What is specialized C++ nodes in Python, are for the most part just plain functions in Julia. There is nothing special about them. That is why they can be used anywhere. In Python frameworks each of these C++ nodes need to contain information about how auto-differentiation is done. That is why they are tailor made for one specific ML library. In Julia the whole automatic differentiation stuff is perform code transformations on regular Julia code. Are you familiar with LISP, and the idea of code as data? That is what we are talking about here. A regular syntax graph of regular Julia code gets manipulated to do autodiff. You are not organizing C++ nodes in a network.
    - swagonomixxx 6y ago
      > A regular syntax graph of regular Julia code gets manipulated to do autodiff. You are not organizing C++ nodes in a network. That does sound very different than Python. Admittedly, I haven't used Julia, because I never had to for professional purposes. For personal projects, I use PyTorch, because I find it easy to use, intuitive, and I have a lot of experience with it. Maybe one day I will be forced to use Julia and will appreciate what it brings :)
    - mark_l_watson 6y ago
      That was a great explanation that makes me appreciate Julia more, thanks. I don’t much use Julia, but it is a beautiful language. I started a tiny side project for using Julia for non-numerical things like querying SPARQL endpoints, text processing, etc. Julia could be a universal, use for almost everything language.
  - dnautics 6y ago
    Let's say you write some code that handles numbers differently, say a library that implements quaternion math. For doing graphics manipulations. Then you try to use it with tensorflow maybe for some ai-driven optimization. Is it going to work? Probably not out of the box. You will have to do some munging of quaternion data type to shove them into tensorflow tensors. Do the same thing in Julia with Julia's flux, it probably will, and maybe even with the gpu That's the difference.
  - coldtea 6y ago
    >I don't understand why it's not possible. You're asking if it's possible in the language. I don't see anything stopping you from, as you say, writing your own framework, or simply doing it in TF. Perhaps my ignorance of Julia is showing :) It's not that it's not possible. Yeah, you could always "write your own framework". It's that it's not readily available by the language, ready-made to compose, without resorting to external libs on C/C++/Fortran, and so on. So, "not possible" not in the sense that "it's not possible to write a program that solves the halting problem". More like "it's not possible to write a web app in assembly". Which means, yeah, it's possible, technically, but you wouldn't like it, it would be full of holes, a pain to maintain, and not a good time investment.
  - mirekrusin 6y ago
    They key concept that allows for this high degree of composability is multiple dispatch; it’s like if you were able to define trait for any function on any parameter type or to overwrite any existing function to specialise/generalise on one or more parameter types. Ie instead of wrapping different libraries to glue them together to work on your data, you update existing functions to accept your data, ie you blend your thing into existing things instead of wrapping. And once blended your data works not only with that library but also with libraries that use this library without extra effort and with very terse final code.
- jl2718 6y ago
  Maybe Numba with Enzyme will catch up to Julia’s features before Julia catches up to Python’s users. I think the reason why people use python is because it’s the only universally-accepted language for coding interviews.
  - pjmlp 6y ago
    Only interviews for Python related jobs. I am not asking anyone to write Python when it isn't part of the job.
    - jl2718 6y ago
      If you are only asking interviewees to demonstrate skills related to the job, it seems like you are deeply in the minority.
anon_tor_12345 6y ago
It is possible https://numba.pydata.org/numba-doc/latest/cuda-reference/kernel.html https://numba.pydata.org/numba-doc/latest/cuda-reference/ker... which basically calls NVPTX https://github.com/numba/numba/blob/ec6fa07c12c3703a52c2b4ac8a225a68539c3cc8/numba/cuda/compiler.py#L71 https://github.com/numba/numba/blob/ec6fa07c12c3703a52c2b4ac... which is exactly what CUDA does anyway; CUDA code is actually a frontend for an ISA https://llvm.org/docs/NVPTXUsage.html https://llvm.org/docs/NVPTXUsage.html
- swagonomixxx 6y ago
  Thanks, I wasn't aware this existed!

selectodude 6y ago

The outcome is more important than the code. While that sounds very elegant, what does it actually add?

amkkma 6y ago
https://www.reddit.com/r/Julia/comments/keuumi/bayesian_neural_ordinary_differential_equations/ https://www.reddit.com/r/Julia/comments/keuumi/bayesian_neur... See the paper. It allows for the ecosystem to have a combinatorial explosion of possibilities as each domain expert works on their own package. I can't foresee every application, but the emergence is the point. This is just a crazy example to show the bounds of what's possible.
socialdemocrat 6y ago
The Julia approach gives quicker development. Some of these ML libraries such as Flux are so small that almost anyone can read the code and learn how it works and make modifications. It is not for anyone to jump into TensorFlow. The irony here is that tiny Julia libraries give the same power as much larger Python libraries which require highly specialized and trained developers to evolve and maintain. With Julia it is much easier to grow the eco system because you can make lots of relatively small packages which can then be combined in almost any way. The challenge for the Julia community today is really to make people new to the environment aware of this. I have encountered people who thought Flux couldn't do anything because it was so small. He was so used to the huge monolithic libraries in Python, that it did not occur to him that adding something like an new activation function is literally one line of code. He is used to that requiring adding various C++ nodes. Wrap those up and God knows what more steps you need when you extend something like TensorFlow. I don't think most people realize how powerful Julia is. They are not used to being able to combine libraries the way you can do in Julia. A big part of the effort for the Julia community will simply be to write more tutorial and introduction which better introduce beginners to these kinds of capabilities. Once you know Julia it is pretty obvious. But grab people's attention and make them realize Julia could be a solution to their problem, I think one needs a lot of sort of shallow and quick intros to these kinds of things.
- 6y ago