5 ms·

Optimization (or any gradient based algorithm) is a good use case for AD, but I don’t see why Julia’s approaches are any better than Python’s, eg autograd, thea

Optimization (or any gradient based algorithm) is a good use case for AD, but I don’t see why Julia’s approaches are any better than Python’s, eg autograd, theano, pytorch etc. And sure that wouldn’t work with arbitrary Cython modules because Cython was designed as a Pythonic syntax over the Python C-API, and it just happened to become popular for numerical work. I don’t think that’s a strong argument, though, because anything small enough to be usable with AD can be rewritten without too much time lost, whereas those massive Fortran routines with iterative algorithms wouldn’t produce useful gradients in any language.

ChrisRackauckas 8y ago
>And sure that wouldn’t work with arbitrary Cython modules because Cython was designed as a Pythonic syntax over the Python C-API, and it just happened to become popular for numerical work. >I don’t think that’s a strong argument, though, because anything small enough to be usable with AD can be rewritten without too much time lost, whereas those massive Fortran routines with iterative algorithms wouldn’t produce useful gradients in any language. No. In Julia you can just stick the entire delay differential equation solver into the AD functions and get a gradient for parameter estimation. Saying you cannot use an arbitrary Cython code is a limitation, and saying you cannot put a random large code into AD is a limitation. It wouldn't be an issue if these weren't already solved problems, but having a performant software with simple and available AD is not something that's unreasonable anymore. If you use Julia, it's just something you can expect to work.
- marmaduke 8y ago
  No you can’t, at least, not if you want it work. This has nothing to do with Julia though and I didn’t say you can’t run AD over the solver but that that generally will not produce good gradient estimates unless the solver is written with AD in mind. DDEs are a mixed case where I’d expect some parts to be workable but not in general. Another example is the FFT, totally worthless to use AD.
Certhas 8y ago
In python I can chose whether to use autograd or numba, I can't JIT the autograded function. Seriously, read this issue: https://github.com/HIPS/autograd/issues/47 https://github.com/HIPS/autograd/issues/47
- marmaduke 8y ago
  I’m aware of that, but high performance, fat vector AD would be done with eg Theano or PyTorch, not autograd.
  - Certhas 8y ago
    That's the problem: You're locked into one particular library. You can not combine libraries without sacrificing massive performance. There is just no way around that. The numba story in that github issue mirrors my own experience: Excitement! This works! It's fast! Ok here are some limitations that I can work around. Hmm I would really like to use this library, in principle it should be possible to JIT its output/make it JIT compatible. In practice this turns out to be way to subtle. Ok I'm giving up, either I reimplement an algorithm directly in my hotloop or run slow code. So yeah, as long as your problems do not cross the domain of one package, Python and its ecosystem is great. Stay with it. But I fully expect that we'll see a lot more innovation in Julia. Already now there are classes of problems for which no Python solution exists but which actually have library support in Julia. That's really really remarkable. Despite my misgivings about the state of tutorials, the release handling process and the aproach to tooling, this is why I switched already. I also hope that all these aspects will improve post 1.0 massively. It's genuinely a liberation to no longer be confined to silos of DSLs that do not allow for low cost abstractions. It's fine if you don't need it. I just don't understand why you hang out in a thread about Julia insisting that I don't need it either and could just use numba + Python when that's exactly what I've been using prior to Julia.
    - marmaduke 8y ago
      > That's the problem: You're locked into one particular library. But that's not a issue resolved by any particular language; Julia appears to be free of lock-in because it hasn't had time to develop multiple, exclusive approaches to the same problems. Perhaps Julia builds into the language the ultimate performance solutions, so ok, then, for example, wait until there are N different web frameworks, and there you will find your silos. Python has many approaches to making things fast, which is why there are silos. /shrug > I switched already I probably would too if I was still a grad student. > why you hang out in a thread about Julia I was perusing whilst waiting for my Python code to complete, when I saw someone suggesting Python is already quite OK, catching some hate. I'm more than happy for the Julia community, but I think it's helpful to get exchanges accurate and critical.

ChrisRackauckas 8y ago

>And sure that wouldn’t work with arbitrary Cython modules because Cython was designed as a Pythonic syntax over the Python C-API, and it just happened to become popular for numerical work. >I don’t think that’s a strong argument, though, because anything small enough to be usable with AD can be rewritten without too much time lost, whereas those massive Fortran routines with iterative algorithms wouldn’t produce useful gradients in any language. No. In Julia you can just stick the entire delay differential equation solver into the AD functions and get a gradient for parameter estimation. Saying you cannot use an arbitrary Cython code is a limitation, and saying you cannot put a random large code into AD is a limitation. It wouldn't be an issue if these weren't already solved problems, but having a performant software with simple and available AD is not something that's unreasonable anymore. If you use Julia, it's just something you can expect to work.

marmaduke 8y ago
No you can’t, at least, not if you want it work. This has nothing to do with Julia though and I didn’t say you can’t run AD over the solver but that that generally will not produce good gradient estimates unless the solver is written with AD in mind. DDEs are a mixed case where I’d expect some parts to be workable but not in general. Another example is the FFT, totally worthless to use AD.

Certhas 8y ago

In python I can chose whether to use autograd or numba, I can't JIT the autograded function. Seriously, read this issue: https://github.com/HIPS/autograd/issues/47 https://github.com/HIPS/autograd/issues/47

marmaduke 8y ago
I’m aware of that, but high performance, fat vector AD would be done with eg Theano or PyTorch, not autograd.
- Certhas 8y ago
  That's the problem: You're locked into one particular library. You can not combine libraries without sacrificing massive performance. There is just no way around that. The numba story in that github issue mirrors my own experience: Excitement! This works! It's fast! Ok here are some limitations that I can work around. Hmm I would really like to use this library, in principle it should be possible to JIT its output/make it JIT compatible. In practice this turns out to be way to subtle. Ok I'm giving up, either I reimplement an algorithm directly in my hotloop or run slow code. So yeah, as long as your problems do not cross the domain of one package, Python and its ecosystem is great. Stay with it. But I fully expect that we'll see a lot more innovation in Julia. Already now there are classes of problems for which no Python solution exists but which actually have library support in Julia. That's really really remarkable. Despite my misgivings about the state of tutorials, the release handling process and the aproach to tooling, this is why I switched already. I also hope that all these aspects will improve post 1.0 massively. It's genuinely a liberation to no longer be confined to silos of DSLs that do not allow for low cost abstractions. It's fine if you don't need it. I just don't understand why you hang out in a thread about Julia insisting that I don't need it either and could just use numba + Python when that's exactly what I've been using prior to Julia.
  - marmaduke 8y ago
    > That's the problem: You're locked into one particular library. But that's not a issue resolved by any particular language; Julia appears to be free of lock-in because it hasn't had time to develop multiple, exclusive approaches to the same problems. Perhaps Julia builds into the language the ultimate performance solutions, so ok, then, for example, wait until there are N different web frameworks, and there you will find your silos. Python has many approaches to making things fast, which is why there are silos. /shrug > I switched already I probably would too if I was still a grad student. > why you hang out in a thread about Julia I was perusing whilst waiting for my Python code to complete, when I saw someone suggesting Python is already quite OK, catching some hate. I'm more than happy for the Julia community, but I think it's helpful to get exchanges accurate and critical.