5 ms·

Meh. I haven't been able to get any speedup out of XLA, and it wasn't for the lack of trying. I will take TF seriously when it's at least as fast as PyTorch, an

by m0zg 7y ago

Meh. I haven't been able to get any speedup out of XLA, and it wasn't for the lack of trying. I will take TF seriously when it's at least as fast as PyTorch, and frugal enough with GPU ram to process batches of the same size as PyTorch. Right now, PyTorch simply blows it out of the water, and it doesn't have any advanced compiler backend or anything. Just lots good old fashioned performance engineering elbow grease, nothing fancy.

svantana 7y ago
Same here, I guess it's a complex function of hardware and what you're trying to do. Personally I ended up writing my own autodiffing tensor library in C++, because all existing solutions had abysmal performance on my problem (lots of local updates in large tensors). The speedup is >50x compared to TF, pytorch, julia, jax.
- thecleaner 7y ago
  > Personally I ended up writing my own autodiffing tensor library in C++ But this will not be a general library right ? You must have only included certain subset of functions of TF or PyTorch or whatever. Autodiffing is also included in certain proprietary libraries like ones from NAG. But I doubt its possible to achieve 50x speedup without compromising on functionality.
  - svantana 7y ago
    Of course it's a herculean task to write a library with that many features. But I don't think that's the issue, it's more that the devs of TF can't possibly optimize for every use case. For me, I knew what kind of ops I needed, so I could focus on getting those as fast as possible.
- mamcx 7y ago
  > lots of local updates in large tensors It mean you have efficient "crud" operators? This interest me because I'm building a relational language and toy with the idea of tensors as the "table" structure, but get rid of that because updates...
lamchob 7y ago
This works fine until it doesn't. See this paper (and the discussion) on why we need better tools: https://news.ycombinator.com/item?id=20301619 https://news.ycombinator.com/item?id=20301619 In short, you don't want to spend weeks on hand-optimized kernels, every time you want to implement a new type of layer for a large-scale network, or on an experimental target hardware.
- m0zg 7y ago
  As someone who has spent weeks on hand-optimized kernels, I've yet to see a compiler based system that comes anywhere near my kernels. I've tried them all, because writing and optimizing assembly on multiple platforms is hard AF. The highest performance approach (on CPU) seems to be JIT assembly (as used in Intel OpenVINO). On Intel, that beats the living Jesus out of everything else, including my work. On ARM it's a free for all, and there's no clear leader, especially in quantized inference. On GPU whatever it is NVIDIA is doing is the right thing to do. I'm skeptical that XLA/MLIR/TVM-like approaches can come close to, let alone exceed, the performance of hand-tuned kernels, for the same reason why hand-tuned assembly beats the shit of what compiler generates most of the time. I've yet to see it happen in practice. And you just need a few of those kernels, strategically placed where most of the computation happens, as per Pareto principle. For something like TF Google has the resources to get that done. It just chooses not to, to sell you more GPU-hours.
Loq 7y ago
Would you be able to sketch what makes PyTorch better than TF? And maybe why Google are sticking with TF -- it's not like Google lack technical ability to do / copy what PyTorch does.
- m0zg 7y ago
  It's basically imperative GPU accelerated NumPy with autograd and really nice libraries. You can be productive in it in a day (much less if you already know numpy), and if something goes wrong, it tells you what went wrong. It's also easily twice as fast as TF on GPU, trivial to use on multi-GPU setups, has an easy to use dataset interface, and allows you to run batches that are twice as large (which means faster training). Google certainly has _technical_ capacity to do what PyTorch does, but not the _organizational_ capacity to scrap TF and start over. So they're trying to half-ass it with TF 2.0. It still sucks though.
- chillee 7y ago
  There's a discussion here of sorts: https://www.reddit.com/r/MachineLearning/comments/cvcbu6/d_why_is_pytorch_as_fast_as_and_sometimes_faster/ https://www.reddit.com/r/MachineLearning/comments/cvcbu6/d_w... Perhaps Google could copy PyTorch. But Tensorflow has a lot of overhead, and for both political/technical reasons, there's no easy path to go from Tensorflow to Pytorch. You could just as easily ask: Why is Google sticking with {Hangouts/(Allo,Duo)/Angular} instead of doing/copying {Zoom/WhatsApp/React}? It's not like Google lacks the technical ability.

by m0zg 7y ago

svantana 7y ago

Same here, I guess it's a complex function of hardware and what you're trying to do. Personally I ended up writing my own autodiffing tensor library in C++, because all existing solutions had abysmal performance on my problem (lots of local updates in large tensors). The speedup is >50x compared to TF, pytorch, julia, jax.

thecleaner 7y ago
> Personally I ended up writing my own autodiffing tensor library in C++ But this will not be a general library right ? You must have only included certain subset of functions of TF or PyTorch or whatever. Autodiffing is also included in certain proprietary libraries like ones from NAG. But I doubt its possible to achieve 50x speedup without compromising on functionality.
- svantana 7y ago
  Of course it's a herculean task to write a library with that many features. But I don't think that's the issue, it's more that the devs of TF can't possibly optimize for every use case. For me, I knew what kind of ops I needed, so I could focus on getting those as fast as possible.
mamcx 7y ago
> lots of local updates in large tensors It mean you have efficient "crud" operators? This interest me because I'm building a relational language and toy with the idea of tensors as the "table" structure, but get rid of that because updates...

lamchob 7y ago

This works fine until it doesn't. See this paper (and the discussion) on why we need better tools: https://news.ycombinator.com/item?id=20301619 https://news.ycombinator.com/item?id=20301619 In short, you don't want to spend weeks on hand-optimized kernels, every time you want to implement a new type of layer for a large-scale network, or on an experimental target hardware.

m0zg 7y ago
As someone who has spent weeks on hand-optimized kernels, I've yet to see a compiler based system that comes anywhere near my kernels. I've tried them all, because writing and optimizing assembly on multiple platforms is hard AF. The highest performance approach (on CPU) seems to be JIT assembly (as used in Intel OpenVINO). On Intel, that beats the living Jesus out of everything else, including my work. On ARM it's a free for all, and there's no clear leader, especially in quantized inference. On GPU whatever it is NVIDIA is doing is the right thing to do. I'm skeptical that XLA/MLIR/TVM-like approaches can come close to, let alone exceed, the performance of hand-tuned kernels, for the same reason why hand-tuned assembly beats the shit of what compiler generates most of the time. I've yet to see it happen in practice. And you just need a few of those kernels, strategically placed where most of the computation happens, as per Pareto principle. For something like TF Google has the resources to get that done. It just chooses not to, to sell you more GPU-hours.

Loq 7y ago

Would you be able to sketch what makes PyTorch better than TF? And maybe why Google are sticking with TF -- it's not like Google lack technical ability to do / copy what PyTorch does.

m0zg 7y ago
It's basically imperative GPU accelerated NumPy with autograd and really nice libraries. You can be productive in it in a day (much less if you already know numpy), and if something goes wrong, it tells you what went wrong. It's also easily twice as fast as TF on GPU, trivial to use on multi-GPU setups, has an easy to use dataset interface, and allows you to run batches that are twice as large (which means faster training). Google certainly has _technical_ capacity to do what PyTorch does, but not the _organizational_ capacity to scrap TF and start over. So they're trying to half-ass it with TF 2.0. It still sucks though.
chillee 7y ago
There's a discussion here of sorts: https://www.reddit.com/r/MachineLearning/comments/cvcbu6/d_why_is_pytorch_as_fast_as_and_sometimes_faster/ https://www.reddit.com/r/MachineLearning/comments/cvcbu6/d_w... Perhaps Google could copy PyTorch. But Tensorflow has a lot of overhead, and for both political/technical reasons, there's no easy path to go from Tensorflow to Pytorch. You could just as easily ask: Why is Google sticking with {Hangouts/(Allo,Duo)/Angular} instead of doing/copying {Zoom/WhatsApp/React}? It's not like Google lacks the technical ability.