7 ms·

This seems really cool, but I don't get why they would pour work into this while simultaneously building a C++ front-end for PyTorch[1]. Per the blog post, both

This seems really cool, but I don't get why they would pour work into this while simultaneously building a C++ front-end for PyTorch[1]. Per the blog post, both frameworks have the goal of empowering ML researchers to iterate on ML models in such a way that it becomes easier to reason about performance than it would be talking to a bunch of dynamically linked object files behind an interpreter. Facebook is a huge company with lots of money, but production-ready ML frameworks are a HUGE undertaking. I don't get how tech companies can be simultaneously recruiting and doing interviews year-round, paying huge salaries and then putting them under a layer of management that thinks two different C++ ML frameworks with the same goals is a good idea. Facebook's work on Tensor Comprehensions, Halide (not originally FB but they have contributed heavily), Glow, and PyTorch all contributed to the ML space by offering alternatives (with innovative UX/technical differences) to the Tensorflow ecosystem. Not all of these contributions had novelty, but I respect FB's choice to work on something for the sole purpose of not having it's direction beholden to the whims of Alphabet (see: Swift for Tensorflow). I just don't see what this adds which FB isn't already working on within a different project. What am I missing? 1 = https://pytorch.org/cppdocs/ https://pytorch.org/cppdocs/

whimsicalism 6y ago
Flashlight is much lower level and gives more fine-grained performance control. For instance, I don't think there is really any way to do real-time speech recognition that is fast with PyTorch because of how it is architected. From my understanding, Tensor Comprehensions and Halide are both very tentative research projects. > not having it's direction beholden to the whims of Alphabet (see: Swift for Tensorflow). I don't think this is an accurate recreation of the history that led to FB working on pytorch.
- dwrodri 6y ago
  This is a very interesting claim. I find it credible because it stands to reason that projects like DeepSpeed[1] and TorchScript[2] wouldn't need to exist if inference performance of research PyTorch models was satisfactory for production, but often case it isn't. It appears as though Flashlight is built on ArrayFire. I haven't seen how gradients are managed in arrayfire-ml, but perhaps it is the case that the autograd implementation in PyTorch was a bottleneck and this is a ground up approach. Editing as I didn't address your second point. I can neither confirm or deny the motivations for creating Torch being related to FB's desire to depend on an Alphabet-managed codebase. I know there are lots of reasons why programmers prefer the UX of the PyTorch Python API (I do as well), and there are probably other reasons I can't recall off the top of my head. I am only saying that PyTorch is contributing to the ML ecosystem already by the sole virtue that it isn't a Google product. [1] = https://github.com/microsoft/DeepSpeed https://github.com/microsoft/DeepSpeed [2] = https://pytorch.org/docs/stable/jit.html https://pytorch.org/docs/stable/jit.html
  - ipsum2 6y ago
    PyTorch is based off of Torch, which was first released in 2002, predating TensorFlow by 13 years. https://en.wikipedia.org/wiki/Torch_(machine_learning) https://en.wikipedia.org/wiki/Torch_(machine_learning)
  - liuliu 6y ago
    Torch (Lua) predates TensorFlow and is Lecun's pet project for a few years at that point already. But Lua as a language is unpopular at a time. PyTorch would be a welcome addition then. But even if no PyTorch (nor Caffe2) in an alternative timeline, I would imagine FB would be stuck with Lua Torch for quite some time.
    - tastyminerals2 6y ago
      Don’t forget that Lua Torch backend is C while Pytorch is C++ mostly.
- deleted 6y ago
  [deleted]
- sp-matcha 6y ago
  Adding to the above - Tensor Comprehensions was path-finding research and is no longer maintained. The git repo is frozen (archived) as a research artifact. Halide is still quite active, and was used in products at Adobe and Google circa 2016-2017. Not sure about the current state of industry usage though.
  - mgraczyk 6y ago
    Halide is still used by Google in a few different places.
  - whimsicalism 6y ago
    I'm unfamiliar with the word "path-finding" in this context. From the usage, is it similar to groundbreaking?
- lunixbochs 6y ago
  I ship an app (Talon) which runs many kinds of wav2letter ASR models on consumer CPUs. It has real-time inference pipelines for both pytorch and flashlight. (I wrote the pytorch code). Performance of both inference pipelines is fine and comparable between the frameworks for me. I'm not sure what you're talking about wrt speech recognition performance. Maybe for training? But there aren't that many cpu bound components there, and you can write those in native code.
  - whimsicalism 6y ago
    Fair enough, I've followed your stuff a bit and you definitely know more about Flashlight than I do. What would you say the motivation is for yet another ML framework, but in C++ this time?
UncleOxidant 6y ago
I tend to agree. If they said they had cleaned up the C++ backend that PyTorch uses and are releasing it separately on it's own with some usability enhancements as Flashlight, I could see that - they wouldn't have to maintain some other new thing. But this is apparently a completely separate thing.
coliveira 6y ago
Have you ever thought that it is not good for a big company like FB to be dependent on a single language/technology? This is the reason why these companies will invest in multiple solutions in different languages.
julienb_sea 6y ago
The documentation for flashlight focuses heavily on low level customizability and rapid rebuilding. My suspicion is that pytorch is fundamentally not designed around rapid iteration of the core model code. Flashlight targets a different use case. As far as Facebook engineering resources are concerned, I believe facebook takes the approach that supporting open source high visibility projects is itself a recruiting draw for pulling talented engineers into their umbrella.
davnicwil 6y ago
> huge company with lots of money... [yet] management that thinks two different C++ ML frameworks with the same goals is a good idea I don't see a contradiction here, in fact it makes sense. You have some goals, and a couple of promising approaches. It's hard to say which will work better, but there's enough budget and people to work on them to just try them both and see. I've heard anecdotes of similar strategies at banks, who have sufficient budget to hire two parallel teams to build literally the same product, sometimes without even knowing about each other. At the end, the one that ends up being faster/better gets used. I guess it's like a microcosm of free market competition within an org, as opposed to top down planning.
kwertyoowiyop 6y ago
Oh sweet summer child, it’s so refreshing that you think big famous companies always act logically!
trhway 6y ago
> paying huge salaries and then putting them under a layer of management that thinks two different C++ ML frameworks with the same goals is a good idea. FB makes $1.3M/employee while average pay is $120K, i.e. less than 10% - thus the employees are actually very cheap and FB can waste a lot of employee resource, like doing 10 ML frameworks, and even though ML related employees earn $1M+ it still wouldn't move the needle financially for FB.
brobdingnagians 6y ago
The end really helped me with understanding their motivations: > Flashlight’s modular internals make it a powerful research framework for research frameworks. > We’re already using Flashlight at Facebook in our research focused on developing a fast speech recognition pipeline, a threaded and customizable train-time relabeling pipeline for iterative pseudo-labeling, and a differentiable beam search decoder. > Our ongoing research is further accelerated by the ability to integrate external platform APIs for new hardware or compiler toolchains and achieve instant interoperability with the rest of Flashlight. They found it easier and faster to make a framework optimized for their research than to try to iterate on a larger and more complex codebase. I think there is less cognitive overhead and less things to change when experimenting with a new idea, and since they've optimized for fast rebuild times, it is much faster to try out new things. They get native integration with the C++ ecosystem for free and since their team all know C++ well, it makes sense to just do it in plain C++.
- dwrodri 6y ago
  I'm a bit late to the reply, but I wanted to thank you for the succinct reponse. This is the exact sort of thing I was looking for when I made my comment.
Narew 6y ago
btw, swift for tensorflow have been discontinued https://github.com/tensorflow/swift https://github.com/tensorflow/swift

whimsicalism 6y ago

Flashlight is much lower level and gives more fine-grained performance control. For instance, I don't think there is really any way to do real-time speech recognition that is fast with PyTorch because of how it is architected. From my understanding, Tensor Comprehensions and Halide are both very tentative research projects. > not having it's direction beholden to the whims of Alphabet (see: Swift for Tensorflow). I don't think this is an accurate recreation of the history that led to FB working on pytorch.

dwrodri 6y ago
This is a very interesting claim. I find it credible because it stands to reason that projects like DeepSpeed[1] and TorchScript[2] wouldn't need to exist if inference performance of research PyTorch models was satisfactory for production, but often case it isn't. It appears as though Flashlight is built on ArrayFire. I haven't seen how gradients are managed in arrayfire-ml, but perhaps it is the case that the autograd implementation in PyTorch was a bottleneck and this is a ground up approach. Editing as I didn't address your second point. I can neither confirm or deny the motivations for creating Torch being related to FB's desire to depend on an Alphabet-managed codebase. I know there are lots of reasons why programmers prefer the UX of the PyTorch Python API (I do as well), and there are probably other reasons I can't recall off the top of my head. I am only saying that PyTorch is contributing to the ML ecosystem already by the sole virtue that it isn't a Google product. [1] = https://github.com/microsoft/DeepSpeed https://github.com/microsoft/DeepSpeed [2] = https://pytorch.org/docs/stable/jit.html https://pytorch.org/docs/stable/jit.html
- ipsum2 6y ago
  PyTorch is based off of Torch, which was first released in 2002, predating TensorFlow by 13 years. https://en.wikipedia.org/wiki/Torch_(machine_learning) https://en.wikipedia.org/wiki/Torch_(machine_learning)
- liuliu 6y ago
  Torch (Lua) predates TensorFlow and is Lecun's pet project for a few years at that point already. But Lua as a language is unpopular at a time. PyTorch would be a welcome addition then. But even if no PyTorch (nor Caffe2) in an alternative timeline, I would imagine FB would be stuck with Lua Torch for quite some time.
  - tastyminerals2 6y ago
    Don’t forget that Lua Torch backend is C while Pytorch is C++ mostly.
deleted 6y ago
[deleted]
sp-matcha 6y ago
Adding to the above - Tensor Comprehensions was path-finding research and is no longer maintained. The git repo is frozen (archived) as a research artifact. Halide is still quite active, and was used in products at Adobe and Google circa 2016-2017. Not sure about the current state of industry usage though.
- mgraczyk 6y ago
  Halide is still used by Google in a few different places.
- whimsicalism 6y ago
  I'm unfamiliar with the word "path-finding" in this context. From the usage, is it similar to groundbreaking?
lunixbochs 6y ago
I ship an app (Talon) which runs many kinds of wav2letter ASR models on consumer CPUs. It has real-time inference pipelines for both pytorch and flashlight. (I wrote the pytorch code). Performance of both inference pipelines is fine and comparable between the frameworks for me. I'm not sure what you're talking about wrt speech recognition performance. Maybe for training? But there aren't that many cpu bound components there, and you can write those in native code.
- whimsicalism 6y ago
  Fair enough, I've followed your stuff a bit and you definitely know more about Flashlight than I do. What would you say the motivation is for yet another ML framework, but in C++ this time?

UncleOxidant 6y ago

I tend to agree. If they said they had cleaned up the C++ backend that PyTorch uses and are releasing it separately on it's own with some usability enhancements as Flashlight, I could see that - they wouldn't have to maintain some other new thing. But this is apparently a completely separate thing.

coliveira 6y ago

Have you ever thought that it is not good for a big company like FB to be dependent on a single language/technology? This is the reason why these companies will invest in multiple solutions in different languages.

julienb_sea 6y ago

The documentation for flashlight focuses heavily on low level customizability and rapid rebuilding. My suspicion is that pytorch is fundamentally not designed around rapid iteration of the core model code. Flashlight targets a different use case. As far as Facebook engineering resources are concerned, I believe facebook takes the approach that supporting open source high visibility projects is itself a recruiting draw for pulling talented engineers into their umbrella.

davnicwil 6y ago

> huge company with lots of money... [yet] management that thinks two different C++ ML frameworks with the same goals is a good idea I don't see a contradiction here, in fact it makes sense. You have some goals, and a couple of promising approaches. It's hard to say which will work better, but there's enough budget and people to work on them to just try them both and see. I've heard anecdotes of similar strategies at banks, who have sufficient budget to hire two parallel teams to build literally the same product, sometimes without even knowing about each other. At the end, the one that ends up being faster/better gets used. I guess it's like a microcosm of free market competition within an org, as opposed to top down planning.

kwertyoowiyop 6y ago

Oh sweet summer child, it’s so refreshing that you think big famous companies always act logically!

trhway 6y ago

> paying huge salaries and then putting them under a layer of management that thinks two different C++ ML frameworks with the same goals is a good idea. FB makes $1.3M/employee while average pay is $120K, i.e. less than 10% - thus the employees are actually very cheap and FB can waste a lot of employee resource, like doing 10 ML frameworks, and even though ML related employees earn $1M+ it still wouldn't move the needle financially for FB.

brobdingnagians 6y ago

The end really helped me with understanding their motivations: > Flashlight’s modular internals make it a powerful research framework for research frameworks. > We’re already using Flashlight at Facebook in our research focused on developing a fast speech recognition pipeline, a threaded and customizable train-time relabeling pipeline for iterative pseudo-labeling, and a differentiable beam search decoder. > Our ongoing research is further accelerated by the ability to integrate external platform APIs for new hardware or compiler toolchains and achieve instant interoperability with the rest of Flashlight. They found it easier and faster to make a framework optimized for their research than to try to iterate on a larger and more complex codebase. I think there is less cognitive overhead and less things to change when experimenting with a new idea, and since they've optimized for fast rebuild times, it is much faster to try out new things. They get native integration with the C++ ecosystem for free and since their team all know C++ well, it makes sense to just do it in plain C++.

dwrodri 6y ago
I'm a bit late to the reply, but I wanted to thank you for the succinct reponse. This is the exact sort of thing I was looking for when I made my comment.

Narew 6y ago

btw, swift for tensorflow have been discontinued https://github.com/tensorflow/swift https://github.com/tensorflow/swift