8 ms·

Word is that internally at Google, among a few teams, and then also externally, Trax/Jax are putting up real competition to Tensorflow. Some teams have moved of

Word is that internally at Google, among a few teams, and then also externally, Trax/Jax are putting up real competition to Tensorflow. Some teams have moved off of tensorflow entirely. Combined with the better research capabilities of PyTorch, the future of tensorflow is not bright. Given that, Tensorflow still provides the highest performance with regards to production usage, and has tons of legacy code strewn throughout the web. I would argue that this is not the fault of Tensorflow, but rather the hazard of being the first implementation in an extremely complex space. Seems like usually there needs to be some sacrificial lamb in software domains. Somewhat like Map/Reduce was quickly replaced by Spark, which has no real competitors.

zitterbewegung 7y ago
I thought Theano was the first real deep learning implementation. What is the difference between that and tensorflow ?
- jhj 7y ago
  The first would be SN/Lush from circa 1987, of which Torch (from NEC Research), and then PyTorch are direct descendants in a lot of ways. https://leon.bottou.org/projects/lush https://leon.bottou.org/projects/lush
- codingslave 7y ago
  True, I was more so referencing first implementation to see wide scale adoption. But maybes there some holes in my narrative...
  - danieldk 7y ago
    Actually, Torch (2002) is older than Theano (2007). Moreover, both were already used widely in the research community. Tensorflow was actually late in the grand scheme of things, but replaced Theano and to some extend Torch (which used Lua for its public API). I think it's fairer to say that Tensorflow is the first implementation that had wide adoption outside academia.
m0zg 7y ago
>> highest performance with regards to production usage Not really, no. I've been using TensorRT for that quite successfully. If you can work around its limitations, I don't think anything can compete, at least not on modern NVIDIA GPUs. And yes, I'm aware TF can use TensorRT as well. But why drag in all of TF if you just want inference?
maiybe 7y ago
Overall, I've seen similar movement away from Tensorflow in my social circle of research scientists/engineers. One area I'd push back on is that "this is not the fault of Tensorflow." An area of weakness for Tensorflow is that it solves a number of DL problems with a specialized API call. That's not an asset, that's a liability. LSTMs were always a pain point. So much so that for Tensorflow projects, I gave up and insisted on traditional feedforward approaches like CNNs + MLPs or ResNets when LSTMs would be viable. Mostly identical performance with decent speed boosts from avoiding recurrence, and the simpler code reduced maintenance by non-ML engineers. As soon as you branch out of standard DL bread and butter models, you spend frustratingly long periods of time tracking down obscure solutions in a part of the API space that had its own hard-to-follow logic. Every time I'd point out that it's hard to do something either in forums or HN directly, I'd get a response that its easy to do with [insert-random-api] function call. In the end, it's my opinion that Tensorflow will lose out to JAX and Pytorch, by no fault other than its own complicated construction.
- sseveran 7y ago
  I agree with this, although I think it was a conscious and deliberate choice with TF 2.0. We have given up on TF for all future work which is sad since I really appreciate a number of the pieces that surround the core. I think they made a choice to emphasize the support of already developed models and make the experience great for novices will be a decision that they will come to regret. We found so many issues when we tried to port some of our existing models to TF 2.0. The sad part was that there were GitHub issues for all of them. Personally I think Tensorflow has already lost and we just need to let it play out over the next few years. One interesting wrinkle is that since Trax, Jax and Flax utilize pieces of Tensorflow the TF team can probably claim good internal adoption numbers depending on how they count.
- lowdose 7y ago
  Every obscure solution in Tensorflow also has a change to break at an upgrade, I'm glad I moved to pytorch.
logicchains 7y ago
>I would argue that this is not the fault of Tensorflow, but rather the hazard of being the first implementation in an extremely complex space. Seems like usually there needs to be some sacrificial lamb in software domains. I'd agree if Google didn't have a history of building things with (arguably) unnecessarily complex APIs, like Angular1. I remember when Angular and React were new, seeing an Angular "cheatsheat" that was around 14 pages; the equivalent React cheatsheet was only 2 pages. Now, I do love the idea of Tensorflow 1, essentially a functional DSL to explicitly construct computation graphs, but Google's implementation of that idea was suboptimal: hard-to-follow error messages, not intuitive, multiple APIs to do the same thing (and continual API breakage as new APIs are introduced), difficult to debug. And even if the graph "compiled" correctly, it could still fail at execution time. It's like they were building a programming language but lacked anyone with language design or PL theory background. Which makes sense given that anyone passionate about language design might prefer to work for somewhere closer to the cutting edge like Microsoft (C#, F#, Typescript, F*...), Facebook (Bucklescript, Hack), Apple (Swift) or Mozilla (Rust). Google does have its own languages, Dart and Go, but they're notable for ignoring and rejecting respectively cutting-edge PL theory (e.g. not disallowing null pointers, and in Go's case not even supporting parametric polymorphism). The day-to-day languages used at Google are also not particularly appealing to a PL enthusiast: Python, Java and non-modern C++. Google software often also seems to care more about enforcing their idea of "best practices" on the user than about user experience. Tensorflow's C++ support is an example of this: it requires using Babel. As Babel doesn't easily support integration into an existing C++ project, this essentially means you have to change your whole project over to Babel just to use Tensorflow C++, which is a huge amount of effort to go to just to use a library. Especially when it's probably quicker to just rewrite the model in PyTorch, which provides a simple header file and static library for linking, the standard way of distributing C++ libraries. PyTorch also provides a nicer C++ API, because it's not confined to the ancient C++ standards that Google enforces, so can provide a modern API that's not full of macros (macros in modern C++ are considered bad practice; should only be used when there's absolutely no alternative). To be fair, Google is really good at engineering language runtimes. Dart, Go and Tensorflow are all impressive works of engineering. They just seem to lack the organisational DNA for making them really nice to use, which maybe makes sense given the main source of their revenues is search/AdWords, the success of which is primarily driven by superior science/engineering. Compared to e.g. Facebook and Microsoft, that were/are in the business of making pretty things that users like to use (operating system, word processor, website). Or even comparing to Apple: people pay a huge premium for Apple phones over Android phones, in spite of their worse hardware, because of their more appealing design.
DrScientist 7y ago
> Somewhat like Map/Reduce was quickly replaced by Spark, which has no real competitors. I wouldn't say that - what about Flink? (https://flink.apache.org/ https://flink.apache.org/)
- sandGorgon 7y ago
  Actually Dask on the python side. Much higher performance if you were doing pyspark. Because of the serialisation cost
BenGosub 7y ago
This is an example of Peter Thiel's "the last mover advantage" principle at work.
joaogui1 7y ago
Any idea about flax? And how do flax and Trax interact?
zelly 7y ago
That's a shame, I liked tensorflow much better than the imperative alternatives. I guess it comes with the territory. Scientific software has to be dumbed down on the CS side because the users (grad students) typically never seriously programmed.

zitterbewegung 7y ago

I thought Theano was the first real deep learning implementation. What is the difference between that and tensorflow ?

jhj 7y ago
The first would be SN/Lush from circa 1987, of which Torch (from NEC Research), and then PyTorch are direct descendants in a lot of ways. https://leon.bottou.org/projects/lush https://leon.bottou.org/projects/lush
codingslave 7y ago
True, I was more so referencing first implementation to see wide scale adoption. But maybes there some holes in my narrative...
- danieldk 7y ago
  Actually, Torch (2002) is older than Theano (2007). Moreover, both were already used widely in the research community. Tensorflow was actually late in the grand scheme of things, but replaced Theano and to some extend Torch (which used Lua for its public API). I think it's fairer to say that Tensorflow is the first implementation that had wide adoption outside academia.

m0zg 7y ago

>> highest performance with regards to production usage Not really, no. I've been using TensorRT for that quite successfully. If you can work around its limitations, I don't think anything can compete, at least not on modern NVIDIA GPUs. And yes, I'm aware TF can use TensorRT as well. But why drag in all of TF if you just want inference?

maiybe 7y ago

Overall, I've seen similar movement away from Tensorflow in my social circle of research scientists/engineers. One area I'd push back on is that "this is not the fault of Tensorflow." An area of weakness for Tensorflow is that it solves a number of DL problems with a specialized API call. That's not an asset, that's a liability. LSTMs were always a pain point. So much so that for Tensorflow projects, I gave up and insisted on traditional feedforward approaches like CNNs + MLPs or ResNets when LSTMs would be viable. Mostly identical performance with decent speed boosts from avoiding recurrence, and the simpler code reduced maintenance by non-ML engineers. As soon as you branch out of standard DL bread and butter models, you spend frustratingly long periods of time tracking down obscure solutions in a part of the API space that had its own hard-to-follow logic. Every time I'd point out that it's hard to do something either in forums or HN directly, I'd get a response that its easy to do with [insert-random-api] function call. In the end, it's my opinion that Tensorflow will lose out to JAX and Pytorch, by no fault other than its own complicated construction.

sseveran 7y ago
I agree with this, although I think it was a conscious and deliberate choice with TF 2.0. We have given up on TF for all future work which is sad since I really appreciate a number of the pieces that surround the core. I think they made a choice to emphasize the support of already developed models and make the experience great for novices will be a decision that they will come to regret. We found so many issues when we tried to port some of our existing models to TF 2.0. The sad part was that there were GitHub issues for all of them. Personally I think Tensorflow has already lost and we just need to let it play out over the next few years. One interesting wrinkle is that since Trax, Jax and Flax utilize pieces of Tensorflow the TF team can probably claim good internal adoption numbers depending on how they count.
lowdose 7y ago
Every obscure solution in Tensorflow also has a change to break at an upgrade, I'm glad I moved to pytorch.

logicchains 7y ago

>I would argue that this is not the fault of Tensorflow, but rather the hazard of being the first implementation in an extremely complex space. Seems like usually there needs to be some sacrificial lamb in software domains. I'd agree if Google didn't have a history of building things with (arguably) unnecessarily complex APIs, like Angular1. I remember when Angular and React were new, seeing an Angular "cheatsheat" that was around 14 pages; the equivalent React cheatsheet was only 2 pages. Now, I do love the idea of Tensorflow 1, essentially a functional DSL to explicitly construct computation graphs, but Google's implementation of that idea was suboptimal: hard-to-follow error messages, not intuitive, multiple APIs to do the same thing (and continual API breakage as new APIs are introduced), difficult to debug. And even if the graph "compiled" correctly, it could still fail at execution time. It's like they were building a programming language but lacked anyone with language design or PL theory background. Which makes sense given that anyone passionate about language design might prefer to work for somewhere closer to the cutting edge like Microsoft (C#, F#, Typescript, F*...), Facebook (Bucklescript, Hack), Apple (Swift) or Mozilla (Rust). Google does have its own languages, Dart and Go, but they're notable for ignoring and rejecting respectively cutting-edge PL theory (e.g. not disallowing null pointers, and in Go's case not even supporting parametric polymorphism). The day-to-day languages used at Google are also not particularly appealing to a PL enthusiast: Python, Java and non-modern C++. Google software often also seems to care more about enforcing their idea of "best practices" on the user than about user experience. Tensorflow's C++ support is an example of this: it requires using Babel. As Babel doesn't easily support integration into an existing C++ project, this essentially means you have to change your whole project over to Babel just to use Tensorflow C++, which is a huge amount of effort to go to just to use a library. Especially when it's probably quicker to just rewrite the model in PyTorch, which provides a simple header file and static library for linking, the standard way of distributing C++ libraries. PyTorch also provides a nicer C++ API, because it's not confined to the ancient C++ standards that Google enforces, so can provide a modern API that's not full of macros (macros in modern C++ are considered bad practice; should only be used when there's absolutely no alternative). To be fair, Google is really good at engineering language runtimes. Dart, Go and Tensorflow are all impressive works of engineering. They just seem to lack the organisational DNA for making them really nice to use, which maybe makes sense given the main source of their revenues is search/AdWords, the success of which is primarily driven by superior science/engineering. Compared to e.g. Facebook and Microsoft, that were/are in the business of making pretty things that users like to use (operating system, word processor, website). Or even comparing to Apple: people pay a huge premium for Apple phones over Android phones, in spite of their worse hardware, because of their more appealing design.

DrScientist 7y ago

> Somewhat like Map/Reduce was quickly replaced by Spark, which has no real competitors. I wouldn't say that - what about Flink? (https://flink.apache.org/ https://flink.apache.org/)

sandGorgon 7y ago
Actually Dask on the python side. Much higher performance if you were doing pyspark. Because of the serialisation cost

BenGosub 7y ago

This is an example of Peter Thiel's "the last mover advantage" principle at work.

joaogui1 7y ago

Any idea about flax? And how do flax and Trax interact?

zelly 7y ago

That's a shame, I liked tensorflow much better than the imperative alternatives. I guess it comes with the territory. Scientific software has to be dumbed down on the CS side because the users (grad students) typically never seriously programmed.