4 ms·

Also, "Reasons to Switch from TensorFlow to CNTK": https://docs.microsoft.com/en-us/cognitive-toolkit/reasons-to-switch-from-tensorflow-to-cntk https://docs.mic

Also, "Reasons to Switch from TensorFlow to CNTK": https://docs.microsoft.com/en-us/cognitive-toolkit/reasons-to-switch-from-tensorflow-to-cntk https://docs.microsoft.com/en-us/cognitive-toolkit/reasons-t...

thr0waway1239 9y ago
"Many other toolkits create examples that appears to be identical to the original implementation of a paper, and then offer a converted model for people to download and evaluate. This is highly irresponsible in our opinion. In CNTK we pay a lot of attention in tracking down bugs and make sure the toolkit can be used to actually train the model from scratch and achieve state-of-the-art accuracy. One example is the Inception V3 network (https://arxiv.org/abs/1512.00567 https://arxiv.org/abs/1512.00567) originally developed by a few researchers at Google. Tensorflow shared the training script for Inception V3, and offered pre-trained models to download. However, no one were able to retrain the model and achieve the same accuracy with their script. The best accuracy that were achieved by a third party (Keras in this case) is about 0.6% worse that what the original paper reported. Researchers in the CNTK team worked hard and were able to train a CNTK Inception V3 model with 5.972% top 5 error, even better than the original paper reported! The training script is shared as an example (https://github.com/Microsoft/CNTK/tree/master/Examples/Image/Classification/GoogLeNet https://github.com/Microsoft/CNTK/tree/master/Examples/Image...) and you may verify it yourself." Ha ha, that does read like sour grapes. Is the idea that they will somehow provoke comparison with the 800lb gorilla, even at the expense of looking juvenile? What a bunch of whiny cry babies!
- KirinDave 9y ago
  Uhh... Classic hacker news right here. I'm fairly sure you don't realize what a major problem reproducible results is for the scientific and ML community, but it's big. A commitment to this and transparency is a pretty big deal.
- eob 9y ago
  As a former ML researcher, I read your quoted text and came away with the exact opposite conclusion. There is an incredible need as a community to work toward reproducible results. Pull a researcher aside, and I bet you every one will have at least one experience of trying to reproduce some reported--sometimes even lauded--result but being unable to. The good news is this kind of cultural fix is really invariant of compute platform or modeling framework. It's just nice that MSFT appears to be making it a priority.
  - cscurmudgeon 9y ago
    I have to agree with your comment fully. Read the comment above and went WTF.
75dvtwin 9y ago
I did not know the below situation with tensor flow. MS contributions to OSS, at least in this instant, appear a lot more transparent and not-self-centered, compared to Google's "...It was made very clear from the first day of TensorFlow’s announcement, that Google created two TensorFlow versions: a public version and an internal version. As a TensorFlow user, one either must tolerate the slow speed of the public version, or pay to run the TensorFlow job on Google’s cloud. ..."
- sja 9y ago
  I think that part is misleading. Vijay Vasudevan (a member of the TensorFlow team) has repeatedly put down the notion that the internal TensorFlow code is significantly different than what we see: https://www.reddit.com/r/MachineLearning/comments/696dzy/d_is_tensorflow_the_fastest_deep_learning_library/dh5cy84/ https://www.reddit.com/r/MachineLearning/comments/696dzy/d_i... Obviously, this is taking the word of someone who is incentivized to get as many people using TF as possible, but I haven't been given a reason to not believe him. I believe that the public TensorFlow repo is very close to what they use internally. That said, I'm sure there is a huge amount of internal tooling (for easily spinning up clusters internally, profiling, probably an automatic device placer) that we don't get to see. But that has more to do with the fact that its designed with Google's specific infrastructure in mind than it has to do with "hoarding the good stuff".
  - vrv 9y ago
    Thanks for mentioning that Sam, I appreciate it. Speaking for just myself: I personally don't care what tools or frameworks people use to get work done and have repeatedly suggested people use whatever works best for them. It also wouldn't make any sense to hoard any good stuff internally if we wanted to provide a useful framework that people wanted to use externally. We actually don't have a huge amount of tooling internally that we hold back. The only tooling I really use is the [timeline](https://github.com/tensorflow/tensorflow/blob/f488419cd6d9256b25ba25cbe736097dfeee79f9/tensorflow/python/platform/benchmark.py#L248 https://github.com/tensorflow/tensorflow/blob/f488419cd6d925...) for debugging performance (not the EEG tool in the whitepaper, I've never used it). That's available externally though, as you can see. On some of the comments in this thread in general, I'm pretty sad at the lack of scientific rigor in the community, and that goes for any person who publishes code that differs from the results they claim, regardless of affiliation. I am happy about projects like OpenAI's RL baselines, as I know others are too. Most of the papers and articles comparing performance of frameworks have lots of bugs and aren't even comparing the same model computation between frameworks. In fact, CNTK's article points to an external benchmark showing CNTK in a good light, but those benchmarks have bugs in them rendering the comparison incorrect (we've been sending PRs to fix them). I find it disappointing that the culture of the organization promotes calling others out for being 'irresponsible' except when it suits them. The TensorFlow team hasn't published many articles comparing performance directly to others because it's honestly a lot of hard work to verify that you are comparing fairly. Of course, I do think TensorFlow needs to improve performance out of the box for people, and the team is working on that.

thr0waway1239 9y ago

"Many other toolkits create examples that appears to be identical to the original implementation of a paper, and then offer a converted model for people to download and evaluate. This is highly irresponsible in our opinion. In CNTK we pay a lot of attention in tracking down bugs and make sure the toolkit can be used to actually train the model from scratch and achieve state-of-the-art accuracy. One example is the Inception V3 network (https://arxiv.org/abs/1512.00567 https://arxiv.org/abs/1512.00567) originally developed by a few researchers at Google. Tensorflow shared the training script for Inception V3, and offered pre-trained models to download. However, no one were able to retrain the model and achieve the same accuracy with their script. The best accuracy that were achieved by a third party (Keras in this case) is about 0.6% worse that what the original paper reported. Researchers in the CNTK team worked hard and were able to train a CNTK Inception V3 model with 5.972% top 5 error, even better than the original paper reported! The training script is shared as an example (https://github.com/Microsoft/CNTK/tree/master/Examples/Image/Classification/GoogLeNet https://github.com/Microsoft/CNTK/tree/master/Examples/Image...) and you may verify it yourself." Ha ha, that does read like sour grapes. Is the idea that they will somehow provoke comparison with the 800lb gorilla, even at the expense of looking juvenile? What a bunch of whiny cry babies!

KirinDave 9y ago
Uhh... Classic hacker news right here. I'm fairly sure you don't realize what a major problem reproducible results is for the scientific and ML community, but it's big. A commitment to this and transparency is a pretty big deal.
eob 9y ago
As a former ML researcher, I read your quoted text and came away with the exact opposite conclusion. There is an incredible need as a community to work toward reproducible results. Pull a researcher aside, and I bet you every one will have at least one experience of trying to reproduce some reported--sometimes even lauded--result but being unable to. The good news is this kind of cultural fix is really invariant of compute platform or modeling framework. It's just nice that MSFT appears to be making it a priority.
- cscurmudgeon 9y ago
  I have to agree with your comment fully. Read the comment above and went WTF.

75dvtwin 9y ago

I did not know the below situation with tensor flow. MS contributions to OSS, at least in this instant, appear a lot more transparent and not-self-centered, compared to Google's "...It was made very clear from the first day of TensorFlow’s announcement, that Google created two TensorFlow versions: a public version and an internal version. As a TensorFlow user, one either must tolerate the slow speed of the public version, or pay to run the TensorFlow job on Google’s cloud. ..."

sja 9y ago
I think that part is misleading. Vijay Vasudevan (a member of the TensorFlow team) has repeatedly put down the notion that the internal TensorFlow code is significantly different than what we see: https://www.reddit.com/r/MachineLearning/comments/696dzy/d_is_tensorflow_the_fastest_deep_learning_library/dh5cy84/ https://www.reddit.com/r/MachineLearning/comments/696dzy/d_i... Obviously, this is taking the word of someone who is incentivized to get as many people using TF as possible, but I haven't been given a reason to not believe him. I believe that the public TensorFlow repo is very close to what they use internally. That said, I'm sure there is a huge amount of internal tooling (for easily spinning up clusters internally, profiling, probably an automatic device placer) that we don't get to see. But that has more to do with the fact that its designed with Google's specific infrastructure in mind than it has to do with "hoarding the good stuff".
- vrv 9y ago
  Thanks for mentioning that Sam, I appreciate it. Speaking for just myself: I personally don't care what tools or frameworks people use to get work done and have repeatedly suggested people use whatever works best for them. It also wouldn't make any sense to hoard any good stuff internally if we wanted to provide a useful framework that people wanted to use externally. We actually don't have a huge amount of tooling internally that we hold back. The only tooling I really use is the [timeline](https://github.com/tensorflow/tensorflow/blob/f488419cd6d9256b25ba25cbe736097dfeee79f9/tensorflow/python/platform/benchmark.py#L248 https://github.com/tensorflow/tensorflow/blob/f488419cd6d925...) for debugging performance (not the EEG tool in the whitepaper, I've never used it). That's available externally though, as you can see. On some of the comments in this thread in general, I'm pretty sad at the lack of scientific rigor in the community, and that goes for any person who publishes code that differs from the results they claim, regardless of affiliation. I am happy about projects like OpenAI's RL baselines, as I know others are too. Most of the papers and articles comparing performance of frameworks have lots of bugs and aren't even comparing the same model computation between frameworks. In fact, CNTK's article points to an external benchmark showing CNTK in a good light, but those benchmarks have bugs in them rendering the comparison incorrect (we've been sending PRs to fix them). I find it disappointing that the culture of the organization promotes calling others out for being 'irresponsible' except when it suits them. The TensorFlow team hasn't published many articles comparing performance directly to others because it's honestly a lot of hard work to verify that you are comparing fairly. Of course, I do think TensorFlow needs to improve performance out of the box for people, and the team is working on that.