7 ms·

This article echoes my experience as well. I was working on some core NLP models for a larger tech company and wanted to experiment with Keras. I had my models

• Ar-Curunir 8y ago
  > Kind of reminds me of assembly programmers that thought C wasn't for l33t 10xx pwner programmers. It's funny because this the same attitude C/C++ programmers have towards developers using other languages now...
  • thewizardofaus 8y ago
    Pffft what are programming languages??? If you aren't writing code in straight up binary then you aren't a real h@k3r
    • tomrod 8y ago
      I think there is an emacs command to convert between that and butterfly wing flaps directing cosmic rays to flip bits. (h/t xckd)
  • Risord 8y ago
    It’s pretty universal thing...
• hknd 8y ago
  From my opinion: Getting started with Tensor Flow, and having a model designed within a day and training within another is also possible. This mostly depends on your model and your data, and (imho) not on the framework of choice. For all, Keras/PyTorch/Tensorflow, you'll need to learn the API - but if you have any ML background, that should be straight forward.
  • stared 8y ago
    Yes, for all practical problems data is the biggest challenge. Though, debugging matters. In TF it is easy to get errors and spend a lot of time searching for them. In PyTorch it is straightforward. It matters the most when the network, or cost function, is not standard (think: YOLO architecture). E.g. when I wanted to write some differentiable decision tree it took me way longer in TF (I already knew) than with PyTorch, having its tutorial on another pane.
    • lord_ring_111 8y ago
      Tensorflow also does not work without needing to build urself for my 2010 or so cpu. So i ended up trying pytorch and i am glad i did. Liking it better than esoteric errors.
    • bitL 8y ago
      TensorFlow needs some Deep Learning-based assistant to identify common cause of errors you might see on the AST level. Cryptic errors are its weakness and an AI trained to spot correlations between Python AST and error might be very helpful.
• danieldk 8y ago
  I was also told that doing it the real way using Tensorflow would be the way to go and I agree with that sentiment if my problem was Google scale which it wasn't. Use the right tool for the job. Keras can get you to a working model faster. However, I am not sure what the current situation is, but in the past it was not possible to dump and freeze Keras' Tensorflow graphs. This can be a problem if you want to embed a model in a non-Python application. This attitude of "real deep learning engineers use Tensorflow" Real engineers use whatever they need to use. But I think that you are overstating the difficulty of Tensorflow. Over the last 6 months, we have hired a couple of students for a research project. Since we standardized on Tensorflow, they had to implement new models in Tensorflow. All of them were up to speed in Tensorflow pretty quickly (they mostly do RNNs and seq2seq learning).
  • candiodari 8y ago
    > dump and freeze Keras' Tensorflow graphs You can get a direct reference to the graphs if you want, that will let you do anything tensorflow lets you do. I think this is what you want: # This assumes your model is ready to be called with .predict() sess = keras.get_session() graph = sess.graph graph_dev = graph.as_graph_def() frozen_graph = tf.graph_util.convert_variables_to_constants( sess, graph_def, nodes_to_output) encoded_frozen_graph = frozen_graph.SerializeToString()
    • danieldk 8y ago
      That didn't work before, but admittedly, the last time I tried was probably 1.5 years ago.
  • Eridrus 8y ago
    > in the past it was not possible to dump and freeze Keras' Tensorflow graphs. This was never true. There was no obvious Keras API for this, but you could build a model with the Keras API, then use the TF API to save it. The inference API would be the TF API (i.e. you'd need to find the names of all your input and output tensors and use those with Session.run).
    • 8y ago
• amelius 8y ago
  > Kind of reminds me of assembly programmers that thought C wasn't for l33t 10xx pwner programmers. The problem with TensorFlow is mainly that you, as a user, have to build a data-dependency graph. This is something a C compiler can do very well, but Python is not so suitable for that. So, in my view, TensorFlow chose the wrong substrate for their "more efficient" library. Instead, they should have developed their own language, where the whole data-flow graph determination could be implicit, and not a concern for the programmer. However, computing a data-flow graph as-you-go (by the library, not the user), like (I think) is done in some libraries, is quite a good approach, since the overhead is quite small (percentage-wise) compared to the large tensor operations that can be performed in highly optimized code.
  • sanxiyn 8y ago
    > So, in my view, TensorFlow chose the wrong substrate for their "more efficient" library. Instead, they should have developed their own language, where the whole data-flow graph determination could be implicit, and not a concern for the programmer. You just described Swift for TensorFlow.
    • codetrotter 8y ago
      > You just described Swift for TensorFlow This one? https://www.tensorflow.org/api_docs/swift/ https://www.tensorflow.org/api_docs/swift/
    • SalimoS 8y ago
      isn't work in progress ? https://github.com/tensorflow/swift/blob/master/docs/WhySwiftForTensorFlow.md https://github.com/tensorflow/swift/blob/master/docs/WhySwif...
    • amelius 8y ago
      Well, apparently :) They say: We believe that machine learning tools are so important that they deserve a first-class language and a compiler. However, I'd like to see some numbers on how more efficient it is to build a graph in advance, given that the lion's share of the computations will be in tensor math anyway (which can be heavily optimized, and is independent of the graph).
• bitL 8y ago
  Try to run multiple models/ensemble training on many computers with many GPUs to pick up the best performing model or combo. TensorFlow so far has probably the easiest approach for it. That might be reason for the attitude "real deep learning engineers use Tensorflow", as other approaches either don't scale that well or you can't even model something you need for your bleeding-edge billion $-making approach, despite other frameworks being much much simpler/more natural and a joy to use.
  • wodenokoto 8y ago
    Funny, I dumped tensor flow a few years ago because it wasn't possible to do bleeding edge stuff.
    • bitL 8y ago
      Yeah, for many bleeding edge things you still need PyTorch ;-)
      • riku_iki 8y ago
        Maybe you have a chance to provide example?
  • mlthoughts2018 8y ago
    Most of the tools that TensorFlow offers for multi-gpu and distributed model training will "just work" directly with Keras models too, or with really minor tweaks. You can even easily mix and match pure TensorFlow code (like explicitly setting the device with a device placement context manager) with Keras code. See e.g. [0] and [1] linked below. For model ensembling, it's even easier. After training, in Keras you could simply load your multiple models and create a new Model() object that does nothing but use a merge layer (with mode set to averaging) to average across multiple input models, even if the models share layers or have other crazy constraints. Writing that final ensemble is extremely easy in Keras. In my experience researching and productionizing very deep Keras models for an image processing use case that has moderately tight performance constraints, Keras has proved to scale extremely well and the code remains dead simple the whole time. [0]: < https://blog.keras.io/keras-as-a-simplified-interface-to-tensorflow-tutorial.html#multi-gpu-and-distributed-training https://blog.keras.io/keras-as-a-simplified-interface-to-ten... > [1]: < https://www.tensorflow.org/programmers_guide/estimators#creating_estimators_from_keras_models https://www.tensorflow.org/programmers_guide/estimators#crea... >
• inputcoffee 8y ago
  As others are pointing out, TF isn't that hard. Or, rather, it is hard but the difficulty is from getting an intuition for what part of this weird multi layer net is producing this weird behavior and is it an artefact or something interesting, and is the connectivity complete and is should I change the learning rate and activation functions? The real reason to use Tensorflow is the same reason you might use a Go framework instead of Rails: in your heart you have this hope that this thing will one day grow into a really large project and support lots of people and that will be easier with this scalable, optimized code. Its not even that you'll hit Google scale, its that you'll hit popular scale and still serve the whole thing out of your Digital Ocean droplet.
  • rahimnathwani 8y ago
    "Its not even that you'll hit Google scale, its that you'll hit popular scale and still serve the whole thing out of your Digital Ocean droplet." Are you saying that model inference is slower or less efficient for a model built and trained in Keras, than the same model architecture built directly in tensorflow?
    • inputcoffee 8y ago
      Actually, with Tensorflow as a Keras backend, I would expect them to be the same. I am not sure where the performance difference between TF and TF as a backend come from. I do think that pure TF would be easier to scale up over multiple servers etc. but that's only because I don't know how it would work in Keras. Maybe its easy.
      • eanzenberg 8y ago
        Its pretty straightforward to convert a keras model to a tf graph, as long as you used a tf backend in keras.
    • crucialfelix 8y ago
      I would think the difference would be from the data input pipeline, efficiency in batching, updating online models. The inference itself would be the exact same.
• sandGorgon 8y ago
  Did you look at Tensorflow Estimators ? They are a new high-level API with built in support for distributed training. https://www.tensorflow.org/programmers_guide/estimators https://www.tensorflow.org/programmers_guide/estimators
  • alfalfasprout 8y ago
    Yes, they're pretty ugly TBH. All they've done is provide some decent "canned" estimators but for anything custom you're still using the base tensorflow API. Not to mention feeding in something like numpy arrays > 2GB is a huge pain (their Dataset API doesn't fully work).
    • sandGorgon 8y ago
      Interesting. So do you recommend Keras+Tf as well, or drop Tensorflow altogether ?
      • alfalfasprout 8y ago
        Keras + MxNet is far better. It's faster, you get multi-gpu out of the box, and it's reaaaally easy to export to ONNX for fast inference elsewhere.
• Eridrus 8y ago
  I used Keras a few years ago, and really liked it and still recommend it to people, even contributing some code back to it, but I don't think it obviated the need to know TF, since eventually you want to do something that's not in the Keras toolbox, and then you need to understand what it's doing under the hood.
• eanzenberg 8y ago
  >>I was also told that doing it the real way using Tensorflow would be the way to go and I agree with that sentiment if my problem was Google scale which it wasn't. In fact I would argue that most workloads around the world are not Google scale and neither are most Google workloads. You can convert the Keras model to TF pretty easily if you need to, as long as you use the TF backend. I did this, and converted the string preprocessing in TF so the model could be used in TF serving taking only the string as input.