5 ms·

I like these kind of "Hello, world!" examples for TensorFlow. As a TensorFlow beginner, I need all the references I can get. Here is what I need right now: "He

I like these kind of "Hello, world!" examples for TensorFlow. As a TensorFlow beginner, I need all the references I can get. Here is what I need right now: "Hello, we meet again!". I can build a neural net model, and train (albeit, often badly) a model, but saving and restoring the trained weights so that I can run the model again is giving me fits. I am clearly missing something fundamental about how to restore a TensorFlow NN model. For your next tutorial, may I suggest: 1) a list of do's and don'ts for constructing a savable/restorable model, and 2) a wee bit of example code. Of course, now that I have discovered Keras I'm moving away from low-level direct TensorFlow. But I suspect I'm not the only one a bit foggy about the whole save/restore work flow.

pmalynin 10y ago
As a person that uses TensorFlow for his day job: I find that saving and restoring are of the weirder things with TensorFlow, you can either go all out an decide to save out all the variables, or only the ones needed for the model. You usually don't want to save out gradients (which are also variables) since they take up a bunch of space and aren't actually that useful to restore. Now on the other, what are model variables -- do you want to save model variables + the moving averages ... or just the averages. But then when you're loading you'll have to "shadow" the moving averages to the real variables that actually run in your model. Good news though, most of the scaffolding code you can write once and re-use it over and over again.
- jostmey 10y ago
  +1
  - jostmey 10y ago
    I would put together a tutorial on how models are saved and restored if I understood it, which is to say it still confuses me
- minimaxir 10y ago
  In Keras, it's just a simple model.save() [to a hdf5 file] and load_model(). This includes both the weights and the architecture. Models with a few million parameters result in a file around ~50MB, which is still reasonable for modern production use cases.
  - glial 10y ago
    Keras makes using deep learning for simple-ish use cases sooooo easy.
    - matheweis 10y ago
      I second this - I'm really excited about Keras being integrated into the core of Tensorflow (other than the chance it might lose the Torch compatibility).
hcrisp 10y ago
This save/restore tutorial was shared early by someone who is writing similar TensorFlow primers: https://blog.metaflow.fr/tensorflow-saving-restoring-and-mixing-multiple-models-c4c94d5d7125#.s3ktly5kg https://blog.metaflow.fr/tensorflow-saving-restoring-and-mix...
- dbcurtis 10y ago
  Great! Thanks, bookmarked.
maaaats 10y ago
> As a TensorFlow beginner, I need all the references I can get. I think that having to use a duck typed language makes it so much harder than it has to be. The API is huge, and you get really no help from the IDE. If I could do tf.train.GradientDescentOptimizer as in the example, and then get autocomplete to what I can do next with that object it would really help. Or list functions that takes that kind of object as input. Now, one is searching in the dark.
- Jach 10y ago
  This sounds like a problem with your IDE than with the language. Vim doesn't have this problem. See https://www.youtube.com/watch?v=TNMjbaimk9g https://www.youtube.com/watch?v=TNMjbaimk9g and I could probably find older systems that predate the plugins mentioned in the video.
allenlavoie 10y ago
A lot of the best practices for initializing/saving/restoring/etc. are handled automatically if you wrap your model in an Estimator[1] using a model_fn. It also enforces a "clean" model specification (in a new Graph) and decoupling of the input pipeline from the model. However, Estimators are not in TensorFlow core, which means the API isn't fixed quite yet. [1] https://www.tensorflow.org/extend/estimators https://www.tensorflow.org/extend/estimators

pmalynin 10y ago

As a person that uses TensorFlow for his day job: I find that saving and restoring are of the weirder things with TensorFlow, you can either go all out an decide to save out all the variables, or only the ones needed for the model. You usually don't want to save out gradients (which are also variables) since they take up a bunch of space and aren't actually that useful to restore. Now on the other, what are model variables -- do you want to save model variables + the moving averages ... or just the averages. But then when you're loading you'll have to "shadow" the moving averages to the real variables that actually run in your model. Good news though, most of the scaffolding code you can write once and re-use it over and over again.

jostmey 10y ago
+1
- jostmey 10y ago
  I would put together a tutorial on how models are saved and restored if I understood it, which is to say it still confuses me
minimaxir 10y ago
In Keras, it's just a simple model.save() [to a hdf5 file] and load_model(). This includes both the weights and the architecture. Models with a few million parameters result in a file around ~50MB, which is still reasonable for modern production use cases.
- glial 10y ago
  Keras makes using deep learning for simple-ish use cases sooooo easy.
  - matheweis 10y ago
    I second this - I'm really excited about Keras being integrated into the core of Tensorflow (other than the chance it might lose the Torch compatibility).

hcrisp 10y ago

This save/restore tutorial was shared early by someone who is writing similar TensorFlow primers: https://blog.metaflow.fr/tensorflow-saving-restoring-and-mixing-multiple-models-c4c94d5d7125#.s3ktly5kg https://blog.metaflow.fr/tensorflow-saving-restoring-and-mix...

dbcurtis 10y ago
Great! Thanks, bookmarked.

maaaats 10y ago

> As a TensorFlow beginner, I need all the references I can get. I think that having to use a duck typed language makes it so much harder than it has to be. The API is huge, and you get really no help from the IDE. If I could do tf.train.GradientDescentOptimizer as in the example, and then get autocomplete to what I can do next with that object it would really help. Or list functions that takes that kind of object as input. Now, one is searching in the dark.

Jach 10y ago
This sounds like a problem with your IDE than with the language. Vim doesn't have this problem. See https://www.youtube.com/watch?v=TNMjbaimk9g https://www.youtube.com/watch?v=TNMjbaimk9g and I could probably find older systems that predate the plugins mentioned in the video.

allenlavoie 10y ago

A lot of the best practices for initializing/saving/restoring/etc. are handled automatically if you wrap your model in an Estimator[1] using a model_fn. It also enforces a "clean" model specification (in a new Graph) and decoupling of the input pipeline from the model. However, Estimators are not in TensorFlow core, which means the API isn't fixed quite yet. [1] https://www.tensorflow.org/extend/estimators https://www.tensorflow.org/extend/estimators