5 ms·

I'm continually impressed by the Keras team's API design decisions. They clearly understand that usability should be the focus, and even just the small snippet

I'm continually impressed by the Keras team's API design decisions. They clearly understand that usability should be the focus, and even just the small snippet they post gets me excited about using the API. Compare this to CMLE's hyper-parameter tuning API: https://cloud.google.com/ml-engine/docs/tensorflow/using-hyperparameter-tuning https://cloud.google.com/ml-engine/docs/tensorflow/using-hyp... It's unclear though from the Readme how this tool works with distributed compute. I assume much of that would be abstracted away, and I'm not sure how you'd make it flexible enough to deal with all the various possible setups. One other aside: I assume that the Keras team has basically decided that Tensorflow is the only first class citizen in their ecosystem?

omrjml 7y ago
Francois Chollet has hinted at distributed being addressed. https://twitter.com/fchollet/status/1142515054647971840?s=19 https://twitter.com/fchollet/status/1142515054647971840?s=19
- jamesblonde 7y ago
  Will it be distributed like "distributed tensorflow" (which requires to supply all IPs/ports in a TF_CONFIG env variable!).
jamesblonde 7y ago
You can't just abstract away the distributed part of it. As far as I can see, it can't be easily distributed with existing data parallel frameworks (Spark,Flink,Beam,Dataflow). My other issue is that you are modifying the base (Mnist) code - and you will need to rewrite it again when you want to do (distributed) training. My final issue is that hyperparameters are spread throughout the code. Yes, this is better than having an external YAML file (more notebook friendly), but it is inferior to centralizing them in a single object/dict in the code. See my comments below for details.
dgacmu 7y ago
It's very easy - observe that each time the model is built it samples from the hyperparameters. So it's easy to have N workers all working in different hyperparameters. To coordinate the testing of hyperparameters, you use something like vizier, which is a service that the workers can call into to get their hyperparameter assignment. If you have a distributed model you have to make sure all of your workers on a single variant get the same HPs. Again, vizier. (Hi, I'm job 25 worker 3, what are my HPs?)

omrjml 7y ago

Francois Chollet has hinted at distributed being addressed. https://twitter.com/fchollet/status/1142515054647971840?s=19 https://twitter.com/fchollet/status/1142515054647971840?s=19

jamesblonde 7y ago
Will it be distributed like "distributed tensorflow" (which requires to supply all IPs/ports in a TF_CONFIG env variable!).

jamesblonde 7y ago

You can't just abstract away the distributed part of it. As far as I can see, it can't be easily distributed with existing data parallel frameworks (Spark,Flink,Beam,Dataflow). My other issue is that you are modifying the base (Mnist) code - and you will need to rewrite it again when you want to do (distributed) training. My final issue is that hyperparameters are spread throughout the code. Yes, this is better than having an external YAML file (more notebook friendly), but it is inferior to centralizing them in a single object/dict in the code. See my comments below for details.

dgacmu 7y ago

It's very easy - observe that each time the model is built it samples from the hyperparameters. So it's easy to have N workers all working in different hyperparameters. To coordinate the testing of hyperparameters, you use something like vizier, which is a service that the workers can call into to get their hyperparameter assignment. If you have a distributed model you have to make sure all of your workers on a single variant get the same HPs. Again, vizier. (Hi, I'm job 25 worker 3, what are my HPs?)