3 ms·

If I have to do MNIST with my custom images (either for training or inference) (png), what steps do I have to follow in here to get it working ?

xboost 9y ago
Hi, thanks for the comment, here's an example for a configuration file for a convolutional denoising autoencoder, where some preprocessing is applied during the input pipelines https://github.com/polyaxon/polyaxon/blob/master/examples/configs_examples/yaml_configs/denoising_conv_autoencoder.yml#L8 https://github.com/polyaxon/polyaxon/blob/master/examples/co...
- option_greek 9y ago
  Thanks xboost. Could you elaborate more on that. Is there a way to convert images files to TF records directly (as a pipeline parameter). Great project by the way! The easier it becomes to setup end to end training, the more easier it will be to use custom data.
  - xboost 9y ago
    Sure, we are preparing some ways to automotate datasets creation and versionning. For now, the way to feed data is through the built-in Tensorflow numpy/pandas input functions, there are some examples where this pipeline configuration is used. Otherwise there are different pipeline modules that could be used to feed the data especially for training where TF records could be faster. The way to create a TF record is still manuall, here's an image data converter that was used to create the mnist dataset: https://github.com/polyaxon/polyaxon/blob/master/polyaxon/datasets/converters.py https://github.com/polyaxon/polyaxon/blob/master/polyaxon/da... More data converters will be available soon. Once you have a record, numpy array, or a pandas Dataframe, you can basically use any operation/layer on your data by providing the feature name and the list of operation to apply. In general only operation that are necessary for data augmentation should be done on the input data pipelines, otherwise everything should be done during the creation of the TF records, to minimize the computations. One last thing for reinforcement learnign, the way we feed data is through the feed_dict, because an interaction with an environement is necessary.

xboost 9y ago

Hi, thanks for the comment, here's an example for a configuration file for a convolutional denoising autoencoder, where some preprocessing is applied during the input pipelines https://github.com/polyaxon/polyaxon/blob/master/examples/configs_examples/yaml_configs/denoising_conv_autoencoder.yml#L8 https://github.com/polyaxon/polyaxon/blob/master/examples/co...

option_greek 9y ago
Thanks xboost. Could you elaborate more on that. Is there a way to convert images files to TF records directly (as a pipeline parameter). Great project by the way! The easier it becomes to setup end to end training, the more easier it will be to use custom data.
- xboost 9y ago
  Sure, we are preparing some ways to automotate datasets creation and versionning. For now, the way to feed data is through the built-in Tensorflow numpy/pandas input functions, there are some examples where this pipeline configuration is used. Otherwise there are different pipeline modules that could be used to feed the data especially for training where TF records could be faster. The way to create a TF record is still manuall, here's an image data converter that was used to create the mnist dataset: https://github.com/polyaxon/polyaxon/blob/master/polyaxon/datasets/converters.py https://github.com/polyaxon/polyaxon/blob/master/polyaxon/da... More data converters will be available soon. Once you have a record, numpy array, or a pandas Dataframe, you can basically use any operation/layer on your data by providing the feature name and the list of operation to apply. In general only operation that are necessary for data augmentation should be done on the input data pipelines, otherwise everything should be done during the creation of the TF records, to minimize the computations. One last thing for reinforcement learnign, the way we feed data is through the feed_dict, because an interaction with an environement is necessary.