There are a few things that people getting started on ML do not know - deep networks are heavily dependent on random initialization - for production scale, it i

There are a few things that people getting started on ML do not know - deep networks are heavily dependent on random initialization - for production scale, it is generally a good idea to have multiple instances of the network initialized and use early-stopping as a way of weeding out poorly initialized network (which can be done in a nice parallel way). You can do either data or model parallelism - TensorFlow 2.x has nice utilities for data parallel. DeepSpeed from Microsoft helps with model parallel training. Slow parameter sync between nodes is a bottleneck.