3 ms·

Anyone training multiple ML networks on a single machine?

1 points by jzombie 3y ago

I have a Mac M1 Max that I'm training 3 TensorFlow networks at once and there seems to be plenty of juice left. Curious if there are any downsides to doing so.

sargstuff 3y ago
Training concurrently or non-concurrently? (aka in parallel or in series) Is training set 'sufficiently large' relative to intended use of a trained Tensorflow network? Are they all using/modifying the same training data input/output concurrently? Is Max just a terminal server for each of the tensorflow networks?
- jzombie 3y ago
  Concurrently. Each network has two sets of inputs, and I'm running about 480,000 data rows per channel, with about 23 features and 10 targets (these are not LLMs). Each network is training on the same set of data, with slightly different architectures. I'm using Keras Tuner to help find the best architecture for each. They are being trained directly on the Max (not sure if I understood that question).
  - sargstuff 3y ago
    Slow things down by running on gpu?[0] According to [0], apple doesn't support multiple-gpu's :( Multi-gpu setup (say one gpu per addon card) could allow for each tensorflow network to be run on it's own 'hardware box'/gpu card (and/or distribute/farm out the training to other hardware), In which case, the typical 'max hardware' just becomes a tty user interface layer to various gpu plug-in cards / processes distributed/farmed out. [0] : https://stackoverflow.com/questions/75236134/training-with-tensorflow-is-much-slower-using-gpu-rather-than-cpu-on-m1-max https://stackoverflow.com/questions/75236134/training-with-t... distributed tensorflow: Scaling TensorFlow Models on kubernetes : https://iamondemand.com/blog/scaling-tensorflow-models-on-kubernetes/ https://iamondemand.com/blog/scaling-tensorflow-models-on-ku... Introduction to distributed TensorFlow on Kubernets : https://techblog.cisco.com/blog/tensorflow-on-k8s https://techblog.cisco.com/blog/tensorflow-on-k8s https://medium.com/utility-machine-learning/a-guide-to-distributed-tensorflow-part-1-549b6b6fb55 https://medium.com/utility-machine-learning/a-guide-to-distr... https://medium.com/utility-machine-learning/a-guide-to-distributed-tensorflow-part-2-b459770e397b https://medium.com/utility-machine-learning/a-guide-to-distr...
    - jzombie 3y ago
      I haven't made a comparison with GPU vs CPU. On GPU, my epochs were averaging about 20 minutes to complete, regardless of how many I had running concurrently. There isn't any other hardware involved other than this one machine; I imagine there was some throttling involved, but I didn't notice it, really, but I think it's because my model isn't that complex. ------- When I decided to try to do this, I had about 40% GPU usage while training one epoch, and the CPU load was around 35%. When I added a parallel run, I was getting around 25% GPU load on the other process, and the CPU load went up about 10%. The third run, I was getting about 9% GPU load on the process and the CPU load went up by another 5%. ------- This isn't something that I plan on doing continuously either, but I was partly debugging some things and figured I could squeeze in some extra runs. ------- Thanks for the links to the distributed TensorFlow. I may eventually get around to doing that but at the moment I'm merely trying to validate an idea that I have at minimal cost, and happen to have this machine around practically doing nothing else.

1 points by jzombie 3y ago

I have a Mac M1 Max that I'm training 3 TensorFlow networks at once and there seems to be plenty of juice left. Curious if there are any downsides to doing so.

sargstuff 3y ago

Training concurrently or non-concurrently? (aka in parallel or in series) Is training set 'sufficiently large' relative to intended use of a trained Tensorflow network? Are they all using/modifying the same training data input/output concurrently? Is Max just a terminal server for each of the tensorflow networks?

jzombie 3y ago
Concurrently. Each network has two sets of inputs, and I'm running about 480,000 data rows per channel, with about 23 features and 10 targets (these are not LLMs). Each network is training on the same set of data, with slightly different architectures. I'm using Keras Tuner to help find the best architecture for each. They are being trained directly on the Max (not sure if I understood that question).
- sargstuff 3y ago
  Slow things down by running on gpu?[0] According to [0], apple doesn't support multiple-gpu's :( Multi-gpu setup (say one gpu per addon card) could allow for each tensorflow network to be run on it's own 'hardware box'/gpu card (and/or distribute/farm out the training to other hardware), In which case, the typical 'max hardware' just becomes a tty user interface layer to various gpu plug-in cards / processes distributed/farmed out. [0] : https://stackoverflow.com/questions/75236134/training-with-tensorflow-is-much-slower-using-gpu-rather-than-cpu-on-m1-max https://stackoverflow.com/questions/75236134/training-with-t... distributed tensorflow: Scaling TensorFlow Models on kubernetes : https://iamondemand.com/blog/scaling-tensorflow-models-on-kubernetes/ https://iamondemand.com/blog/scaling-tensorflow-models-on-ku... Introduction to distributed TensorFlow on Kubernets : https://techblog.cisco.com/blog/tensorflow-on-k8s https://techblog.cisco.com/blog/tensorflow-on-k8s https://medium.com/utility-machine-learning/a-guide-to-distributed-tensorflow-part-1-549b6b6fb55 https://medium.com/utility-machine-learning/a-guide-to-distr... https://medium.com/utility-machine-learning/a-guide-to-distributed-tensorflow-part-2-b459770e397b https://medium.com/utility-machine-learning/a-guide-to-distr...
  - jzombie 3y ago
    I haven't made a comparison with GPU vs CPU. On GPU, my epochs were averaging about 20 minutes to complete, regardless of how many I had running concurrently. There isn't any other hardware involved other than this one machine; I imagine there was some throttling involved, but I didn't notice it, really, but I think it's because my model isn't that complex. ------- When I decided to try to do this, I had about 40% GPU usage while training one epoch, and the CPU load was around 35%. When I added a parallel run, I was getting around 25% GPU load on the other process, and the CPU load went up about 10%. The third run, I was getting about 9% GPU load on the process and the CPU load went up by another 5%. ------- This isn't something that I plan on doing continuously either, but I was partly debugging some things and figured I could squeeze in some extra runs. ------- Thanks for the links to the distributed TensorFlow. I may eventually get around to doing that but at the moment I'm merely trying to validate an idea that I have at minimal cost, and happen to have this machine around practically doing nothing else.