4 ms·

Concurrently. Each network has two sets of inputs, and I'm running about 480,000 data rows per channel, with about 23 features and 10 targets (these are not LL

Concurrently. Each network has two sets of inputs, and I'm running about 480,000 data rows per channel, with about 23 features and 10 targets (these are not LLMs). Each network is training on the same set of data, with slightly different architectures. I'm using Keras Tuner to help find the best architecture for each. They are being trained directly on the Max (not sure if I understood that question).

sargstuff 3y ago
Slow things down by running on gpu?[0] According to [0], apple doesn't support multiple-gpu's :( Multi-gpu setup (say one gpu per addon card) could allow for each tensorflow network to be run on it's own 'hardware box'/gpu card (and/or distribute/farm out the training to other hardware), In which case, the typical 'max hardware' just becomes a tty user interface layer to various gpu plug-in cards / processes distributed/farmed out. [0] : https://stackoverflow.com/questions/75236134/training-with-tensorflow-is-much-slower-using-gpu-rather-than-cpu-on-m1-max https://stackoverflow.com/questions/75236134/training-with-t... distributed tensorflow: Scaling TensorFlow Models on kubernetes : https://iamondemand.com/blog/scaling-tensorflow-models-on-kubernetes/ https://iamondemand.com/blog/scaling-tensorflow-models-on-ku... Introduction to distributed TensorFlow on Kubernets : https://techblog.cisco.com/blog/tensorflow-on-k8s https://techblog.cisco.com/blog/tensorflow-on-k8s https://medium.com/utility-machine-learning/a-guide-to-distributed-tensorflow-part-1-549b6b6fb55 https://medium.com/utility-machine-learning/a-guide-to-distr... https://medium.com/utility-machine-learning/a-guide-to-distributed-tensorflow-part-2-b459770e397b https://medium.com/utility-machine-learning/a-guide-to-distr...
- jzombie 3y ago
  I haven't made a comparison with GPU vs CPU. On GPU, my epochs were averaging about 20 minutes to complete, regardless of how many I had running concurrently. There isn't any other hardware involved other than this one machine; I imagine there was some throttling involved, but I didn't notice it, really, but I think it's because my model isn't that complex. ------- When I decided to try to do this, I had about 40% GPU usage while training one epoch, and the CPU load was around 35%. When I added a parallel run, I was getting around 25% GPU load on the other process, and the CPU load went up about 10%. The third run, I was getting about 9% GPU load on the process and the CPU load went up by another 5%. ------- This isn't something that I plan on doing continuously either, but I was partly debugging some things and figured I could squeeze in some extra runs. ------- Thanks for the links to the distributed TensorFlow. I may eventually get around to doing that but at the moment I'm merely trying to validate an idea that I have at minimal cost, and happen to have this machine around practically doing nothing else.

sargstuff 3y ago

Slow things down by running on gpu?[0] According to [0], apple doesn't support multiple-gpu's :( Multi-gpu setup (say one gpu per addon card) could allow for each tensorflow network to be run on it's own 'hardware box'/gpu card (and/or distribute/farm out the training to other hardware), In which case, the typical 'max hardware' just becomes a tty user interface layer to various gpu plug-in cards / processes distributed/farmed out. [0] : https://stackoverflow.com/questions/75236134/training-with-tensorflow-is-much-slower-using-gpu-rather-than-cpu-on-m1-max https://stackoverflow.com/questions/75236134/training-with-t... distributed tensorflow: Scaling TensorFlow Models on kubernetes : https://iamondemand.com/blog/scaling-tensorflow-models-on-kubernetes/ https://iamondemand.com/blog/scaling-tensorflow-models-on-ku... Introduction to distributed TensorFlow on Kubernets : https://techblog.cisco.com/blog/tensorflow-on-k8s https://techblog.cisco.com/blog/tensorflow-on-k8s https://medium.com/utility-machine-learning/a-guide-to-distributed-tensorflow-part-1-549b6b6fb55 https://medium.com/utility-machine-learning/a-guide-to-distr... https://medium.com/utility-machine-learning/a-guide-to-distributed-tensorflow-part-2-b459770e397b https://medium.com/utility-machine-learning/a-guide-to-distr...

jzombie 3y ago
I haven't made a comparison with GPU vs CPU. On GPU, my epochs were averaging about 20 minutes to complete, regardless of how many I had running concurrently. There isn't any other hardware involved other than this one machine; I imagine there was some throttling involved, but I didn't notice it, really, but I think it's because my model isn't that complex. ------- When I decided to try to do this, I had about 40% GPU usage while training one epoch, and the CPU load was around 35%. When I added a parallel run, I was getting around 25% GPU load on the other process, and the CPU load went up about 10%. The third run, I was getting about 9% GPU load on the process and the CPU load went up by another 5%. ------- This isn't something that I plan on doing continuously either, but I was partly debugging some things and figured I could squeeze in some extra runs. ------- Thanks for the links to the distributed TensorFlow. I may eventually get around to doing that but at the moment I'm merely trying to validate an idea that I have at minimal cost, and happen to have this machine around practically doing nothing else.