4 ms·

Slow things down by running on gpu?[0] According to [0], apple doesn't support multiple-gpu's :( Multi-gpu setup (say one gpu per addon card) could allow for e

Slow things down by running on gpu?[0] According to [0], apple doesn't support multiple-gpu's :( Multi-gpu setup (say one gpu per addon card) could allow for each tensorflow network to be run on it's own 'hardware box'/gpu card (and/or distribute/farm out the training to other hardware), In which case, the typical 'max hardware' just becomes a tty user interface layer to various gpu plug-in cards / processes distributed/farmed out. [0] : https://stackoverflow.com/questions/75236134/training-with-tensorflow-is-much-slower-using-gpu-rather-than-cpu-on-m1-max https://stackoverflow.com/questions/75236134/training-with-t... distributed tensorflow: Scaling TensorFlow Models on kubernetes : https://iamondemand.com/blog/scaling-tensorflow-models-on-kubernetes/ https://iamondemand.com/blog/scaling-tensorflow-models-on-ku... Introduction to distributed TensorFlow on Kubernets : https://techblog.cisco.com/blog/tensorflow-on-k8s https://techblog.cisco.com/blog/tensorflow-on-k8s https://medium.com/utility-machine-learning/a-guide-to-distributed-tensorflow-part-1-549b6b6fb55 https://medium.com/utility-machine-learning/a-guide-to-distr... https://medium.com/utility-machine-learning/a-guide-to-distributed-tensorflow-part-2-b459770e397b https://medium.com/utility-machine-learning/a-guide-to-distr...

jzombie 3y ago
I haven't made a comparison with GPU vs CPU. On GPU, my epochs were averaging about 20 minutes to complete, regardless of how many I had running concurrently. There isn't any other hardware involved other than this one machine; I imagine there was some throttling involved, but I didn't notice it, really, but I think it's because my model isn't that complex. ------- When I decided to try to do this, I had about 40% GPU usage while training one epoch, and the CPU load was around 35%. When I added a parallel run, I was getting around 25% GPU load on the other process, and the CPU load went up about 10%. The third run, I was getting about 9% GPU load on the process and the CPU load went up by another 5%. ------- This isn't something that I plan on doing continuously either, but I was partly debugging some things and figured I could squeeze in some extra runs. ------- Thanks for the links to the distributed TensorFlow. I may eventually get around to doing that but at the moment I'm merely trying to validate an idea that I have at minimal cost, and happen to have this machine around practically doing nothing else.

jzombie 3y ago

I haven't made a comparison with GPU vs CPU. On GPU, my epochs were averaging about 20 minutes to complete, regardless of how many I had running concurrently. There isn't any other hardware involved other than this one machine; I imagine there was some throttling involved, but I didn't notice it, really, but I think it's because my model isn't that complex. ------- When I decided to try to do this, I had about 40% GPU usage while training one epoch, and the CPU load was around 35%. When I added a parallel run, I was getting around 25% GPU load on the other process, and the CPU load went up about 10%. The third run, I was getting about 9% GPU load on the process and the CPU load went up by another 5%. ------- This isn't something that I plan on doing continuously either, but I was partly debugging some things and figured I could squeeze in some extra runs. ------- Thanks for the links to the distributed TensorFlow. I may eventually get around to doing that but at the moment I'm merely trying to validate an idea that I have at minimal cost, and happen to have this machine around practically doing nothing else.