35 ms·
A lot of the optimisation at this level is getting data into the right place at the right time, without killing the network. Its also a group effort to provide
by KaiserPro 3y ago
A lot of the optimisation at this level is getting data into the right place at the right time, without killing the network.
Its also a group effort to provide simple to use primitives that "normal" ML people can use, even if they've never used hyper scale clusters before.
So you need a good scheduler, that understand dependencies (no, the k8s scheduler(s) are shit for this, plus it wont scale past 1k nodes without eating all of your network bandwidth), then you need a dataloader that can provide the dataset access, then you need the IPC that allows sharing/joining of GPUs together.
all of that needs to be wrapped up into a python interface that fairly simple to use.
Oh and it needs to be secure, pass an FCC audit (ie you need to prove that no user data is being used) have a high utilisation efficiency and uptime.
the model stuff is the cherry on the top
- jvanderbot 3y agoOk, but back to my main question, how do I get into this?
- willsmith72 3y agoIt looks more like an infra problem than ML. "Software architect"s mixed with devops/infra/sre people
- jvanderbot 3y agoWell since I'm not a ML engineer of any kind - that's good!
- zooq_ai 3y agoat the end of the day, you are still moving, storing and manipulating 1's and 0's, whether you are a front end engineer or a backend engineer or systems engieer or an ML engineer or an infra engineer
- elbear 3y agoyeah, but how do you get the hiring managers to see things in the same way? :)
- zooq_ai 3y agowell at least I fit my resume to match the 'job description' because at the end of the day it's all hallucinations and 'real' software engineers that has core computer science skills can literally do anything
- claytonjy 3y agocan you say more about the network issues with thousands of k8s nodes? I'm regularly running 2-3000 nodes in a GKE cluster, majority have GPUs, is this something I need to be worrying about?
- KaiserPro 3y agoOnly if you are paying for the network bandwidth. for example if there are nodes spanning more than one zone, and you pay for that traffic, you might want to think about moving stuff to a single zone. For other settings, moving to something like opencue might be better (caveats apply)