6 ms·

Wait why is renting a GPU in the cloud not a solution? You can even try multiple options and see which ones are capable enough for your use case. Look into som

Wait why is renting a GPU in the cloud not a solution? You can even try multiple options and see which ones are capable enough for your use case. Look into some barebones cloud GPU services, for example Lambda Labs which is significantly cheaper than AWS/GCP but offers basically nothing besides the machine with a GPU. You could even try something like Vast in which people rent out their personal GPU machines for cheap. Not something I'd use for uhhh...basically anything corporate, but for a personal project with no data security or uptime issues it would probably work great.

chaxor 3y ago
Cloud has a variable price (up to the whim of whatever they decide the price to be that day) so it's uncertain, but typically it is far" more expensive for this type of application. So when faced with 1) probably far more expensive, or 2) single price that will be cheaper, is always available, and has far more uses, I think most would choose 2) for self-hosting. Cloud is very rarely* a good option.
anonzzzies 3y ago
I have not seen many setups that wouldn’t pay itself back (including energy in my case) within a year (sometimes even 6 months) with buying vs renting. For something that pays itself back that fast, and that is without renting it out myself, just training with it, I cannot see how I would want to rent one. Edit; on Lambda labs, the only exception seems to be the H100; it would be 1.5 years or so, but even 2 years would still fast enough. I have an A100 which paid itself back; thinking of getting another one.
- fennecfoxy 3y ago
  I think the downside to buying hardware as well is that compared to other tech, this LLM/ML stuff is moving very quickly, people are great at quantising now (whereas I only really saw it done before for the coral edge TPUs etc). Someone could buy an H100 to run the biggest and bestest stuff right now, but we could find that a model gets shrunk down to run on a consumer card within a year or two with equivalent performance. I suppose it makes sense if someone wants to be on the bleeding edge all the time.
fnord77 3y ago
"but offers basically nothing besides the machine with a GPU" they must offer distributed storage, that can accommodate massive models, though? how else would you have multiple GPUs working on a single training model?
itake 3y ago
My annoyance was managing state. I’d have to spend hours installing tools, downloading data, updating code, then when I want to go to bed I have to package it up and store as much as I can on s3 before shutting off the $$ server.
- izuchukwu 3y ago
  You should check out https://brev.dev https://brev.dev. You can rent GPUs, pause instances, use your own AWS/GCP accounts to make use of your credits, and the CLI lets you use your GPU as if it’s on your local machine.
- throwaway6734 3y ago
  Docker?
- akiselev 3y ago
  You can try skypilot: https://skypilot.readthedocs.io/en/latest/ https://skypilot.readthedocs.io/en/latest/ It handles storage, setup, etc for machine learning work loads across several providers - which helps a lot if you need one of the instances that rarely have capacity like 8x A100 pods.
- orloffm 3y ago
  Aren't Vortex AI Workbooks on Google Cloud exactly the tool to solve this?
- sneak 3y ago
  This is what containers solve. Don't waste time manually installing things. Store state in a database via the app on a different host.
  - ElectricalUnion 3y ago
    I used to think that the 300MB containers I shipped to prd were reasonably unoptimized. Those AMD ROCm containers are like 14GiB compressed.
  - kunwon1 3y ago
nomel 3y ago
Besides some great tooling out there if you wanted to roll your own, you can literally rent windows/linux computers, with persistent disks. If you have good internet, you can even use it as a gaming PC, as I do.
- haliskerbas 3y ago
  What sort of frame rate and cost do you get? I have the highest tier GeForce now subscription and it sometimes drops to horrible conditions.
- itake 3y ago
  Is there an easy way to off-board the persistent disk to cheaper machines when you don't need the gpus? Like imagine, setting up and installing everything with the gpu attached, but when you're not using the gpu or all the cpu cores, you can disconnect them. If you have docs on how to do this, please let me know.
  - helsinkiandrew 3y ago
    With AWS (and probably most other cloud VPC services) the disk is remote from the hardware so you can halt the CPU and just pay for the storage until you restart. AWS also provides accessible datasets of training data: https://aws.amazon.com/marketplace/search/results?trk=868d8747-614e-4d4d-9fb6-fd5ac02947a8&sc_channel=el&FULFILLMENT_OPTION_TYPE=DATA_EXCHANGE&CONTRACT_TYPE=OPEN_DATA_LICENSES&filters=FULFILLMENT_OPTION_TYPE%2CCONTRACT_TYPE https://aws.amazon.com/marketplace/search/results?trk=868d87...
  - cmgbhm 3y ago
    https://towardsdatascience.com/stop-duplicating-deep-learning-training-datasets-with-amazon-ebs-multi-attach-d9f61fdc1de4 https://towardsdatascience.com/stop-duplicating-deep-learnin... This is basically the building blocks.
ginko 3y ago
Having to connect to a gpu over the internet seems extremely cumbersome. Stuff like this should be as easy as running a local program with an accelerator.

chaxor 3y ago

Cloud has a variable price (up to the whim of whatever they decide the price to be that day) so it's uncertain, but typically it is far" more expensive for this type of application. So when faced with 1) probably far more expensive, or 2) single price that will be cheaper, is always available, and has far more uses, I think most would choose 2) for self-hosting. Cloud is very rarely* a good option.

anonzzzies 3y ago

I have not seen many setups that wouldn’t pay itself back (including energy in my case) within a year (sometimes even 6 months) with buying vs renting. For something that pays itself back that fast, and that is without renting it out myself, just training with it, I cannot see how I would want to rent one. Edit; on Lambda labs, the only exception seems to be the H100; it would be 1.5 years or so, but even 2 years would still fast enough. I have an A100 which paid itself back; thinking of getting another one.

fennecfoxy 3y ago
I think the downside to buying hardware as well is that compared to other tech, this LLM/ML stuff is moving very quickly, people are great at quantising now (whereas I only really saw it done before for the coral edge TPUs etc). Someone could buy an H100 to run the biggest and bestest stuff right now, but we could find that a model gets shrunk down to run on a consumer card within a year or two with equivalent performance. I suppose it makes sense if someone wants to be on the bleeding edge all the time.

fnord77 3y ago

"but offers basically nothing besides the machine with a GPU" they must offer distributed storage, that can accommodate massive models, though? how else would you have multiple GPUs working on a single training model?

itake 3y ago

My annoyance was managing state. I’d have to spend hours installing tools, downloading data, updating code, then when I want to go to bed I have to package it up and store as much as I can on s3 before shutting off the $$ server.

izuchukwu 3y ago
You should check out https://brev.dev https://brev.dev. You can rent GPUs, pause instances, use your own AWS/GCP accounts to make use of your credits, and the CLI lets you use your GPU as if it’s on your local machine.
throwaway6734 3y ago
Docker?
akiselev 3y ago
You can try skypilot: https://skypilot.readthedocs.io/en/latest/ https://skypilot.readthedocs.io/en/latest/ It handles storage, setup, etc for machine learning work loads across several providers - which helps a lot if you need one of the instances that rarely have capacity like 8x A100 pods.
orloffm 3y ago
Aren't Vortex AI Workbooks on Google Cloud exactly the tool to solve this?
sneak 3y ago
This is what containers solve. Don't waste time manually installing things. Store state in a database via the app on a different host.
- ElectricalUnion 3y ago
  I used to think that the 300MB containers I shipped to prd were reasonably unoptimized. Those AMD ROCm containers are like 14GiB compressed.
- kunwon1 3y ago

nomel 3y ago

Besides some great tooling out there if you wanted to roll your own, you can literally rent windows/linux computers, with persistent disks. If you have good internet, you can even use it as a gaming PC, as I do.

haliskerbas 3y ago
What sort of frame rate and cost do you get? I have the highest tier GeForce now subscription and it sometimes drops to horrible conditions.
itake 3y ago
Is there an easy way to off-board the persistent disk to cheaper machines when you don't need the gpus? Like imagine, setting up and installing everything with the gpu attached, but when you're not using the gpu or all the cpu cores, you can disconnect them. If you have docs on how to do this, please let me know.
- helsinkiandrew 3y ago
  With AWS (and probably most other cloud VPC services) the disk is remote from the hardware so you can halt the CPU and just pay for the storage until you restart. AWS also provides accessible datasets of training data: https://aws.amazon.com/marketplace/search/results?trk=868d8747-614e-4d4d-9fb6-fd5ac02947a8&sc_channel=el&FULFILLMENT_OPTION_TYPE=DATA_EXCHANGE&CONTRACT_TYPE=OPEN_DATA_LICENSES&filters=FULFILLMENT_OPTION_TYPE%2CCONTRACT_TYPE https://aws.amazon.com/marketplace/search/results?trk=868d87...
- cmgbhm 3y ago
  https://towardsdatascience.com/stop-duplicating-deep-learning-training-datasets-with-amazon-ebs-multi-attach-d9f61fdc1de4 https://towardsdatascience.com/stop-duplicating-deep-learnin... This is basically the building blocks.

ginko 3y ago

Having to connect to a gpu over the internet seems extremely cumbersome. Stuff like this should be as easy as running a local program with an accelerator.