3 ms·

Had the same reaction so had Grok create a script to: - find a free GPU droplet on digital ocean - fire it up - pull in a snapshot of the model + extra files

Had the same reaction so had Grok create a script to: - find a free GPU droplet on digital ocean - fire it up - pull in a snapshot of the model + extra files/packages etc - set up a ssh tunnel so that the localhost:8000 routes to the above Then I just configured OpenCode to use the above and was off to the races. Works out to be about ~$2/hr all said and done which isn't bad as I only pay when I'm using it (but could get expensive with 24/7 running)

b112 2mo ago
Awesome, I used Claude to write a small python script to do the same with Linode's API. The only difference is I setup a persistent drive, and with Linode you can boot off of it. So my biggest start up lag is ~ 2 minutes to deploy + boot, then maybe 2 more to warm the model. I actually dislike LLMs. But I'm a realist, and on-demand compute like this is massive cost saving measure. (persistent drives are relatively cheap, compared to a box with several GPUs.. or even one. I find it worth the expense)
- alexpotato 2mo ago
  > the only difference is I setup a persistent drive, Going to try this out vs the snapshot! I also really like this experiment b/c it's a mix of LLMs and old school IaaC/DevOps.
embedding-shape 2mo ago
> Works out to be about ~$2/hr all said What GPU you end up with for that price? Vast.ai (https://cloud.vast.ai/?priceInstanceHourlyMax=2 https://cloud.vast.ai/?priceInstanceHourlyMax=2) has a bunch of setups available to reach 192GB VRAM under $2 :) Quick skim showed 4x48, 2x96 and 8x24, all for under 2 buckaroos or around there.
- alexpotato 2mo ago
  It's actually a bit less than $2 but I rounded up just to make the math easier. And yeah, did a lot of work with Vast AI at a past job and it's pretty wild the variety of prices/hardware that they have.
  - embedding-shape 2mo ago
    > It's actually a bit less than $2 but I rounded up just to make the math easier. That's cool, what actual GPU though? I'm still curious :P
    - alexpotato 2mo ago
      gpu-6000adax1-48gb RTX 6000 Ada GPU Droplet - 1X

b112 2mo ago

Awesome, I used Claude to write a small python script to do the same with Linode's API. The only difference is I setup a persistent drive, and with Linode you can boot off of it. So my biggest start up lag is ~ 2 minutes to deploy + boot, then maybe 2 more to warm the model. I actually dislike LLMs. But I'm a realist, and on-demand compute like this is massive cost saving measure. (persistent drives are relatively cheap, compared to a box with several GPUs.. or even one. I find it worth the expense)

alexpotato 2mo ago
> the only difference is I setup a persistent drive, Going to try this out vs the snapshot! I also really like this experiment b/c it's a mix of LLMs and old school IaaC/DevOps.

embedding-shape 2mo ago

> Works out to be about ~$2/hr all said What GPU you end up with for that price? Vast.ai (https://cloud.vast.ai/?priceInstanceHourlyMax=2 https://cloud.vast.ai/?priceInstanceHourlyMax=2) has a bunch of setups available to reach 192GB VRAM under $2 :) Quick skim showed 4x48, 2x96 and 8x24, all for under 2 buckaroos or around there.

alexpotato 2mo ago
It's actually a bit less than $2 but I rounded up just to make the math easier. And yeah, did a lot of work with Vast AI at a past job and it's pretty wild the variety of prices/hardware that they have.
- embedding-shape 2mo ago
  > It's actually a bit less than $2 but I rounded up just to make the math easier. That's cool, what actual GPU though? I'm still curious :P
  - alexpotato 2mo ago
    gpu-6000adax1-48gb RTX 6000 Ada GPU Droplet - 1X