3 ms·

I don't want to deploy an app, I just want to play around with LLMs and don't want to go out and buy an expensive PC with a highend GPU just now. Is Fly.io a go

I don't want to deploy an app, I just want to play around with LLMs and don't want to go out and buy an expensive PC with a highend GPU just now. Is Fly.io a good way to go? What about alternatives?

leourbina 3y ago
Paperspace is a great way to go for this. You can start by just using their notebook product (similar to Collab), and you get to pick which type of machine/GPU it runs on. Once you have the code you want to run, you can rent machines on demand: https://www.paperspace.com/notebooks https://www.paperspace.com/notebooks
- janalsncm 3y ago
  I used paperspace for a while. Pretty cheap for mid tier gpu access (A6000 for example). There were a few things that annoyed me though. For one, I couldn’t access free GPUs with my team account. So I ended up quitting and buying a 4090 lol.
mrkurt 3y ago
You might actually be better off building a gaming rig and using that. The datacenter GPUs are silly expensive, because this is how NVIDIA price discriminates. The consumer, game GPUs work really well and you can buy them for almost as cheap as you can lease datacenter ones.
mrcwinn 3y ago
https://ollama.com/ https://ollama.com/ - Easy setup, run locally, free.
- UncleOxidant 3y ago
  Yeah, but I've got an RTX1070 in my circa 2017 PC. How well is that going to work?
  - thangngoc89 3y ago
    It's slow but still decent since it has 8GB of RAM.
  - jeswin 3y ago
    You mean GTX 1070. There's no RTX 1070.
nojs 3y ago
I can recommend runpod.io after a few months of usage - very easy to spin up different GPU configurations for testing and the pricing is simple and transparent. Using TheBloke docker images you can get most local models up and running in a few minutes.
ignoramous 3y ago
> What about alternatives? Custom models? Apart from the Big 3 (in no particular order): - https://together.ai/ https://together.ai/ - https://replicate.com/ https://replicate.com/ - https://anyscale.com/ https://anyscale.com/ - https://baseten.co/ https://baseten.co/ - https://modal.com/ https://modal.com/ - https://banana.dev/ https://banana.dev/ - https://runpod.io/ https://runpod.io/ - https://bentoml.com/ https://bentoml.com/ - https://brev.dev/ https://brev.dev/ - https://octo.ai/ https://octo.ai/ - https://cerebrium.ai/ https://cerebrium.ai/ ...
- ayewo 3y ago
  > Apart from the Big 3 ... Who are the big 3 in this context?
  - gk1 3y ago
    OpenAI, Anthropic, Cohere
mrb 3y ago
Use https://vast.ai https://vast.ai and rent a machine for as long as you need (minutes, hours, days). You pick the OS image, and you get a root shell to play with. An RTX 4090 currently costs $0.50 per hour. It literally took me less than 15 minutes to sign up for the first time a few weeks ago. For comparison, the first time experience on Amazon EC2 is much worse. I had tried to get a GPU instance on EC2 but couldn't reserve it (cryptic error message). Then I realized as a first-time EC2 user my default quota simply doesn't allow any GPU instances. After contacting support and waiting 4-5 days I eventually got a response my quota was increased, but I still can't launch a GPU instance... apparently my quota is still zero. At this point I gave up and found vast.ai. I don't know if Amazon realizes how FRUSTRATING their useless default quotas are for first-time EC2 users.
- janalsncm 3y ago
  Pretty much had the same experience with EC2 GPUs. No permission, had to contact support. Got permission a day later. I wanted to run on A100 ($30/hour, 8GPU minimum) but they were out of them that night. I tried again next day, same thing. So I gave up and used RunPod.io.
dathinab 3y ago
main question, do you need a A100? some use cases do so. but if not there are much cheaper consumer GPU based choices but then maybe you anyway just use it for 1-2 hours in total in which case the price difference might just not matter

I don't want to deploy an app, I just want to play around with LLMs and don't want to go out and buy an expensive PC with a highend GPU just now. Is Fly.io a good way to go? What about alternatives?

leourbina 3y ago

Paperspace is a great way to go for this. You can start by just using their notebook product (similar to Collab), and you get to pick which type of machine/GPU it runs on. Once you have the code you want to run, you can rent machines on demand: https://www.paperspace.com/notebooks https://www.paperspace.com/notebooks

janalsncm 3y ago
I used paperspace for a while. Pretty cheap for mid tier gpu access (A6000 for example). There were a few things that annoyed me though. For one, I couldn’t access free GPUs with my team account. So I ended up quitting and buying a 4090 lol.

mrkurt 3y ago

You might actually be better off building a gaming rig and using that. The datacenter GPUs are silly expensive, because this is how NVIDIA price discriminates. The consumer, game GPUs work really well and you can buy them for almost as cheap as you can lease datacenter ones.

mrcwinn 3y ago

https://ollama.com/ https://ollama.com/ - Easy setup, run locally, free.

UncleOxidant 3y ago
Yeah, but I've got an RTX1070 in my circa 2017 PC. How well is that going to work?
- thangngoc89 3y ago
  It's slow but still decent since it has 8GB of RAM.
- jeswin 3y ago
  You mean GTX 1070. There's no RTX 1070.

nojs 3y ago

I can recommend runpod.io after a few months of usage - very easy to spin up different GPU configurations for testing and the pricing is simple and transparent. Using TheBloke docker images you can get most local models up and running in a few minutes.

ignoramous 3y ago

> What about alternatives? Custom models? Apart from the Big 3 (in no particular order): - https://together.ai/ https://together.ai/ - https://replicate.com/ https://replicate.com/ - https://anyscale.com/ https://anyscale.com/ - https://baseten.co/ https://baseten.co/ - https://modal.com/ https://modal.com/ - https://banana.dev/ https://banana.dev/ - https://runpod.io/ https://runpod.io/ - https://bentoml.com/ https://bentoml.com/ - https://brev.dev/ https://brev.dev/ - https://octo.ai/ https://octo.ai/ - https://cerebrium.ai/ https://cerebrium.ai/ ...

ayewo 3y ago
> Apart from the Big 3 ... Who are the big 3 in this context?
- gk1 3y ago
  OpenAI, Anthropic, Cohere

mrb 3y ago

Use https://vast.ai https://vast.ai and rent a machine for as long as you need (minutes, hours, days). You pick the OS image, and you get a root shell to play with. An RTX 4090 currently costs $0.50 per hour. It literally took me less than 15 minutes to sign up for the first time a few weeks ago. For comparison, the first time experience on Amazon EC2 is much worse. I had tried to get a GPU instance on EC2 but couldn't reserve it (cryptic error message). Then I realized as a first-time EC2 user my default quota simply doesn't allow any GPU instances. After contacting support and waiting 4-5 days I eventually got a response my quota was increased, but I still can't launch a GPU instance... apparently my quota is still zero. At this point I gave up and found vast.ai. I don't know if Amazon realizes how FRUSTRATING their useless default quotas are for first-time EC2 users.

janalsncm 3y ago
Pretty much had the same experience with EC2 GPUs. No permission, had to contact support. Got permission a day later. I wanted to run on A100 ($30/hour, 8GPU minimum) but they were out of them that night. I tried again next day, same thing. So I gave up and used RunPod.io.

dathinab 3y ago

main question, do you need a A100? some use cases do so. but if not there are much cheaper consumer GPU based choices but then maybe you anyway just use it for 1-2 hours in total in which case the price difference might just not matter