3 ms·

I'm running Vicuna on a free 4core Oracle VPS, and it's perfectly usable for a Discord bot. Responses rarely take more than 15 seconds with <256 max token limit

I'm running Vicuna on a free 4core Oracle VPS, and it's perfectly usable for a Discord bot. Responses rarely take more than 15 seconds with <256 max token limit, and the responses are much more entertaining than GPT 3.5. I'm not using the streaming API my server software[0] offers, but if I did it would probably load somewhere between the speeds of GPT-3.5 and GPT-4. It's more or less the same time a human would take to compose the same message. So... not exactly a serious use-case. But it's what I'm using, and now I'm saving 10s of dollars on inferencing costs per month! [0] https://github.com/go-skynet/LocalAI https://github.com/go-skynet/LocalAI I'm also using this to improve acceleration - https://cloudmarketplace.oracle.com/marketplace/en_US/adf.task-flow?tabName=O&adf.tfDoc=%2FWEB-INF%2Ftaskflow%2Fadhtf.xml&application_id=125935163&adf.tfId=adhtf https://cloudmarketplace.oracle.com/marketplace/en_US/adf.ta...

paulgb 3y ago
Now I’m curious what your bot does!
- smoldesu 3y ago
  It's one of those "say a keyword with a question, get a response" type bots. I added in a few other "prompt sources" though, where it grabs the first part of an RSS entry or HN comment and tries to autocomplete the rest. Mostly just a boring testbed for me to play with models, for free, with friends.
fareesh 3y ago
TIL Oracle has VPS offerings with a free tier. Are they any good? Is the free-tier time limited? This use-case is alright for a toy I guess - which is the extent that I was originally expecting these things to be useful for.
- smoldesu 3y ago
  They're okay. This isn't the place for a full review of their offerings (especially considering everyone's mixed feelings on Oracle), but I'm confident that it's better than most 1core/$5 deals you'll find elsewhere. > Are they any good? Yep, free tier allows you to spec up to 24gb of RAM without paying, which is cool. The bottleneck is really the disk speed, but that's not an issue with mmaped models. There's enough cached memory that it loads instantly, so it's good-ish for this use case. > Is the free-tier time limited? No, but there are a lot of strings attached: - The cores are vCPUs, not dedi (duh) - You can't create new instances when demand is high (unless you add a credit card) - Technically Oracle reserves the right to shut down the instance if demand gets really high (although I haven't heard any stories about this personally) Proceed with caution. It's still a great place to start before you shell out $1/hr for dedi GPU rackspace.
  - tylergetsay 3y ago
    They will shut down the VPS if there's no activity on it, not sure how they detect this though
    - ukuina 3y ago
      CPU idle % over a rolling time window (say, 15 minutes).

paulgb 3y ago

Now I’m curious what your bot does!

smoldesu 3y ago
It's one of those "say a keyword with a question, get a response" type bots. I added in a few other "prompt sources" though, where it grabs the first part of an RSS entry or HN comment and tries to autocomplete the rest. Mostly just a boring testbed for me to play with models, for free, with friends.

fareesh 3y ago

TIL Oracle has VPS offerings with a free tier. Are they any good? Is the free-tier time limited? This use-case is alright for a toy I guess - which is the extent that I was originally expecting these things to be useful for.

smoldesu 3y ago
They're okay. This isn't the place for a full review of their offerings (especially considering everyone's mixed feelings on Oracle), but I'm confident that it's better than most 1core/$5 deals you'll find elsewhere. > Are they any good? Yep, free tier allows you to spec up to 24gb of RAM without paying, which is cool. The bottleneck is really the disk speed, but that's not an issue with mmaped models. There's enough cached memory that it loads instantly, so it's good-ish for this use case. > Is the free-tier time limited? No, but there are a lot of strings attached: - The cores are vCPUs, not dedi (duh) - You can't create new instances when demand is high (unless you add a credit card) - Technically Oracle reserves the right to shut down the instance if demand gets really high (although I haven't heard any stories about this personally) Proceed with caution. It's still a great place to start before you shell out $1/hr for dedi GPU rackspace.
- tylergetsay 3y ago
  They will shut down the VPS if there's no activity on it, not sure how they detect this though
  - ukuina 3y ago
    CPU idle % over a rolling time window (say, 15 minutes).