5 ms·

This would be great if I had a beefy machine with 32GB ram and 16 cores. The alternatives presented are to use either Together or Standard api's to run llama-c

This would be great if I had a beefy machine with 32GB ram and 16 cores. The alternatives presented are to use either Together or Standard api's to run llama-code. Both have a free tier but eventually you wind up paying. Has anyone calculated the financials and determined that the cost of these alternatives is less than copilot? Cost aside, there's a concern of quality. I should just buy a super computer..

ramesh31 3y ago
>This would be great if I had a beefy machine with 32GB ram and 16 cores. Honestly this is a pretty standard workstation setup now. The days of 16GB being sufficient are pretty much over. RAM is cheap. And you can run the 7b model with well under that much.
sestinj 3y ago
If you run Ollama locally, the cost is zero. With Replicate, you pay about $0.005 per second, so with an average of say 15 seconds per request to generate the full response, and 25 inputs per day, this comes out to ~$50/month. More expensive than classic Copilot, but also solving a different, potentially more valuable problem. Also likely that costs drop as we're seeing across the board. And at a minimum, the free tiers probably get you all the experimentation you need before figuring out which model is best :) (including Continue's free trial for GPT-4)
- agcat 3y ago
  Curious to learn about the different, potentially more valuable problem that you mentioned about? Is this for Replicate OR Continue?
whimsicalism 3y ago
Yes, you likely will either have to pay a monthly fee or own compute yourself. Where did this expectation of free compute come from? Copilot costs too.
- Havoc 3y ago
  >Where did this expectation of free compute come from? VC funded growth & users at any cost and figure out profitability later model? i.e. hn gang
freedomben 3y ago
Perfect reason to build a desktop IMHO :-) You'd be surprised how affordable it can be. Mine sits on a tailnet with my laptop and handles most of the workload for me. Especially while I'm at home, I barely even notice that my dev environment is not running "locally." Also doubles great as a host for audiobookshelf, jellyfin, archivebox, and more. I have virt manager set up too (client on laptop, virtmanager service running on desktop) so I can easily spin up all sorts of VMs. I love it. I spent about $2k but I bought top of the line.
hmottestad 3y ago
No need for a super computer. The 7B model should run on fairly reasonably priced GPUs. So if you have a desktop PC you can probably get a second hand GPU. My MacBook Pro runs the larger model fine, wouldn’t call it a super computer, but I do think it’s a rather expensive machine.

ramesh31 3y ago

>This would be great if I had a beefy machine with 32GB ram and 16 cores. Honestly this is a pretty standard workstation setup now. The days of 16GB being sufficient are pretty much over. RAM is cheap. And you can run the 7b model with well under that much.

sestinj 3y ago

If you run Ollama locally, the cost is zero. With Replicate, you pay about $0.005 per second, so with an average of say 15 seconds per request to generate the full response, and 25 inputs per day, this comes out to ~$50/month. More expensive than classic Copilot, but also solving a different, potentially more valuable problem. Also likely that costs drop as we're seeing across the board. And at a minimum, the free tiers probably get you all the experimentation you need before figuring out which model is best :) (including Continue's free trial for GPT-4)

agcat 3y ago
Curious to learn about the different, potentially more valuable problem that you mentioned about? Is this for Replicate OR Continue?

whimsicalism 3y ago

Yes, you likely will either have to pay a monthly fee or own compute yourself. Where did this expectation of free compute come from? Copilot costs too.

Havoc 3y ago
>Where did this expectation of free compute come from? VC funded growth & users at any cost and figure out profitability later model? i.e. hn gang

freedomben 3y ago

Perfect reason to build a desktop IMHO :-) You'd be surprised how affordable it can be. Mine sits on a tailnet with my laptop and handles most of the workload for me. Especially while I'm at home, I barely even notice that my dev environment is not running "locally." Also doubles great as a host for audiobookshelf, jellyfin, archivebox, and more. I have virt manager set up too (client on laptop, virtmanager service running on desktop) so I can easily spin up all sorts of VMs. I love it. I spent about $2k but I bought top of the line.

hmottestad 3y ago

No need for a super computer. The 7B model should run on fairly reasonably priced GPUs. So if you have a desktop PC you can probably get a second hand GPU. My MacBook Pro runs the larger model fine, wouldn’t call it a super computer, but I do think it’s a rather expensive machine.