3 ms·
Personally I was testing with TogetherAI because I don't have the specs for a local 70b. Using quantized versions helps (Ollama's downloads 4-bit by default, yo
by sestinj 3y ago
Personally I was testing with TogetherAI because I don't have the specs for a local 70b. Using quantized versions helps (Ollama's downloads 4-bit by default, you can get down to 2), but it would still require a higher-end Mac. Highly recommend Together, it runs quite quickly and is $0.9/million tokens
- d_sc 3y agoare there any docs on setting up togetherAI with continue.dev? would be interested in checking that out as an alternative to OpenAI for experimenting with larger models that won't run/run well on a m1 max.
- sestinj 3y agoDefinitely, here is a brief reference page for the Together provider: https://continue.dev/docs/reference/Model%20Providers/togetherllm https://continue.dev/docs/reference/Model%20Providers/togeth..., and a higher-level explanation of model configuration here: https://continue.dev/docs/model-setup/overview https://continue.dev/docs/model-setup/overview
- vwkd 3y agoWhat’s the advantage of Together? The price is about the price of GPT 3.5 Turbo ($1/mil tokens is $0.001/thousand tokens), which has the advantage of wide ecosystem and support.