3 ms·

In this post yesterday https://justine.lol/lex/ https://justine.lol/lex/ there is this quote: > The new highlighter and chatbot interface has made llamafile so

In this post yesterday https://justine.lol/lex/ https://justine.lol/lex/ there is this quote: > The new highlighter and chatbot interface has made llamafile so pleasant for me to use, combined with the fact that open weights models like gemma 27b it have gotten so good, that it's become increasingly rare that I'll feel tempted to use Claude these days. Leaving me tempted more than ever to see if I can integrate some sort of LLM workflow locally. I would only consider doing it locally, and I've an older computer, so I didn't think this would be possible till reading that post yesterday. The only thing I thought was - how would I work it in to Emacs? And then today, this post. It looks very well integrated. Has anyone any experience using gemma 27b it with llamafile and gptel? I know very little about the whole space, really.

whartung 2y ago
I'm on an Intel iMac, and llama can't leverage its GPU. So, it's 1990s slow. It's literally like talking to a machine in the 90s, from the slow response time to the 1200-2400 baud output. It's easy to give tasks to, hard to have a conversation with. Just paste the task in and let it churn, come back to it later.
- anthk 2y ago
  90s? Are you sure?
  - jasonm23 2y ago
    80s isn't it.
omneity 2y ago
Hearing Gemma 27B called “so good I’ll rarely reach out to Claude” (paraphrased) is quite a surprising sentence to hear. Perhaps for simpler workflows? My experience with the whole Gemma line 27B included has been less than stellar, especially in instruction following ability.
alwayslikethis 2y ago
Gemma 27b is pretty censored, I prefer mistral-small 22b. Works fairly well with ollama and gptel but it still doesn't compare to the commercial offerings. llama3.1 70b is quite a bit smarter but doesn't fit into my VRAM. I expect llama3.1 405b would come quite close to commercial offerings.

whartung 2y ago

I'm on an Intel iMac, and llama can't leverage its GPU. So, it's 1990s slow. It's literally like talking to a machine in the 90s, from the slow response time to the 1200-2400 baud output. It's easy to give tasks to, hard to have a conversation with. Just paste the task in and let it churn, come back to it later.

anthk 2y ago
90s? Are you sure?
- jasonm23 2y ago
  80s isn't it.

omneity 2y ago

Hearing Gemma 27B called “so good I’ll rarely reach out to Claude” (paraphrased) is quite a surprising sentence to hear. Perhaps for simpler workflows? My experience with the whole Gemma line 27B included has been less than stellar, especially in instruction following ability.

alwayslikethis 2y ago

Gemma 27b is pretty censored, I prefer mistral-small 22b. Works fairly well with ollama and gptel but it still doesn't compare to the commercial offerings. llama3.1 70b is quite a bit smarter but doesn't fit into my VRAM. I expect llama3.1 405b would come quite close to commercial offerings.