5 ms·

I eat my initial words, this works really well on my macbook air M1 and feels comparable of GPT3.5 - which is actually an amazing feat! Question: is there some

I eat my initial words, this works really well on my macbook air M1 and feels comparable of GPT3.5 - which is actually an amazing feat! Question: is there something like this, but with the "function calling api" finetuning? 95% of my uses nowadays deal with input/output of structured data (JSON, basically), and I kind of fear that OpenAI will no longer support thos specialized models after a few months... I know its probably not that glorious compared with multimodal chat UIs, but a workhose like nothing else for automation!

brucethemoose2 3y ago
Yes! https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md https://github.com/ggerganov/llama.cpp/blob/master/grammars/... Its actually better than a specialized model, during token generation it constrains the possible output tokens to an arbitrary grammar (like, say, JSON syntax). So it will work "perfectly" with any model with a basic understanding of the format. Kobold.cpp and text-generation-ui already support this, and both will run on your mac.
- FieryTransition 3y ago
  Does that mean you could make any model at least always produce syntactically correct code as output?
  - brucethemoose2 3y ago
    Yes. The model has no choice, as the syntax is applied as the model is "picking" probable output tokens, not parsing complete output like some other methods. Though its possible it would output syntactically correct nonsense.
    - amilios 3y ago
      How does this actually work though, since the model could e.g. abruptly end the generation giving something syntactically invalid? Doesn't it need to look at the whole output at some stage?
      - brucethemoose2 3y ago
        All generated output depends on all the previous output. The model "looks" at (mostly) everything every time. Generation never actually stops, the model just emits a special stop token when stopping is most likely next token. Hence the grammar implementation can prevent this stop token from being emitted prematurely. There was some discussion of models getting "stuck" where there is no syntactically correct token to emit. Some proposals included a "backspace token" IIRC, but I dunno what they actually did. You can look through the discussion in the PR.
        amilios 3y ago
        Oh yeah that's true! Just block the stop token. But yes, my thought is that there are scenarios where it can get "stuck" as you said. I'll look at the PR, thanks!
      - uoaei 3y ago
        At every generation step you mask the tokens so that only syntactically valid outputs are possible. This includes the <end> token.
    - FieryTransition 3y ago
      That's very awesome, I feel like it would be fun to make a feedback loop with the LLM outputting syntactically valid programs which also log the status of various invariants and pre/post conditions as the program runs to validate correctness and maybe even train a LLM this way. Would be interesting to see how far it would get writing programs, and wrt. the problem of stopping too early on a token, it could resume with some extra context from where it stopped. Maybe construct program such that it is made by building blocks which can fit into context, such that each block/function has a pre/post condition and invariants are inside it, which the LLM will try to test against each time it runs. I think I just found my next side-project, I know there are many similar tools, but I haven't seen anything trying to couple the compiler, running the program, creating pre/post and invariant checking against emitted code yet. Would be interesting to test this hypothesis to see if the llm can actually build a program this way. I think about it like humans have a short-term memory as well, and for each sub-structure of a program, the llm would have to work in a similar way. Then reserve a bit of context to control the long-term memory for general goals, or create something like an AST of thoughts which it would recurse through, while reasoning about it.
Karrot_Kream 3y ago
How many tokens / sec are you getting on an M1 Air? Curious since I'm at work and can't try this on my Air yet hah.
- anonyfox 3y ago
  feels roughly like the same speed as GPT3.5 in the browser UI
- Patrick_Devine 3y ago
  I'm getting >30 tokens/sec using it with ollama and an M2 Pro. That might be a little slow though because I have a background finetuning job running.
  - minzi 3y ago
    Bit of a tangential question here, but any recommendations on how to get started fine tuning this model (or ones like it)? I feel like there are a million different tutorial and ways of doing it when I google.
    - Karrot_Kream 3y ago
      Take a look at the QLoRA repo https://github.com/artidoro/qlora/ https://github.com/artidoro/qlora/ which has an example finetuning Llama. Made by the authors of the QLoRA paper.
    - brucethemoose2 3y ago
      https://github.com/OpenAccess-AI-Collective/axolotl https://github.com/OpenAccess-AI-Collective/axolotl This is a wrapper around many training methods, and it has yielded many excellent community finetunes already.
- brucethemoose2 3y ago
  Its the same speed as llama 7B, so very quick.
- benkaiser 3y ago
olso 3y ago
Hey, how did you try it on M1? I don't see any MPS support. https://github.com/mistralai/mistral-src/issues/2 https://github.com/mistralai/mistral-src/issues/2 edit: nevermind https://ollama.ai https://ollama.ai
- anonyfox 3y ago
  `ollama run mistral` <-- literally thats it
  - iamcreasy 3y ago
    For anyone wondering - no need to download anything from Mistral website. Download ollama and use this command.
  - deleted 3y ago
    [deleted]
- KennyBlanken 3y ago
  Heads up that this app is extremely obnoxious and upon launch installs itself in to your startup items for zero fucking good reason.
  - justinmayer 3y ago
    Quite true. I actually submitted a pull request [1] over a month ago to make it clear to potential users what will happen when Ollama is launched for the first time, but based on the complete lack of response from Ollama developers, I get the distinct impression that they are reluctant to draw attention to these important details. Unfortunate. [1]: https://github.com/jmorganca/ollama/pull/395 https://github.com/jmorganca/ollama/pull/395
- jsnow119 3y ago
  How come one blog post with the torrent magnet of mistral says I need 24g vram but ollama says I only need 8g regular ram minimum? If I wanted to experiment with working with models and finetuning offline should I just save for a 32g vram+ video card?
recroad 3y ago
I'm stuck at when doing pip install. NotADirectoryError: [Errno 20] Not a directory: 'hipconfig'
btwillard 3y ago
Our project Outlines provides JSON output in a near optimal way that also works for all types of pre-trained transformer-based LLMs: https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines Our approach also extends to EBNF grammars and LALR parsing. There's an example of that in the repository. It builds off of the Lark library, so you can use existing grammar specifications instead of starting from scratch.
- nacs 3y ago
  Outlines looks like an actively maintained alternative to Guidance. Do you have any plans to support llama.cpp?
  - btwillard 3y ago
    We've considered it, but I haven't started any work on it. Feel free to create an issue to track the status, if only to find out who else might be interested and/or working on it.

brucethemoose2 3y ago

Yes! https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md https://github.com/ggerganov/llama.cpp/blob/master/grammars/... Its actually better than a specialized model, during token generation it constrains the possible output tokens to an arbitrary grammar (like, say, JSON syntax). So it will work "perfectly" with any model with a basic understanding of the format. Kobold.cpp and text-generation-ui already support this, and both will run on your mac.

FieryTransition 3y ago
Does that mean you could make any model at least always produce syntactically correct code as output?
- brucethemoose2 3y ago
  Yes. The model has no choice, as the syntax is applied as the model is "picking" probable output tokens, not parsing complete output like some other methods. Though its possible it would output syntactically correct nonsense.
  - amilios 3y ago
    How does this actually work though, since the model could e.g. abruptly end the generation giving something syntactically invalid? Doesn't it need to look at the whole output at some stage?
    - brucethemoose2 3y ago
      All generated output depends on all the previous output. The model "looks" at (mostly) everything every time. Generation never actually stops, the model just emits a special stop token when stopping is most likely next token. Hence the grammar implementation can prevent this stop token from being emitted prematurely. There was some discussion of models getting "stuck" where there is no syntactically correct token to emit. Some proposals included a "backspace token" IIRC, but I dunno what they actually did. You can look through the discussion in the PR.
      - amilios 3y ago
        Oh yeah that's true! Just block the stop token. But yes, my thought is that there are scenarios where it can get "stuck" as you said. I'll look at the PR, thanks!
    - uoaei 3y ago
      At every generation step you mask the tokens so that only syntactically valid outputs are possible. This includes the <end> token.
  - FieryTransition 3y ago
    That's very awesome, I feel like it would be fun to make a feedback loop with the LLM outputting syntactically valid programs which also log the status of various invariants and pre/post conditions as the program runs to validate correctness and maybe even train a LLM this way. Would be interesting to see how far it would get writing programs, and wrt. the problem of stopping too early on a token, it could resume with some extra context from where it stopped. Maybe construct program such that it is made by building blocks which can fit into context, such that each block/function has a pre/post condition and invariants are inside it, which the LLM will try to test against each time it runs. I think I just found my next side-project, I know there are many similar tools, but I haven't seen anything trying to couple the compiler, running the program, creating pre/post and invariant checking against emitted code yet. Would be interesting to test this hypothesis to see if the llm can actually build a program this way. I think about it like humans have a short-term memory as well, and for each sub-structure of a program, the llm would have to work in a similar way. Then reserve a bit of context to control the long-term memory for general goals, or create something like an AST of thoughts which it would recurse through, while reasoning about it.

Karrot_Kream 3y ago

How many tokens / sec are you getting on an M1 Air? Curious since I'm at work and can't try this on my Air yet hah.

anonyfox 3y ago
feels roughly like the same speed as GPT3.5 in the browser UI
Patrick_Devine 3y ago
I'm getting >30 tokens/sec using it with ollama and an M2 Pro. That might be a little slow though because I have a background finetuning job running.
- minzi 3y ago
  Bit of a tangential question here, but any recommendations on how to get started fine tuning this model (or ones like it)? I feel like there are a million different tutorial and ways of doing it when I google.
  - Karrot_Kream 3y ago
    Take a look at the QLoRA repo https://github.com/artidoro/qlora/ https://github.com/artidoro/qlora/ which has an example finetuning Llama. Made by the authors of the QLoRA paper.
  - brucethemoose2 3y ago
    https://github.com/OpenAccess-AI-Collective/axolotl https://github.com/OpenAccess-AI-Collective/axolotl This is a wrapper around many training methods, and it has yielded many excellent community finetunes already.
brucethemoose2 3y ago
Its the same speed as llama 7B, so very quick.
benkaiser 3y ago

olso 3y ago

Hey, how did you try it on M1? I don't see any MPS support. https://github.com/mistralai/mistral-src/issues/2 https://github.com/mistralai/mistral-src/issues/2 edit: nevermind https://ollama.ai https://ollama.ai

anonyfox 3y ago
`ollama run mistral` <-- literally thats it
- iamcreasy 3y ago
  For anyone wondering - no need to download anything from Mistral website. Download ollama and use this command.
- deleted 3y ago
  [deleted]
KennyBlanken 3y ago
Heads up that this app is extremely obnoxious and upon launch installs itself in to your startup items for zero fucking good reason.
- justinmayer 3y ago
  Quite true. I actually submitted a pull request [1] over a month ago to make it clear to potential users what will happen when Ollama is launched for the first time, but based on the complete lack of response from Ollama developers, I get the distinct impression that they are reluctant to draw attention to these important details. Unfortunate. [1]: https://github.com/jmorganca/ollama/pull/395 https://github.com/jmorganca/ollama/pull/395
jsnow119 3y ago
How come one blog post with the torrent magnet of mistral says I need 24g vram but ollama says I only need 8g regular ram minimum? If I wanted to experiment with working with models and finetuning offline should I just save for a 32g vram+ video card?

recroad 3y ago

I'm stuck at when doing pip install. NotADirectoryError: [Errno 20] Not a directory: 'hipconfig'

btwillard 3y ago

Our project Outlines provides JSON output in a near optimal way that also works for all types of pre-trained transformer-based LLMs: https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines Our approach also extends to EBNF grammars and LALR parsing. There's an example of that in the repository. It builds off of the Lark library, so you can use existing grammar specifications instead of starting from scratch.

nacs 3y ago
Outlines looks like an actively maintained alternative to Guidance. Do you have any plans to support llama.cpp?
- btwillard 3y ago
  We've considered it, but I haven't started any work on it. Feel free to create an issue to track the status, if only to find out who else might be interested and/or working on it.