7 ms·

Mistral-8x7B-Chat

131 points by birriel 3y ago

antirez 3y ago
llama.cpp support needed ASAP, not just quantized, but I hope it will be albe to run in a way that keeps the same two experts for minimum N tokens generated and re-valuate the expert to select only then, so that it will be possible to do the inference in low ram doing something similar to "paging".
- kristianp 3y ago
  There is an issue for it: https://github.com/ggerganov/llama.cpp/issues/4381 https://github.com/ggerganov/llama.cpp/issues/4381
  - rgbrgb 3y ago
    what a sick project to be able to attract a billionaire programmer [0] and c royalty. [0]: https://github.com/ggerganov/llama.cpp/issues/4216#issuecomment-1826393578 https://github.com/ggerganov/llama.cpp/issues/4216#issuecomm...
    - jaipilot747 3y ago
      For those out of the loop, who are the billionaire programmer and C royalty people in this link?
      - kostadin 3y ago
        Tobi Shopify and antirez (redis etc)
      - antirez 3y ago
        Tobi is the founder of Shopify.
      - KennyBlanken 3y ago
        I also looked at the threads and had no idea who they were referencing.
      - OJFord 3y ago
        The linked comment is authored by the CEO of Shopify; the top-level comment in this thread is the .. what would you say, author/originator/founder of Redis. (And maybe/presumably more widely known in C-land for other projects or contributions, idk.)
  - kristianp 3y ago
    I'm looking forward to a 4-bit quantized version that I can run on my Intel Thinkpad with 32GB RAM, should be almost usable if its as fast as a 14B size model. Whether it's any good is another matter, I guess the leaderboard on HuggingFace will be updated at some point.
- tarruda 3y ago
  > so that it will be possible to do the inference in low ram doing something similar to "paging". I was also hoping for that, but my initial research suggests that you need to fit everything into VRAM (or RAM when doing llama.cpp CPU inference). The upside is that it should be as fast as running a single 7B model, which on llama.cpp should be in the ballpark of 10 tok/sec when 4-bit quantized. For local usage, I'd say you will have an "OK" experience in a recent laptop with 64GB RAM, which is more realistic than having the necessary VRAM in a Nvidia GPU.
  - tarruda 3y ago
    Correction, it should have the performance of a single 12B parameter model: "Concretely, Mixtral has 45B total parameters but only uses 12B parameters per token. It, therefore, processes input and generates output at the same speed and for the same cost as a 12B model." https://mistral.ai/news/mixtral-of-experts/ https://mistral.ai/news/mixtral-of-experts/
a_wild_dandan 3y ago
PSA: This is broken for Apple silicon. It relies on the CUDA-only `flash_attn` package. Alas, I wanted to try Mistral's MoE on my MacBook. :( Understandable though! It's a new model with an incredibly bizarre release. My phalanges are crossed.
- king_magic 3y ago
  I think "broken" is a bit of an exaggeration, when the actual problem is "just not compatible with your hardware".
lannisterstark 3y ago
Man this LLM stuff gets released faster than I can keep up. Is there a centralized list somewhere that tests "use this for x purpose, use that for y?"
- brucethemoose2 3y ago
  Even the OSS trainers/finetuners themselves don't really know the use case for their models. They just hot drop them and move on, lol.
  - htrp 3y ago
    They train and drop them.... and rely on you to figure it out.
- lxe 3y ago
  This is a finetune of the torrent-dropped moe model from a few days ago. Not a "release" per se, but rather a display of community excitement over this.
- behnamoh 3y ago
  > Is there a centralized list somewhere that tests "use this for x purpose, use that for y?" Yeah, "don't use these models for production, use OpenAI for production, ignore Claude/Gemini/etc.".
  - holoduke 3y ago
    Why not. For simple things like converting, translating its quite handy and the costs are with you. I use open models for removing backgrounds from images. Also use models to translate texts. Openai would be too expensive for my usecases.
  - PostOnce 3y ago
    "Put all your eggs in one basket" isn't the traditional advice. Any one of these companies can at any time change their API, pricing, access rules, or even swap the model out for a dumber, cheaper one at the same price. You'll have no recourse if you don't have several backends available, or control your own. At a minimum, you should have available several hot-swappable backends/APIs if you want to remain viable in an indeterminate future.
simonw 3y ago
I thought the new MoE Mistral model was called Mixtral.
- irthomasthomas 3y ago
  Correct. Makes for easy monitoring of hf. https://huggingface.co/models?search=mixtral https://huggingface.co/models?search=mixtral The pace is exciting, eh? How long till llm-mixtral? Btw. The other day I uploaded ~4k chats from your llm dB to the code interpreter and had it label them. Worked pretty well. Only gave them one label each at first, then I started working on more complex labeling, but then my poor assistant ran out of steam and the interpreter session expired. So much pleasure and pain! Love your llm cli though. Thanks.
lab 3y ago
We just made it available on OpenRouter: https://openrouter.ai/models/fireworks/mixtral-8x7b-fw-chat https://openrouter.ai/models/fireworks/mixtral-8x7b-fw-chat Chat playground: https://openrouter.ai/playground?models=fireworks/mixtral-8x7b-fw-chat https://openrouter.ai/playground?models=fireworks/mixtral-8x...
- behnamoh 3y ago
  Man, I liked OpenRouter but most models run so much slower on OR that I had to switch to other solutions. Is there a reason for horribly slow Meta Llama 2 models?
Jimbabwe 3y ago
There’s probably a better place to ask this highly specific technical question, but I’m avoiding Reddit these days so just throwing it out I guess. I’ve been trying to run these in a container but it’s verrrry slow, I believe, because of the lack of gpu help. All the instructions I find are for nvidia gpus and my server is a qnap tvs-473e with an embedded amd cpu/gpu (I know, I know). The only good news is that I’ve upgraded the ram to 32gb, and I have a 1TB ssd. Any idea of how I can get my own self-hosted LLM/chat service on this funky hardware? The nvidia/docker option requires installing the nvidia runtime alongside docker, but I can’t find an amd equivalent. Thanks. Sorry for the wall of text nobody cares about.
- brucethemoose2 3y ago
  > qnap tvs-473e Specs say this runs an AMD RX-421BD. This is a 2015 AMD CPU with 2 bulldozer cores and a tiny IGP. ...To be blunt, you would be much better off running LLMs on your phone. Even an older phone. Or literally whatever device you are reading HN on. But if you insist, the runtime you want in MLC-LLM's Vulkan runtime.
  - refulgentis 3y ago
    This. Sibling llama.cpp comment is standard "I know llama.cpp, I assume that's 80% of the universe instead of .8%, and I assume that's all anyone needs. So I know just enough to be dangerous with ppl looking for advice". You'll see it over and over again when you're looking for help, be careful, it's 100% a blind alley in your case. It's very likely you'll be disappointed by MLC as well, simultaneously it's your only real option. You definitely won't hit 1 tkn/sec, and honestly, id bet 0.1 tkn / sec
    - Jimbabwe 3y ago
      Thanks. Yeah I’ve noticed that llama.cpp is definitely the recommendation du jour, but this thread gave me hope: https://www.reddit.com/r/LocalLLaMA/comments/13p8zq2/update_opencl_is_merged_amd_gpus_now_work_with https://www.reddit.com/r/LocalLLaMA/comments/13p8zq2/update_... Sorry for reddit link.
vizzah 3y ago
This model is better by many other contenders, but still far from GPT4. "what famous brands are there which change one letter from a common word to make a non-existent, but a catchy name, such as "musiq" instead of "music".. etc?" There are several brands that have played with words by changing a letter or adding a letter to create a new and memorable name. Here are a few examples: Qatar Airways - This airline's name was inspired by the country of Qatar, but with the letter "Q" replaced by "Q" to create a unique and catchy name. Lush Cosmetics - This cosmetics company's name is a play on the word "lush," which means "profuse" or "luxuriant." Pinterest - This social media site's name is a combination of the words "pin" and "interest," but with the letter "e" replaced by "i" to create a more unique and memorable name. Tumblr - This social media platform's name is a play on the words "tumble" and "blogger," but with the letter "l" replaced by "r" to create a more unique and catchy name. --------- that is compared to hilarious Yi 34B Chat below: =) There are several famous brands that have used this strategy of changing one letter from a common word to create a unique and catchy name. Here are a few examples: eBay - Instead of "cBay," which would be a reference to "crab," eBay was named after the first three letters of the word "baY" and the "e" from the word "electronic," as in "electronic marketplace." Google - Originally named "BackRub," the company changed its name to Google, a play on the word "googol," which refers to the number one followed by one hundred zeros, expressing the vastness of the internet. Kodak - The name Kodak is derived from the word "cod" with an "k" added. It was chosen because it is short, easy to remember, and not suggestive of the photographic nature of the business.
- brucethemoose2 3y ago
  Here is my response from Yi, which is actually pretty good: One well-known example is the brand Apple. The company's name comes from a play on the phrase "Apple of My Eye." Another example is Google, which is derived from the mathematical term "googolplex," which refers to the number 1 followed by 100 zeroes. It was originally meant to be a play on the word "googol" (1 followed by 100 zeroes), but a spelling mistake led to the creation of the new word "google." Other examples include PepsiCo, which combines the words "pepsin" and "kola," and Amazon, which was inspired by the world's largest river, the Amazon River. In the case of "musiq," it seems like a made-up name without any specific reference or known origin. However, some possible variations could be created using existing words, such as "dramatize" instead of "dramatize" or "fashin" instead of "fashion."
behnamoh 3y ago
Somewhere between shiny Google releases and Mistral's magnet link tweet, there's gotta be a sweet spot where you release the model but also have enough decency to tell people how to use it optimally. Mistral, if you're reading this, I'm talking about you.
tempaway47641 3y ago
Every day: "Cheeseface just dropped the Blippy-7B model which is almost as good as the twinamp 34B model on the SwagCube benchmark when run locally as int8 and this shows that the gains made by the skibidi-70B model will probably filter down to the baseline Eras models in the next few weeks"
- MarcoZavala 3y ago
  [dead]
- brucethemoose2 3y ago
  Thats giving the community way too much organization credit. Everyone just seems to be running experiments independently and then randomly drop some results, with basically no documentation. Sometimes the motivation is clearly VC money or paper exposure, but sometimes there is no apparent motivation... Or even no model card. Then when something works, others copy the script. Not that I dont enjoy it. I find the sea of finetune generations fascinating.
  - sebosp 3y ago
    Now I feel I have to read again Haroun and the Sea of Stories from Rushdie thanks to your last comment <3 that's one of my favorite childhood reads
- deleted 3y ago
  [deleted]
deleted 3y ago
[deleted]
kristianp 3y ago
It's available at [1], I haven't tried their api yet. [1] https://app.fireworks.ai/models https://app.fireworks.ai/models
sabareesh 3y ago
Does anyone know how to load this into ollama

131 points by birriel 3y ago

llama.cpp support needed ASAP, not just quantized, but I hope it will be albe to run in a way that keeps the same two experts for minimum N tokens generated and re-valuate the expert to select only then, so that it will be possible to do the inference in low ram doing something similar to "paging".

kristianp 3y ago
There is an issue for it: https://github.com/ggerganov/llama.cpp/issues/4381 https://github.com/ggerganov/llama.cpp/issues/4381
- rgbrgb 3y ago
  what a sick project to be able to attract a billionaire programmer [0] and c royalty. [0]: https://github.com/ggerganov/llama.cpp/issues/4216#issuecomment-1826393578 https://github.com/ggerganov/llama.cpp/issues/4216#issuecomm...
  - jaipilot747 3y ago
    For those out of the loop, who are the billionaire programmer and C royalty people in this link?
    - kostadin 3y ago
      Tobi Shopify and antirez (redis etc)
    - antirez 3y ago
      Tobi is the founder of Shopify.
    - KennyBlanken 3y ago
      I also looked at the threads and had no idea who they were referencing.
    - OJFord 3y ago
      The linked comment is authored by the CEO of Shopify; the top-level comment in this thread is the .. what would you say, author/originator/founder of Redis. (And maybe/presumably more widely known in C-land for other projects or contributions, idk.)
- kristianp 3y ago
  I'm looking forward to a 4-bit quantized version that I can run on my Intel Thinkpad with 32GB RAM, should be almost usable if its as fast as a 14B size model. Whether it's any good is another matter, I guess the leaderboard on HuggingFace will be updated at some point.
tarruda 3y ago
> so that it will be possible to do the inference in low ram doing something similar to "paging". I was also hoping for that, but my initial research suggests that you need to fit everything into VRAM (or RAM when doing llama.cpp CPU inference). The upside is that it should be as fast as running a single 7B model, which on llama.cpp should be in the ballpark of 10 tok/sec when 4-bit quantized. For local usage, I'd say you will have an "OK" experience in a recent laptop with 64GB RAM, which is more realistic than having the necessary VRAM in a Nvidia GPU.
- tarruda 3y ago
  Correction, it should have the performance of a single 12B parameter model: "Concretely, Mixtral has 45B total parameters but only uses 12B parameters per token. It, therefore, processes input and generates output at the same speed and for the same cost as a 12B model." https://mistral.ai/news/mixtral-of-experts/ https://mistral.ai/news/mixtral-of-experts/

a_wild_dandan 3y ago

PSA: This is broken for Apple silicon. It relies on the CUDA-only `flash_attn` package. Alas, I wanted to try Mistral's MoE on my MacBook. :( Understandable though! It's a new model with an incredibly bizarre release. My phalanges are crossed.

king_magic 3y ago
I think "broken" is a bit of an exaggeration, when the actual problem is "just not compatible with your hardware".

lannisterstark 3y ago

Man this LLM stuff gets released faster than I can keep up. Is there a centralized list somewhere that tests "use this for x purpose, use that for y?"

brucethemoose2 3y ago
Even the OSS trainers/finetuners themselves don't really know the use case for their models. They just hot drop them and move on, lol.
- htrp 3y ago
  They train and drop them.... and rely on you to figure it out.
lxe 3y ago
This is a finetune of the torrent-dropped moe model from a few days ago. Not a "release" per se, but rather a display of community excitement over this.
behnamoh 3y ago
> Is there a centralized list somewhere that tests "use this for x purpose, use that for y?" Yeah, "don't use these models for production, use OpenAI for production, ignore Claude/Gemini/etc.".
- holoduke 3y ago
  Why not. For simple things like converting, translating its quite handy and the costs are with you. I use open models for removing backgrounds from images. Also use models to translate texts. Openai would be too expensive for my usecases.
- PostOnce 3y ago
  "Put all your eggs in one basket" isn't the traditional advice. Any one of these companies can at any time change their API, pricing, access rules, or even swap the model out for a dumber, cheaper one at the same price. You'll have no recourse if you don't have several backends available, or control your own. At a minimum, you should have available several hot-swappable backends/APIs if you want to remain viable in an indeterminate future.

simonw 3y ago

I thought the new MoE Mistral model was called Mixtral.

irthomasthomas 3y ago
Correct. Makes for easy monitoring of hf. https://huggingface.co/models?search=mixtral https://huggingface.co/models?search=mixtral The pace is exciting, eh? How long till llm-mixtral? Btw. The other day I uploaded ~4k chats from your llm dB to the code interpreter and had it label them. Worked pretty well. Only gave them one label each at first, then I started working on more complex labeling, but then my poor assistant ran out of steam and the interpreter session expired. So much pleasure and pain! Love your llm cli though. Thanks.

lab 3y ago

We just made it available on OpenRouter: https://openrouter.ai/models/fireworks/mixtral-8x7b-fw-chat https://openrouter.ai/models/fireworks/mixtral-8x7b-fw-chat Chat playground: https://openrouter.ai/playground?models=fireworks/mixtral-8x7b-fw-chat https://openrouter.ai/playground?models=fireworks/mixtral-8x...

behnamoh 3y ago
Man, I liked OpenRouter but most models run so much slower on OR that I had to switch to other solutions. Is there a reason for horribly slow Meta Llama 2 models?

Jimbabwe 3y ago

There’s probably a better place to ask this highly specific technical question, but I’m avoiding Reddit these days so just throwing it out I guess. I’ve been trying to run these in a container but it’s verrrry slow, I believe, because of the lack of gpu help. All the instructions I find are for nvidia gpus and my server is a qnap tvs-473e with an embedded amd cpu/gpu (I know, I know). The only good news is that I’ve upgraded the ram to 32gb, and I have a 1TB ssd. Any idea of how I can get my own self-hosted LLM/chat service on this funky hardware? The nvidia/docker option requires installing the nvidia runtime alongside docker, but I can’t find an amd equivalent. Thanks. Sorry for the wall of text nobody cares about.

brucethemoose2 3y ago
> qnap tvs-473e Specs say this runs an AMD RX-421BD. This is a 2015 AMD CPU with 2 bulldozer cores and a tiny IGP. ...To be blunt, you would be much better off running LLMs on your phone. Even an older phone. Or literally whatever device you are reading HN on. But if you insist, the runtime you want in MLC-LLM's Vulkan runtime.
- refulgentis 3y ago
  This. Sibling llama.cpp comment is standard "I know llama.cpp, I assume that's 80% of the universe instead of .8%, and I assume that's all anyone needs. So I know just enough to be dangerous with ppl looking for advice". You'll see it over and over again when you're looking for help, be careful, it's 100% a blind alley in your case. It's very likely you'll be disappointed by MLC as well, simultaneously it's your only real option. You definitely won't hit 1 tkn/sec, and honestly, id bet 0.1 tkn / sec
  - Jimbabwe 3y ago
    Thanks. Yeah I’ve noticed that llama.cpp is definitely the recommendation du jour, but this thread gave me hope: https://www.reddit.com/r/LocalLLaMA/comments/13p8zq2/update_opencl_is_merged_amd_gpus_now_work_with https://www.reddit.com/r/LocalLLaMA/comments/13p8zq2/update_... Sorry for reddit link.

vizzah 3y ago

This model is better by many other contenders, but still far from GPT4. "what famous brands are there which change one letter from a common word to make a non-existent, but a catchy name, such as "musiq" instead of "music".. etc?" There are several brands that have played with words by changing a letter or adding a letter to create a new and memorable name. Here are a few examples: Qatar Airways - This airline's name was inspired by the country of Qatar, but with the letter "Q" replaced by "Q" to create a unique and catchy name. Lush Cosmetics - This cosmetics company's name is a play on the word "lush," which means "profuse" or "luxuriant." Pinterest - This social media site's name is a combination of the words "pin" and "interest," but with the letter "e" replaced by "i" to create a more unique and memorable name. Tumblr - This social media platform's name is a play on the words "tumble" and "blogger," but with the letter "l" replaced by "r" to create a more unique and catchy name. --------- that is compared to hilarious Yi 34B Chat below: =) There are several famous brands that have used this strategy of changing one letter from a common word to create a unique and catchy name. Here are a few examples: eBay - Instead of "cBay," which would be a reference to "crab," eBay was named after the first three letters of the word "baY" and the "e" from the word "electronic," as in "electronic marketplace." Google - Originally named "BackRub," the company changed its name to Google, a play on the word "googol," which refers to the number one followed by one hundred zeros, expressing the vastness of the internet. Kodak - The name Kodak is derived from the word "cod" with an "k" added. It was chosen because it is short, easy to remember, and not suggestive of the photographic nature of the business.

brucethemoose2 3y ago
Here is my response from Yi, which is actually pretty good: One well-known example is the brand Apple. The company's name comes from a play on the phrase "Apple of My Eye." Another example is Google, which is derived from the mathematical term "googolplex," which refers to the number 1 followed by 100 zeroes. It was originally meant to be a play on the word "googol" (1 followed by 100 zeroes), but a spelling mistake led to the creation of the new word "google." Other examples include PepsiCo, which combines the words "pepsin" and "kola," and Amazon, which was inspired by the world's largest river, the Amazon River. In the case of "musiq," it seems like a made-up name without any specific reference or known origin. However, some possible variations could be created using existing words, such as "dramatize" instead of "dramatize" or "fashin" instead of "fashion."

behnamoh 3y ago

Somewhere between shiny Google releases and Mistral's magnet link tweet, there's gotta be a sweet spot where you release the model but also have enough decency to tell people how to use it optimally. Mistral, if you're reading this, I'm talking about you.

tempaway47641 3y ago

Every day: "Cheeseface just dropped the Blippy-7B model which is almost as good as the twinamp 34B model on the SwagCube benchmark when run locally as int8 and this shows that the gains made by the skibidi-70B model will probably filter down to the baseline Eras models in the next few weeks"

MarcoZavala 3y ago
[dead]
brucethemoose2 3y ago
Thats giving the community way too much organization credit. Everyone just seems to be running experiments independently and then randomly drop some results, with basically no documentation. Sometimes the motivation is clearly VC money or paper exposure, but sometimes there is no apparent motivation... Or even no model card. Then when something works, others copy the script. Not that I dont enjoy it. I find the sea of finetune generations fascinating.
- sebosp 3y ago
  Now I feel I have to read again Haroun and the Sea of Stories from Rushdie thanks to your last comment <3 that's one of my favorite childhood reads
deleted 3y ago
[deleted]

deleted 3y ago

[deleted]

kristianp 3y ago

It's available at [1], I haven't tried their api yet. [1] https://app.fireworks.ai/models https://app.fireworks.ai/models

sabareesh 3y ago

Does anyone know how to load this into ollama