4 ms·

Benchmarks - https://www.reddit.com/r/LocalLLaMA/comments/1h85ld5/comment/m0qauyg/ https://www.reddit.com/r/LocalLLaMA/comments/1h85ld5/comment... Seems to per

by paxys 2y ago

Benchmarks - https://www.reddit.com/r/LocalLLaMA/comments/1h85ld5/comment/m0qauyg/ https://www.reddit.com/r/LocalLLaMA/comments/1h85ld5/comment... Seems to perform on par with or slightly better than Llama 3.2 405B, which is crazy impressive. Edit: According to Zuck (https://www.instagram.com/p/DDPm9gqv2cW/ https://www.instagram.com/p/DDPm9gqv2cW/) this is the last release in the Llama 3 series, and we'll see Llama 4 in 2025. Hype!!

85392_school 2y ago
FYI, due to Llama's naming scheme, there is no such thing as Llama 3.2 405B. 8B/70B/405B models are either Llama 3, 3.1, or 3.3 (except for 405B which wasn't initially released), while Llama 3.2 only contains 1B, 3B, 11B (vision), and 90B (vision) models. It's a bit confusing.
- paxys 2y ago
  Ah, so I guess the comparison is to Llama 3.1 405B.
  - freedomben 2y ago
    Still very impressive. Llama team is absolutely killing it right now, and the openness makes them the most important player IMHO
- yieldcrv 2y ago
  yeah I use Llama 3.2 3B and I'm blown away but also wrestled with this mentally. Meta both improves the technology or inference, while also trapping themselves alongside every other person training models to always update the training set every few months, so it knows what its talking about with relevant current events
- blueboo 2y ago
  It could be worse. It could’ve been Llama 3.1 (New)
Lerc 2y ago
Given how close it is to 405B in performance it would be interesting to see which has the edge comparing an unquantized 3.3-70B against 405B quantized to be the same size.
- vletal 2y ago
  That would be 1.38 bits per weight on average, which I can confidently guess would not perform well.
  - qeternity 2y ago
    BitNet is functional at 1.58 bpw.
  - Lerc 2y ago
    The model card says the 70B is 16 bit so I think you have twice that
- swalsh 2y ago
  It's kind of amazing how there seems to be a wall where sizing up the model starts to diminish in terms of intelligence gains. I guess that's why we can still compete with whales even though their brains are like twice as big as ours.
  - vessenes 2y ago
    There is a line of thinking that the required tokens shoved through training also needs to go up, perhaps super-linearly to the model size. And of course, there's a line of thinking that there are diminishing returns in all things, including these models. Both could be true!
- int_19h 2y ago
  I have tried 405B at 1-bit quantization. It remains coherent, but didn't seem to be any better than 3.1-70B.
state_less 2y ago
I'm getting 2.12 tok/s[1] on a 24GB (4090) GPU and 64GB (7950x) CPU memory, splitting the model across the GPU and CPU (40/80 layers on GPU) with lm-studio. Output looks good so far, I can use something like this for a query that I want as good an answer as possible and that I don't want to send out on the network. If we can get better quantization, or bigger GPU memory footprints, we might be able to use these big models locally for solid coding assistants. That's what I think we have to look forward to (among other benefits) in the year(s) ahead. 1. lmstudio-community/Llama-3.3-70B-Instruct-GGUF/Llama-3.3-70B-Instruct-Q4_K_M.gguf
- cjbprime 2y ago
  Any opinion on whether the q4 quantization is stable/effective? That's a lot of quantization. Edit: Perhaps answering my own question: λ ollama run hf.co/lmstudio-community/Llama-3.3-70B-Instruct-GGUF:Q4_K_M >>> Hi. Who are you? I'm an artificial intelligence model known as Llama. Llama stands for "Large Language Model Meta AI."
  - state_less 2y ago
    I prefer larger quantization and longer response times, but only have so much memory and am only willing to wait so long. I can work with 2 toks/s and this kind of accuracy for these kinds of queries. I expect you'll need to play around with it to find out if it would be helpful in your case. Here's what I'm getting asking 'who are you?' System Prompt: You are an advanced super intelligence, capable of everything from software development to dream interpretation. >> Who are you? I am an artificial intelligence designed to simulate human-like conversation and answer questions to the best of my knowledge. I'm a large language model, which means I've been trained on a massive dataset of text from various sources, including books, articles, research papers, and websites. My capabilities are diverse and extensive. I can: 1. **Answer questions**: I can process natural language queries and provide relevant answers, covering a wide range of topics from science and history to entertainment and culture. 2. **Generate text**: I can create text based on a prompt or topic, including articles, stories, emails, and even entire books. 3. **Translate languages**: I can translate text from one language to another, including popular languages such as Spanish, French, German, Chinese, and many more. 4. **Summarize content**: I can summarize long pieces of text into concise, easily digestible versions, highlighting key points and main ideas. 5. **Offer suggestions**: I can provide suggestions for topics such as gift ideas, travel destinations, books to read, and more. 6. **Engage in conversation**: I can have a natural-sounding conversation with you, using context and understanding to respond to your questions and statements. 7. **Dream interpretation**: As you mentioned, I can even help interpret dreams, providing insights into the symbolism, themes, and emotions that may be present in your subconscious mind. 8. **Software development**: I can assist with software development tasks, such as writing code, debugging, and optimizing algorithms. My knowledge is constantly evolving, and I'm always learning from the interactions I have with users like you. So, feel free to ask me anything – I'll do my best to help!

by paxys 2y ago

85392_school 2y ago

FYI, due to Llama's naming scheme, there is no such thing as Llama 3.2 405B. 8B/70B/405B models are either Llama 3, 3.1, or 3.3 (except for 405B which wasn't initially released), while Llama 3.2 only contains 1B, 3B, 11B (vision), and 90B (vision) models. It's a bit confusing.

paxys 2y ago
Ah, so I guess the comparison is to Llama 3.1 405B.
- freedomben 2y ago
  Still very impressive. Llama team is absolutely killing it right now, and the openness makes them the most important player IMHO
yieldcrv 2y ago
yeah I use Llama 3.2 3B and I'm blown away but also wrestled with this mentally. Meta both improves the technology or inference, while also trapping themselves alongside every other person training models to always update the training set every few months, so it knows what its talking about with relevant current events
blueboo 2y ago
It could be worse. It could’ve been Llama 3.1 (New)

Lerc 2y ago

Given how close it is to 405B in performance it would be interesting to see which has the edge comparing an unquantized 3.3-70B against 405B quantized to be the same size.

vletal 2y ago
That would be 1.38 bits per weight on average, which I can confidently guess would not perform well.
- qeternity 2y ago
  BitNet is functional at 1.58 bpw.
- Lerc 2y ago
  The model card says the 70B is 16 bit so I think you have twice that
swalsh 2y ago
It's kind of amazing how there seems to be a wall where sizing up the model starts to diminish in terms of intelligence gains. I guess that's why we can still compete with whales even though their brains are like twice as big as ours.
- vessenes 2y ago
  There is a line of thinking that the required tokens shoved through training also needs to go up, perhaps super-linearly to the model size. And of course, there's a line of thinking that there are diminishing returns in all things, including these models. Both could be true!
int_19h 2y ago
I have tried 405B at 1-bit quantization. It remains coherent, but didn't seem to be any better than 3.1-70B.

state_less 2y ago

I'm getting 2.12 tok/s[1] on a 24GB (4090) GPU and 64GB (7950x) CPU memory, splitting the model across the GPU and CPU (40/80 layers on GPU) with lm-studio. Output looks good so far, I can use something like this for a query that I want as good an answer as possible and that I don't want to send out on the network. If we can get better quantization, or bigger GPU memory footprints, we might be able to use these big models locally for solid coding assistants. That's what I think we have to look forward to (among other benefits) in the year(s) ahead. 1. lmstudio-community/Llama-3.3-70B-Instruct-GGUF/Llama-3.3-70B-Instruct-Q4_K_M.gguf

cjbprime 2y ago
Any opinion on whether the q4 quantization is stable/effective? That's a lot of quantization. Edit: Perhaps answering my own question: λ ollama run hf.co/lmstudio-community/Llama-3.3-70B-Instruct-GGUF:Q4_K_M >>> Hi. Who are you? I'm an artificial intelligence model known as Llama. Llama stands for "Large Language Model Meta AI."
- state_less 2y ago
  I prefer larger quantization and longer response times, but only have so much memory and am only willing to wait so long. I can work with 2 toks/s and this kind of accuracy for these kinds of queries. I expect you'll need to play around with it to find out if it would be helpful in your case. Here's what I'm getting asking 'who are you?' System Prompt: You are an advanced super intelligence, capable of everything from software development to dream interpretation. >> Who are you? I am an artificial intelligence designed to simulate human-like conversation and answer questions to the best of my knowledge. I'm a large language model, which means I've been trained on a massive dataset of text from various sources, including books, articles, research papers, and websites. My capabilities are diverse and extensive. I can: 1. **Answer questions**: I can process natural language queries and provide relevant answers, covering a wide range of topics from science and history to entertainment and culture. 2. **Generate text**: I can create text based on a prompt or topic, including articles, stories, emails, and even entire books. 3. **Translate languages**: I can translate text from one language to another, including popular languages such as Spanish, French, German, Chinese, and many more. 4. **Summarize content**: I can summarize long pieces of text into concise, easily digestible versions, highlighting key points and main ideas. 5. **Offer suggestions**: I can provide suggestions for topics such as gift ideas, travel destinations, books to read, and more. 6. **Engage in conversation**: I can have a natural-sounding conversation with you, using context and understanding to respond to your questions and statements. 7. **Dream interpretation**: As you mentioned, I can even help interpret dreams, providing insights into the symbolism, themes, and emotions that may be present in your subconscious mind. 8. **Software development**: I can assist with software development tasks, such as writing code, debugging, and optimizing algorithms. My knowledge is constantly evolving, and I'm always learning from the interactions I have with users like you. So, feel free to ask me anything – I'll do my best to help!