3 ms·

attempt 1 with llama 3.1 405b what version llama are you I am an instance of Llama 2, an artificial intelligence model developed by Meta AI. how many paramet

by shmatt 2y ago

attempt 1 with llama 3.1 405b what version llama are you I am an instance of Llama 2, an artificial intelligence model developed by Meta AI. how many parameters I am a Llama 2 model with 7 billion parameters. Attempt 2 which llama are you exactly I am a variant of the LLaMA model, specifically LLaMA 1.5B. Here's a brief overview of the different LLaMA models: LLaMA 7B: This is the largest and most powerful LLaMA model, with 7 billion parameters. It's the most advanced and capable model in the LLaMA family. LLaMA 3B: This model has 3 billion parameters and is a smaller but still very capable version of the LLaMA model. LLaMA 1.5B: This is the model I'm based on, with 1.5 billion parameters. It's a smaller and more efficient version of the LLaMA model, but still very knowledgeable and capable. LLaMA 650M: This is the smallest LLaMA model, with 650 million parameters. It's a more lightweight and efficient version of the model, but still useful for many tasks. is 3.1 this bad at hallucinations? or are you just serving something else This just keeps on giving when does your training data end " My training data is a massive corpus of text that was available to me up to a certain point in time. My training data is cut off at December 2021."

Onawa 2y ago
People continually asking the models what their abilities and knowledge are demonstrates a disconnect in the capabilities of the raw models and what people are expecting them to be capable of. Many of the answers that models give in regards to their capabilities are actually from the system prompts provided from the chat interface providers.
- Zambyte 2y ago
  I asked mistral-nemo:12b-instruct-2407-q8_0 > Who are you? In my own machine without a system prompt and it said: > I am a language model developed through a collaboration between Mistral AI and NVIDIA. Often times this information will be included in the instruct fine tuning. The same from qwen2:7b-instruct-q8_0: > I am a large language model created by Alibaba Cloud. I'm called Qwen. All I did was change the model and re-run the prompt in Open WebUI, so if there was a system prompt, it would have been used for both.
reissbaker 2y ago
Hmm, yeah models typically don't know much about themselves and hallucinate when asked. We can probably fix this with a system prompt; it's the real 3.1-405B though. Try asking it for some code and compare it with llama-3-8b in comparison!
- reaperman 2y ago
  What quantization is used for it? I see you can run models that use up to approximately 640GB but the full non-quantized 405B uses ~810GB of VRAM (native weights for that model are fp16 before quantization).
  - reissbaker 2y ago
    * Models we run on our own infra (the ones with boot times) always run at whatever quantization the repo uses — typically unquantized, although if you paste in a link to a 4-bit AWQ repo of course that's what we'll run. * For always-on proxied models, we're dependent on the inference provider. Typically we use Together AI, but for Llama 3.1 we found some show-stopping context length related bugs that caused us to switch to Fireworks. Since we proxy 405B to Fireworks, that one runs at FP8 (that's what Fireworks uses). This may change in the future and in general we prefer to match the underlying repo quantization when possible; proxying is a bit of a hack in order to keep more models warmed up and reduce boot times (and to run 405B for now, although we're working on getting larger clusters and supporting distributed multi-node inference). As we grow, hopefully we'll be able to reduce our dependency on third party proxies. I'm also curious to try running models on AMD, since that would allow for much larger single-node models — for example, an MI300X has 192GB per card, so 8x of those would run 405B at fp16. Obviously there are some support issues with AMD so TBD how well it'll work but it's an interesting option.
mattnewton 2y ago
You need a system prompt. The raw model isn't going to see much data about itself yet in the training corpus - it hasn't been released yet when the corpus was assembled! It'd be like asking a child who was never spoken to before their name. How would the child know until someone told it?

by shmatt 2y ago

Onawa 2y ago

People continually asking the models what their abilities and knowledge are demonstrates a disconnect in the capabilities of the raw models and what people are expecting them to be capable of. Many of the answers that models give in regards to their capabilities are actually from the system prompts provided from the chat interface providers.

Zambyte 2y ago
I asked mistral-nemo:12b-instruct-2407-q8_0 > Who are you? In my own machine without a system prompt and it said: > I am a language model developed through a collaboration between Mistral AI and NVIDIA. Often times this information will be included in the instruct fine tuning. The same from qwen2:7b-instruct-q8_0: > I am a large language model created by Alibaba Cloud. I'm called Qwen. All I did was change the model and re-run the prompt in Open WebUI, so if there was a system prompt, it would have been used for both.

reissbaker 2y ago

Hmm, yeah models typically don't know much about themselves and hallucinate when asked. We can probably fix this with a system prompt; it's the real 3.1-405B though. Try asking it for some code and compare it with llama-3-8b in comparison!

reaperman 2y ago
What quantization is used for it? I see you can run models that use up to approximately 640GB but the full non-quantized 405B uses ~810GB of VRAM (native weights for that model are fp16 before quantization).
- reissbaker 2y ago
  * Models we run on our own infra (the ones with boot times) always run at whatever quantization the repo uses — typically unquantized, although if you paste in a link to a 4-bit AWQ repo of course that's what we'll run. * For always-on proxied models, we're dependent on the inference provider. Typically we use Together AI, but for Llama 3.1 we found some show-stopping context length related bugs that caused us to switch to Fireworks. Since we proxy 405B to Fireworks, that one runs at FP8 (that's what Fireworks uses). This may change in the future and in general we prefer to match the underlying repo quantization when possible; proxying is a bit of a hack in order to keep more models warmed up and reduce boot times (and to run 405B for now, although we're working on getting larger clusters and supporting distributed multi-node inference). As we grow, hopefully we'll be able to reduce our dependency on third party proxies. I'm also curious to try running models on AMD, since that would allow for much larger single-node models — for example, an MI300X has 192GB per card, so 8x of those would run 405B at fp16. Obviously there are some support issues with AMD so TBD how well it'll work but it's an interesting option.

mattnewton 2y ago

You need a system prompt. The raw model isn't going to see much data about itself yet in the training corpus - it hasn't been released yet when the corpus was assembled! It'd be like asking a child who was never spoken to before their name. How would the child know until someone told it?