4 ms·
can anyone comment on running the 2b quantized llama 70b on consumer cards like the 4090?
by skavi 3y ago
can anyone comment on running the 2b quantized llama 70b on consumer cards like the 4090?
- beacon294 3y agoI have quantized Llama-2-chat-70b to 2q_k with llama.cpp then run on a 4090 for 50/83 layers. I was able to get 1.3 tokens per second. It answers this question, which smaller models have trouble with: <s>[INST]Hello. The functions addition, multiplication, and exponentiation can be considered each as a higher order of the previous. For example, 2+2+2 = 2*3 which is significant because there are three twos, which is the multiplication factor. Likewise 2*2*2=2^3, again, the power number represents the count of the numbers being multiplied. What is this concept and what is the next higher order function?[/INST] The concept you are referring to is called the "iterative" or "recursive" nature of arithmetic operations. Each operation can be seen as a repeated application of the previous operation, with the multiplication factor representing the number of times the operation is applied. For example, the addition of two numbers, 2+2, can be seen as a repeated application of the multiplication operation, where the multiplicand is 2 and the multiplier is 3 (i.e., 2*3 = 6, which is the result of the addition). Similarly, the multiplication of two numbers, 2*2, can be seen as a repeated application of the exponentiation operation, where the base is 2 and the power is 3 (i.e., 2^3 = 8, which is the result of the multiplication). The next higher order function would be the "tetration" operation, which is the iterative application of the exponential function. In other words, tetration(n,x) = x^(x^(x^(...(x^n)...))), where n is the number of times the exponential function is applied. For example, tetration(2,2) = 2^(2^(2^(2))), which can be simplified to 2^16, or 65536. It's worth noting that this concept can be extended even further to higher levels of abstraction, such as "hyper-operations" and "super -operations", but these are more advanced and less commonly used concepts in mathematics.*
- beacon294 3y agoThe answer seems kind of low quality, but it got further than others. It could probably self correct.
- taneq 3y agoI don’t know too many humans who could answer that half as well.
- quickthrower2 3y agoTo be fair it is a memory/recite test mostly. “Define this”
- epcoa 3y ago"For example, the addition of two numbers, 2+2, can be seen as a repeated application of the multiplication operation," I bet most humans ever born did not know what the word "multiplication" meant. Not sure how your metric is useful.
- philipswood 3y agoSometimes, to achieve some perspective, I also zoom out and declare normal over the estimated +-120,000,000,000 people who have ever lived.
- epcoa 3y agoI think the point went over your head (which just further proves mine). Hint: Perhaps read the original output more carefully: "For example, the addition of two numbers, 2+2, can be seen as a repeated application of the multiplication operation,". Or are you counted in the majority of that 120,000,000,000?
- philipswood 3y agoParis in the the spring
- skavi 3y agoto clarify, is that with https://huggingface.co/relaxml/Llama-2-70b-chat-E8P-2Bit https://huggingface.co/relaxml/Llama-2-70b-chat-E8P-2Bit
- beacon294 3y agoIt is Llama-2-70b-chat. I quantized it to 2q_k using `quantize` with llama.cpp.
- skykooler 3y agoNote that it got all the math wrong. 2+2 is a repetition of the succession operator, not multiplication; 2+2 equals 4, not 6; multiplication of two numbers is repeated addition, not exponentiation; 2*2 equals 4, not 8; and tetration(2,2) is 4, not 65536. This is actually the invariant in that n-ation(2,2) equals 4 for all natural numbers n.
- Philpax 3y agoTo be clear, this is not using QuIP#, and is thus not particularly relevant to the discussion at hand.
- lhl 3y agoIt's been on my list to do a proper shootout of all the various new quant formats floating around (my list here: https://llm-tracker.info/books/llms/page/quantization-overview https://llm-tracker.info/books/llms/page/quantization-overvi...) but a lot of them don't have very good production code yet (eg, a few months ago, when I tried OmniQuant, some of the important bits of code wasn't even included and had to be gotten directly from the authors: https://llm-tracker.info/books/llms/page/omniquant https://llm-tracker.info/books/llms/page/omniquant). If you're looking for the best widely deployed quant format atm, it's probably ExLlamaV2's EXL2 - it supports arbitrary bpw w/ a calibration file, and also 8-bit kvcache support. I haven't tested EXL2 much at lower bpws though. Note, both llama.cpp and AirLLM allow layer offloading to system memory (or in AirLLM's case, even to disk?!). r/LocalLlama probably is the best place to search for if you're looking for people's experiences w/ quants. I know some people have been testing, like: https://www.reddit.com/r/LocalLLaMA/comments/17klaa5/tested_exllamav2s_max_context_on_24gb_with_70b/ https://www.reddit.com/r/LocalLLaMA/comments/17klaa5/tested_...
- 0xDEADFED5 3y agoi humbly request you to add mlc-llm to your quant test when/if you get around to doing it
- lhl 3y agoSure, I think their quant format is pretty basic, something similar to bnb q4 - my plan will to be scripting a framework for testing, so should do that as well since the omniquant implementation is in mlc-llm anyways.
- acosmism 3y agoi was trying to get this to work with mlc-llm. i'd appreciate any pointers
- acosmism 3y agomore specifically on a non-cuda gpu - mali on orangepi via opencl
- 3y ago
- thulle 3y agoBetter output than the smaller llamas in my limited testing, but it's surprisingly slow: Output generated in 101.74 seconds (0.98 tokens/s, 100 tokens, context 82, seed 532878022) Output generated in 515.46 seconds (0.99 tokens/s, 511 tokens, context 27, seed 660997525) Checking nvidia-smi it stalls at ~130W (out of ~470 W max) power usage, ~25% GPU usage and ~10% memory bandwidth usage. There's fairly much traffic on the pci-bus though, and the python process is stable at 100% usage of one core. GPU possibly limited by some thing handled in python? Pausing the GPU-accelerated video-decoding of a twitch stream it get a surprisingly large boost: Output generated in 380.42 seconds (1.34 tokens/s, 511 tokens, context 26, seed 648992918)