8 ms·
Run llama3 locally with 1M token context
- deleted 2y ago[deleted]
- superkuh 2y ago> Note: using a 256k context window requires at least 64GB of memory. Using a 1M+ context window requires significantly more (100s of GBs). The ram requirements at 200k seem to be using an assumed 16 bits? With a 4 bit quantization it's more like ~12GB of kv-cache for for 200k sequence length. Unless I'm missing something?
- datadrivenangel 2y agoHow much does this do to performance though?
- deleted 2y ago[deleted]
- sebzim4500 2y agoAre people quantizing the activations? I was under the impression that people generally quantize the parameters but leave the activations unchanged.
- brrrrrm 2y agoYou can quantize KV caches
- consumer451 2y agoPardon my ignorance, but I am a dilettante. I was just wondering about this earlier today. How much unlocking of a more active intelligence from LLMs, will far larger context length provide? Would virtually unlimited context move us much closer to an LLM where training is continuous? Is that a bit of a holy grail? I assume not, but would love to know why.
- mritchie712 2y agocontext length doesn't unlock intelligence, just more information. e.g. adding info that wasn't part of training (or wasn't heavily weighted in training).
- consumer451 2y agoRight, I should have been more clear than the words "active intelligence." But as one use of this... would unlimited, say 1 to 10 billion tokens of context, used as a system prompt, with "just" 32k left for the user, allow a model to be updated every day, in between actual training? (This is in the future, where training only takes a month, or much less.) I guess part of what I really don't understand is how context tokens compare to training weights, as far as value to the final response. Would a giant context window muddle the value of weights? (Maybe what I am missing is the human-feedback on the training weights? If the giant system prompt I am imagining is garbage, then that would be bad.)
- nl 2y agoIn-context learning (ICL) is a thing. People aren't entirely sure how it works[1]. LLMs are very effective at few-shot learning via so yes, for all practical purposes yes, large context windows do allow for continuous learning. Note that the context needs to be loaded and processed on every request to the LLM though - so all that additional information has to be "retaught" each time. [1] https://openreview.net/pdf?id=992eLydH8G https://openreview.net/pdf?id=992eLydH8G "These results indicate that the equivalence between ICL (ed: In-context-learning) and GD (ed: Gradient Descent) is an open hypothesis, requires nuanced considerations, and calls for further studies.
- consumer451 2y agoThank you so much for your response. It's amazing what typing a little acronym like "ICL" can do as far as sharing knowledge. This is so cool! Also, your link appears to exactly address my question. It's late here, but I am very excited to do my best at understanding that paper in the morning.
- Workaccount2 2y agoI've been using Gemini 1.5 with 1M tokens and it's a totally different game. You can essentially "train" the model on whatever hyper specific stuff you have on hand. Don't know how to work with my test system? No problem, here is an 800 page reference manual. Now you know.
- rahimnathwani 2y agoI'm curious what your setup is? I mean what platform, how much RAM and which particular version of the model?
- lennxa 2y agoThey are talking about Google's Gemini, not running locally.
- rahimnathwani 2y agoOops I confused Gemini and Gemma.
- theolivenbaum 2y agoWhat kind of latency do you observe when using Gemini with such large contexts?
- dmazin 2y agoInference takes 30+ seconds before it starts outputting even the first token.
- danielbln 2y agoHow's recall over 1M tokens?
- Workaccount2 2y agoAround 250k tokens its around 15-20 seconds. When you really pack it full (which I have only done just to test it) it can take a minute before it outputs anything. It also sometimes glitches out with massive contexts, and just stops generation mid sentence. Asking it to repeat the response clears it up though.
- haolez 2y agoCan we expect that the 1M version has the same level of intelligence as the vanilla version? Across the whole 1M tokens without degradation? What are the trade offs?
- haolez 2y agoWell, according to LocalLLaMa, it's surprisingly good: https://www.reddit.com/r/LocalLLaMA/s/ypFd2Xlnvf https://www.reddit.com/r/LocalLLaMA/s/ypFd2Xlnvf
- mehulashah 2y agoOur results are varied on long contexts. True — one can put a lot of stuff in there and the thing groks a bunch of it. Being able to synthesize specific answers from “long tail” facts in the context can be difficult. YMMV.
- benreesman 2y agoIt’s just so clear the math is wrong on these things. You’ve got an apparent contradiction: SGD (AdamW at 1e-6 give or take) works. So we’ve got extremely abundant local maxima up to epsilon_0, but, it always lands in the same place, so there are abundant “well-studied” minima, likewise symmetrical up to epsilon_1, both of which are roughly: “start debugging if above we can tell”. The maxima have meaningful curvature tensors at or adjacent to them: AdamW works. But joker in the deck: control vectors work. So you’re in a quasi-Euclidean region. In fact all the useful regions are about exactly the same, the weights are actually complex valued, everyone knows this part…. The conserved quantity up to let’s call it phi is compression ratio. Maybe in a year or two when Altman is in jail and Mme. Su gives George cards that work, well crunch numbers more interesting than how much a googol FMA units cost and Emmy Nother gets some damned credit for knowing this a century ago.
- benreesman 2y agoRare rule violation on the downvote thing. Make a dissenting case or leave it the fuck alone. I’ve been pretty laid back about the OpenAI bot thing but I’m over it.
- benreesman 2y ago@dang there are the same number of points on me finally accusing big Azure IPv4 blocks of manipulating the site as there are downvotes on an argument south of preprint but north of what passes for AI math here. This is by no means the worst example this month. You run the best moderation team on the Internet, but no one at OpenAI (including Fidji) will flat deny they’re doing it, and it’s just obvious. I know you’re doing yeoman’s work like always. Have someone let @sama that at least one person is going to start making charts. Not here.
- khaki54 2y agoBen, honestly some of your comments are extremely abstruse to the point where people can't tell if you are serious or not. That's my take anyway (I don't have downvote power)
- ukuina 2y agoTested this on Apple Silicon at 128k context, very poor results with random character hallucinations.
- growfeather 2y agoThanks for testing it. I wonder if there’s a reasonable line? 64k? How much RAM did the 128k context require? (Maybe future models might be better at that level)
- nicognaw 2y agoIf you got random tokens on Apple Silicon Macs, you should check your stderr for Metal’s error code, like 5 for out of metal memory. Metal lets you use half of your unified memory as metal memory.
- rahimnathwani 2y agoIs this better than the large context window versions of dolphin-llama3? e.g. https://ollama.com/library/dolphin-llama3:8b-256k-v2.9-q6_K https://ollama.com/library/dolphin-llama3:8b-256k-v2.9-q6_K
- deleted 2y ago[deleted]
- labrador 2y ago"run locally" is meaningless if you don't state the hardware requirements unless you don't mind wasting the readers time making him or her figure it out only find out it takes a $20,000 computer. Or is this just assumed?
- boredemployee 2y agoman, exactly. it gets on my nerves.
- labrador 2y agoI asked Claude Opus about it and got an estimate. It's going to take over 100GB (A 128 GB RAM stick can cost around $1,500) to run locally and probably mulitple Nvidia A100's ($10,000 per) to run locally, so if you're rich or work is supplying the hardware or cloud it's possible. What are the hardware requirements to run the model I gave you? Based on the provided document, the hardware requirements for running the llama3-gradient model depend on the desired context window size: To use a 256k context window, at least 64GB of memory is required. To use a 1M+ context window, significantly more memory (100GB+) is needed. The exact requirements are not specified, but it's clear that running this model with very large context windows requires a machine with a substantial amount of RAM, likely a high-end workstation or server rather than a typical consumer PC. The computational demands also likely necessitate a powerful CPU and/or GPU. CopyMWwhat type of CPU and what type of GPU's and how many would it take?The document does not provide specific details on the CPU or GPU requirements for running the llama3-gradient model. However, based on the memory requirements mentioned and the nature of the model, we can make some educated guesses: CPU: For optimal performance, a high-end, multi-core server-grade CPU would likely be necessary, such as an Intel Xeon or AMD EPYC processor with a high core count (e.g., 16+ cores). GPU: Running large language models often benefits greatly from the use of one or more powerful GPUs. High-end consumer GPUs like the Nvidia GeForce RTX 3090 (24GB VRAM) or data center GPUs like the Nvidia A100 (40GB-80GB VRAM) would likely be suitable. The number of GPUs required would depend on the specific implementation and the desired performance. A single high-end GPU might be sufficient, but using multiple GPUs could speed up processing. Keep in mind that these are estimates based on the provided information. The actual hardware requirements may vary depending on the specific implementation, optimizations, and performance requirements. Running such a model with very large context windows would likely require a powerful, expensive system with ample RAM, fast storage, and robust cooling. Detailed benchmarks or recommendations from the model's developers would provide more definitive guidance on the exact hardware needed.
- tankado 2y ago"Note: using a 256k context window requires at least 64GB of memory. Using a 1M+ context window requires significantly more (100GB+)."