12 ms·
M2 Ultra can run 128 streams of Llama 2 7B in parallel
- deleted 3y ago[deleted]
- nvm0n2 3y agoWith these improvements llama.cpp/ggml is really becoming a pretty competitive serving stack even for large scale cloud hosted AI. I wonder how ggerganov finds the time to do all this, does anyone know if he's being sponsored?
- killthebuddha 3y agoHe founded a startup https://ggml.ai/ https://ggml.ai/
- selectodude 3y agoDoesn’t seem like much of a business model there.
- vineyardmike 3y agoIDK, I can see a future. It’s a one-man (for now) business, so minimal costs to consider. If he can swing consulting using the .cpp projects as advertising, that sounds like a good business. Additionally, I can imagine companies investing and paying for the open source work to expand access to their licensed models. Use the same interface as people use LLAMA but upgrade to BetterModel, fully compatible. Additionally, I could believe this is simply a build up to a future Acquihire, which is the most lucrative way to be hired.
- simonw 3y agoHe also raised a bunch of money, so I guess the plan is to figure out the business model as he goes.
- leetharris 3y agoFor applications that aren't latency sensitive my company has found that Apple hardware inference is far less expensive than the competition when calculating for electricity usage. I wish they would make a cloud offering.
- crooked-v 3y agoYou could always rack mount some Mac Minis.
- ninkendo 3y agoIt’s sad that racking and maintaining your own physical hardware is becoming such a lost art… I appreciate the up-front simplicity of cloud offerings as much as anyone, but there’s something to be said for owning your own hardware and avoiding the continual rent payments you’re sending the cloud providers. The wisdom is that cloud providers are better at infra than you, and that the economies of scale make it better to piggy back on what they’re doing, but… AWS is the most profitable part of Amazon for a reason. They’re overcharging you.
- cherryswitch 3y agoSometimes the flexibility and time savings is worth the added upfront cost. Similar to how companies like to hire consultants or lease office space. Being able to walk away is better for short term, because companies value profits in the short term.
- ilc 3y agoFor most orgs: AWS is not overcharging IMHO. When you look at the cost of the hardware + hosting. Yes, it certainly looks and feels that way. But if you've dealt with corporate IT, and had to deal with 3-6 month lead times on getting hardware, or politics to get your hands on hardware to get stuff done. AWS is cheap. It gives you velocity. If your company is large enough that it can offer the elasticity of resources that Amazon offers or even 1/4 of it... and you have an IT org that will let it happen. Yes, AWS is a waste. But with AWS... when a project dies, you can wipe its costs out, people won't hold onto hardware so they have hardware for the next project, etc... Trust me. I've been IT, I can spec and build rack systems. I am a software dev. And I've been a dev most all my career. For 90%+ of orgs... they don't have the maturity and skills to handle that type of infra without substantially distracting from their primary business.
- Metus 3y agoDo these implementations use the neural engine? I saw that there was a stable diffusion implementation using the neural engine and I found that my macbook noticably did not run hot, as opposed to an average Teams call.
- snitty 3y agoIt doesn't. You need to generate models for use on the neural engine, which apple did for Stable Diffusion, but this is just taking advantage of lots of fast RAM and lots and lots of threads, if I understand it correctly.
- ramesh31 3y agoIt uses Metal acceleration, and takes advantage of the shared memory architecture, meaning it's basically a GPU with 196GB VRAM. Trading space (VRAM) for time (FLOPs), it can beat the performance of an RTX4080 here.
- GaggiX 3y agoAutoregressive transformer models are usually memory bound, whereas SD is compute bound, so perhaps the difference lies here. Also the reason why SD runs so much faster on the GPU than on the CPU.
- ninkendo 3y agoM1 has (fast) unified memory between GPU and CPU, so something being memory bound ought not to have much bearing on whether it belongs on CPU or GPU… at least in theory. I’m a total noob here though so I may be wrong.
- mistrial9 3y agoLLaMA2 is English-centric fwiw, censored for safety, and black-box on its contents. Happy to be proved wrong
- mensetmanusman 3y agoThink of the censoring as comedy. It doesn’t like Kanye music and will tell you to listen to the song ‘Happy’ instead. That’s pretty hilariously dystopian and therefore quite funny.
- extasia 3y agoYou're right, none of that is disputed though?
- crooked-v 3y agoOn the second point, there's an uncensored version available and assorted trained-for-purpose derivatives of it.
- mistrial9 3y agolinks please!
- mschuster91 3y agoYou can run it on ollama: https://ollama.ai/blog/run-llama2-uncensored-locally https://ollama.ai/blog/run-llama2-uncensored-locally
- nomel 3y agoThese are not the uncensored models. It's a fine tune of the censored models [1]. > Filter refusals and bias from the dataset -> finetune the model -> release. The alignment tax should still exist, maybe doubly so. [1] https://erichartford.com/uncensored-models#heading-lets-get-down-to-business-uncensoring-wizardlm https://erichartford.com/uncensored-models#heading-lets-get-...
- gcr 3y agoFor folks curious what this is, this seems to be a caching optimization for saving time on parallel streams of text. The benchmark is incidental. Most individual users likely have just one un-batchable conversation going at once with llama-cpp, and I think it’s unclear whether this PR improves that case much. Also note that the demo video is sped up to fit inside GitHub attachment limits. Your observed speed may vary. :)
- fudged71 3y agoSo this will only work on, say, one prompt being run many times at once?
- gpderetta 3y agoI always wondered of pipelining/parallelization can help with AutoGPT-like tasks were a supervisor AI delegates subtasks to sub instances.
- londons_explore 3y agoSpeed-wise, on a single stream, I have no need for it to generate text faster than I can read. However, for scripts that try to use hundreds/thousands of invocations to solve some problem (eg. "write me a whole book"), the parallelism will be great (but obviously the script has to be written with that in mind).
- lukeschlather 3y agoI've been curious about using LLMs for large-scale refactoring. Prompts like anywhere you find `FooBarBaz(blip, kap)` replace it with `new newThing(blip).bump(kap)` I don't know how reliable it is, but it seems like if you can easily run this on commodity hardware it could totally replace most IDE refactoring tools, although obviously the IDE refactoring is more reliable, it seems like this could be made simple and flexible, and possibly just as reliable as IDEs. But also it could enable some interesting things that you could never do with an IDE refactoring tool.
- potatoman22 3y agoLLMs aren't very good at string operations. They struggle with things like character counts, extracting substrings, and replacements.
- ugh123 3y agoIs it possible that in a few years time, only Mac silicon and PCs with high-end GPUs will be required to run "In-home LLMs" affordably? If we get closer to a either "AGI" (whatever the hell that is) or at least a reasonably useful AutoGen/BabyAGI-like system that become popular to use at home, those machines will be the only ones capable of running advanced LLMs without having to pay OpenAI, Microsoft, Amazon/AWS, etc inordinate sums of money to do what consumers will deem a utility some day.
- anotherhue 3y ago> Is it possible that in a few years time, only Mac silicon and PCs with high-end GPUs will be required to run "In-home LLMs" affordably? No. They work well on the apple chips thanks to the integrated memory and the large size of the models. I know of no reason why an x86 chip could not be designed in a similar way if desired. IANAChipDesigner but I have worked for one of them.
- behnamoh 3y agoFWIW, while Apple silicon can _run_ huge models thanks to the unified memory (not to be confused with shared memory), the inference is pretty slow compared to dedicated GPUs, so it's a tradeoff. The significance of this PR is that inference speed can—at least in certain applications—be sped up using parallel decoding.
- behnamoh 3y agoI'm waiting for someone to comment "use the page title/why did you change the title/etc.". It's frustrating when you find something important on a page and type that as the title, and then the post gets flagged because it violates HN rules. For comparison, this is the actual title of the page, but do you think this would increase people's awareness about the fascinating fact I highlighted in the title? llama : custom attention mask + parallel decoding + no context swaps #3228
- solardev 3y agoYeah, I really hate the "no changing titles" rule. I can understand something like "don't sensationalize", but many articles just have poor titles that lack context. What does that accomplish aside from discouraging readership and discussion?
- mortenjorck 3y agoThe rule is officially "don't editorialize" which I would interpret (perhaps incorrectly) as allowing a little leeway in surfacing a buried lede so long as it's presented in neutral language. Something like "Amazing Llama 2 7B performance on M2 Ultra" would obviously fail that test, but the current title of "M2 Ultra can run 128 streams of Llama 2 7B in parallel" seems to follow the spirit of the rule, at least as I read it.
- solardev 3y agoI think the guidelines[1] say not just "don't editorialize", but (emphasis added): > Otherwise please use the original title, unless it is misleading or linkbait; don't editorialize. [1] https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- mtillman 3y agoExcept the original title in my experience is always longer than they allow.
- eurekin 3y agoOk, impressive! What are real world use cases for 7B family of models? Is anyone using them for anything productive?
- deleted 3y ago[deleted]
- anotherjesse 3y ago7B coding models? Having massive amounts of questionable code :)
- Havoc 3y agoThey're perfectly fine for story telling and basic chatbot duty. Also generating basic code boilerplate works just fine.
- potatoman22 3y agoThey make good classifiers when fine tuned
- eurekin 3y agoInteresting! I'd have one use case for classification: user text (from a jira issue) mapped to the team responsible for the fix. Can you share some tutorials? I only just managed to get this working on windows/cuda: https://colab.research.google.com/drive/1vk8i01apaSp59GVV2yInxOV15QwCwMrg#scrollTo=OtXBD2kreFmp https://colab.research.google.com/drive/1vk8i01apaSp59GVV2yI... It's been a royal pain to setup
- deleted 3y ago[deleted]
- teen 3y agoIt still can't run 10 year old games though (dota 2)
- sgjohnson 3y agoI’ve heard that there are some issues of gaming performance on M1/2 Ultra specifically (due to it being just 2x M2 Max in the same package), however my M2 Max MacBook absolutely runs Dota 2, and runs it very, very well. Like 180-200fps average well.
- dagmx 3y agoYou’re purposefully conflating hardware capability and product availability/priority. The hardware can run AAA games today. You might as well say the same thing about the PS5 because it can’t run an Atari game.
- ngcc_hk 3y agoWonder about CUDA issue. Some can but most cannot ?
- mrb 3y agoFor those wondering why the M2 Ultra is so fast, or the M1 & M2 series in general, it's because inference's main bottleneck is memory bandwidth, not compute power. And the M2 Ultra has a bandwidth of 800 GB/s which is about 8 times faster than an average modern desktop CPU (dual-channel DDR4-6400 offers a bandwidth of 102 GB/s). This high bandwidth is really a result of Apple having designed a unified memory architecture for the M1 and M2 chips. Typically on a laptop or desktop, the CPU and GPU have distinct memory systems: high-bandwidth (but relatively low-capacity) graphics memory, and relatively low-bandwidth (but high-capacity) CPU memory. Apple decided to simplify that and instead implemented a single high-bandwidth memory system shared by the CPU and GPU. The only downside is that such high-bandwidth memory had to be tightly integrated in the M2 package, so the maximum capacity is limited. For example whether you spend 5,600 USD (cheapest Mac Studio machine with M2 Utra and 192 GB) or $10k+ (maxed out Mac Pro), you will only ever get 192 GB RAM max. For that amount, a PC could get 1024 GB RAM (5× more!) But on the other hand, if your workload, like inference, doesn't need more than 192 GB, then that's great. Personally I think Apple made the right tradeoff here. 800 GB/s of memory bandwidth on a general purpose CPU, on a single socket, has never been done before (to my knowledge.)
- kristianp 3y agoCompare with an rtx 4090, which has memory bandwidth of 1,008 GB/s, but only 24GB of gddr. The 4090 is cheaper. https://www.techpowerup.com/gpu-specs/geforce-rtx-4090.c3889 https://www.techpowerup.com/gpu-specs/geforce-rtx-4090.c3889
- mrb 3y agoI agree, GPUs can still generate more token per second per dollar, but what is new and great about the high-end M1 and M2 is Apple offering this much memory bandwidth on a general purpose CPU, thus immediately available to all software running on the CPU.
- boesboes 3y agoUnless you calculate power usage I’d bet
- deleted 3y ago[deleted]
- bick 3y agoAnyone here wanting to try an M1 Mac mini 16GB in the cloud for free this month just send me an email (click handle). Now that we've moved on to the M2, I've got a hand full of the M1 available FREE for trying. You can also try an M2 Pro, Max, or Ultra but for that you'll need to subscribe. https://www.macweb.com/macinthecloud https://www.macweb.com/macinthecloud