4 ms·
As someone entirely at sea with the rapid pace of development in this sphere: 1. Is this a new LLM from Apple? 2. Is this a way to optimize running LLMs like
by hannofcart 4y ago
As someone entirely at sea with the rapid pace of development in this sphere:
1. Is this a new LLM from Apple?
2. Is this a way to optimize running LLMs like Llama locally on M1 macs?
3. Something else altogether?
- uoaei 4y ago2. A Transformer is a core building block of LLMs. > [T]he device spec for this reference implementation is M1 or newer chips for the Mac and A14 and newer chips for the iPhone and iPad
- jeffbee 4y agoIt's none of those things. It is tweaks of other existing code to run better on apple's hardware. This other article is far more informative than the repo: https://machinelearning.apple.com/research/neural-engine-transformers https://machinelearning.apple.com/research/neural-engine-tra...
- sheepscreek 4y ago#2. A way to optimize running LLMs locally on Apple Silicon (including iPhones) I am just a little better informed. As I understand it, their code improves model performance and memory consumption using PyTorch and Huggingface libraries.
- singularity2001 4y agoit's python code so not for iphone (?)
- sheepscreek 4y agoSorry - I should have been more clear. Their examples compared the A* CPU performance and the repo includes Swift only code samples. But they’ve also made it possible to use them with traditional tooling (torch, huggingface). Hope that helps explain it.
- simonh 4y agoIt this provides optimised hardware acceleration on Apple Silicon for low level machine learning libraries. PyTorch is supported for example, it’s a machine learning library with GPU acceleration that’s been around for 6 years now. It’s used in a few commercial projects, including Tesla Autopilot. It can be used for natural language processing, image manipulation, and possibly to build an LLM I suppose, but as a low level library it just gives the base tech to build such systems from.
- JCharante 4y agoMy understanding is that llama.cpp is a port of meta’s llama release that uses PyTorch. Then I hope we can run llama with Apple Neural Engine.
- sebzim4500 4y agollama.cpp does not use PyTorch, it uses a niche ML library made by its creator that only runs on the CPU.
- JCharante 4y agoI’m under the impression that llama.cpp is a fork of llama (which uses pytorch)