3 ms·
CPU isnt realistic, regardless what the marketers tell you.
by resource_waste 2y ago
CPU isnt realistic, regardless what the marketers tell you.
- simonw 2y agoiPhones have GPUs. I've successfully run Mistral on mine using this app: https://llm.mlc.ai/docs/deploy/ios.html https://llm.mlc.ai/docs/deploy/ios.html
- resource_waste 2y agoPlease don't say 'what is technically possible' It blurs what is realistically useful. Also 'GPUs' No. This is a major red flag man. They do not. The marketers told you this, and you believed them.
- simonw 2y agoI don't get it. The iPhone does have a GPU, and it can be used to accelerate AI workloads. This isn't theoretical. You can see this for yourself by installing https://apps.apple.com/us/app/mlc-chat/id6448482937 https://apps.apple.com/us/app/mlc-chat/id6448482937 and turning off your wifi and running the quite capable Mistral 7B Instruct LLM on your phone. I've even used it to answer simple questions while I was offline and only had my phone with me. What am I missing here?
- resource_waste 2y agoWhat like 50 token responses? Buddy I'm cranking out 4k. Also LOL at the GPU thing. Its insane how well Apple marketing worked.
- simonw 2y agoI didn't say an iPhone was as good as a desktop with a dedicated graphics card. I said that an iPhone has a GPU and is capable of running a small but capable LLM. Most people don't know that's possible yet. Why do keep indicating that iPhones don't have a GPU?
- regularfry 2y agoI can and do run phi-2/3 on cpu, which are in the same size class.