3 ms·
> Would be amusing if they were to release a Mac Pro with 300+ GB and dominate the LLM serving space Is there any framework that can batch LLMs on Metal? I don
by brucethemoose2 3y ago
> Would be amusing if they were to release a Mac Pro with 300+ GB and dominate the LLM serving space
Is there any framework that can batch LLMs on Metal? I don't think GGML or MLC have it yet.
Otherwise that is just another reason they wouldn't be good for LLM hosting at this moment.
Anyway, the real disruptor is Intel. They could theoretically barge in with a 2x48GB Arc card and undercut the market that AMD/Nvidia refuse to dive into because of their Pro card clients.
- astrange 3y agoIt's coming to llama.cpp soon. https://github.com/ggerganov/llama.cpp/pull/3228 https://github.com/ggerganov/llama.cpp/pull/3228
- brucethemoose2 3y agoAmazing! Its really a shame that Apple (and AMD/Intel or pretty much any other infrence vendor) are not directly contributing to llama.cpp. The feature set is amazing and growing at a stunning pace.
- nomel 3y agoI imagine the legal concerns around it are part of the reason. Where I am, our legal team has been very direct, that anything llama based cannot touch a work computer.
- stringtoint 3y agoCurious to know why this is the case? I haven't played around with LLAMA yet but I figured it being open-source would make it more trustworthy than models provided by OpenAI.
- nomel 3y ago> it being open-source Absolutely not. The LLAMA license [1] is clear that it's not open source. It's for non-commercial, research only, and only by explicit permission from Meta. The weights were leaked, on 4chan [2], illegally, according to the license. Very very few people are using it legally. This interpretation is clear from its wording, and also matches the interpretation of our flock of lawyers. [1] https://github.com/facebookresearch/llama/issues/266 https://github.com/facebookresearch/llama/issues/266 [2] https://levelup.gitconnected.com/metas-chatgpt-is-now-illegally-free-for-everyone-5b58524caeb5 https://levelup.gitconnected.com/metas-chatgpt-is-now-illega...
- astrange 3y agoLlama 2 doesn't use that license and llama.cpp isn't limited to using those models anyway.
- deleted 3y ago[deleted]
- nomel 3y agoTrue. That gives these companies a little less than two months to start contributing, with the easier license restrictions in mind. But, Llama 2 is not open source [1]. [1] https://blog.opensource.org/metas-llama-2-license-is-not-open-source/ https://blog.opensource.org/metas-llama-2-license-is-not-ope...
- brucethemoose2 3y agoI wonder what legal teams (such as y'alls legal dept) think of the new license.
- stringtoint 3y agoYou're correct, I confused it with LLAMA being more business-friendly (at least in theory) than some other models out there.
- deleted 3y ago[deleted]