4 ms·

Oh, this is a llama.cpp frontend. Y'all should have lead with that! I saw this on HN before, but I thought it was another from-scratch llama implementation...

Oh, this is a llama.cpp frontend. Y'all should have lead with that! I saw this on HN before, but I thought it was another from-scratch llama implementation... Which is fine, but much less interesting to me, as a from-scratch implementation probably not as fast/feature packed as llama.cpp or the TVM implementation. Keeping up with llama.cpp's rapid evolution is very difficult, and there's a need for projects like this.

UncleOxidant 3y ago
Where does one find this TVM implementation you mention?
- capableweb 3y ago
  Maybe they're talking about https://github.com/mlc-ai/mlc-llm https://github.com/mlc-ai/mlc-llm which is used for web-llm (https://github.com/mlc-ai/web-llm https://github.com/mlc-ai/web-llm)? Seems to be using TVM.
  - brucethemoose2 3y ago
    Yep. Its very fast on Vulkan, and from what I understand fast on metal, but its not as feature packed as llama.cpp yet.
mgreg 3y ago
I'm actually using Ollama for it's Rest API endpoint. Llama.cpp does now have it's server implementation. Unfortunately they do have different endpoints and behave a little differently. * https://github.com/jmorganca/ollama/blob/main/docs/api.md https://github.com/jmorganca/ollama/blob/main/docs/api.md * https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md https://github.com/ggerganov/llama.cpp/blob/master/examples/...
- lhl 3y ago
  I put together a list of OpenAI API compatibility layers for local LLMs recently: https://llm-tracker.info/books/llms/page/openai-api-compatibility https://llm-tracker.info/books/llms/page/openai-api-compatib... Some like c0sogi/llama-api are pretty neat because they support concurrency, and supports multiple backends (llama.cpp and Exllama, although it could be expanded). While you might lose out on some low-level configurability, being able to easily swap between OpenAI and local models is a big win in my book.
  - mgreg 3y ago
    Very cool and thanks for sharing. To me a killer feature would be easily running different models simultaneously such as one for embeddings and another for completion (e.g. Chat). This likely can be done already by specifying the model parameter in Ollama (and others) but I've not explored it much yet.
  - d4rkp4ttern 3y ago
    Ooba is my favorite. It automatically converts chats to single prompts using the model-specific (finicky) formatting specs (e.g [INST] [/INST] etc for llama2) so that you can directly submit chat dialogs to the endpoint. This is a subtle point not obvious to many and I wrote about it here — https://langroid.github.io/langroid/blog/2023/09/19/language-models-completion-and-chat-completion/ https://langroid.github.io/langroid/blog/2023/09/19/language...
deleted 3y ago
[deleted]
jmorgan 3y ago
There's a ton of cool opportunity in the runtime layer. I've been keeping my eye on the compiler-based approaches. From what I've gathered many of the larger "production" inference tools use compilers: - https://github.com/openai/triton https://github.com/openai/triton - https://github.com/NVIDIA/TensorRT https://github.com/NVIDIA/TensorRT TVM and other compiler-based approaches seem to really perform really well and make supporting different backends really easy. A good friend who's been in this space for a while told me llama.cpp is sort of a "hand crafted" version of what these compilers could output, which I think speaks to the craftmanship Georgi and the ggml team have put into llama.cpp, but also the opportunity to "compile" versions of llama.cpp for other model architectures or platforms.
- factibicongue 3y ago
  [flagged]
andy99 3y ago
Hi, fyi I am working on a from-scratch implementation (currently llama2 on linux focused) in Fortran, CPU only but in my initial test about as fast as llama.cpp. Currently has fp16 and 4-bit quantization and I hope this week to finish support for ggml files - I based it off of Karpathy's llama2.c and so it uses that format now which is not great. Llama.cpp is the leader and has more diverse hardware support, for CPU inference and simplicity (complexity of llama.cpp had exploded) there is still room for competition I believe. https://github.com/rbitr/llama2.f90 https://github.com/rbitr/llama2.f90 Once I make it a bit easier to use I want to promote it more.

UncleOxidant 3y ago

Where does one find this TVM implementation you mention?

capableweb 3y ago
Maybe they're talking about https://github.com/mlc-ai/mlc-llm https://github.com/mlc-ai/mlc-llm which is used for web-llm (https://github.com/mlc-ai/web-llm https://github.com/mlc-ai/web-llm)? Seems to be using TVM.
- brucethemoose2 3y ago
  Yep. Its very fast on Vulkan, and from what I understand fast on metal, but its not as feature packed as llama.cpp yet.

mgreg 3y ago

I'm actually using Ollama for it's Rest API endpoint. Llama.cpp does now have it's server implementation. Unfortunately they do have different endpoints and behave a little differently. * https://github.com/jmorganca/ollama/blob/main/docs/api.md https://github.com/jmorganca/ollama/blob/main/docs/api.md * https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md https://github.com/ggerganov/llama.cpp/blob/master/examples/...

lhl 3y ago
I put together a list of OpenAI API compatibility layers for local LLMs recently: https://llm-tracker.info/books/llms/page/openai-api-compatibility https://llm-tracker.info/books/llms/page/openai-api-compatib... Some like c0sogi/llama-api are pretty neat because they support concurrency, and supports multiple backends (llama.cpp and Exllama, although it could be expanded). While you might lose out on some low-level configurability, being able to easily swap between OpenAI and local models is a big win in my book.
- mgreg 3y ago
  Very cool and thanks for sharing. To me a killer feature would be easily running different models simultaneously such as one for embeddings and another for completion (e.g. Chat). This likely can be done already by specifying the model parameter in Ollama (and others) but I've not explored it much yet.
- d4rkp4ttern 3y ago
  Ooba is my favorite. It automatically converts chats to single prompts using the model-specific (finicky) formatting specs (e.g [INST] [/INST] etc for llama2) so that you can directly submit chat dialogs to the endpoint. This is a subtle point not obvious to many and I wrote about it here — https://langroid.github.io/langroid/blog/2023/09/19/language-models-completion-and-chat-completion/ https://langroid.github.io/langroid/blog/2023/09/19/language...

deleted 3y ago

[deleted]

jmorgan 3y ago

There's a ton of cool opportunity in the runtime layer. I've been keeping my eye on the compiler-based approaches. From what I've gathered many of the larger "production" inference tools use compilers: - https://github.com/openai/triton https://github.com/openai/triton - https://github.com/NVIDIA/TensorRT https://github.com/NVIDIA/TensorRT TVM and other compiler-based approaches seem to really perform really well and make supporting different backends really easy. A good friend who's been in this space for a while told me llama.cpp is sort of a "hand crafted" version of what these compilers could output, which I think speaks to the craftmanship Georgi and the ggml team have put into llama.cpp, but also the opportunity to "compile" versions of llama.cpp for other model architectures or platforms.

factibicongue 3y ago
[flagged]

andy99 3y ago

Hi, fyi I am working on a from-scratch implementation (currently llama2 on linux focused) in Fortran, CPU only but in my initial test about as fast as llama.cpp. Currently has fp16 and 4-bit quantization and I hope this week to finish support for ggml files - I based it off of Karpathy's llama2.c and so it uses that format now which is not great. Llama.cpp is the leader and has more diverse hardware support, for CPU inference and simplicity (complexity of llama.cpp had exploded) there is still room for competition I believe. https://github.com/rbitr/llama2.f90 https://github.com/rbitr/llama2.f90 Once I make it a bit easier to use I want to promote it more.