4 ms·
Anyone more knowledgeable in this space please explain what is meant by inference? From what I know. LLaMa is built in Python and assuming PyTorch, does this p
by mattfrommars 4y ago
Anyone more knowledgeable in this space please explain what is meant by inference?
From what I know. LLaMa is built in Python and assuming PyTorch, does this port in Rust make use of Python process or is it what LLaMA algorithm is fully written in Rust?
- tantony 4y agoIt uses "ggml", a tensor library written in C to do the math. The models weights are converted to a format that can be used by GGML. The higher level structure of the model is created in Rust be leveraging ggml data structures and the weights are passed into the C library. PyTorch is only needed for reading the original weights while converting them.