3 ms·
No. The C code can read in a model weight, take input, and return output, but it runs on CPU, not GPU. It also can't run any other models, unlike PyTorch. The m
by sanxiyn 3y ago
No. The C code can read in a model weight, take input, and return output, but it runs on CPU, not GPU. It also can't run any other models, unlike PyTorch. The model is hardcoded to Llama 2.
- deleted 3y ago[deleted]
- noduerme 3y agoWhoa. So no attempts at using a GPU, and still performs that fast. That's bloody impressive. Kind of scary, actually. Thanks for explaining.