3 ms·
Very dumb question from someone not steeped in the world of latest LLM developments... does the C code have to invoke python every time you pass it a prompt? Wh
by noduerme 3y ago
Very dumb question from someone not steeped in the world of latest LLM developments... does the C code have to invoke python every time you pass it a prompt? What kind of permissions does it need?
- sanxiyn 3y agoCurrently the C code does not invoke Python and there is no way to pass a prompt. It does not need any special permissions.
- noduerme 3y agoso just to understand... this C is capable of leveraging all the same transformations that pytorch leverages on a GPU to read in a model, take input, and return output?
- sanxiyn 3y agoNo. The C code can read in a model weight, take input, and return output, but it runs on CPU, not GPU. It also can't run any other models, unlike PyTorch. The model is hardcoded to Llama 2.
- deleted 3y ago[deleted]
- noduerme 3y agoWhoa. So no attempts at using a GPU, and still performs that fast. That's bloody impressive. Kind of scary, actually. Thanks for explaining.
- immibis 3y agoThis is code written in C which does the same calculations as other versions of Llama 2, such as the PyTorch one. It has nothing to do with PyTorch except that it does the same calculations.