3 ms·

Do you think it's possible also to create a trainer in pure C, instead of using python?

karpathy 3y ago
In principle easy and possible, just not exactly useful. Would just involve adding the backward pass. But I’m not sure that this is something many people would want.
immibis 3y ago
Of course it's possible. The question is whether anyone finds it worth doing. ML algorithms are, at their core, not particularly complicated code. But they are still tricky code, because if you get them wrong you will find that you spent 500 GPU-years turning random numbers that cause the model to output gibberish into other random numbers that cause the model to output different yet semantically identical gibberish. Writing them in a more abstract languages has advantages - like automatic differentiation. You could explicitly tell the computer how to compute the output and its derivative, or you could tell the computer how to compute the output, and let it also compute the derivative by itself. Having all your weights in one object is also awfully convenient; you can write something like `weights -= error * deriv * learning_rate` instead of iterating over each individual weight (and a large model contains many different sets of weights, not just a single NxMxPxQ matrix) This is good for the rapid iteration that ML research demands. However, once you have selected a model, I'm sure you can get performance advantages by coding it in a low level and eliminating inefficiencies. For example, you should be able to use the weight update equation from above by using fused multiply-accumulate, and the Python framework might not realize that.
DougBTX 3y ago
This is C++ rather than C, but a substantial portion of PyTorch is written in C++, and they provide a C++ interface: https://pytorch.org/tutorials/advanced/cpp_frontend.html https://pytorch.org/tutorials/advanced/cpp_frontend.html In other words, you can absolutely use PyTorch without Python.
voz_ 3y ago
Just compile the Python

In principle easy and possible, just not exactly useful. Would just involve adding the backward pass. But I’m not sure that this is something many people would want.

immibis 3y ago

Of course it's possible. The question is whether anyone finds it worth doing. ML algorithms are, at their core, not particularly complicated code. But they are still tricky code, because if you get them wrong you will find that you spent 500 GPU-years turning random numbers that cause the model to output gibberish into other random numbers that cause the model to output different yet semantically identical gibberish. Writing them in a more abstract languages has advantages - like automatic differentiation. You could explicitly tell the computer how to compute the output and its derivative, or you could tell the computer how to compute the output, and let it also compute the derivative by itself. Having all your weights in one object is also awfully convenient; you can write something like `weights -= error * deriv * learning_rate` instead of iterating over each individual weight (and a large model contains many different sets of weights, not just a single NxMxPxQ matrix) This is good for the rapid iteration that ML research demands. However, once you have selected a model, I'm sure you can get performance advantages by coding it in a low level and eliminating inefficiencies. For example, you should be able to use the weight update equation from above by using fused multiply-accumulate, and the Python framework might not realize that.

DougBTX 3y ago

This is C++ rather than C, but a substantial portion of PyTorch is written in C++, and they provide a C++ interface: https://pytorch.org/tutorials/advanced/cpp_frontend.html https://pytorch.org/tutorials/advanced/cpp_frontend.html In other words, you can absolutely use PyTorch without Python.

voz_ 3y ago

Just compile the Python