2 ms·
What I'm most interested in with abstraction is how easy it is to change something that doesn't fit neatly inside of the abstraction framework. It looks like th
by andy99 3y ago
What I'm most interested in with abstraction is how easy it is to change something that doesn't fit neatly inside of the abstraction framework. It looks like the model is pretty flexible as it's just plain pytorch, I couldn't immediately tell about other aspects of the training - for example they have their own optimizer, what if I want to change something?
There are lots of "just one line of python" type frameworks that are fine if you want to do the one thing in the demo but are more complicated than just writing it yourself if you have to change something.
- mabrowning 3y agoWe do have reference optimizers implemented for use on our system and available in the `cerebras_pytorch` package, but this isn't because those are the only ones supported; instead, no vanilla pytorch optimizer is currently `torch.compile()` compatible. The main difference is that we pre-initialize the optimizer state instead of doing it lazily in the first `step()` You can read more here https://docs.cerebras.net/en/latest/wsc/tutorials/custom-optimizer.html https://docs.cerebras.net/en/latest/wsc/tutorials/custom-opt...