4 ms·

Looks like hardware vs. software abstraction. Considereing the perspective of an LLM startup: Would you rather write 20k LOC of complex code that would make you

by kkzz99 3y ago

Looks like hardware vs. software abstraction. Considereing the perspective of an LLM startup: Would you rather write 20k LOC of complex code that would make you be able to more easily switch hardware platforms - or - write 600 LOC of less complex code and be pinned to a single provider?

minimaxir 3y ago
Or use a more battle-harded abstraction like Hugging Face transformers and get both a) minimal LOC in the end-application and b) portability.
- omneity 3y ago
  To be honest, HF compat comes at the cost of abysmal performance compared to an inference solution such as vLLM.
  - LatticeAnimal 3y ago
    New to the space — I have been using HF for both training and inference. Do you have recommendations for a different framework for training? Accelerate seems fantastic for scaling up once I need to
    - omneity 3y ago
      HF has been pretty good for training in my experience. Both accelerate, deepspeed and other improvements are available and the ecosystem is vibrant. One thing you might want to look into is to write your own training loop rather than using Autotrainer, but that's if you want to get you hands dirty and get finer control over how you call layers, your loss function etc. Inference is where it falls short really, and solutions like vLLM are much much faster.
- jackblemming 3y ago
  Stay far away from HuggingFace if you can. Battle-hardened if you only do the absolute simplest and boring stuff. Look at the number of open issues and skim through a few source code files and you’ll understand. Like LangChain, they were at the right place at the right time. That doesn’t make them good.
  - b4ckup 3y ago
    Good to hear that I'm not the only one thinking that. I read a lot of their code for one of my pet projects and I thought that maybe it's me that doesn't get it because I don't write that much python.
  - minimaxir 3y ago
    In the context of this blog post (reimplementing nanoGPT as-is without shenanigans, although training is a separate issue), transformers is straightforward enough. I do agree that going beyond the documentation demos does make things more complicated, but IMO still less so than other implementations speaking from experience. AI is still sometimes necessarily complex. It's certainly an order of magnitude easier to use something like transformers or diffusers than the original implementations provided by the original model trainers, and has a few good optimizations out of the box. That's different from LangChain which is complex for the sake of being complex.
- kkzz99 3y ago
  We are talking about very big models which training requires an enormous amount of hardware. Not sure how scalable HuggingFace transformer for training such models is.
  - minimaxir 3y ago
    transformers outsources those types of tasks to accelerate, also developed by Hugging Face: https://huggingface.co/docs/accelerate/index https://huggingface.co/docs/accelerate/index I can't speak to the efficacy of it at large scale, though.
- tbalsam 3y ago
  HF is okay for off the shelf prototypes and very quick, multi-hour hacks, and I greatly appreciate their contributions to the community, but unfortunately in my (and sadly, many other people's experiences), their code is a continuum of devilish nightmares for anything beyond that.
andy99 3y ago
What I'm most interested in with abstraction is how easy it is to change something that doesn't fit neatly inside of the abstraction framework. It looks like the model is pretty flexible as it's just plain pytorch, I couldn't immediately tell about other aspects of the training - for example they have their own optimizer, what if I want to change something? There are lots of "just one line of python" type frameworks that are fine if you want to do the one thing in the demo but are more complicated than just writing it yourself if you have to change something.
- mabrowning 3y ago
  We do have reference optimizers implemented for use on our system and available in the `cerebras_pytorch` package, but this isn't because those are the only ones supported; instead, no vanilla pytorch optimizer is currently `torch.compile()` compatible. The main difference is that we pre-initialize the optimizer state instead of doing it lazily in the first `step()` You can read more here https://docs.cerebras.net/en/latest/wsc/tutorials/custom-optimizer.html https://docs.cerebras.net/en/latest/wsc/tutorials/custom-opt...
tbalsam 3y ago
I think you're missing a crucial difference here: > ... write 20k LOC of complex code ... Vs the all important: > ... write _and maintain_ 20k LOC of complex code ... The answer to the latter, for me, is a hard no.

by kkzz99 3y ago

minimaxir 3y ago

Or use a more battle-harded abstraction like Hugging Face transformers and get both a) minimal LOC in the end-application and b) portability.

omneity 3y ago
To be honest, HF compat comes at the cost of abysmal performance compared to an inference solution such as vLLM.
- LatticeAnimal 3y ago
  New to the space — I have been using HF for both training and inference. Do you have recommendations for a different framework for training? Accelerate seems fantastic for scaling up once I need to
  - omneity 3y ago
    HF has been pretty good for training in my experience. Both accelerate, deepspeed and other improvements are available and the ecosystem is vibrant. One thing you might want to look into is to write your own training loop rather than using Autotrainer, but that's if you want to get you hands dirty and get finer control over how you call layers, your loss function etc. Inference is where it falls short really, and solutions like vLLM are much much faster.
jackblemming 3y ago
Stay far away from HuggingFace if you can. Battle-hardened if you only do the absolute simplest and boring stuff. Look at the number of open issues and skim through a few source code files and you’ll understand. Like LangChain, they were at the right place at the right time. That doesn’t make them good.
- b4ckup 3y ago
  Good to hear that I'm not the only one thinking that. I read a lot of their code for one of my pet projects and I thought that maybe it's me that doesn't get it because I don't write that much python.
- minimaxir 3y ago
  In the context of this blog post (reimplementing nanoGPT as-is without shenanigans, although training is a separate issue), transformers is straightforward enough. I do agree that going beyond the documentation demos does make things more complicated, but IMO still less so than other implementations speaking from experience. AI is still sometimes necessarily complex. It's certainly an order of magnitude easier to use something like transformers or diffusers than the original implementations provided by the original model trainers, and has a few good optimizations out of the box. That's different from LangChain which is complex for the sake of being complex.
kkzz99 3y ago
We are talking about very big models which training requires an enormous amount of hardware. Not sure how scalable HuggingFace transformer for training such models is.
- minimaxir 3y ago
  transformers outsources those types of tasks to accelerate, also developed by Hugging Face: https://huggingface.co/docs/accelerate/index https://huggingface.co/docs/accelerate/index I can't speak to the efficacy of it at large scale, though.
tbalsam 3y ago
HF is okay for off the shelf prototypes and very quick, multi-hour hacks, and I greatly appreciate their contributions to the community, but unfortunately in my (and sadly, many other people's experiences), their code is a continuum of devilish nightmares for anything beyond that.

andy99 3y ago

What I'm most interested in with abstraction is how easy it is to change something that doesn't fit neatly inside of the abstraction framework. It looks like the model is pretty flexible as it's just plain pytorch, I couldn't immediately tell about other aspects of the training - for example they have their own optimizer, what if I want to change something? There are lots of "just one line of python" type frameworks that are fine if you want to do the one thing in the demo but are more complicated than just writing it yourself if you have to change something.

mabrowning 3y ago
We do have reference optimizers implemented for use on our system and available in the `cerebras_pytorch` package, but this isn't because those are the only ones supported; instead, no vanilla pytorch optimizer is currently `torch.compile()` compatible. The main difference is that we pre-initialize the optimizer state instead of doing it lazily in the first `step()` You can read more here https://docs.cerebras.net/en/latest/wsc/tutorials/custom-optimizer.html https://docs.cerebras.net/en/latest/wsc/tutorials/custom-opt...

tbalsam 3y ago

I think you're missing a crucial difference here: > ... write 20k LOC of complex code ... Vs the all important: > ... write _and maintain_ 20k LOC of complex code ... The answer to the latter, for me, is a hard no.