5 ms·

Or use a more battle-harded abstraction like Hugging Face transformers and get both a) minimal LOC in the end-application and b) portability.

omneity 3y ago
To be honest, HF compat comes at the cost of abysmal performance compared to an inference solution such as vLLM.
- LatticeAnimal 3y ago
  New to the space — I have been using HF for both training and inference. Do you have recommendations for a different framework for training? Accelerate seems fantastic for scaling up once I need to
  - omneity 3y ago
    HF has been pretty good for training in my experience. Both accelerate, deepspeed and other improvements are available and the ecosystem is vibrant. One thing you might want to look into is to write your own training loop rather than using Autotrainer, but that's if you want to get you hands dirty and get finer control over how you call layers, your loss function etc. Inference is where it falls short really, and solutions like vLLM are much much faster.
jackblemming 3y ago
Stay far away from HuggingFace if you can. Battle-hardened if you only do the absolute simplest and boring stuff. Look at the number of open issues and skim through a few source code files and you’ll understand. Like LangChain, they were at the right place at the right time. That doesn’t make them good.
- b4ckup 3y ago
  Good to hear that I'm not the only one thinking that. I read a lot of their code for one of my pet projects and I thought that maybe it's me that doesn't get it because I don't write that much python.
- minimaxir 3y ago
  In the context of this blog post (reimplementing nanoGPT as-is without shenanigans, although training is a separate issue), transformers is straightforward enough. I do agree that going beyond the documentation demos does make things more complicated, but IMO still less so than other implementations speaking from experience. AI is still sometimes necessarily complex. It's certainly an order of magnitude easier to use something like transformers or diffusers than the original implementations provided by the original model trainers, and has a few good optimizations out of the box. That's different from LangChain which is complex for the sake of being complex.
kkzz99 3y ago
We are talking about very big models which training requires an enormous amount of hardware. Not sure how scalable HuggingFace transformer for training such models is.
- minimaxir 3y ago
  transformers outsources those types of tasks to accelerate, also developed by Hugging Face: https://huggingface.co/docs/accelerate/index https://huggingface.co/docs/accelerate/index I can't speak to the efficacy of it at large scale, though.
tbalsam 3y ago
HF is okay for off the shelf prototypes and very quick, multi-hour hacks, and I greatly appreciate their contributions to the community, but unfortunately in my (and sadly, many other people's experiences), their code is a continuum of devilish nightmares for anything beyond that.

omneity 3y ago

To be honest, HF compat comes at the cost of abysmal performance compared to an inference solution such as vLLM.

LatticeAnimal 3y ago
New to the space — I have been using HF for both training and inference. Do you have recommendations for a different framework for training? Accelerate seems fantastic for scaling up once I need to
- omneity 3y ago
  HF has been pretty good for training in my experience. Both accelerate, deepspeed and other improvements are available and the ecosystem is vibrant. One thing you might want to look into is to write your own training loop rather than using Autotrainer, but that's if you want to get you hands dirty and get finer control over how you call layers, your loss function etc. Inference is where it falls short really, and solutions like vLLM are much much faster.

jackblemming 3y ago

Stay far away from HuggingFace if you can. Battle-hardened if you only do the absolute simplest and boring stuff. Look at the number of open issues and skim through a few source code files and you’ll understand. Like LangChain, they were at the right place at the right time. That doesn’t make them good.

b4ckup 3y ago
Good to hear that I'm not the only one thinking that. I read a lot of their code for one of my pet projects and I thought that maybe it's me that doesn't get it because I don't write that much python.
minimaxir 3y ago
In the context of this blog post (reimplementing nanoGPT as-is without shenanigans, although training is a separate issue), transformers is straightforward enough. I do agree that going beyond the documentation demos does make things more complicated, but IMO still less so than other implementations speaking from experience. AI is still sometimes necessarily complex. It's certainly an order of magnitude easier to use something like transformers or diffusers than the original implementations provided by the original model trainers, and has a few good optimizations out of the box. That's different from LangChain which is complex for the sake of being complex.

kkzz99 3y ago

We are talking about very big models which training requires an enormous amount of hardware. Not sure how scalable HuggingFace transformer for training such models is.

minimaxir 3y ago
transformers outsources those types of tasks to accelerate, also developed by Hugging Face: https://huggingface.co/docs/accelerate/index https://huggingface.co/docs/accelerate/index I can't speak to the efficacy of it at large scale, though.

tbalsam 3y ago

HF is okay for off the shelf prototypes and very quick, multi-hour hacks, and I greatly appreciate their contributions to the community, but unfortunately in my (and sadly, many other people's experiences), their code is a continuum of devilish nightmares for anything beyond that.