3 ms·
The Annotated GPT-2
- lee101 4y agoSo good, also for some good resources on how to train these networks: * checkout mosaic ml, it has some recent improvements like Alibi that are important https://docs.mosaicml.com/en/v0.3.1/method_cards/alibi.html https://docs.mosaicml.com/en/v0.3.1/method_cards/alibi.html * Huggingface trainer * Pytorch lightning (still has some great training improvements like LR find and saving top k best checkpoints ect) * Deepspeed/megatron/bloom For inference: try a few alternatives to OpenAI like https://text-generator.io/ https://text-generator.io/ or hosting your own network/huggingface pipelines (can use a feature extractor pipeline and a text generation pipeline to support getting embeddings and generating from the same one model on the GPU)
- jaskyle 4y agoThis is a great article for anyone working with language models!