3 ms·
FLAN-T5 11b is the most competitive public model that has undergone instruction finetuning. It has better few shot and instruction following performance than mu
by yunyu 4y ago
FLAN-T5 11b is the most competitive public model that has undergone instruction finetuning. It has better few shot and instruction following performance than much bigger models like GPT-NeoX: https://huggingface.co/google/flan-t5-xxl https://huggingface.co/google/flan-t5-xxl, and you can deploy it on 24GB VRAM with bfloat16 inference
- K0balt 4y agoI have a naive question. When people talk about needing x amount of vram are they meaning all on one card, or can it be distributed across multiple cards? Assuming power consumption is not an issue, Can these loads be effectively run on farms of systems using older cards with smaller vram sizes? How about for training? Are the hardware requirements fundamentally different apart from scale?
- yunyu 4y agoModel parallelism is possible for inference and you can host extremely large models across multiple accelerators (see DeepSpeed ZeRO), but the inference speed is several times slower compared to just having all of it on one accelerator due to communication bottlenecks and overhead (parallelism refers to memory needed for parameters, not the actual computation in most cases). Certain models may require storage formats like bfloat16 to run efficiently that may not be supported on older hardware. Parallelism is also supported (and more necessary actually) for training, but since backprop is expensive it typically requires many multiples of memory requirements needed for inference.