5 ms·
I believe that the 1.5B model weights take up the 6GB themselves. Presumably f32 weight values? Since they are all needed for an evaluation pass they will eat u
by dodobirdlord 7y ago
I believe that the 1.5B model weights take up the 6GB themselves. Presumably f32 weight values? Since they are all needed for an evaluation pass they will eat up 6GB on the GPU off the bat. Not too surprising that everything else can't fit in 2GB, since that's going to have to fit the entire model architecture and all intermediary values.
- Dylan16807 7y agoWhat exactly are these intermediate values composed of that makes them notable in size compared to the model itself? Are there resources I should read for how a model like this executes? Will this model work with half precision weights? Is it very awkward to use "brain" 16 bit floats?