3 ms·
175B * 16 bits = 350GB, but it does compress a bit. GPT-J-6B, which you can download at https://github.com/kingoflolz/mesh-transformer-jax https://github.com/k
by sanxiyn 5y ago
175B * 16 bits = 350GB, but it does compress a bit.
GPT-J-6B, which you can download at https://github.com/kingoflolz/mesh-transformer-jax https://github.com/kingoflolz/mesh-transformer-jax, is 6B parameters but weighs 9GB. It does decompress to 12GB as expected. Assuming the same compression ratio, download size would be 263GB, not 350GB.