llama-65b on a 4-bit quantize sizes down to about 39GB - you can run that on a 48GB A6000 (~$4.5K) or on 2 x 24GB 3090s (~$1500 used). llama-30b (33b really but

llama-65b on a 4-bit quantize sizes down to about 39GB - you can run that on a 48GB A6000 (~$4.5K) or on 2 x 24GB 3090s (~$1500 used). llama-30b (33b really but who's counting) quantizes down to 19GB (17GB w/ some optimization), so that'll fit comfortably on a 24GB GPU. A 4-bit quantize of a 1000B model should be <600GB, so would fit on a regular 8x80 DGX system.