>How long before a consumer component machine for less than $10K can do inference for a gpt 3.5 turbo quality model? Uh two months ago? LLaMA-13B is GPT 3.5 Tu

>How long before a consumer component machine for less than $10K can do inference for a gpt 3.5 turbo quality model? Uh two months ago? LLaMA-13B is GPT 3.5 Turbo quality and runs locally on my Android phone's CPU at acceptable speeds. You can even run LLaMA-65B (which far surpasses GPT 3.5) faster than GPT 3.5 Turbo with two $200 24GB Nvidia Tesla P40 cards, since in 4bit the model is only 39GB with no output quality loss. Total cost $400 plus some junker used PC with two spare PCIe 4 x16 lanes.