Agreed with the sentiments here that this article gets a lot of the facts wrong, and I'll add one: the cost for electricity when self-hosting is dramatically lo

Agreed with the sentiments here that this article gets a lot of the facts wrong, and I'll add one: the cost for electricity when self-hosting is dramatically lower than the article says. The math assumes that each of the Tesla T4s will be using their full TDP (70W each) 24 hours a day, 7 days a week. In reality, GPUs throttle down to a low power state when not in use. So unless you're conversing with your LLM literally 24 hours a day, it will be using dramatically less power. Even when actively doing inference, my GPU doesn't quite max out its power usage. Your self-hosted LLM box is going to use maybe 20-30% of the power this article suggests it will. Source: I run LLMs at home on a machine I built myself.