3 ms·

Because GPUs are so power hungry that in a conventional data center, a fully loaded HPE Apollo 6500 (which has 8x H100 cards) uses more power than a rack’s tota

Because GPUs are so power hungry that in a conventional data center, a fully loaded HPE Apollo 6500 (which has 8x H100 cards) uses more power than a rack’s total power budget. Running an LLM just for fun requires tremendous amount of energy. Training one from scratch is even more energy intensive. Real AI applications are not much behind either. If you are processing any serious amount of data with AI, you use a lot of energy, again. A smallish water cooled “super computer” class CPU cluster needs “half a megawatt” to operate. As I said elsewhere, you can’t build a multi-megawatt data center “just like that”. Making the power you need available there is a multi-year, multi-institution endeavor on many cases. Similarly, you can’t “just force” the power grid to modernize, because live infrastructure servicing millions moves very slowly for good reasons (like multi-day boot up sequences if it shuts down).

infecto 2y ago
What is an LLM for fun compared to “real AI”. Hard to take your word for anything based on that. Regardless, you absolutely can force the grid to modernize which is what is happening now with DC colocating near new and existing plants along with large inflows of investment into generators.
- bayindirh 2y ago
  I work as a sysadmin at an academic HPC center, and some projects we support use AI for real causes like marine life and water health, and other ecosystem related work. These are production systems which provide real benefits and returns to humanity, and tries to secure our and other species future and on this planet. When compared to these projects, an LLM is just a shiny toy which can do many tasks with worse quality, with no guarantee of correctness and with much higher energy usage. Other numbers I gave is fresh from our data center, about the systems I work with directly. You can dismiss all my comment just because I call a big, energy wasting NN a toy, but nothing I say on this comment is wrong, and I’m reporting them from the trenches directly. > DC colocating near new and existing plants along with large inflows of investment into generators. You can get tons of servers in six weeks, install them in two days, and get them running in a day, tops (for us it’s shorter). If you can get your generators manufactured and delivered to your location in a year, you’re lucky.

infecto 2y ago

What is an LLM for fun compared to “real AI”. Hard to take your word for anything based on that. Regardless, you absolutely can force the grid to modernize which is what is happening now with DC colocating near new and existing plants along with large inflows of investment into generators.

bayindirh 2y ago
I work as a sysadmin at an academic HPC center, and some projects we support use AI for real causes like marine life and water health, and other ecosystem related work. These are production systems which provide real benefits and returns to humanity, and tries to secure our and other species future and on this planet. When compared to these projects, an LLM is just a shiny toy which can do many tasks with worse quality, with no guarantee of correctness and with much higher energy usage. Other numbers I gave is fresh from our data center, about the systems I work with directly. You can dismiss all my comment just because I call a big, energy wasting NN a toy, but nothing I say on this comment is wrong, and I’m reporting them from the trenches directly. > DC colocating near new and existing plants along with large inflows of investment into generators. You can get tons of servers in six weeks, install them in two days, and get them running in a day, tops (for us it’s shorter). If you can get your generators manufactured and delivered to your location in a year, you’re lucky.