5 ms·
It's orders of magnitude cheaper to serve requests with conventional methods than directly with LLM. My back-of-envelope calculation says, optimistically, it ta
by ychen306 11mo ago
It's orders of magnitude cheaper to serve requests with conventional methods than directly with LLM. My back-of-envelope calculation says, optimistically, it takes more than 100 GFLOPs to generate 10 tokens using a 7 billion parameter LLM. There are better ways to use electricity.
- ls-a 11mo agoTry to convince the investors. The way the industry is headed is not necessarily related to what is most optimal. That might be the future whether we like it or not. Losing billions seems to be the trend.
- sramam 11mo agoI work in enterprise IT and sometimes wonder if we should add the equivalent energy calculations of human effort - both productive and unproductive - that underlies these "output/cost" comparisons. I realize it sounds inhuman, but so is working in enterprise IT! :)
- ethmarks 11mo agoI agree wholeheartedly. It irks me when people critique automation because it uses large amounts of resources. Running a machine or a computer almost always uses far less resources than a human would to do the same task, so long as you consider the entire resource consumptions. Growing the food that a human eats, running the air conditioning for their home, powering their lights, fueling their car, charging their phone, and all the many many things necessary to keep a human alive and productive in the 21st century are a larger resource cost than almost any machine/system that performs the same work. From an efficiency perspective, automation is almost always the answer. The actual debate comes from the ethical perspective (the innate value of human life).
- runarberg 11mo agoI suspect you may be either underestimating how efficient our brains are at computing or severely underestimating how much energy these AI models take to train and run. Even including our system of comfort like refrigerated blueberries in January and AC cooling a 40° C heat down to 25° C (but excluding car commutes, because please work from home or take public transit) the human is still far far more energy efficient in e.g. playing go then alpha-go. With LLMs this isn’t even close (and we can probably factor in that stupid car commute, because LLMs are just that inefficient).
- zelphirkalt 11mo agoHm, that gives me an idea: The next human vs engine matches in chess, go, and so on, should be set at a specific level of energy consumption of the engines, that's close or approximately that of an extremely good human player, like a world champion or at least grand master. Let's see how engines keep up then!
- ethmarks 11mo agoThat sounds delightful. Get a Raspberry Pi or something connected to a power supply capped at 20 watts (approximate electricity consumption of the human brain). It has to be able to run its algorithm in less than the time limit per turn for speed chess. Then you'd have to choose an algorithm based on if it produces high-quality guesses before arriving at its final answer so that if it runs out of time it can still make a move. I wonder if this is already a thing?
- nradov 11mo agoSure, but we can start with an LLM to build V1 (or at least a demo) faster for certain problem domains. Then apply traditional coding techniques as an efficiency optimization later after establishing product-market fit.