3 ms·

This is a false dichotomy. Bleeding edge research work should not be hindered by premature optimization concerns. Take quantization for example. Before people

by hnfong 3y ago

This is a false dichotomy. Bleeding edge research work should not be hindered by premature optimization concerns. Take quantization for example. Before people were able to train a model with the usual floating point precisions, nobody knew that INT8 or q4 quantization was feasible. (In fact, nobody would have a full precision model to compare performance with.) Also, the idea that energy efficiency should be a top concern basically undermines the whole idea of developing new technology. It's obvious that if fancy things are to come out of the research, it's going to cost more energy to run than not running anything at all. That itself is an argument that if we don't want energy usage to keep ramping up, we should shut down ALL research that potentially give us new energy-depleting toys. So, really, I'm personally not concerned with "one-off" resource usage if they advance human understanding of the state of the art. Since energy actually costs money, capitalist pressures will make people think of ways to save energy (and time). The moralistic arguments are just misguided in the big picture. IMHO it feels like luddites putting on the environmentalist hat here. Instead of shaming machine learning researchers over their energy use, it's probably more effective from a energy use standpoint (for example) to ban "proof of work" schemes in cryptocurrency.

katella 3y ago
Do you hold the same opinion for crypto?
mratsim 3y ago
Video encoding uses a *lot* of energy, Youtube, Netflix and friends. And capitalistic force are not fast enough and d not take environmental impact into account has carbon taxes are not there or not high enoigh or lobbied against or not measurable and gamed.
- tonyhb 3y ago
  CRT TVs used a lot of energy. It’s getting better.
uoaei 3y ago
> Since energy actually costs money, capitalist pressures will make people think of ways to save energy (and time). This and similar facile arguments are getting tiresome. They seem predicated on nothing but the most basic Econ 101 understanding of value. Nothing exists in that idealized world -- in reality, complex mechanisms keep this kind of excess spend relevant (marketing and public image, "first to market" concerns, sunk cost spending, and a million more) regardless of more material considerations, so to lean on this trope is not really up to the standards of HN discussion in my book.
imjonse 3y ago
As you know, energy efficiency is not related to research and training only, but increasingly to inference and productionizing of the models. This is mostly what this course emphasizes too. It's not about luddites putting on the environmental hat, if environment was the main concern your capitalist pressure couldn't care less (\o/ externalities). It's about not draining phones batteries and not racking up datacenter bills.
_xivi 3y ago
> Bleeding edge research work should not be hindered by premature optimization concerns Except nothing here is hindering the ability of scientists to develop whatever new technologies they want in their labs. > shut down ALL research > shaming machine learning researchers You're being sensationalistic. They're not banning super computers. AI research isn't facing an existential threat.
- hashtag-til 3y ago
  > shut down ALL research > shaming machine learning researchers It’s quite hard to make any suggestion to a researcher these days, especially AI researchers who think they know it all. Even calling research work as “research” is not well received many times :) This is one of the reasons I’d like this bubble to burst soon, so that we can focus on applying the resources we got efficiently, rather than pursing - many times pointless - “bleeding-edge” research.
rlt 3y ago
Couldn’t cryptocurrency proponents make a similar argument?
- arthurcolle 3y ago
  Yes they could. Does that make it harder to reason about?
- WithinReason 3y ago
  No, how does a lot of crypto mining infrastructure now reduce the cost of cryptomining 5 years down the line? I will say though that at this point in time it's ridiculous how little research has gone into reducing the energy use of networks.
- abecedarius 3y ago
  OK, I will: it's illiberal to ban a computation on the grounds of the pollution emitted by some ways to generate energy. If the pollution is bad, you want to discourage the bad way to generate it, and encourage the good way. If this makes a computation more expensive (in those places where less-pollutey power is less available), that will discourage that computation if and only if it's not valuable enough to the person choosing whether to run it. A human may emit more CO2 than a computer doing the same work. As AI improves, this will happen more and more until they dominate humans by this measure. Then you'll have established a principle justifying a ban of biological humans. If you believe in human rights, it's mostly bad to put prior restraint on people's use of computers, just as it's bad to restrict writing, talk, and privacy. The restriction of this kind that makes the most sense to me is on the development of superhuman general intelligence, until we understand better how to avoid that ending humanity.
derangedHorse 3y ago
Instead of shaming machine learning researchers over their energy use, it's probably more effective from a energy use standpoint (for example) to ban “dirty” energy plants. Banning a class of high energy consuming algorithms used towards a non-violent technological development you disagree with makes absolutely no sense to me.
mathisfun123 3y ago
> Bleeding edge research work should not be hindered by premature optimization concerns. isn't the ingenuinity in doing things efficiently rather than just turning off the "scarce resources" setting? really i wish someone would explain to me what is the substance of "bleeding edge research" wherein nothing has been optimized except the flow of grant dollars from DoE/NSF to NVIDIA. like what is the research in "we burned O(1,000,000) compute core hours to train a model". i was recently in a discussion with ANL people about rick stevens next big idea - "the trillion parameter consortium" - where he plans to just commandeer aurora for a month. lol.
matt_daemon 3y ago
> Also, the idea that energy efficiency should be a top concern basically undermines the whole idea of developing new technology. I’m sorry but this is such a dumb argument. Of course new research will burn energy - there’s no question about that. To suggest that ML is like any other development, when it’s well documented how much power these algorithms (LLMs for example) use is kind of bad faith. The argument is not that we should halt technological progress, but in doing so always be considering its impact on the planet and evaluating if it is really worth it or not.

by hnfong 3y ago

katella 3y ago

Do you hold the same opinion for crypto?

mratsim 3y ago

Video encoding uses a *lot* of energy, Youtube, Netflix and friends. And capitalistic force are not fast enough and d not take environmental impact into account has carbon taxes are not there or not high enoigh or lobbied against or not measurable and gamed.

tonyhb 3y ago
CRT TVs used a lot of energy. It’s getting better.

uoaei 3y ago

> Since energy actually costs money, capitalist pressures will make people think of ways to save energy (and time). This and similar facile arguments are getting tiresome. They seem predicated on nothing but the most basic Econ 101 understanding of value. Nothing exists in that idealized world -- in reality, complex mechanisms keep this kind of excess spend relevant (marketing and public image, "first to market" concerns, sunk cost spending, and a million more) regardless of more material considerations, so to lean on this trope is not really up to the standards of HN discussion in my book.

imjonse 3y ago

As you know, energy efficiency is not related to research and training only, but increasingly to inference and productionizing of the models. This is mostly what this course emphasizes too. It's not about luddites putting on the environmental hat, if environment was the main concern your capitalist pressure couldn't care less (\o/ externalities). It's about not draining phones batteries and not racking up datacenter bills.

_xivi 3y ago

> Bleeding edge research work should not be hindered by premature optimization concerns Except nothing here is hindering the ability of scientists to develop whatever new technologies they want in their labs. > shut down ALL research > shaming machine learning researchers You're being sensationalistic. They're not banning super computers. AI research isn't facing an existential threat.

hashtag-til 3y ago
> shut down ALL research > shaming machine learning researchers It’s quite hard to make any suggestion to a researcher these days, especially AI researchers who think they know it all. Even calling research work as “research” is not well received many times :) This is one of the reasons I’d like this bubble to burst soon, so that we can focus on applying the resources we got efficiently, rather than pursing - many times pointless - “bleeding-edge” research.

rlt 3y ago

Couldn’t cryptocurrency proponents make a similar argument?

arthurcolle 3y ago
Yes they could. Does that make it harder to reason about?
WithinReason 3y ago
No, how does a lot of crypto mining infrastructure now reduce the cost of cryptomining 5 years down the line? I will say though that at this point in time it's ridiculous how little research has gone into reducing the energy use of networks.
abecedarius 3y ago
OK, I will: it's illiberal to ban a computation on the grounds of the pollution emitted by some ways to generate energy. If the pollution is bad, you want to discourage the bad way to generate it, and encourage the good way. If this makes a computation more expensive (in those places where less-pollutey power is less available), that will discourage that computation if and only if it's not valuable enough to the person choosing whether to run it. A human may emit more CO2 than a computer doing the same work. As AI improves, this will happen more and more until they dominate humans by this measure. Then you'll have established a principle justifying a ban of biological humans. If you believe in human rights, it's mostly bad to put prior restraint on people's use of computers, just as it's bad to restrict writing, talk, and privacy. The restriction of this kind that makes the most sense to me is on the development of superhuman general intelligence, until we understand better how to avoid that ending humanity.

derangedHorse 3y ago

Instead of shaming machine learning researchers over their energy use, it's probably more effective from a energy use standpoint (for example) to ban “dirty” energy plants. Banning a class of high energy consuming algorithms used towards a non-violent technological development you disagree with makes absolutely no sense to me.

mathisfun123 3y ago

> Bleeding edge research work should not be hindered by premature optimization concerns. isn't the ingenuinity in doing things efficiently rather than just turning off the "scarce resources" setting? really i wish someone would explain to me what is the substance of "bleeding edge research" wherein nothing has been optimized except the flow of grant dollars from DoE/NSF to NVIDIA. like what is the research in "we burned O(1,000,000) compute core hours to train a model". i was recently in a discussion with ANL people about rick stevens next big idea - "the trillion parameter consortium" - where he plans to just commandeer aurora for a month. lol.

matt_daemon 3y ago

> Also, the idea that energy efficiency should be a top concern basically undermines the whole idea of developing new technology. I’m sorry but this is such a dumb argument. Of course new research will burn energy - there’s no question about that. To suggest that ML is like any other development, when it’s well documented how much power these algorithms (LLMs for example) use is kind of bad faith. The argument is not that we should halt technological progress, but in doing so always be considering its impact on the planet and evaluating if it is really worth it or not.