10 ms·

TinyML and Efficient Deep Learning Computing

239 points by samuel246 3y ago

jalcazar 3y ago
These TinyML courses on edx look good https://www.edx.org/professional-certificate/harvardx-tiny-machine-learning https://www.edx.org/professional-certificate/harvardx-tiny-m...
matt_daemon 3y ago
I feel like this should be at the forefront of thinking on ML. Seeing how many computing resources Big Cloud companies are planning due to demand for ML infrastructure is pretty concerning from an energy use standpoint.
- choppaface 3y ago
  DeepSpeed and Hugginface Accelerate already include a lot of these features and they’re indeed already used in production. This class would be a great intro to said features given most people barely have access to one GPU let alone hundreds. Moreover, the class also covers topics related to using and hacking with foundational models of all you have is the model versus a cluster. Energy used to train foundation models is indeed extreme but the gross expenditure is more a function of corporate spending and competition than deep learning technology.
- hnfong 3y ago
  This is a false dichotomy. Bleeding edge research work should not be hindered by premature optimization concerns. Take quantization for example. Before people were able to train a model with the usual floating point precisions, nobody knew that INT8 or q4 quantization was feasible. (In fact, nobody would have a full precision model to compare performance with.) Also, the idea that energy efficiency should be a top concern basically undermines the whole idea of developing new technology. It's obvious that if fancy things are to come out of the research, it's going to cost more energy to run than not running anything at all. That itself is an argument that if we don't want energy usage to keep ramping up, we should shut down ALL research that potentially give us new energy-depleting toys. So, really, I'm personally not concerned with "one-off" resource usage if they advance human understanding of the state of the art. Since energy actually costs money, capitalist pressures will make people think of ways to save energy (and time). The moralistic arguments are just misguided in the big picture. IMHO it feels like luddites putting on the environmentalist hat here. Instead of shaming machine learning researchers over their energy use, it's probably more effective from a energy use standpoint (for example) to ban "proof of work" schemes in cryptocurrency.
  - katella 3y ago
    Do you hold the same opinion for crypto?
  - mratsim 3y ago
    Video encoding uses a *lot* of energy, Youtube, Netflix and friends. And capitalistic force are not fast enough and d not take environmental impact into account has carbon taxes are not there or not high enoigh or lobbied against or not measurable and gamed.
    - tonyhb 3y ago
      CRT TVs used a lot of energy. It’s getting better.
  - uoaei 3y ago
    > Since energy actually costs money, capitalist pressures will make people think of ways to save energy (and time). This and similar facile arguments are getting tiresome. They seem predicated on nothing but the most basic Econ 101 understanding of value. Nothing exists in that idealized world -- in reality, complex mechanisms keep this kind of excess spend relevant (marketing and public image, "first to market" concerns, sunk cost spending, and a million more) regardless of more material considerations, so to lean on this trope is not really up to the standards of HN discussion in my book.
  - imjonse 3y ago
    As you know, energy efficiency is not related to research and training only, but increasingly to inference and productionizing of the models. This is mostly what this course emphasizes too. It's not about luddites putting on the environmental hat, if environment was the main concern your capitalist pressure couldn't care less (\o/ externalities). It's about not draining phones batteries and not racking up datacenter bills.
  - _xivi 3y ago
    > Bleeding edge research work should not be hindered by premature optimization concerns Except nothing here is hindering the ability of scientists to develop whatever new technologies they want in their labs. > shut down ALL research > shaming machine learning researchers You're being sensationalistic. They're not banning super computers. AI research isn't facing an existential threat.
- cosmo13 3y ago
  Initiatives like green data centers are there which are designed to minimize environmental impact
- passion__desire 3y ago
  If we care about energy use so much, why is Las Vegas gambling industry allowed to exist. Or for that matter Shein. It's like we are doing premature optimization of one aspect of economy (tech), while other industries are given free rein. https://time.com/6247732/shein-climate-change-labor-fashion/ https://time.com/6247732/shein-climate-change-labor-fashion/
  - matt_daemon 3y ago
    I agree and we should be concerned about those industries too.
    - passion__desire 3y ago
      Work from Home could be planet saving (saving so much of fuel and time wastage) but top management needs to feel important. Why aren't journalist raising these talking points?
- wseqyrku 3y ago
  > is pretty concerning from an energy use standpoint I would agree that efficient LMs should be a focus, but a framing like that is too pretentious and misses the point IMO. I'd expect Apple to heavily double down on this point though, because that's what they always do (see the latest Apple event for more).
CamperBob2 3y ago
$25 billion endowment, and we get the audio quality of a 1930s-era wire recorder. Annoying.
- zxexz 3y ago
  OOC, what are you listening to? The youtube playlist has 2 videos for each lecture, a zoom recording focusing on the slides with the professor visible on camera, and a classroom recording - the latter definitely has better audio quality, but the audio quality of both is far better than your average OCW lecture from 10 years ago.
  - gozzoo 3y ago
    It's ironic that the audio quality of most AI online courses is terrible.I'm looking forward to the moment when AI can be used to improve the audio of these lectures, especially older ones like the Feynman lectures.[1] [1] https://www.youtube.com/watch?v=-kFOXP026eE&list=PLS3_1JNX8dEh5YcO-Y05stU0u_T9nqIlF https://www.youtube.com/watch?v=-kFOXP026eE&list=PLS3_1JNX8d...
    - CamperBob2 3y ago
      It did occur to me that improving the audio might be a good final exam. Better suited to a DSP class than an ML class, though.
- gozzoo 3y ago
  with subtitles it's bearable
tysam_and 3y ago
I'm curious if anyone knows people who would help me bring some efficient ML project work I've done to spheres like Africa. I've visited a couple times and loved being there so much, but definitely feel extremely disjoint from that part of the world. It would be really nice to me to help make some of those connections. <3 :'))))
- YAmpedUp 3y ago
  Hi Tysam_ Thanks for your intreset in helping others with your knowlegde , i would greatly recommend you to deeplearningindaba https://deeplearningindaba.com/mentorship/ https://deeplearningindaba.com/mentorship/ , its one of the biggest ml communities in africa and has a host of students, companies and mentors
  - tysam_and 3y ago
    Thank you, very much appreciated, this means a ton. Thank you for helping me make the connection, I will look into this. <3 :'))))
- EddTheSDET 3y ago
  Could you write to a university near where you’d like to work and ask them for some advice too?
  - tysam_and 3y ago
    I think you may have replied to the wrong comment (I might be misunderstanding, however).
    - joshvm 3y ago
      I think the suggestion is to try to collaborate with local universities. From experience, a lot of research/fieldwork in Africa is only practical with local experts and communities. Also look into attending or supporting the next Deep Learning Indaba. This year has just finished I think. It's one of the largest African ML communities and they have a mentorship program. https://deeplearningindaba.com/2023/ https://deeplearningindaba.com/2023/
jimmySixDOF 3y ago
I highly recommend the tinyML Talks to anyone interested in pushing edge compute past the last mile they have a huge library and a full upcoming schedule most of the slides are available so there is a huge amount of content it's amazing what you can get running on embedded systems. https://quip.com/MENbAvuQkrb0 https://quip.com/MENbAvuQkrb0
facu17y 3y ago
seems that the course doesn't include Google's highly efficiemt "step by step distillation" method, which was discussed on HN yesterday
tayo42 3y ago
They dont have this semesters problems online do they? Just the last semesters? Wondering if i missed it?

239 points by samuel246 3y ago

These TinyML courses on edx look good https://www.edx.org/professional-certificate/harvardx-tiny-machine-learning https://www.edx.org/professional-certificate/harvardx-tiny-m...

matt_daemon 3y ago

I feel like this should be at the forefront of thinking on ML. Seeing how many computing resources Big Cloud companies are planning due to demand for ML infrastructure is pretty concerning from an energy use standpoint.

choppaface 3y ago
DeepSpeed and Hugginface Accelerate already include a lot of these features and they’re indeed already used in production. This class would be a great intro to said features given most people barely have access to one GPU let alone hundreds. Moreover, the class also covers topics related to using and hacking with foundational models of all you have is the model versus a cluster. Energy used to train foundation models is indeed extreme but the gross expenditure is more a function of corporate spending and competition than deep learning technology.
hnfong 3y ago
This is a false dichotomy. Bleeding edge research work should not be hindered by premature optimization concerns. Take quantization for example. Before people were able to train a model with the usual floating point precisions, nobody knew that INT8 or q4 quantization was feasible. (In fact, nobody would have a full precision model to compare performance with.) Also, the idea that energy efficiency should be a top concern basically undermines the whole idea of developing new technology. It's obvious that if fancy things are to come out of the research, it's going to cost more energy to run than not running anything at all. That itself is an argument that if we don't want energy usage to keep ramping up, we should shut down ALL research that potentially give us new energy-depleting toys. So, really, I'm personally not concerned with "one-off" resource usage if they advance human understanding of the state of the art. Since energy actually costs money, capitalist pressures will make people think of ways to save energy (and time). The moralistic arguments are just misguided in the big picture. IMHO it feels like luddites putting on the environmentalist hat here. Instead of shaming machine learning researchers over their energy use, it's probably more effective from a energy use standpoint (for example) to ban "proof of work" schemes in cryptocurrency.
- katella 3y ago
  Do you hold the same opinion for crypto?
- mratsim 3y ago
  Video encoding uses a *lot* of energy, Youtube, Netflix and friends. And capitalistic force are not fast enough and d not take environmental impact into account has carbon taxes are not there or not high enoigh or lobbied against or not measurable and gamed.
  - tonyhb 3y ago
    CRT TVs used a lot of energy. It’s getting better.
- uoaei 3y ago
  > Since energy actually costs money, capitalist pressures will make people think of ways to save energy (and time). This and similar facile arguments are getting tiresome. They seem predicated on nothing but the most basic Econ 101 understanding of value. Nothing exists in that idealized world -- in reality, complex mechanisms keep this kind of excess spend relevant (marketing and public image, "first to market" concerns, sunk cost spending, and a million more) regardless of more material considerations, so to lean on this trope is not really up to the standards of HN discussion in my book.
- imjonse 3y ago
  As you know, energy efficiency is not related to research and training only, but increasingly to inference and productionizing of the models. This is mostly what this course emphasizes too. It's not about luddites putting on the environmental hat, if environment was the main concern your capitalist pressure couldn't care less (\o/ externalities). It's about not draining phones batteries and not racking up datacenter bills.
- _xivi 3y ago
  > Bleeding edge research work should not be hindered by premature optimization concerns Except nothing here is hindering the ability of scientists to develop whatever new technologies they want in their labs. > shut down ALL research > shaming machine learning researchers You're being sensationalistic. They're not banning super computers. AI research isn't facing an existential threat.
cosmo13 3y ago
Initiatives like green data centers are there which are designed to minimize environmental impact
passion__desire 3y ago
If we care about energy use so much, why is Las Vegas gambling industry allowed to exist. Or for that matter Shein. It's like we are doing premature optimization of one aspect of economy (tech), while other industries are given free rein. https://time.com/6247732/shein-climate-change-labor-fashion/ https://time.com/6247732/shein-climate-change-labor-fashion/
- matt_daemon 3y ago
  I agree and we should be concerned about those industries too.
  - passion__desire 3y ago
    Work from Home could be planet saving (saving so much of fuel and time wastage) but top management needs to feel important. Why aren't journalist raising these talking points?
wseqyrku 3y ago
> is pretty concerning from an energy use standpoint I would agree that efficient LMs should be a focus, but a framing like that is too pretentious and misses the point IMO. I'd expect Apple to heavily double down on this point though, because that's what they always do (see the latest Apple event for more).

CamperBob2 3y ago

$25 billion endowment, and we get the audio quality of a 1930s-era wire recorder. Annoying.

zxexz 3y ago
OOC, what are you listening to? The youtube playlist has 2 videos for each lecture, a zoom recording focusing on the slides with the professor visible on camera, and a classroom recording - the latter definitely has better audio quality, but the audio quality of both is far better than your average OCW lecture from 10 years ago.
- gozzoo 3y ago
  It's ironic that the audio quality of most AI online courses is terrible.I'm looking forward to the moment when AI can be used to improve the audio of these lectures, especially older ones like the Feynman lectures.[1] [1] https://www.youtube.com/watch?v=-kFOXP026eE&list=PLS3_1JNX8dEh5YcO-Y05stU0u_T9nqIlF https://www.youtube.com/watch?v=-kFOXP026eE&list=PLS3_1JNX8d...
  - CamperBob2 3y ago
    It did occur to me that improving the audio might be a good final exam. Better suited to a DSP class than an ML class, though.
gozzoo 3y ago
with subtitles it's bearable

tysam_and 3y ago

I'm curious if anyone knows people who would help me bring some efficient ML project work I've done to spheres like Africa. I've visited a couple times and loved being there so much, but definitely feel extremely disjoint from that part of the world. It would be really nice to me to help make some of those connections. <3 :'))))

YAmpedUp 3y ago
Hi Tysam_ Thanks for your intreset in helping others with your knowlegde , i would greatly recommend you to deeplearningindaba https://deeplearningindaba.com/mentorship/ https://deeplearningindaba.com/mentorship/ , its one of the biggest ml communities in africa and has a host of students, companies and mentors
- tysam_and 3y ago
  Thank you, very much appreciated, this means a ton. Thank you for helping me make the connection, I will look into this. <3 :'))))
EddTheSDET 3y ago
Could you write to a university near where you’d like to work and ask them for some advice too?
- tysam_and 3y ago
  I think you may have replied to the wrong comment (I might be misunderstanding, however).
  - joshvm 3y ago
    I think the suggestion is to try to collaborate with local universities. From experience, a lot of research/fieldwork in Africa is only practical with local experts and communities. Also look into attending or supporting the next Deep Learning Indaba. This year has just finished I think. It's one of the largest African ML communities and they have a mentorship program. https://deeplearningindaba.com/2023/ https://deeplearningindaba.com/2023/

jimmySixDOF 3y ago

I highly recommend the tinyML Talks to anyone interested in pushing edge compute past the last mile they have a huge library and a full upcoming schedule most of the slides are available so there is a huge amount of content it's amazing what you can get running on embedded systems. https://quip.com/MENbAvuQkrb0 https://quip.com/MENbAvuQkrb0

facu17y 3y ago

seems that the course doesn't include Google's highly efficiemt "step by step distillation" method, which was discussed on HN yesterday

tayo42 3y ago

They dont have this semesters problems online do they? Just the last semesters? Wondering if i missed it?