I wrote a blog post on the difficulties of memory-efficient training, which seems relevant: http://mitchgordon.me/machine/learning/2020/01/13/do-we-really-need-

I wrote a blog post on the difficulties of memory-efficient training, which seems relevant: http://mitchgordon.me/machine/learning/2020/01/13/do-we-really-need-model-compression.html http://mitchgordon.me/machine/learning/2020/01/13/do-we-real... The methods discussed there take a different angle at the problem.