Nvidia's apex.amp does just this - attempts half-precision training where possible, and adjusts scaling factors during training. I achieved a very significant s

Nvidia's apex.amp does just this - attempts half-precision training where possible, and adjusts scaling factors during training. I achieved a very significant speedup using this on my RTX 2080 Super, though it doesn't support Windows that well.