3 ms·
One neat trick is that you can extend GPT-2 117M's context window from 1024 up to 30k on a TPU, since TPUs can allocate up to 300GB of memory for backprop. http
by sillysaurusx 7y ago
One neat trick is that you can extend GPT-2 117M's context window from 1024 up to 30k on a TPU, since TPUs can allocate up to 300GB of memory for backprop. https://twitter.com/gwern/status/1218001309435072513 https://twitter.com/gwern/status/1218001309435072513
It's not quite 1M words, but a 30k context window is big enough for e.g. most midi songs.