11 ms·
This was done using preemptible TPUv3's, but you can get similar results with TPUv2's. It would just take about 3 times longer. I'm excited about the multi-TPU
by sillysaurusx 7y ago
This was done using preemptible TPUv3's, but you can get similar results with TPUv2's. It would just take about 3 times longer.
I'm excited about the multi-TPU technique. We've been calling it "swarm training". https://www.gwern.net/GPT-2#gcp https://www.gwern.net/GPT-2#gcp
It's cheaper than a TPU pod by a fair margin, and you can scale your usage up and down rather than having to do a 1-year commitment for $1M/yr.
Here's a typical tensorboard example: https://tensorboard.dev/experiment/MEXO9GEWR0Wjdb6bz6yjsw/ https://tensorboard.dev/experiment/MEXO9GEWR0Wjdb6bz6yjsw/
We're able to process about 1 billion GPT-2 tokens every 12 hours, which seems about equivalent to a TPU pod. It's hard to get a direct comparison because so little data is available. (Each TPU is currently getting 256 tokens/sec, and we've been running with 88 TPUs in the swarm.)
As for the sample quality, I'm pretty happy with it. It seems good at philosophical debates, trolling, writing prompts, and several other tasks.
The dataset was provided by the creator of https://www.reddit.com/r/SubSimulatorGPT2/ https://www.reddit.com/r/SubSimulatorGPT2/ and was gathered via pushshift.io. It's about 10GB.
Beyond that, I wrote some code to dump samples using a rolling context window. You'll notice that this file is one continuous sample. You can try it yourself:
git clone https://github.com/shawwn/gpt-2
cd gpt-2
sudo pip3 install -r requirements.txt
python3 download_model.py 117M
python3 src/generate_samples.py --prompt "What a lovely day today!" --model_name 117M --length 256 --step 64 --top_p 0.90 --temperature 1.0
Once it's running, it will look like someone is rapidly typing into your terminal. (--step 1 is even cooler in that regard, since it looks like it's typing out one word at a time.)
Sometimes it will get stuck in a loop. If so, you can reset the context window manually while it's running by executing `rmdir commands/clear_context`
The command system is kind of a funny hack. I wanted a way of interacting with the AI training while it's running, and using `rmdir` seemed like the easiest way to "press a button". You can also do things like attach a debugger, save a snapshot, change the learning rate, and so on. Bonus: it works in Colab, since you can remove directories while a cell is executing.
- sansnomme 7y agoHow much did the parallel training cost?
- sillysaurusx 7y agoNothing, thanks to TFRC. https://www.tensorflow.org/tfrc https://www.tensorflow.org/tfrc Without TFRC it's about $4k per day. Which is orders of magnitude less expensive than a TPU pod.