4 ms·
Notably, the 345M model (1.5 GB on disk) is big enough that it's pushing the limits of conventional GPUs, and an alternative method of finetuning the model (htt
by minimaxir 8y ago
Notably, the 345M model (1.5 GB on disk) is big enough that it's pushing the limits of conventional GPUs, and an alternative method of finetuning the model (https://github.com/nshepperd/gpt-2/commit/47df6da611716b4826e3397cd68d711c6951c8e5 https://github.com/nshepperd/gpt-2/commit/47df6da611716b4826...) has to be used to prevent the GPU from going OOM.
I'm working on tools to streamline GPT-2 text generation: I'm currently porting the code above to gpt-2-simple (https://github.com/minimaxir/gpt-2-simple https://github.com/minimaxir/gpt-2-simple) to allow easy finetuning/generation, and am also working on a way to quickly build an API/client for easily deploying GPT-2 to production and generating text at scale and cost effectively. Even the 117M model, managing CPU and RAM performance is tricky.
But given the incredible results of just the 117M model (e.g. Hacker News titles from a retrained 117M model: https://github.com/minimaxir/hacker-news-gpt-2 https://github.com/minimaxir/hacker-news-gpt-2), I'm eager to put the 345M model through its paces.
- yeldarb 8y agoCan you elaborate on the expected hardware requirements? (Both for fine tuning and inference)
- czr 8y agoIt seems to train fine on Colab without that flag, so that's a decent fallback for anyone running into memory issues on a local machine.
- minimaxir 8y agoIf you did a fresh pull of that fork using a Colab notebook that does so, it will set the flag automatically if you specify the 345M model (I will follow that behavior in my own package). If it didn't set the flag, it gets messy: https://twitter.com/minimaxir/status/1124742105421631488 https://twitter.com/minimaxir/status/1124742105421631488
- jchw 8y ago>(e.g. Hacker News titles from a retrained 117M model: https://github.com/minimaxir/hacker-news-gpt-2 https://github.com/minimaxir/hacker-news-gpt-2) Wow, thats great. “The Bullshit Bubble” “Fuck you, Bootstrap” “We should give up on America” - they’re practically comedy, yet very believable too.
- Judgmentality 8y ago"How to Make a T-Shirt in 10,000 hours" is my favorite.
- mkl 8y agoI had to do a search to make sure that or something analogous wasn't actually real! It seems incredibly on target.
- nicwilson 8y agoI think its just edged by "Why I'm ditching ebike hoses for barometers" for me Though "Hacker makes an infinitely scaleable cup of coffee" gives it a good run for its money.
- zrm 8y agoHas anybody tried feeding it comedy to begin with to see what it spits back out?
- minimaxir 8y agoThere was an attempt on Reddit but it didn't turn out well, likely because there wasn't enough input data: https://www.reddit.com/r/MachineLearning/comments/bgvzdu/d_jokes_generated_via_gpt2/ https://www.reddit.com/r/MachineLearning/comments/bgvzdu/d_j... (I have an idea for a more proper approach)
- nprateem 8y agoIt'd probably produce a crap sitcom. But then again so does the BBC these days.
- ddalex 8y agoI can provide more details next week once the article is out, but I've fed it a corpus of one liners and it started to produce weird and sometimes very funny liners :)
- CYHSM 8y agoI tried it on The Office quotes [1] and it seems to generate some more or less humorous examples [1] https://www.reddit.com/r/MachineLearning/comments/bmn0og/p_language_model_gpt2_finetuned_on_the_office/ https://www.reddit.com/r/MachineLearning/comments/bmn0og/p_l...
- applecrazy 8y agoOne example from your dataset that gave me a laugh: Hiring technical debt (or "unsortable overtime") How do you hire technical debt?? Edit: another one- “In 2009, Africa power creation was switched on for the Google Earth Darth Vader Imperial Warplane Propaganda” I guess Google is diversifying :P
- dannyw 8y agoOutsource a project to a contracting agency and you’ve hired technical debt.
- antpls 8y agoI guess it went through that path : -Hiring -> Software Developer -Software Developer -> Code -Code -> Technical Debt You only have to hire a Software Developer :)
- de_watcher 8y agoIt has found a shortcut.
- burtonator 8y agoThis thing really does scare me and fascinate me at the same time. There's so many nefarious things you could do with this... Generate fake news. spam google. etc. One valuable use could be to generate comedy and parody. You could also make it to sabotage others too. You could set it lose on nazi forums and have them argue with bots constantly.
- nprateem 8y agoTrue. Authors at The Onion should be afraid.
- lopmotr 8y agoI think people are over-worried about fake material. We've been able to say lies since we became human. We've been able to print lies since printing. We can already edit photos. People routinely make fake screenshots of messages to defame others. Newspapers sometimes print fake stories based on lies given by their sources or which they fail to vet. But provenance still exists to prove things, not how convincing it looks.
- narrator 8y agoIf you wanted to run a propaganda department though, you needed to be a government or a large news organization. Now it will be within the reach of 4chan.
- ben_w 8y agoIf evidence alone was enough, there wouldn’t be any debate about antivaxxers, climate change, the Iraq WMDs, the 9/11 and moon landing conspiracies, Holocaust denial, the correct way to bring peace to the Israel/Palestine conflict, the safety of being white in a Muslim community or being black around an American cop, Brexit, any of the Assange drama, or if Snowden was acting appropriately by leaking what he leaked. And those are just the disagreements I’ve witnessed. Just because lies and propaganda have always been around, it doesn’t mean they were never a problem.
- 8y ago
- minimaxir 8y agoUpdate: 345M model support successfully added to gpt-2-simple and new release pushed to PyPI, plus I've updated the corresponding Colaboratory Notebook accordingly (https://colab.research.google.com/drive/1VLG8e7YSEwypxU-noRNhsv5dW4NfTGce https://colab.research.google.com/drive/1VLG8e7YSEwypxU-noRN...) At a glance it's hard to see if the 345M model is "better" at the moment (it's all qualitative), but I'll be doing more testing. Unfortunately, the 345M model might be slightly too resource intensive for the API/client use case I wanted to do, so I'll likely be sticking with the 117M model for now.
- M4v3R 8y ago> Show HN: Blogia, a blogging framework for Ruby Wow, this sounded so believable I had to check if this exists or not. It seems it doesn't exist, yet :).
- leod 8y agoI haven't heard of gradient checkpointing yet, thank you for the link! Do you know how it compares to gradient accumulation? The latter basically reduces the batch size, but takes the sum of multiple gradients before actually performing an update, thereby having the same effect as the original batch size. The generated titles are great! You can put them into hncynic (https://github.com/leod/hncynic https://github.com/leod/hncynic) to get closer to a fully generated HN experience.
- indalo 8y agowow! I just had a blast putting titles into that. the results are amazing. kudos!
- gwern 8y agoGradient accumulation and gradient checkpointing are orthogonal. You might want to use them simultaneously. If I had to compare them, I'd say that accumulation is about working on a minibatch datapoint by datapoint and faking being able to run an entire large minibatch in a single shot, while checkpointing is about working on a model layer by layer and faking being able to run an entire model in a single shot. The problem with GPT-2-335M and why nshepperd had to mess with gradient checkpointing is that the GPT-2-335M model will literally not fit in your standard 11GB GPU (and from Twitter comments about people trying it on the new 16GB Google Colab instances, it's unclear if 16GB would be enough either!). You can't even run minibatch n=1. It doesn't fit. It OOMs. The model itself is only a gigabyte or so, the problem is that the self-attention layers, when run, use up a huge amount of memory for their intermediate steps, which must be stored in order to trace everything backwards through each step for the backprop part of training. (Right now I believe nshepperd's code punts on doing gradient accumulation simultaneous with gradient checkpointing, so we've just been reducing the learning rate, which is sort of similar to faking large minibatches with gradient accumulation.) Fortunately, because the self-attention layers are so small and cheap to compute, they work well with gradient checkpointing. They're cheap to recompute on the fly, so it's more important to save memory and allow training at all. (This is also how OpenAI is training the Sparse Transformers which are enormous; they haven't said either way, but I assume this is how they trained the larger GPT-2s like the 1.5b parameter version, because I can't imagine what hardware would fit even a single GPT-2 1.5b without tricks.)
- bazza451 8y agoThis is really cool. Just to pick your brain (I’m an ML noob and it triggered an idea in my head). I was wondering if GPT-2 could be used to generate small paraphrases for an input sentence for search suggestions? e.g “I’m going to the mall today” -> “I will be going to mall today”.