29 ms·
OpenAI Releases Largest GPT-2 Text Generation Model
- epoch_100 7y agoPaper: https://d4mucfpksywv.cloudfront.net/papers/GPT_2_Report.pdf https://d4mucfpksywv.cloudfront.net/papers/GPT_2_Report.pdf Code: https://github.com/openai/gpt-2 https://github.com/openai/gpt-2
- sillysaurusx 7y agoIf anyone wants to fine-tune the 1.5B model, I ported the gpt-2 code to TPUs. You can fine-tune it in Colab. Snapshots are 5.8GB. notebook: https://twitter.com/theshawwn/status/1191800180192010246 https://twitter.com/theshawwn/status/1191800180192010246 code: https://github.com/shawwn/gpt-2 https://github.com/shawwn/gpt-2 It's a fork of nshepperd's gpt-2 codebase (https://github.com/nshepperd/gpt-2 https://github.com/nshepperd/gpt-2) which lets you fine-tune 117M and 345M on GPUs. For a tutorial on how to fine-tune GPT-2, see http://gwern.net/GPT-2 http://gwern.net/GPT-2
- zitterbewegung 7y agoCool this is awesome ! I’m going to try to retrain this with a twitter dataset called sentiment140 ( I have already processed it with gpt2 345M).
- MasterScrat 7y agoIs your fine-tuned model available somewhere?
- zitterbewegung 7y agoI can provide it to you. I have only done 355M. I was trying this for 1.5B but ran into memory issues .
- MasterScrat 7y agoI would be very interested! My email is on my profile.
- zitterbewegung 7y agoEmail sent
- sillysaurusx 7y agoSorry about the memory issue! I’ll have a fix up later today. Some info: https://twitter.com/theshawwn/status/1192038627854946304?s=21 https://twitter.com/theshawwn/status/1192038627854946304?s=2...
- rm_-rf_slash 7y agoAt a credibility score of 6.91/10, many people will rightly judge that the full GPT-2 model will remain insufficient for malicious use in creating fake news. However, even the smaller models are already good enough for spamming/trolling/astroturfing. It doesn’t take a Shakespearean soliloquy to convince people of a point. Just enough of a flood of short 1-3 sentence pro/con comments on a forum can drastically affect the perceived public opinion of an issue. Those comments can then spur real people to reply, which could result in an ultimately organic but directed propaganda vector. Propaganda directors will carefully craft something for people to look at, and the GPT-2 bots will move people’s eyes in that direction. You can see the same happen on r/subsimulatorgpt2, where the longer titles and prompts and replies eventually sprawl into incoherence, but the shorter sentences from the finetuned bots in the comments section are effectively indistinguishable from the kinds of short comments you would find on their respective subreddits. Or in other words, the malicious uses for GPT-2 won’t be a tidal wave, but a flash flood.
- gambler 7y agoFunny how instead of admitting that social media is trash and doesn't facilitate meaningful communication, thus making it impossible to distinguish between humans and low-level bots, Silicon Valley types keep pretending the problem lies everyone except with their systems. The AI it too good, the people are too stupid, etc.
- rm_-rf_slash 7y agoThere are too many edge cases to throw out the baby with the bathwater though. Reddit has a lot of really great niche communities that work hard to avoid toxicity (although to be fair I don’t go to anything news/politics related there, just memes and machine learning for me). Facebook, likewise, is a trash company with a trash product, but has completely replaced the town notice boards and meeting halls in rural communities, and is indispensable for working with them. I’ve been thinking hard enough to get steam blowing out of my ears trying to figure out a way to slice out that essential function from Facebook and put it on a platform that isn’t predatory and dangerous, but I still can’t square that circle. So in the meantime, if people want to engage and stay engaged with rural communities, Facebook is a necessary evil.
- chaz6 7y agoSurely we are not far off models capable of submission-quality essays that will enable a new generation of cheating.
- JRKrause 7y agoFrom my observation, even the largest GPT-2 model has difficulty retaining any long-range relationship information. In the "unicorn" writing example that was published originally, the model 'forgets' where the researchers are (climbing a mountain versus being beside a lake iirc) after just a few sentences. Because of this, it's hard to imagine models of this type being able to write long-form coherent papers. Now if we could somehow constrain the generated text to conform to a predefined graph structure that isn't forgotten so quickly...
- jsinai 7y agoMaybe the problem is that most of these models seem to rely on sequential information (even the transformer needs this for forward generation of text) to encode long range information. But I can’t remember the last time I relied on sequentially remembering the ordering of tokens in order to complete an essay or hell even reply to an email. Structurally we retain some kind of hierarchical information (topic, places, names, events) about text. Is there any active research looking into text generation models which do this? Maybe some kind of query that is made in a learned vector space and which is not temporarily dependent but rather “spatially” - as in these are the facts about the text being generated so far.
- flancian 7y agoI'm interested on this as well. I have been trying to fine-tune GPT-2 on genre fiction to work as a sort of "fiction replicator". Stylistically it actually seems to do quite reasonably, but it lacks narrative cohesion. This problem, as you point out, is corpus agnostic. I thought of trying to keep track of characters and key interactions outside of the model, but I haven't figured out how to make these two models interact reliably -- outside of just having the first component generate prompts for the second model in a kind of cooperative setting. Is there a known way to set up transformer to do infix generation? That is: give it a start and end prompt, and an estimated number of tokens to fill in between. That seems like it should be doable and could improve things, but I haven't found any work on this problem yet and haven't had the time (and potentially don't have the skills) to look deeply myself yet.
- ionwake 7y agoSorry for asking but is there an example output and an example input?
- minimaxir 7y agoThe original GPT-2 blog post has famous demos of the 1.5B model: https://openai.com/blog/better-language-models/ https://openai.com/blog/better-language-models/
- jboren_gpt2 7y agoI created an Instagram account from a Google Colab instance. https://medium.com/@jboren_20569/tensorrandos-gpt-2-musings-for-automatically-posted-to-instagram-4ddacba96bbe https://medium.com/@jboren_20569/tensorrandos-gpt-2-musings-...
- Voloskaya 7y agoYou can check this interactive autocompletion demo (you provide the input), that I believe is based on the 774M model: https://transformer.huggingface.co/doc/gpt2-large https://transformer.huggingface.co/doc/gpt2-large
- lotyrin 7y agoThey've added https://transformer.huggingface.co/doc/gpt2-xl https://transformer.huggingface.co/doc/gpt2-xl
- bungula 7y agoYou can interact with the full model here: https://talktotransformer.com/ https://talktotransformer.com/
- minimaxir 7y agoHuh, he updated that to the full model quickly.
- deleted 7y ago[deleted]
- buboard 7y ago> (CTEC) found that extremist groups can use GPT-2 for misuse, specifically by fine-tuning GPT-2 models on four ideological positions: white supremacy, Marxism, jihadist Islamism, and anarchism. CTEC demonstrated that it’s possible to create models that can generate synthetic propaganda for these ideologies I wonder how they tested that
- TaylorAlexander 7y agowow I’d love to read the Marxist and anarchist texts it has produced. I wonder if they used good source material. I would bet it gets the talking points but can’t convey the subtlety.
- buboard 7y agoI bet anyone who bothers to read them twice realizes it's nonsensical
- endtime 7y agoAnti-Foos already think that real primary Foo sources are nonsense (and they aren't the target audience anyway), and confirmation bias will take care of the pro-Foos.
- dragonwriter 7y agoIf it's not moving opinion, it's not effective propaganda. Something that people will agree with only to the extent it restates what they already believe isn't working as propaganda.
- endtime 7y agoNot sure I agree. Some propaganda probably just wants to rile up the base. It's also possible to change which issues seem most salient/urgent/etc. without actually changing anyone's mind. E.g. Democrats typically care about police brutality, and the environment, but they probably thought more about the former a few years ago and the latter more in 2019, even though AFAIK the actual issues haven't changed much.
- 490d0aff0ee8 7y agoTangent rant. I'm skimming over some of the code at https://github.com/openai/gpt-2/blob/master/src/model.py https://github.com/openai/gpt-2/blob/master/src/model.py and I can't help but feel frustrated at how unreadable this stuff is. 1. Why is it acceptable to have single-letter variable names everywhere? 2. There's little to almost no documentation in the code itself. It's unclear what the parameters of any given function mean. 3. There are magic constants everywhere. 4. Function names are so terse... ("gelu", "attn")
- slimsag 7y agoIn my experience, this is the norm in the ML scene. Giant globs of unreadable and in no way understandable code -- unless of course you already understand everything. I think this is because the "product" so to speak is often the papers themselves, not the code, but I'm not sure.
- fermenflo 7y agoI agree, a lot of the code could be improved. But some of what you mentioned is fairly standard. Like "Gaussian Error Linear Units being GELU, w/b for weights/biases, etc...
- jsinai 7y agoNot sure how standard that is ...
- steve_musk 7y agoIt’s very standard ML abbreviations.
- high_derivative 7y agoMy professional observation (as ml researcher at big tech): These companies hire a lot of engineers straight out of undergrad/master's degrees. The interviews test leetcode knowledge, and today lots of degrees are heavy on Python-scripted ML homework. The result is companies with billion dollar funding and world-changing goals having a lot of their code look like complete spaghetti. And this is the engineers who are meant to clean up research scientist code. Scientists generally don't feel like it's their responsibility to write strong code. Systems-side teams/orgs have better code, but essentially as soon as you enter the 'ml engineer/research engineer/research scientist' layer, it's doomed.
- oaskmutboard 7y agoI think this could make a great Tinder feature to suggest chat lines.
- YeGoblynQueenne 7y agoOh, I can imagine a few: <input> "If I told you your body is hot, would you hold it against me?" <input> <output> It was hot and the body of a young woman was lying in a bloody hell. Hell was hot and was full of beautiful young women. The body was lying in the entrace of the lobby and there was a small crowd gathering. it was a hot day in hell <output>. Could work on the right person though.
- odkamkfn 7y agoWho would benefit?
- hint23 7y agoYou can try it at: http://textsynth.org http://textsynth.org
- vitorbaptistaa 7y agoPretty impressive! This is what I got (my initial text is in italic): One day I woke up suddenly after a bad dream. I was sitting on my bed alone. As soon as I turned over I found myself lying down in a strange bed. It had no bed, no mattress and no cover for me. I was just lying on that thick carpet. I looked around to discover nothing but a white bed. There was no window, no furniture. No door. The only clue was a single white mirror that stood beside my bed. And there was a single candle lying on my bedside table. I was not worried or scared. I was fascinated. I could see the floor. I could see the walls. And I could see everything that was around me. I could hear the wind. I could see everything from the windows, from the door that was always slightly open. I could even see the light. I was in a dream world and I knew that I was in a dream. But what I wanted to know was what was happening in this world.
- Rebelgecko 7y agoI've always wondered if I could spend a weekend with some neural nets, do some light editing, and try to corner the market on some of the more obscure Amazon ebook categories (e.g. Sasquatch Erotica) It seems like this model would be more than adequate. Other than some basic human anatomy mistakes (e.g. breasts are typically above the waist), it generates some surprisingly explicit stuff (which I've mostly elided) even with my misspelling of "caressed". He slowly carresed her shuddering body back to her body. Her lips were wet and slick. Her breathing slowed and she began to shudder as a hand came down on her breasts. She shivered as it moved between her breasts gently. "Don't worry, I'm here. I'll let her go for you." He stroked down her body slowly as he stroked down her back. "Come here, my beautiful." He took a step down and began to push his body down into her. He gently pulled her closer against himself and held his breath as his hands came down from her waist to her breasts as his mouth came down to her p...
- YeGoblynQueenne 7y ago
- gerash 7y agoSampling realistic text from large pretrained models is non-trivial. I came across this paper in one of ACL 2019 workshops: https://arxiv.org/pdf/1904.09751.pdf https://arxiv.org/pdf/1904.09751.pdf
- clmnt 7y agoWe (Hugging Face) added it to Write With Transformers if you want to try the text generation capabilities of the model: https://transformer.huggingface.co/doc/gpt2-xl https://transformer.huggingface.co/doc/gpt2-xl
- krick 7y agoWow, some samples are frighteningly good. I was impressed by previous models and I don't know if I'm just lucky this time, but... wow. Can anybody who is not into climbing even tell this is all fake? Jain Kim is an experienced climber. In 2006, she became the first woman from Korea to climb all five 8,000 meters (24,064 ft) peaks in the Swiss alpine ski run Alps in 24 hours. In 2009, she made history again by setting the record for the fastest time to climb an 8,000 meter peak with a team from China and South Korea. She made the first ascent of 8,832-meter K2 in China, the second highest mountain in the world, in 2009 and the third highest mountain in Europe. She also is the first female Korean to summit a world-class peak. During her two years as a mountaineering professor at Sogang University in Korea, she established two new routes in the Yalu River area. The first of these routes is a 3,547-meter peak named K2 on Mount Long in China. Her second route is on the same mountain, called the Lomonosov Ridge, at 3,632 meters.
- bonoboTP 7y ago> Can anybody who is not into climbing even tell this is all fake? Hm, I'm pretty sure it's hard to climb five 8000 meter peaks in 24 hours :)
- krick 7y agoYeah-yeah, and "Lomonosov Ridge" is a weird name for a peak in China (and you might even happen to know the Lomonosov Ridge is underwater). Please don't be distracted by details. Maybe the author meant 24 h per peak, maybe his typing is really spurious. Is it really obvious to somebody who doesn't know anything about climbing that this is not true? Come on. What's really fascinating is how truth is mixed up with made up stuff. It's incredible, how much much real-world knowledge it has. - Jain Kim is likely a female name. - Jain Kim is likely a Korean. - K2 is in China and it is the second highest mountain in the world. - Mountaineering is somehow associated with climbing (and Jain Kim is not known for mountaineering, btw). - 8,000 meters is 24,064 ft and this is a formidable height. - climbing something in 24 h is a common pattern for a type of achievement. And more. No, I'm completely sure it would throw off plenty of people.
- k8si 7y agoOmfg can we stop making these things bigger PLEASE Like, who cares?? * What I mean is, text gen models are big enough. We need controllable text generation; like, so it can talk about a specific THING sensibly. Rather than spew statistically plausible nonsense.
- vicaya 7y agoJust like various chat bot incarnations, it's reasonably OK at generating generic sentences without real understanding. I tried several attempts with this leading fragment: "Is borrow checking unique to Rust? Probably not. But". The results are not pretty (the best one is as follows). Not impressed at all with this giant model :| "Is borrow checking unique to Rust? Probably not. But the Rust library is written by and for Rust programmers, so if you're looking for a library that's designed for the language you already know, this might be your first choice. The library is still in a fairly early stage. It does not support the standard libraries. If you're interested in reading about how the library works, the docs are up on GitHub for reference! How can I get this for my project? Get it here: https://github.com/paulp/rustbinder https://github.com/paulp/rustbinder I've just read your blog post and want to use it, what can I do? If you're looking for a project to use this library for, check out the docs for the "RustBinder project". What are the current limitations and how will future changes affect this library? This is still a work in progress. The library currently does not support the standard Rust library. There are a few work in progress"
- ssivark 7y agoWow! That is meaningless but difficult to distinguish at a glance, especially when read by someone not familiar with the subject. Are we going to see such auto generated content take over as the primary pillar of the SEO content farms? ... Kinda like an automated (text) version of Siraj Raval videos :-P
- genidoi 7y agoIt's interesting how the first paragraph in particular reads like a deflection of the original question, somewhat akin to a middle manager avoiding exposing their lack of domain expertise in response to a technical question by giving a vague answer. There were heaps of semantic pathways the network could have chosen such as guessing what a borrower is, or maybe even weighing the significance of "borrower" more than rust in the original question because it has no clue what rust+borrower is; but it instead picked the safer option of deflection. Fascinating stuff.
- rfhjt 7y agoPrompt: "Real things don't exist unconditionally and things that exist unconditionally are not real. However the reality has an essense. It is" Response: "an actual thing, and it is not the thing to which we attach meaning. It is not real because it is not a thing. And therefore, it does not possess the qualities that are inherent in all real things." Just wow. Sure, there are a few logical mistakes here, but this response serves as a good prompt for my bio-GPT. In other words, we usually need some starting points or hints for analysis and discovering these hints is non trivial because whatever we can think of is not very new to us. This GPT just gave me an answer that smells like a serious wisdom and I'll surely dig in that direction to see if this idea has any substance. Edit: what's happening here is while I can't ask this model to give me a short and conscise summary on a topic, I can still interrogate this model and find out what it's seen in the training set. I can't possibly read all these books in the training set, but now I can rapidly navigate in the multidimensional meaning space: I tell it where to start and it says what it seems in close proximity to my prompt. This is a breakthru.
- DiogenesKynikos 7y agoMaybe you've just proven that philosophical tracts can be written without any understanding. If the wording is sophisticated enough, everyone will assume there's some deep, hidden meaning in it. All form and no content.
- deleted 7y ago[deleted]
- deleted 7y ago[deleted]
- rfhjt 7y agoPrompt: The coming global recession is a real possibility and" Response: "The coming global recession is a real possibility and the Fed is playing games, creating artificial market conditions to make a recovery seem possible in the short-term. The Fed has an option to change its monetary policies but it will not make the problem go away, so it is in their best interest to pretend it won't happen." Change and to however and you'll get another stereotype opinion. It really just composes pieces of texts it's seen around the prompt, but it does this really well. Most of the news agencies can now fire most of their monkey typewriters: this GPT will outperform them on every metric.
- RobZombie 7y agoGay men seeking other gay men