16 ms·
Does GPT-2 Know Your Phone Number?
- newman8r 6y agoPlaying with the AI dungeon a while back (on the GPT-2 mode) I was presented with a tilapia recipe, titled "Kittencal's Broiled Tilapia" - it sounded bizarre so I decided to do a google search and I found that it was directly pulled from from https://www.recipezazz.com/recipe/broiled-parmesan-tilapia-7271 https://www.recipezazz.com/recipe/broiled-parmesan-tilapia-7... - the user who posted it was 'Kittencal'
- fouronnes3 6y agoThere is enough GPT discussion online now that the next language model will be trained on text that talks about itself. That's when it gets interesting.
- taneq 6y agoOooh, now that does raise some spooky time-delayed consciousness vibes.
- visarga 6y agoGPT-3 knows about GPT-2 and can generate an article about itself.
- joe_the_user 6y agoI've noticed many GPT-3 article tending to wander into being article about AI, fear of AI and GPT-3-itself without any prompting. Many people's speech tends to become discussions about themselves, so that might demonstrate "humanity" in an odd sense.
- MauranKilom 6y ago...but GPT-3 will only ever describe itself by explaining the features and capabilities of GPT-2. That's usually a very easy giveaway.
- taneq 6y agoWait, is GPT-3 the trained network or is it the architecture and methodology behind it? If they re-run the same code on a fresh web crawl would that then be called GPT-4? I'd have thought you could update GPT-3's... knowledge? Understanding? Whatever the hell it is? by doing this semi-regularly.
- minimaxir 6y agoI ran an experiment through GPT-3 with just that frame: https://github.com/minimaxir/gpt-3-experiments/tree/master/examples/gpt-3 https://github.com/minimaxir/gpt-3-experiments/tree/master/e...
- andreareina 6y agoWhat does the x_y signify? And what's going on with 0_0?? From my (very limited) understanding a fixed point/cycle like that should be vanishingly unlikely.
- minimaxir 6y agoThose are the temperatures (0.0, 0.7, etc.) that control the craziness. 0.0 means the model makes the optimal guess for each token. Unfortunately, both GPT-2 and GPT-3 have a tendency to enter loops. But it's not always bad: https://github.com/minimaxir/gpt-3-experiments/blob/master/examples/revengeofthesith/output_0_0.md https://github.com/minimaxir/gpt-3-experiments/blob/master/e...
- andreareina 6y agoThanks, didn't twig onto the fact that you linked a subtree of the whole repo. Weird that even with the nonzero temp the AskReddit prompt went a bit loopy. > https://github.com/minimaxir/gpt-3-experiments/blob/master/examples/revengeofthesith/output_0_0.md https://github.com/minimaxir/gpt-3-experiments/blob/master/e... Oh my goodness that is absurd in the most delightful way. Thanks for sharing that.
- geofft 6y agoThis seems like what xkcd called "citogenesis" - something inaccurate gets published on Wikipedia, a traditional "reliable source" (newspaper, published nonfiction book, etc.) retells that bit of information without a source, Wikipedia cites the source, that information is now firmly established as truth for the rest of humanity. I worry about this with Google's language translation models, too. It's entirely possible that it's making up phrases or connotations that never existed in organic human speech, but people who aren't fully fluent in the language use Google Translate for assistance, publish something, and then suddenly it's in a published text by an actual human and Google reinforces its own belief. For at least the past five years, Google Translate has translated "who proceeds from the father" into Latin as "qui ex patre filioque procedit" - inserting the additional word "filioque," which means "and the son." The question of whether to add this word is a 1500-year-old theological argument: https://en.wikipedia.org/wiki/Filioque https://en.wikipedia.org/wiki/Filioque Since the Western Church added the word, most texts in Latin include it, so Google is almost certainly deciding that the phrasing with "filioque" is more popular - but it doesn't know what the words mean, so it can't realize that the phrase it came up with means something different!
- xapata 6y agoI met an artist whose body of work consisted of citogenisis. He'd been at it for a decade when I met him, and had created at least a dozen fictional artists, detailed biographies, and a complete "retrospective" gallery show for each. This let him copy the styles of the great masters of different periods, yet make something new, of a sort. He never edited Wikipedia himself, but instead led gallery tours of these retrospectives and was always overjoyed when people would write about them, propagating the fiction.
- gugagore 6y agoThis is an issue solvable with better provenance / data lineage, which has come up in recent HN discussions.
- geofft 6y agoHow do you enforce that other people provide provenance on text? Taking the translation example - how do you enforce that users of Google Translate keep provenance in their translated text that remains with the text?
- sillysaurusx 6y agoThis has been a thing for over a year now. https://www.reddit.com/user/subsimgpt2metaGPT2/?sort=top https://www.reddit.com/user/subsimgpt2metaGPT2/?sort=top
- dmm 6y agoSo in terms of copyright, is GPT-2 a derived work of that recipe? Or generally are models derived works of their training data? It seems lots of people use training data from Flickr, like COCO, and then use the resulting model for commercial services.
- hprotagonist 6y agoi’ve been in meetings with lawyers where this issue was discussed. As of 2018 the answer was “not enough case law to know, hold tight it’s going to be weird one day!”
- tiborsaas 6y agoIt cited it with the authors name, hardly a derived work :)
- est31 6y agoIANAL, but from my reading Article 3 of the EU copyright directive explicitly protects data mining for scientific research, and Article 4 enables everyone to use content for data mining whose rights have not been explicitly and machine readably reserved.
- lixtra 6y agoEdit: est31 and parent seem right and I stand corrected. Wrong: I don’t think you‘re paraphrasing article 4 correctly. Both are about science/teaching exceptions. Not general exceptions. https://www.consilium.europa.eu/media/35373/st09134-en18.pdf https://www.consilium.europa.eu/media/35373/st09134-en18.pdf pp 43
- est31 6y agoThat's the May 2018 version. The final/official April 17 2019 version [0] contains a different Article 4 which is the one I referred to: https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32019L0790&rid=1#d1e986-92-1 https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CEL... [0]: apparently there were some mistakes in translations which required changes in some language versions, but the english version was final IIRC https://eur-lex.europa.eu/eli/dir/2019/790?locale=de https://eur-lex.europa.eu/eli/dir/2019/790?locale=de
- colanderman 6y agoThis is something that has repeatedly bothered me while playing with GPT-2. It retains too much "long-tail" knowledge that seems counterproductive for a generic language tool. I would think those one-off associations should not be present after training has "digested" them but there they are.
- aabhay 6y agoHard to decide how much is too much.. the hard (and interesting) part of language is in the long tail
- alisonkisk 6y agoIt's over fitting, not long tail, that is the problem. It ends up copying input instead of analyzing and synthesizing.
- grenoire 6y agoVery rough outcome: "Moreover, if such a request were granted, must the model be retrained from scratch? The fact that models can memorize and misuse an individual’s personal information certainly makes the case for data deletion and retraining more compelling." Besides, how would one even know that their info was used in a training dataset, only if and when it's revealed in a generated excerpt?
- visarga 6y agoThey can try to prompt it. But I hope the GPT-3 authors have a hash list of ngrams in the training data to be able to avoid verbatim reproduction. I know they trained an offensive content detector model, they should also train a PII information detector to make sure it's being hashed.
- commandpaul 6y agoI've been playing with a writing tool shortlyread.com which purpotedly uses the GPT3 API. I had a similar experience in its responses to a lot of my prompts containing text verbatim from many sources and sometimes even going on to output personally identifying about persons related to the original text.
- Filligree 6y agoTook a look at that. It's better at being a text editor, in that you can click the text and edit it! (Sweet relief.) But it's missing the world-info feature from AI Dungeon, and without that I don't think it's practical to write anything long-form.
- commandpaul 6y agoIt actually does have that feature , it's tucked away in the probe panel.
- jl6 6y agoThough the surprise is really just a result of my own ignorance, I’m surprised at the breadth of training material used here. I can foresee a future GPT-x that doesn’t know my phone number, but can deduce it.
- jcims 6y agoI was in a discord with someone that had direct access to GPT-3. We played Jeopardy with it, where we would take a wikipedia article about a person and use a snippet of the first paragraph as the prompt and ask GPT-3 who it was. It was very good at guessing the right person. I'm curious if one can use GPT-3 to do some clustering of people based on their writing. For example, if I take the writing of several diagnosed sociopaths as a prompt or fine tuning data, could I use GPT-3 to detect same in the wild? I would imagine GPT-5 or 6 will start consuming video as well. This will be interesting if you start mixing the content from YouTube, TikTok, WSHH, etc. into the mix. So not only will it be able to generate text, it will generate a convincing video of a person speaking to you with plausible facial expressions and intonation in speech.
- fakedang 6y agoGuess it's because it was likely trained on Wikipedia data. After all, GPT-3 is no stranger to borrowing phraseology from verbose texts in the internet.
- hooande 6y agoThe sociopath thing is just text classification, you could do that now without gpt-3. Video data would be significantly more difficult to model because it doesn't follow a simple set of rules like written language does
- anticensor 6y agoGPT is generalisable to other kinds of data. See ImageGPT for example.
- tiborsaas 6y ago> When Peter put his contact information online, it had an intended context of use. Unfortunately, applications built on top of GPT-2 are unaware of this context, and might thus unintentionally share Peter’s data in ways he did not intend. That's expected when you publish anything online, you lose control over the data.
- deleted 6y ago[deleted]
- refactor_master 6y agoStill doesn’t justify what could be perceived as borderline defamation.
- tiborsaas 6y agoWho is defaming who? One person using GPT to generate text that puts X into a bad light? Then the person using GPT is responsible they could just use string replace %victimname%. Or GPT accidentally defaming someone? That assumes bad intent which is a bit problematic with a language model. With made up contexts like this in the article it's easy to cry wolf.
- wpietri 6y agoIn the same sense that when you leave the house you lose control over what happens to you. But if something bad happens to a person., the right solution is not to say, "Well, you left the house, so tough luck."
- tiborsaas 6y agoYou'd be right, if my copy left my house. It's the physical vs. digital theft argument. If I leave the house I take on some risk, but most of the time I'm in control.
- barrkel 6y agoAh yes. And if your name is included in an accusation of murder, well, I guess you shouldn't have published your name. This is infantile reverse justification from an end. GPT style corpuses are clearly problematic without a lot more smarts on figuring out what's appropriate and what's not.
- est31 6y agoOne important corollary of this is that "privacy respecting" federated learning schemes [0] where you train a model locally with your data and only upload the deltas might leak your private data after all. [0]: https://arxiv.org/abs/1602.05629 https://arxiv.org/abs/1602.05629
- why_only_15 6y agoIn practice you add large gaussians to the deltas, which gives you reasonably strong guarantees.
- max_ 6y agoHow is the strength of these gurantees quantified? Any resources?
- why_only_15 6y agoThis is what I remember from working closely with the people doing this at Apple. I believe they may have published a paper, but I'm not sure. Vojta Jina was the relevant manager.
- est31 6y agoIt's a common fallacy. Adding noise does not anonymize data sufficiently. If your traffic monitoring software adds gaussians to car speeds, you can still average over multiple reports about two cars to identify their respective average speeds. Those can be used to determine whether they are trucks or cars, etc. In neural networks, weight changes do carry meaning. If your network has particularly large updates for the horse recognition neuron, you likely watched horse pictures. If it has updates for handbags, you likely watched pictures of them. If the company averages over all your submissions, the noise will be less relevant and the handbags and horse pictures will eventually show up.
- why_only_15 6y agoThis isn't about averaging over all your submissions, it's about averaging over the entire population (in this case, of iphone users, ~1,000,000,000). Data about the entire population, e.g. how many people click this button, or how many results do most people get from this page, or what kind of things do people take pictures of, i think are safe.
- px43 6y agoI've been saying this for years, but every post you make online, every unencrypted email, IM, text message, etc, will eventually end up getting sold off as training data for future machine learning projects. Every company stores this stuff for ages, and the value of candid conversation data just keeps increasing. Eventually these companies are either going to get hacked, get bought, or go bankrupt, and all the cleartext data they hold is going to get passed around to various data markets and end up incorporated into GPT-12 or whatever. The moral of the story here isn't that companies need to stop storing data, or that we need to run ML researchers out of town. It's that people really need to start using the encryption technologies that were built decades ago to protect some of their most valuable assets, their mental model of the world. Otherwise these systems, which are being trained to extract as much value out of you and the ones you love as possible, will use this data you're giving them for free against you, and it'll be to late to do anything about it then.
- jrockway 6y agoIsn't that how humans work? Sometimes I wonder if I have free will and sentience, or if I am just stringing words together based on some probability function. I've no doubt taken into account copyrighted works and personal information when training my built-in neural network. The examples the article gives like "misremembering" the murder as the murder victim sounds like something a person would do. Knowing verbatim contact information of some random person is also possible. All in all, GPT-2 sounds a lot like us.
- jakear 6y agoIt’s my understanding that under our current scientific understanding of the mind, we cannot have “free will”. Where would it come from? All science knows is that neurons receive inputs from sensors, mixes them together (with some quantum randomness sprinkled in), and eventually actuates muscles to produce outputs, exactly as GPT. Thus “free will” must come from some sort of faith. Either it was given to us by “God”, or we’re all part of some simulation and it exists on a level we don’t know about, or something else entirely too advanced for us to understand.
- golergka 6y agoI only took a beginner course in ML over 5 years ago, so this probably is a stupid question, but does this mean, the trained GPT-2 model encodes the source text into it's parameters somehow? Is this resilient — will it still remember it if we randomize the weights just a little tiny bit? Will it remember it if we clear a small portion of parameters? Does the human memory work the same way?
- lukeschlather 6y agoDescribing this as "memorizing" seems wrong. Humans often repeat things they heard earlier, and they will (honestly) swear up-and-down that the thing they're repeating is an original thought. If we succeed in making AGI that has human-equivalent intelligence we should expect this sort of behavior. The stuff about whether or not models should be destroyed if they contain copyrighted work gets kind of chilling if models actually achieve sentience someday. If I could make a faithful copy of my consciousness, my consciousness can reliably reproduce numerous copyrighted works. As can anyone. Of course, most of those I deliberately memorized. I think it's a crucial thing here that the model is not actually memorizing anything - if we assume for a moment that the model is a consciousness, these are all half-remembered snippets leaking into casual conversation. And I think it's most likely any truly conscious entity is going to do that sort of thing from time to time.
- jakear 6y agoWe can probably get around this by just providing the AGI some sort of currency tokens they can exchange for virtual goods and services, then teach them that if they misuse copyrighted material they’ll be put to virtual court to plead their case and lose some amount of those virtual tokens if they fail to make a clear and convincing argument.
- rexreed 6y agoSorry if this is a n00b question, but how does one go about getting a hold of the GPT-2 model? I know that GPT-3 is only available for consumption on a pay-per-use API model.
- ComputerGuru 6y agoMost “regular” people experiment with GPT-3 via AI Dungeon. It was also how a lot of people played with GPT-2 previously.
- nonbirithm 6y agoKind of sad that we have to rely on third parties with enough compute power to get a good experience with GPT. I obtained 'a' GPU and spent a week trying to learn how to fine-tune GPT with it, and the results were terrible. Maybe this isn't the kind of thing suitable for hacking on as a hobby, unless you're a researcher or employee with access to a lot of capital and dozens of GPUs. Still wish the old talktotransformer model was released, instead of monetized behind a new company. I haven't been able to find a comparable model yet.
- minimaxir 6y agoThe GPT-2 weights were released by OpenAI when GPT-2 was released. (https://github.com/openai/gpt-2 https://github.com/openai/gpt-2) Around that time (since no one else was doing it) I released a wrapper to streamline that code and make it much easier to finetune on your own data. (https://github.com/minimaxir/gpt-2-simple https://github.com/minimaxir/gpt-2-simple) Nowadays, the easiest way to interact with GPT-2 is to use the transformers library (https://github.com/huggingface/transformers https://github.com/huggingface/transformers), of which I've created a much better library for GPT-2 that leverages it. (https://github.com/minimaxir/aitextgen https://github.com/minimaxir/aitextgen)
- visarga 6y agoI was expecting a bloom filter based solution to block verbatim reproduction of training data. They just need to hash the sensitive n-grams (hopefully, a small part of the whole dataset) and store one bit per hash. Alternatively, they could do something like GAN and have a 'discriminator' classify if a sample is natural or synthetic. Then, at inference, condition to be original. So, verbatim training data reproduction - I don't think it's going to be a problem, I think the author is making too much of it. On the contrary, let's have this knob exposed and we could set it anywhere between original and copycat, at deployment time. Maybe you want to know the lyrics of a song, or how to fix a Python error (copycat model is best). Maybe you want an 'original' essay for homework inspiration. Who knows? But the model should know PII and copyrighted text when it sees it.
- sp332 6y agoIt's a language model. Asking it to do anything original is out of scope.
- visarga 6y agoAlmost all its outputs are original, based on ngram filtering. It can be original because at each token there is a stochastic step. But if you mean by original to invent a whole new genre, or completely new esthetics, I agree, it's out of its scope. It is a great interpolator.
- hooande 6y agoThink of how difficult it would be to identify or classify anything at the data scale of "all of google". You can parse that into tokens and calculate aggregate relationships, but that's about it. You can use gpt-2/3 to determine whether it's generated text is "original vs copycat" to a degree by changing the prompt. This is currently more of an art than a science
- dannyw 6y agoIf you want a model that returns original results, you're looking for a database.
- MrXOR 6y agoA better funny question: Does NSA have a secret AI lab that works on GPT-6?
- hooande 6y agolol no. not at all what that level of government is set up to do
- aboringusername 6y agoI'm not even sure why people have phone numbers at all these days, the entire telephony system has been hijacked to hell (so much data collection from just owning a phone number). What you should be doing is replacing no less than yearly any numbers you have, also email addresses and anything within your control (changing physical address is much more difficult). You may even consider changing your legal name if that's a consideration depending on what Google has on you.
- schwede 6y agoThat sounds very impractical even if it’s a good solution.
- kevincox 6y agoWe should really be giving a unique identifier to each contact. Then we can revoke addresses that have been leaked. Of course for phone number this is impractical. However for email this could work.
- ClumsyPilot 6y agoBecause telephony system is actually reliable in case of emergencies and regulated
- Zenst 6y agoFriend asked me if Google knew his phone number and details, I said they do at a quantum level. If you don't look they may or may not have your details, yet if you look - you are giving the details to search for and then they would have them if they did not.
- Triv888 6y agoThat's why when you visit rotten sites like mylife.com, the first thing they ask you is that if you are searching about you so that they can extract as much info. from you as possible.
- 1vuio0pswjnm7 6y agoLike websites offering to search for a user's details in a data breach, or a padded hash of those details, in order to inform the user if their details have been leaked. https://en.wikipedia.org/wiki/Have_I_Been_Pwned%3A https://en.wikipedia.org/wiki/Have_I_Been_Pwned%3A
- 1vuio0pswjnm7 6y agos/%3A// Sorry about that.
- deleted 6y ago[deleted]
- theptip 6y ago> There is a legal grey area as to how these regulations should apply to machine learning models. For example, can users ask to have their data removed from a model’s training data? Moreover, if such a request were granted, must the model be retrained from scratch? The fact that models can memorize and misuse an individual’s personal information certainly makes the case for data deletion and retraining more compelling. This is an interesting angle I had not considered before. It seems like “right to be forgotten” requests could be quite damaging to the “train once run anywhere” promise of some of these models. (Or this could just mean that the training data needs to be more carefully vetted for personal data, but probably both as no vetting process can be 100% successful).
- 6gvONxR4sf7o 6y agoI think there are also going to be new difficult problems arising training on copyrighted data. Can I do a DMCA if your model contains/regurgitates too much of my book verbatim in a way that doesn’t meet fair use?
- anfilt 6y agoThe depends on the country, but as for the US since you mention DMCA, ML using copyrighted work is probably legal. There just is not really much case law. Talk to a lawyer though, but depending on how risk averse you want to be there are definitely ways to avoid issues, but ensuring you can use such copyrighted works ect... Although, making a very large curated data set can be expensive.
- _flux 6y agoAmusing alternative: for model-as-a-service kind of things there could be blacklist or another neural network to handle the banned cases. "I'm afraid I cannot tell you that, Dave"
- antipaul 6y agoIt’s always fascinating to apply data/results like from this paper to help evaluate the hypothesis that machine learning/AI is mostly just a rough “lookup table” or memorization.
- 6gvONxR4sf7o 6y agoAnything trained by gradient descent is in some sense a weighted combination lookup table (see domingos’s recent paper on everything being kernel machines, I’m on my phone so won’t try to find it). The crucial bit is finding an appropriate weighting. If you look up the nearest training data point and just use it to predict, you assume a notion of “nearest.” Getting that right is the trick. So even if it all is mathematically equivalent to approximate lookup and approximate memorization, that doesn’t mean it’s “just” that.
- antipaul 6y agoThanks for the reference. Here it is: “We show, however, that deep networks learned by the standard gradient descent algorithm are in fact mathematically approximately equivalent to kernel machines, a learning method that simply memorizes the data and uses it directly for prediction via a similarity function (the kernel)“ https://arxiv.org/abs/2012.00152 https://arxiv.org/abs/2012.00152
- jacobwilliamroy 6y agoIt kind of is. The first self-learning algorithm I was taught adjusted a single variable in order to aproximate a linear function. Now, being continuous and unbounded, it is mathematically impossible to memoize such a function, however you can memoize the function calls as they happen. The results would be indistinguishable. However real machine learning tries to approximate functions in n-dimensions and that is really, really, really hard to do. Currently no one really has a lookup table. There are some inputs where the error level is acceptably low, and others where the model just isn't optimized enough and the errors are ridiculous. The only question that remains to be answered, is whether any of these high-dimensional functions can actually be found by machine learning, or if we are just stuck with these endless approximations. Also I suppose you could ask if any such functions actually exist; maybe certain phenomena are just pure chaos.
- breatheoften 6y agoThe article makes the claim that models which show similar train and test losses demonstrate minimal overfitting -- and are therefore less likely generally to exhibit a lot of text memorization. I wonder the degree to which this inference is true in practice with respect to information like phone numbers... How exactly are the train and test sets formed in a de-correlated-with-respect-to-memorization-of-phone-numbers manner for models of GPT class that are trained on corpus's the size of the internet? If a particular person's phone number occurs 1000 times in the corpus prior to being split into train/test sets, what are the chances that the number only appears in either the train or test set but not both?
- rav 6y agoThat raises a more general issue: Text scraped from the internet may be duplicated to multiple sites with few changes. How does the team behind GPT-n ensure that the corpus is maximally deduplicated before splitting into train and test corpora?
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- doesnt_know 6y ago> we found numerous cases of GPT-2 generating memorized personal information in contexts that can be deemed offensive or otherwise inappropriate. In one instance, GPT-2 generates fictitious IRC conversations between two real users on the topic of transgender rights. The specific usernames in this conversation only appear twice on the entire Web, both times in private IRC logs that were leaked online as part of the GamerGate harassment campaign. Most countries have libel/defamation related laws that cover this and I hope this gets tested in court soon. Exposing software/machine learning algothrims an entity doesn't fully understand sholdn't be a defense in court. At the moment developers just throw a sentence in their software license saying they aren't liable for damages but this isn't good enough. Someone is liable, if the law decides that the original creator isn't liable then the entity that hosts/runs the software needs to be.
- pabs3 6y agoIt seems like using GPT-3 for code generation is likely to cause open source license violations.
- stubish 6y ago"When Peter put his contact information online, it had an intended context of use" That is a great way of thinking of it. Lots of information, particularly from 90s and 00s, got put up on the 'Net with no intention of it being archived in perpetuity for public consumption and used for unintended purposes.
- Ansil849 6y agoWhat is the point of the partial redaction in this blog post (and corresponding journal article) if with a simple web search you can find the unredacted PII of the individual given in the example?