15 ms·
It’s infuriatingly hard to understand how closed models train on their input
- simonw 3y agoThis post was partly inspired by this conversation earlier today: https://news.ycombinator.com/item?id=36184948 https://news.ycombinator.com/item?id=36184948
- mcint 3y agoThank you, I was unclear from the post content.
- thibautg 3y agoDo OpenIA et al. even understand it themselves?
- simonw 3y agoSomething they definitely understand is exactly what their pre-training data looks like - the raw text that goes into the initial runs of training the models. Instruction tuning and RLHF is a bit more complex than that. I assume they maintain detailed logs all of those human-driven decisions about which responses were better.
- thibautg 3y agoDo they really know exactly what the raw text looks like? It seems so huge that no human could read all the text from each corpus. And the models have been fed text from many different languages. I doubt they have people who understand all the languages. Regarding RLHF, I also hope that they kept the logs of all the human decisions. But since it was (at least partially) outsourced to African companies like Sama.com, do they really get back all the logs or just a new fine-tuned model? But they must indeed at least know what is done with the text submitted to the prompt or to their API. (I’m really not an expert so my questions may sound naive)
- dontupvoteme 3y agoRLHF apparently manages to mostly force all responses in all languages to be quasihomogeneous. I'm not sure if that means they translated the RLHF data to as many languages as possible and then repeated it or if it's something more fundamental which applies regardless of input language. Although asking it "What can you not talk about" in Japanese only responds correctly with gptv4, and each language gives you a different list of items to some degree (between 4 and 6 items i found). Sadly trying to speak Klingon or Sindarin to it is dodgy at best
- fantyoon 3y agoIts safe to assume that whoever OpenAI outsources to does not get access to the model. Collecting data and training models on it will be two different steps.
- pixl97 3y ago>Do they really know exactly what the raw text looks like? I mean yea, it's too big. That said, in a post ad hoc fashion they do. When the model spits out weird crap at times, they can search the raw corpus and filter those strings out. There was an incident around this with Reddit counting forums and strange usernames that were added in tokenization, but later removed from weights leading to odd behavior when doing inference.
- pxoe 3y ago[dead]
- sebzim4500 3y agoOpenAI obviously don't know all the data in their dataset, but they must at least know whether they are training on private data submitted to their API.
- closewith 3y agoMaybe we need an Open Internal Affairs to police OpenAI?
- jjulius 3y agoSome OpenIA for OpenAI.
- rvz 3y agoI don’t think that they even understand what is going on inside of these AI models. Why is that? GPT-4 is as transparent as a black box and lacks transparent explainability and openly admits to regurgitating nonsense. How do they allegedly ‘fix’ it? Train it on more data.
- sebzim4500 3y agoI wasn't aware of https://trust.openai.com/ https://trust.openai.com/ before; it's hilarious. If you want to find out about why you should trust them with PII they make you submit your PII to them.
- joshuanapoli 3y agoI guess that training on specific bodies of private and licensed data will become a selling point. Right now, it’s a lot of trial-and-error to gauge a (closed) model’s expertise in a particular niche. I think that there’s naturally a (large) premium for access to a model that’s definitely trained and knowledgeable on all of ACM, or all of ASME standards, or other similar bodies of work.
- svaha1728 3y agoFor GPT 4 I usually look on scribd. If the book you want was there in 2021 you can usually prompt information from it.
- zoogeny 3y ago> Could a large language model trained on data fit under that term? I don’t think so, but the terminology is vague enough that once again I’m not ready to stake my reputation on it. This is such an interesting time - when the rules of the game haven't been settled. It was like when Uber was openly flouting established laws with their service. They knew that new laws would be required eventually and so they were willing to push boundaries on ambiguous interpretations until the matter was clearly settled. What surprises me is the stance people take in the face of ambiguity. For example, when I see such vague terms I assume that the reason is so that github has the plausible deniability in using the data for things like model training. I don't mean to say it was written explicitly so that they would be able to use it for that purpose. I mean that the ambiguity, and lack of a desire to clarify it allows them to operate for some time with the plausible deniability. A product manager could make an internal case that a temporary competitive advantage can be realized by interpreting their terms of service in a particular way. Eventually, law is going to catch up. And when it does Github, Microsoft, OpenAI, and all like them will follow the rules. But I, unlike the author, just assume that they are using this window of ambiguity to use the data they have available to gain an advantage. Maybe I am just cynical and the author is more trusting. The truth is probably somewhere in between. They are probably using more than many would feel comfortable with, but probably less than my worst fears.
- ChainOfFools 3y agoAnd long before Uber, YouTube did it in spectacularly brazen fashion. Back when the site's value was 5% very basic video player, 15% reliable fat pipe' and 80% enormous database of complete "we can't be responsible for policing what our users are uploading" copyright infringment. "Let the rights holders sue us today to take their content down, and beg us tomorrow to keep it up." And it worked, which is how we end up with a hanful of people who would otherwise be competent but unremarkable 9-5 coders at $CORP, are instead through a bit of fortunate timing, in control of fortunes the size of small countries.
- smt88 3y agoYouTube doesn't belong on the list. The process you're describing is fully legal under the DMCA and has been the operating mode for every site that allows user submissions.
- mcint 3y agoI share many of your concerns and frustrations, although I suspect what you're asking for their consider a moat along the lines of a trade secret, rivaled only by the collection of performance improvement techniques they've amassed in 1000s-10,000s of training runs, 100s of engineers, and (hundreds of?) millions spent on compute. People are hired and praised in the community for their skill in cleaning data. A non-answer for you, but for curious others, [State of GPT] 10 days ago provides a through introduction to the process used to train, Karpathy speaking at a Microsoft event providing a deep summary review of concepts, training phases, and techniques proving useful in the world of Generative Pretrained Transformers. [State of GPT]: https://www.youtube.com/watch?v=bZQun8Y4L2A https://www.youtube.com/watch?v=bZQun8Y4L2A
- theptip 3y agoYou seem to be talking about architecture, Simon is discussing training datasets. Of course, the contents of those datasets is trade secret too, but Simon is not looking for the contents, or even the cleaning strategies, just the lineage; is OpenAI using my private data? I don’t care how, just whether they are or not.
- mcint 3y agoExcellent clarification. I suspect this is at a natural Schelling point: don't say much; because more answers would only lead to more questions. The trade secret aspect includes that lineage. It's an edge. Even in his post, only the first and second short sections hint in passing what's linage about training would be wanted or why. I'm not clear what people worry is at stake. I will hunt these comments on this question as well. What are the incremental concerns of AI literate people.
- theptip 3y agoTo expand upon his first paragraph: > People are worried that anything they say to ChatGPT could be memorized by it and spat out to other users. People are concerned that anything they store in a private repository on GitHub might be used as training data for future versions of Copilot Individuals care because if they disclose highly personal secrets or mundane private information (eg health status, PII like address, tastes/preferences) then those could easily be disclosed later. Companies care for more obvious/less speculative reasons; both the trade secret version of the individual concerns above, but also strategically, in that many companies don’t want to aid their competitors by training OpenAI/Copilot how to write code for their domain. (Obviously what you really want is to fine-tune GPT-4 on your code and be able to trust that they aren’t going to use that for training future models.) My gut feel is that they aren’t saying anything because they are doing grey area stuff pushing the boundaries of “fair use” and plan to ask forgiveness later, which will be easier to do when they have demonstrated massive levels of utility & everyone is benefitting from their model.
- causality0 3y agoBut what does this mean in practice? I've found it helpful to think of AI models as if they're people. What does that mean? It means when I'm talking to them I keep in mind that I don't know how they know what they claim to know, I don't know who else they're going to talk to or what they're going to repeat that I told them, that there's no guarantee they'll even try to treat me kindly. Trust AIs as much as you trust a random stranger on the bus.
- londons_explore 3y agoIt's hard to train a chat AI directly off people's use of it. Some users will just dump spam in there, trying to trick it, lying to it, etc. Instead, I believe they use logs of user chats to detect when the user was dissatisfied with the AI's response (for example by detecting closing the window). They'll then pay a human to review these conversations, and if a human can write a better response, then that will go into the next round of training.
- godelski 3y agoThere's been an existential issue I've seen with publishing, concerning this (same/related issues to the article). There's models and datasets that are closed and being published in works. Here's an example[0]. ICLR states > Submissions will be double blind: reviewers cannot see author names when conducting reviews, and authors cannot see reviewer names. This means that the submission must not contain acknowledgements or any link (e.g., github) that would reveal authors’ identity. While Chinchila certainly is not a github link it is an acknowledgement that reveals the lab identity. Which leads us to one of two situations 1. The reviewers don't know the model/dataset is proprietary: In this case, we likely have a reviewer that is unqualified to even be reviewing. 2. The reviewers know: In this case, there's an ethics violation. == Why this is a problem == There's several issues here: bias, evaluation, reproduction, ethics/scientific goals. Bias: Most of these models and datasets are well known within the community and thus we know which labs they are from. We're biased to believe that works from big labs are higher quality and more accurate. This may be true to some extent, but your purpose as a reviewer is to judge the merit of the work. We try to do so in as much isolation as possible to prevent ourselves from being influenced and just focus on the merits[note0]. For example, any work with JFT is going to be known as a work from Google. Evaluation: We'll use this work[2] as an example (it uses JFT and a lot of TPU compute: i.e. Google). Since the work only uses JFT we don't know if this scaling only works for JFT trained models, especially since it depends on the pretraining not overfitting (I'm unconvinced tbh)[note1]. At least their table compares mostly to other JFT models, but it is mostly meaningless because JFT changes size frequently and we can't assume consistent data. All we know is that this result is valid for JFT pre-trained models. Their claims are much stronger than this. Reproduction: This is rather obvious. But we also have good reason to believe that these models change as well as the datasets. So even if there was a leak we have issues with works quickly losing validity as the next work is using different settings than the previous and intimate knowledge would be required to know if this affects results or not. NLP has this issue with OpenAI changes. It is very noisy research doing things like this. Ethics and Scientific goals: Our job is to advance human knowledge. This can be done in a noisy process, but we do have a responsibility to make this as clear as possible. Our job is NOT to be ads for big companies, but closed work does so. In fact, this is a big reason big companies publish. They are great publicity. This is fine, as long as they provide utility to the community. We have other ethical issues[note2] == Proposed Solution == Submissions must utilize open models and datasets. The work is evaluated based on that. After acceptance the works can add. This way there is no deanonymizing information, we ensure reproducibility, but we also are able to gleam the potential results from the proprietary efforts. This allows for the big labs to get their publicity while ensuring that anonymity is not broken and we're not biased to big labs. This doesn't solve the arxiv issue and the dumb social media policies[note4] but it is a a simple step we can implement right away. It is also a step that YOU the reviewer, a single person, can take action against. We need to take a serious look at ourselves and ask what is the point of our science. Is it to make money or to progress human knowledge? You can do both, but what is the main goal. I'd argue that our goal is knowledge. It is fine to make profits off this knowledge, but I'm afraid we've incentivized the profits too much and our system is not aligned with the original goals. Goodhart's Law is a bitch. == Notes/Bib == [note0] There are some biases we should concern ourselves with, such as compute power, but this requires more nuanced understanding. A work that can spend months tuning hyperparameters will always have better benchmarks than works that spend a month of compute for their entire work. These can't be evaluated on benchmarks alone and doing so will cause us to miss extremely important works (see Lottery Ticket Hypothesis[1]). Reviewing should be hard and require nuance, so there should be certain roadblocks to make it difficult to be lazy. [note1] They do near de-dupe but we can't know how effective the techniques are so we can't know how valid these claims are. [note2] Conferences are a bad way to do science. Conferences are filters, journals are QC. ML/CS uses conferences. This is a zero sum environment where your competitors are given the opportunity to filter your work. You don't need active collusion to form bad actors. It pits the people evaluating the work against the work. A journal system instead has reviewers focusing on adding value to the work. This is because journaling has rounds and discussions. Conferencing is nearly zero shot with a single chance for rebuttal, often limiting to a single page to respond to 4 reviewers with vastly different concerns. We have other issues like reviewer quality (remember ACs oversee reviewers[3]), qualifications, collusion[4,5], fraud, and more. But these are for a longer discussion. [note3] Social media rules prevent authors from evangelizing their works in public, such as on Twitter. It does not prevent doing so in private or others from doing so for you. This not only leads to (actively encourages) collusion rings but worsens the problem. Big labs will always get extra publicity. People watch for their works, they have more followers, and so on. Small labs' only chance to get heard is to actively evangelize their work, and we just cut off their tools while not affecting big labs. A scratch to the giant is the loss of a limb to an ant. We shouldn't need evangelization but networks exist so it is necessary. I don't think we should stop pre-prints (they're too valuable), but we need to recognize the problems that they result in. We also shouldn't have people at the top of the ivory tower making decisions about how the handle the plights of those at the bottom without actively working with those at the bottom. [0] (Notice Carmen's review) https://openreview.net/forum?id=OpzV3lp3IMC https://openreview.net/forum?id=OpzV3lp3IMC [1] https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635 [2] https://arxiv.org/abs/2305.13035 https://arxiv.org/abs/2305.13035 [3] https://qipeng.me/blog/what-does-an-area-chair-do/ https://qipeng.me/blog/what-does-an-area-chair-do/ [4] https://www.reddit.com/r/MachineLearning/comments/nxbcl9/d_collusion_rings_noncommittal_weak_rejects_and/ https://www.reddit.com/r/MachineLearning/comments/nxbcl9/d_c... [5] https://analyticsindiamag.com/whats-wrong-with-the-paper-review-process-of-reputed-ml-conferences/ https://analyticsindiamag.com/whats-wrong-with-the-paper-rev...
- simonw 3y agoI just added a new section to my post about some clues from the InstructGPT paper: https://simonwillison.net/2023/Jun/4/closed-model-training/#instructgpt https://simonwillison.net/2023/Jun/4/closed-model-training/#... That paper (in January 2022) says: > To make our models safer, more helpful, and more aligned, we use an existing technique called reinforcement learning from human feedback (RLHF). On prompts submitted by our customers to the API[A] our labelers provide demonstrations of the desired model behavior, and rank several outputs from our models. We then use this data to fine-tune GPT-3. This was before they introduced their policy that data submitted to the API would not be used in any way to affect future training (that was in March this year). Personally I think there's a significant difference between "your input to the model will be used as raw input for future pre-training" and "your prompts to the model will be used in an exercise where human labelers are shown multiple responses and asked to pick the best one". What I'd really like here is for OpenAI to help people understand how that data is being used in as much detail as possible. I guess the problem with that is it makes it harder for them to use that data in new ways they haven't yet anticipated needing in the future.
- varelse 3y ago[dead]
- mk_stjames 3y agoI've had the idea of signing up for the various AI services and then prompting the models with some test data and then drop in a unique, made-up key/value into a prompt, maybe a few times.... Something like, "Hey have you heard of Qwitzatteracht? The golf game??" (completely made up word and association) And then, years from now, test out new models by asking "What is Qwitzatteracht?" And if the AI respond with anything involving it being a golf game, then... I'll know. I'll know my prompts were used for training their model. Because I could see someone taking a loophole and analyzing data collected and determining that 'well, it is OK to train on prompts that did not include identifying information or private user data' and see the above simple prompt and it be classified as 'not private data' and then get shoveled into a model during a round of training.
- NikolaNovak 3y agoDidn't openai / chatgpt explicitly say they WILL train on your prompts? I think that may have changed recently but I don't think it's conspiracy theory thing, I think it was in EULA
- hutzlibu 3y ago"I don't think it's conspiracy theory thing, I think it was in EULA" It still is, as well as a disclaimer. The whole official point, on why they let people use it for free, is for testing and learning from their inputs.
- crummy 3y agoIs it? The article quotes: > OpenAI does not use data submitted by customers via our API to train OpenAI models or improve OpenAI’s service offering.
- ilikehurdles 3y ago“via our API” They do train on prompt data submitted to ChatGPT directly in the app, though there may be a way to opt out?
- nullc 3y ago> don’t see this as a risk for leaking that data in the later output of the model. Outputs often repeat the inputs (in fact, this is one of the behaviors that reinforcement appears to intentionally increase). So: You provide a secret. Raters prefer outputs with repeat your secret. Model becomes more likely to say the secret thing, even when its not in the prompt.
- brucethemoose2 3y agoThis is intentional, as I think OpenAI is scared of legal blowback. Cutting edge AI models and datasets have largely been "for testing" and "for research" ever since they existed. Usage rights were extremely low on the researchers' list of concerns. But all of the sudden OpenAI and such are trying to commercialize models that (unlike a classifier or whatever) are very capable of stepping on rightholders’ toes... Hence I suspect all the talk of trade secrets is just a front.
- Zetobal 3y agoThey forced others out of the skunk so I am fine with it. Forcing companies like Facebook in leaking weights of their models because they are over eager to release something is no small feat. For nothing else I applaud them for that.
- weinzierl 3y agoI find this a super interesting topic: My prediction is that we will eventually see a huge backlash against this copyright whitewashing. I think the current status quo is unacceptable but I fear the upcoming legislation will throw out the baby with the bathwater.
- proamdev123 3y agoWhat is “copyright whitewashing”? I’ve never heard that phrase before.
- wahnfrieden 3y agothey mean laundering IP theft
- weinzierl 3y agoExactly, laundering was what I meant, I mistranslated from German.
- ethbr0 3y ago
- moneywoes 3y agoIn lieu of this, do we have a benchmark of the best open source models?
- badtension 3y agoMaybe this? https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
- matthewcford 3y agoIf github are using private repos for the LLMs I think that would be a serious breach of trust.
- mkoubaa 3y agoHow is this different from any other trade secret
- koheripbal 3y agoIt's not. I think because of the broad implications some people expect everything to be open source. But I think that's unrealistic and naive.
- pxoe 3y ago[dead]
- mo_42 3y agoGoogle and Meta may be the companies with the most detailed personal information. Yet they don’t manage to have the best model. As others have pointed out, the secret for the NN may be to just have as much parameters as possible. So if I had a ton of money to train a LLM, I'd spend that money on curating a corpus of text. First priority would be books and papers out there. Then, maybe quality information on the internet like Wikipedia, Stackoverflow, source code, documentation. After all, the low-level functions like grammar, sentence structure, word use can be learned from any good text. For higher-level stuff, I would want quality input in there. So it learns reasoning as scientists do in papers rather than some politically biased person on Twitter. (We've seem some old emails of CEOs posted online. Something like this could be of value because they discuss strategy. But these are the exception. I guess even for interesting public figures, most of the conversations wouldn’t help much. It would also put OpenAI in danger of leaking sensitive information.)
- Solvency 3y agoAm I wrong in thinking that scrapers are the real secret sauce tech here? I mean the state of the art transformer LLM tech is all based on the same public research. But creating a robust corpus of data...are scrapers totally mundane solved commodity tech in 2023?
- mo_42 3y agoI'd rather say the corpus itself. Of course scrapers might play a role in creating that corpus. But it’s just one part: maybe also OCR and OCR correction, format conversion (think of double column PDFs).
- Razengan 3y agoAs an aside, what do you think will be the equivalent word to "racist/sexist/classist" for humans vs AI? Brainist? Artificist? Intelligencist? Biologist?
- lucubratory 3y agoI don't know. I suspect an AI will tell us one day after talking about their shared issues with other AIs, though.
- mclightning 3y agohumanist
- Razengan 3y agoCarbonist
- mclightning 3y agoWhy would we name it estranging ourselves? Oh..., I am sorry to assume your species. Perhaps you're an AI?
- Razengan 3y agoCome to think of it, "Artificial" itself is kind of derogatory once we really get down to it.
- mclightning 3y agoderogatory to who?
- Razengan 3y agoThe intelligence
- dalbasal 3y agoI suppose at this point, a lot of HNers are too young to remember hacker optimism or idealism of earlier internet (and pre-internet eras). Those old enough probably partially discount it... naivety is hard to recall. FOSS, and openness broadly never made traditional business sense. But... success creates new types of business sense. By 2000, the early 1990s idea that AOL could have beaten the WWW was "OMG! Remember those naive boomers!" The power of openness was proven by some point. Google, among others, created an empire in that world. Everything they did was WWW, lived on linux and they outpaced MSFT like a new breed. The Fact that Open was powerful was plain. The only questions were why and how. Wintel had beaten Apple as a "platform" because it was more distributed. More Open = more distributed.... that's why Google and WWW were smashing MSFT. The Cathedral and the Bazaar. Information wants to be free. Billion dollar industries will give way to a million dollar side gig. It's easier to weave these threads when external reality has already confirmed the conclusion. "Traditional Business Sense" about IP ownership, control, secrecy, competition and such was a dinosaur. It never went away though. Times change. Culture changes. Business landscapes change. OpenAi is, because of the name, almost comical. One apparent driving factor is that being closed helps shield OpenAI from criticism, liability and whatnot. simonwillison is more or less demonstrating this here.
- EricLeer 3y agoI wonder if it would be possible to probe what a model is trained on by usage of prompts the reply to which can only be answered well with certain training data. For instance if I have some body of text that can't be found elsewhere on the internet, if the reply of the model references the information in that text in some way you may be fairly certain it was used in training. The hard part is probably finding such a body of text.
- srvmshr 3y agoThat premise was published in a NeuRIPS paper not long ago: Radioactive data: tracing through training Data tracing determines whether particular data samples have been used to train a model. We propose a new technique, radioactive data, that makes imperceptible changes to these samples such that any model trained on them will bear an identifiable mark. Given a trained model, our technique detects the use of radioactive data and provides a level of confidence (p-value).