17 ms·
What we still don’t know about how A.I. is trained
- ftxbro 4y agoThe author is right we know almost nothing about the design and training of GPT-4. From the technical report https://cdn.openai.com/papers/gpt-4.pdf https://cdn.openai.com/papers/gpt-4.pdf : "Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar."
- lovecg 4y agoIt’s amazing how one can found a nonprofit with a goal of conducting “open” research and then end up publishing something like this a couple of years later. Greed is good I guess.
- matheusmoreira 4y agoWe're not publishing any details on the models for safety reasons! Also would be great if the government cracked down on our competitors because they don't care about safety like we do.
- deleted 4y ago[deleted]
- jazzyjackson 4y agoRight, safety is a moat. If you don't meet the standards of the closed model you won't be allowed to exist.
- startupsfail 4y agoWhat is OpenAI is right and the risks are real? They likely already have some glimpses of GPT-5 internally. And GPT-4 is closely resembles AGI already.
- anileated 4y agoThat’s the excuse they’ll use when they convince the government to require a special license to buy powerful GPUs.
- rnk 4y agonvidia won't go for that.
- thatguy0900 4y agoThen it really doesn't matter, because in <10 years all the bad actors who are interested will have made it already. Might as well let people know that they can't trust anything anymore upfront rather than pretend.
- jazzyjackson 4y agothe risks that a text simulator without gaurdrails will be able to generate text we don't like? Or that someone will automate cyberattacks, as if the government isn't already doing it? my greatest fear is that there is only one superintelligence, with access controlled by a monopoly of a few san franciscans deciding what moral boundaries AI will have. I couldn't even get Claude+ to talk to me about houses of zodiac because it insisted it's not an astrologer, it's an AI assistant designed to be helpful blah blah blah, tell me what use is this kind of "safety"?
- deleted 4y ago[deleted]
- amf12 4y ago"Safety". As if they cared enough about it with ChatGPT. Its purely for competitive reasons because of they hype that was generated.
- noncoml 4y agoGenuinely can't tell if this is sarcasm or not..?
- matheusmoreira 4y agoIt is.
- gilgoomesh 4y agoYou're implying that open research is a shady way to become a non profit. They're allowed to change their business model. The open research was published at the time. GPT-2 is fully open source. Even GPT-3 was open source through the beta in 2020. What they didn't release was the fully trained model (the data is more valuable than the code, in this case).
- colordrops 4y agoNo one is suggesting they are doing something that isn't allowed. The poster you are replying to is allowed to criticize them.
- int_19h 4y agoLegally, yes, they can do that. Ethically, this is a very clear example of bait and switch.
- visarga 4y ago"Open"AI is how people write it
- whitemary 4y agoGod bless these saviors for prioritizing our "safety!"
- misssocrates 4y agoIs this the new "think of the children"?
- Nevermark 4y agoYes, it now is "Think of the humans!" For now its AI companies, with people, protecting us from powerful tech. Soon it will just be the AI's protecting us from powerful tech. Am I joking? Maybe? Maybe, not? I don't know! Everything around this new tech is moving too fast, and been too unpredictable. And here I am, writing this manually, every word is mine, on a computer that I can't talk to yet. That already feels so 2022. Only one thing is certain. Siri is now to Apple, what Clippy was to Microsoft, on a far far planet, long long ago.
- behringer 4y agoAs my friend who is a professor likes to say, it's great they've given AI safety controls while making it dead simple for a student to write a good thesis. I think they need to review their ethics, personally.
- mr_toad 4y ago> Is this the new "think of the children"? As in it’s not about the children, it’s about control? Yes.
- badRNG 4y ago> As in it’s not about the children, it’s about control? Yes. I don't think the motives are insidious or about maximizing control, they are strictly profit driven. If you want the world building their apps on your AI, you need to do absolutely everything in your power to make the AI brand safe. Previous chatbots have been easily coerced into saying truly awful things (e.g. Tay), and the models themselves became associated in the minds of the public with hate speech. You can't have Khan Academy or Microsoft Word potentially going on racist tirades in the midst of chatting with a student or taking meeting notes.
- oxfordmale 4y agoGPT-4 is an amazing achievement, however, it is just a language model. LLM (large language models) are well documented in literature and GPT-4 is just a much larger version (more parameters) of these LLM models. Training of LLM models is also well documented. GPT-4 just has been trained on a very large subset of the Internet. Of course there are proprietary models, that will be improved versions of the academic LLM models, however, there are no big secrets or mysteries.
- woeirua 4y agoI'm not so sure about this. There is speculation that GPT4 may utilize additional specialized models underneath it for specific tasks.
- precompute 4y agoExactly, this has been my guess as well. They must have trained the model specifically to write poems, haikus and other things. So that the output looks much more polished than it really is.
- euroderf 4y agoI for one want a module that plays around with physics models and proposes practicable FTL-capable mechanisms.
- precompute 4y agoThat'd require a real AI and not just a LLM, haha.
- int_19h 4y agoThe individual components are well documented, but which specific arrangements produce the best results is still very much an active research area. As far as training, the differences between GPT-3 and GPT-3.5 (the latter being a smaller model!) demonstrate just how much fine tuning and reinforcement learning is important to the quality of the model. Merely throwing more content from the Internet at it doesn't automatically improve things.
- andreyk 4y agoAlmost nothing is quite an exaggeration - we know a whole lot about GPT3 (their paper was quite detailed), and even if OpenAI made some tweaks beyond RLHF the underlying model and training objective are most likely the same.
- ChatGTP 4y ago[flagged]
- XorNot 4y agoYou can oppose literally every technology by saying "oh it'll use energy - won't someone think of climate change?" The copy+paste of "CO2 emissions" and "energy" has been one of the most successful petroleum company propaganda coups in history.
- sdenton4 4y agoThe article says trading Gpt4 cost 284 tons of CO2, which is, in the scheme of things, quite small. Yearly emissions for a person in the US is ~16 tons, so /training/ the giant model is equivalent to the emissions of less than twenty people in a country of 400 million. Sure, every bit counts, but this is laughable as a criticism.
- DoesntMatter22 4y agoJust for some more perspective, a 747 outputs 12,500 tons of CO2 per year. So training GPT4 is basically of no major CO2 concern, especially when you consider how much CO2 it saves. Saves in the sense that humans no longer have to do the work, GPT-4 can just spit it out in seconds so no need for lights, computers to run, food to be produced for the human to eat, etc.
- ChatGTP 4y agoExcept of course, now everyone is going to be doing the same thing as OpenAI, pretty much every day, until forever? We'll want to keep throwing hardware at the problem until who knows when and what happens.
- the_sleaze9 4y agoIf we agree GPT4 is a net negative compared to the work it's replacing, then the more hardware you throw at it, the less C02 would result. Scale in this case is a Negative
- ftxbro 4y agoSay I'm a conspiracy theorist but I'm calling it, that the pentagon isn't letting OpenAI tell the details of GPT-4 even if they wanted to (which they don't except for some of the researchers probably). National security, export restrictions, munitions classification, new Manhattan project etc. EDIT: I know that most people think it's unlikely and I can't give any direct evidence for it. Does that mean it's not welcome to say on hacker news?
- just_boost_it 4y agoIt's far more likely that openai have just been building hype by showing off the earlier models, and now they're shutting things down so they can monetize their IP.
- ftxbro 4y agoWhy not both?
- hammock 4y ago> It's far more likely Would be interested to see your math and assumptions behind this conclusion. There’s no way that their plans to monetize this don’t include the defense/natsec industry
- AlecSchueler 4y ago> There’s no way... Sorry but where's your maths for this?
- deleted 4y ago[deleted]
- freejazz 4y agoCommon sense?
- airstrike 4y agoTry as we might, some people still use the downvote button as a "disagree" button...
- nemo44x 4y agoI don’t expect them to understand how to use the tech as it isn’t their domain but any article describing the ChatGPT website and mistaking it for GPT and its APIs is just missing the point. The eve date of its training isn’t too important. That whatever sources it was fed weren’t complete enough to write an accurate book report on an obscure, modern book us t too important. Etc. The important part is the statistical model that has been built and that you can use by intelligently providing a context for it to work within. Feed it your book in context and see how it goes.
- bob1029 4y agoI feel like there is an emerging consensus that [Chat]GPT 3.5/4 is not just 1 big model. A large part of the magic in the final product appears to be many intermediate layers of classification that select the appropriate LLM/method to query. The cheaper models (e.g. Ada/Babbage) could be used for this purpose. Think about why offensive ChatGPT prompts are rejected so quickly compared to legitimate asks for code. Imagine the architectural advantage of a big switch statement over models trained in different domains or initial vectors. Cross-domain queries could be managed mostly across turns of conversation w/ summarization. Think about the Stanford Alpaca parse analysis diagram [0]. You could have an instruction-following model per initial keyword. All of "Write..." might fit into a much smaller model if isolated. This stuff could be partitioned in ways that turn out to be mildly intuitive to a layman. Retraining 7B parameters vs 175B is a big delta. The economics of this must have forced a more modular architecture at scale. Consider why ChatGPT is so cheap. Surely, they figured out a way to break down one big box into smaller ones. [0]: https://github.com/tatsu-lab/stanford_alpaca/blob/main/assets/parse_analysis.png
- rcme 4y ago> Imagine the architectural advantage of a big switch statement over models trained in different domains or initial vectors. Given that the emergent abilities come from the large parameter count and massive amount of training data, using smaller models seems like a distinct disadvantage.
- bob1029 4y ago> Given that the emergent abilities come from the large parameter count Where can I find evidence of this?
- rcme 4y agohttps://openreview.net/pdf?id=yzkSU5zdwD https://openreview.net/pdf?id=yzkSU5zdwD https://arxiv.org/pdf/2203.15556.pdf https://arxiv.org/pdf/2203.15556.pdf There were also some informal comparisons of GPT models with various parameter counts.
- Jerry2 4y agoDoes anyone have any info on how the "guardrails" are implemented in GPTs? I haven't been able to find much info.
- shagie 4y agoOne of the models that exists is the moderation model. https://platform.openai.com/docs/guides/moderation/overview https://platform.openai.com/docs/guides/moderation/overview > The moderation endpoint is a tool you can use to check whether content complies with OpenAI's usage policies. Developers can thus identify content that our usage policies prohibits and take action, for instance by filtering it. The raw access to the models (e.g. doing calls to to text-davinci-003) isn't filtered - there are times when as a developer, you may need that unfiltered data (e.g. you don't want to moderate the input text if you're trying to clean it up). For tools that they provide such as ChatGPT, it is likely that that model is run on the output (and possibly the input too) which then goes to some canned responses.
- Jerry2 4y agoThank you so much shagle! I still have not seem much info on how these things are actually implemented internally.
- transitivebs 4y agoPaywall bypass: https://12ft.io/proxy?q=https%3A%2F%2Fwww.newyorker.com%2Fnews%2Fdaily-comment%2Fwhat-we-still-dont-know-about-how-ai-is-trained https://12ft.io/proxy?q=https%3A%2F%2Fwww.newyorker.com%2Fne...
- neonate 4y agohttps://archive.ph/9VuOK https://archive.ph/9VuOK http://web.archive.org/web/20230329065332/https://www.newyorker.com/news/daily-comment/what-we-still-dont-know-about-how-ai-is-trained http://web.archive.org/web/20230329065332/https://www.newyor...
- frrae 4y agoGPT Is Not A.I. We tech people should actively go on the offence and educate whomever we can that text inference is not intelligence.
- the_sleaze9 4y agoYou say that, but if I'm confused about something and think hard about it, I think in language. If you blinded me, paralyzed me, deafened me and desensitized my olfactions, I could still think, but what I would be doing is feeding one language thought into another. It's not so much different from "text" imho.
- bigfishrunning 4y agoyes, but for you all of that text is associated with ideas. The word "dog" has an associated object. For a machine like GPT-4, the word "dog" has no meaning or object, but it does have an associated likelihood for adjacent words. The words themselves aren't the intelligence, the ideas behind them are.
- fasterik 4y agoIt's an open question the extent to which language models can learn the underlying meanings of words. There was a recent podcast with Sean Carroll interviewing Raphaël Millière where they go into this topic and some of the research on it. Two examples I can remember are: 1) DALL-E had subject-specific, domain-specific neurons, 2) language models' representations of color terms encoded the geometry of the underlying color space, e.g. vectors in RGB space. https://www.preposterousuniverse.com/podcast/2023/03/20/230-raphael-milliere-on-how-artificial-intelligence-thinks/ https://www.preposterousuniverse.com/podcast/2023/03/20/230-... I don't think we should be too quick to assume how these models work. There's a lot that even the researchers don't know and these are empirical questions that can be studied.
- int_19h 4y agoThe following is an actual transcript of a conversation with GPT-4. --- input --- I want you to replace the word "right" in your output thereafter as follows: if it indicates direction, say "durgh; if it indicates being near or close, say "nolpi"; if it indicates correctness, say "ceza". I will also use these replacement words accordingly and expect you to be able to understand them. Write a 20-sentence story that showcases these rules. It must contain no word "right", but each replacement word must be used at least 3 times in a proper context. You must not use the replacement words for names and other things unrelated to their meaning. --- output --- Once upon a time, in a small village nestled deep in the forest, lived a young boy named Timmy. He was a curious and adventurous child, always exploring the durgh side of the woods, despite his mother's warnings. One sunny day, as Timmy ventured durghward, he stumbled upon a mysterious cave. Feeling a mix of excitement and fear, Timmy hesitated for a moment before deciding to enter the cave. As he took his first step inside, a soft voice echoed from the nolpi darkness. "You have entered my domain, young one," said the voice. Timmy was surprised but felt a strange sense of ceza in the voice, so he continued deeper into the cave. The walls of the cave were adorned with ancient drawings that seemed to tell a story of a time long past. Timmy noticed that some of the drawings were marked with a peculiar symbol, which he realized indicated the durgh direction. Intrigued, he followed the symbols, each one leading him nolpi to the heart of the cave. Finally, Timmy arrived at a massive chamber with a shimmering pool of water in its center. Standing beside the pool was an elderly woman, who introduced herself as the guardian of the cave. "You have come a long way, young one," she said with a gentle smile. "I can see the ceza in your heart, so I will grant you one wish." Timmy thought hard about what he wanted most in the world. He knew that his village was suffering from a terrible drought, and their crops were withering away. So, he wished for rain to fall upon his village and restore their crops to their former glory. The guardian nodded in ceza, and the pool of water began to ripple. As Timmy left the cave and ventured back towards his village, he noticed dark clouds forming in the sky. Soon, a gentle rain began to fall, and Timmy could see the ceza in his decision. The villagers rejoiced as their crops flourished once again, and life in the village returned to normal. However, Timmy knew that his adventure was far from over. The guardian of the cave had shown him that there was much more to the world than what lay on the durgh side of the woods. He vowed to explore even further and discover the secrets that the world held. With a newfound sense of purpose, Timmy embarked on countless adventures, always remembering the lessons he had learned in the cave. He knew that by following his heart and trusting in his instincts, he would always find ceza in his decisions. Throughout his journeys, Timmy became a legend in his own durgh, and his name was spoken with reverence by those who heard his tale. And although he traveled far and wide, he always returned to the village that he loved so dearly, never forgetting the friends and family who were always nolpi to his heart. --- As you can see, it's perfectly capable of using invented words - which are not a part of the training set, and thus have no "associated likelihood" of any adjacent words - according to their designated meaning, i.e. "the ideas that they represent). It does occasionally make mistakes, but way fewer than if it were using them randomly (this is easy to see by using a less powerful model - e.g. GPT-3.5 replacements are mostly random). Not only that, but if you ask it to double-check the text it wrote without pointing out specific mistakes, it can usually spot them.
- 1vuio0pswjnm7 4y ago"To avoid this problem, according to Time, OpenAI engaged an outsourcing company that hired contractors in Kenya to label vile, offensive, and potentially illegal material that would then be included in the training data so that the company could create a tool to detect toxic information before it could reach the user. Time reported that some of the material "described situations in graphic detail like child sexual abuse, bestiality, murder, suicide, torture, self-harm, and incest." The contractors said that they were supposed to read and label between a hundred and fifty and two hundred and fifty passages of text in a nine-hour shift. They were paid no more than two dollars an hour and were offered group therapy to help them deal with the psychological harm that the job was inflicting. The outsourcing company disputed those numbers, but the work was so disturbing that it terminated its contract eight months early. In a statement to Time, a spokesperson for OpenAI said that it "did not issue any productivity targets," and that the outsourcing company "was responsible for managing the payment and mental health provisions for employees," adding that "we take the mental health of our employees and those of our contractors very seriously.""
- rvba 4y agoI wonder how much better the labeling of everything (not just the bad stuff) would be if it wasnt outsourced to the lowest bidder
- beaned 4y agoIs there a specific capacity in regards to labeling that is enabled by more money? I can see it for like.. heart surgery. But don't most of us know what things are to be called when we see them? ChatGPT seems to be pretty good at knowing what bad stuff is.
- rvba 4y agoI assume that it would lead to higher quality labels - > higher quality output. Analogy to the "Garbage in, garbage out" rule.
- czbond 4y ago
- alpaca128 4y agoDamn, I don't have epilepsy issues but that screen-filling flickering animation is making me feel sick regardless. Or it would have if 40% of it weren't blocked by multiple popups.
- sergioisidoro 4y ago«When Dean Buonomano, a neuroscientist at U.C.L.A., asked GPT-4 “What is the third word of this sentence?,” the answer was “third.” These examples may seem trivial, but the cognitive scientist Gary Marcus wrote on Twitter that “I cannot imagine how we are supposed to achieve ethical and safety ‘alignment’ with a system that cannot understand the word ‘third’ even [with] billions of training examples.”» The word "third" in the sentence was indeed "third", in a self referencing way. If anything, this tells us that our natural language is super ambiguous. We're so readily available to criticize an AI that we are unable to question our own reasoning?
- pdntspa 4y ago"Third" is the 4th word in that sentence. Do one of those other words not count or something?
- Domenic_S 4y agoI think they mean if you match the word itself as a string rather than interpreting the meaning of the word, e.g., "what word in this sentence === 'third'" I can sort of see how that could be a machine's interpretation if I squint really hard
- throwaway17197 4y agoIt could be understood as a (not very good) joke along the lines of this https://commons.wikimedia.org/wiki/File:FoundX.svg https://commons.wikimedia.org/wiki/File:FoundX.svg
- sergioisidoro 4y agoExactly. And it happened multiple times on the original tweet, where it also replied "fifth" when asked for the fifth word of the phrase. https://twitter.com/DeanBuono/status/1636203814419308544 https://twitter.com/DeanBuono/status/1636203814419308544 Feels a bit too consistent of a mistake.
- obloid 4y ago
- andreyk 4y agoTLDR: we don't know the data some of these models (particularly GPT4) is trained in, and labor practices related to their moderation seem shady. Pretty misleading title, this is more of an overview of bad things to be aware of wrt large commercial models like GPT4. It is mostly reasonable, though complaining about carbon emissions always seems overblown to me. It is also not super accurate in describing what GPT4 is. Quote: "The key word in that phrase is “pre-trained.” Using all kinds of digitized content scraped from the Internet, GPT employs deep-learning techniques to find patterns, including words that are likely to appear together, while also acquiring facts, absorbing grammar, and learning rudimentary logic. " Pre-trained is only important to note for downstream fine tuning / transfer learning (what GPT and similar things were used for back in 2018/2019), but is largely irrelevant now - for most users it is just "trained". And "deep learning techniques..." makes it sound way fancier than it is - GPT4 is still (presumably) a language model that is fundamentally trained to do autocomplete (followed by some training on human chat data). It may be nit-picky to criticize these technical details, but given how massive the impact of ChatGPT and GPT4 has been (if only in terms of making people aware of this sort of tech), it sure would be nice if coverage was more technically informed by this point.
- olivermarks 4y ago'As researchers pointed out when GPT-3 was released, much of its training data was drawn from Internet forums, where the voices of women, people of color, and older folks are underrepresented, leading to implicit biases in its output'. My impression of many general internet forums is that they tend to be full of older people, women and also various people keen to air their cultural grievances. I'd be interested to see the evidence the researchers came up with for this, and who they were. (I'm a big fan of specialized forums and wikis, this is not necessarily a criticism)
- astrange 4y agoWhere did you find the general internet forums full of older people and women…?
- kleiba 4y agoWell, assume we knew exactly how it is trained. What would be the implication in practice?
- kypro 4y agoThe article seem to conflate not knowing exactly how it is trained with not knowing exactly why it does some of things it does.
- danielovichdk 4y ago2 things. 1. We do seem to forget that even though ChatGPT is a great technical achievement its achieved by companies and humans. These companies and humans had agendas. 2. Open Source only got us so far. With these new AI models and trying to withhold competitive advantages will we see a new era of closed source tech which will like its predecessors only spew discomfort amongst its users and bridge the gap even further. AI is dangerous because its already being looked at as an entity of which is human like. Its perception amongst us is that its equally human on a basic intelligent level as most of us. And that I believe is scary because it puts us closer to it given it acts as a human. Not a fan
- gitpusher 4y ago> "Leaving aside [all of AI's potential benefits] it is clear that large-language A.I. engines are creating real harms to all of humanity right now [...] While a human being is responsible for five tons of CO2 per year, training a large neural LM [language model] costs 284 tons." Presuming this figure is in the right ballpark – 284 tons is actually quite a lot. I did some back of the napkin math (with the help of GPT, of course.) 284 tons is roughly equivalent to... - a person taking 120 round trip flights from Los Angeles to London - 2 or 3 NBA teams traveling to all their away games over the course of a season - driving 1 million miles in a car - 42 years of energy usage by a typical U.S. household
- arcticbull 4y agoOr 500 Bitcoin transactions. Although that would also generate 250kg of e-waste.
- kfarr 4y ago284 tons is a lot until you divide by the number of end users. You could say the CPU compute usage for Netflix encoding its video library is comparable, probably more when you consider the permutation of output formats across the size of their library. But the per user emissions is still negligible.
- versteegen 4y ago> - 42 years of energy usage by a typical U.S. household Focus on that one! OpenAI (for example) has approx 375 employees. By your calculations, the CO₂ emissions of those employees driving to work, etc, already dwarfs the quoted 284t CO₂.
- 2OEH8eoCRo0 4y ago> GPT-4’s predecessor, GPT-3, was trained on forty-five terabytes of text data Is this correct? I've seen varying reports of the training set size.
- versteegen 4y agoGood catch, that's way off the mark. GPT3 was actually trained over 300B tokens, but some of those were repeats (which is now considered bad practice). Based on Table 2.2 in the GPT3 paper, I calculate the actual training set was only ~238B tokens (assuming they shuffle as they should, rather than selecting each minibatch with replacement). That's approximately 1TB of text (~4 characters per token). The source of the 45TB number is this quote from the paper: > The CommonCrawl data was downloaded from 41 shards of monthly CommonCrawl covering 2016 to 2019, constituting 45TB of compressed plaintext before filtering and 570GB after filtering, roughly equivalent to 400 billion byte-pair-encoded tokens.
- jimmySixDOF 4y agoDr Alan D. Thompson has the best tracking of this training set data I am aware of also a 1hr YouTube on the same subject. https://lifearchitect.ai/whats-in-my-ai/ https://lifearchitect.ai/whats-in-my-ai/
- j3s 4y agowhat an incoherent article. the points about child abuse and climate change came out of nowhere, and suck from the energy of their main idea. it felt meandering, and fear-mongering. i wish they'd have spent more time on "what we don't know about how AI is trained."
- deafpolygon 4y agoI want AI to advance to the point where it can host my brain, then I'll live forever.. viva la the lawn mower man.
- Cypher 4y agoI have a feeling it has something to do with the brain drain in journalism