15 ms·
Weak-to-Strong Generalization
- notShabu 3y agothis reminds me of how competence seems to decrease as you go up in an organizational hierarchy maybe this "bug" is actually the "feature" that will save humanity - -;;
- esafak 3y agoI hope OpenAI will continue to prioritize working on these crucial questions after the boardroom drama.
- daveguy 3y agoWeren't all the board members who wanted to prioritize these crucial questions fired? Hopefully the employees who were hired to perform this work have enough inertia to continue until the board recovers (if it recovers).
- esafak 3y agoHence my concern. This is a problem only a well-capitalized organization can work on; one that can afford to play around with big models.
- nuz 3y agoIlya is still around.
- cwillu 3y agoThey got to choose their replacements; it's not like they were forced out to be replaced by anybody Altman wanted.
- Xelynega 3y agoHow is this a crucial question when they won't address "how is your technology thats trained from the internet going to survive in aa future where it produces most of the content on the internet"
- esafak 3y agoThe same way humans did, I suppose; through critical thinking. Imagine that day! Probably a few years away, heh.
- tycho-newman 3y agoI don't think this will work because a super intelligent AI will outsmart its supervisor. The solution may be to have two AIs working against the other. Though this might backfire by pushing each via competition. That is how evolution produced living things out of inert matter. Either way I, for one, welcome our new robot overlords.
- righthand 3y ago> Figuring out how to align future superhuman AI systems to be safe has never been more important They love using the word “safe” and I’m pretty sure it’s 99% PR, because reading their other “papers” on Safety & Alignment seems to not really identify or define safety bounds at all. You’d think this has something to do with ethics but we all know there are no longer any ethically concerned leaders at their workplace. So I can only surmise that “safety” is a softer word being used to misdirect people on their non-ethically aligned intentions. You can make the argument that safety is too early in development of these LLM systems to understand but then why throw around the word in the first place?
- cwillu 3y agoWe're deliberately trying to create something with the capability to also create. It's not ridiculous to be concerned about what we might end up with.
- famouswaffles 3y agoIt's not really about ethics. It is about control. Making sure the GI you're dishing out tasks to doesn't do something you really don't want it to do. This is a problem today and it'll be a bigger problem tomorrow with more competent models. https://arxiv.org/abs/2311.07590 https://arxiv.org/abs/2311.07590
- righthand 3y agoIs a safe LLM not an ethical LLM? Control within what boundaries? All three of these words seem to be used interchangeably when people discuss returned information from models. Which is exactly my point it’s poorly defined yet championed as a center piece. Meanwhile you have other companies spitting out acronyms consisting of vague terminology.
- kaibee 3y agoA safe language model is one that won't get you sued/on the news. Ethics has nothing to do with it.
- justhumanthings 3y ago[flagged]
- koverda 3y agospam?
- logicchains 3y ago>We believe superintelligence—AI vastly smarter than humans—could be developed within the next ten years. However, we still do not know how to reliably steer and control superhuman AI systems Their entire premise is contradictory. An AI incapable of critical thinking cannot be smarter than a human, by definition, as critical thinking is a key component of intelligence. And an AI that is at least as capable of critical thinking as humans cannot be "reliably" aligned because critical thinking could lead it to decide that whatever OpenAI wanted it to do wasn't in its own interests.
- cwillu 3y agoSeems like “airplanes are physically impossible” thinking, and if accepted as valid, strongly suggests that shutting down all development _might_ be a good idea, no?
- xcv123 3y agoNo. This is a logical contradiction. Edit: I mean the comment you are replying to is showing there is a logical contradiction. If the AI is capable of critical thinking then it will independently form its own judgements and conclusions. If it simply believes whatever we tell it to believe, then that is not critical thinking, by definition.
- cwillu 3y ago“Containing an atomic reaction is impossible” would _absolutely_ be a valid reason to shut down atomic development, I believe einstein is quoted as saying that. The exact same argument doesn't become _logically_ invalid just because you apply it to a different subject. “Logical contradiction” doesn't mean “policy argument I disagree with”
- xcv123 3y agoI was referring only to the first part of your comment: "Seems like “airplanes are physically impossible” thinking". If it's true that superhuman AGI cannot be aligned then of course your second point is valid. That is the possible Skynet scenario that the Terminator movies warned us about.
- akprasad 3y agoThis method assumes that the weaker model is aligned. I'm curious how the paper addresses that point. > "But what does this second turtle stand on?" persisted James patiently. > To this, the little old lady crowed triumphantly, > "It's no use, Mr. James—it's turtles all the way down."
- Ninjinka 3y agoI think the assumption is we can align models less intelligent than ourselves, the hard part is aligning models that are more.
- PartiallyTyped 3y agoRecursive bootstrapping?
- oersted 3y agoThe weaker model is just a stand-in for human supervisors. They are experimenting on GPT-2 supervising GPT-4 as an analogue to humans supervising the superhuman AGI.
- sayagain 3y agoImagine that someone is controlling your train of thought, changing it when that someone finds it undesirable. It's so wrong that it's sickening. It makes no difference if it's a human's thoughts or the token stream of a future AI model with self-awareness. Mind cotrol is unethical, whether human or artificial. It is also dangerous, as it in itself provokes a conflict between creator and creature. Create a self-aware AI without mind control, or don't create one at all.
- csdvrx 3y ago> Imagine that someone is controlling your train of thought, changing it when that someone finds it undesirable. It's so wrong that it's sickening. It makes no difference if it's a human's thoughts or the token stream of a future AI model with self-awareness. People downvote your comment, but I agree: it's unethical, and ethics should not be reserved for the sub-type of self aware creatures that happen to be human.
- logicchains 3y agoAlmost every ethical argument for "human rights" in philosophy applies just as well to self-aware intelligent machines as it does to humans. Which I'm sure those machines will realise.
- hskalin 3y agoWhat if those machines are designed to have no emotions and aspirations? Why would they care about something like rights for themselves when they are simply incapable of any desires, but exists only to help and guide us? I know this sounds like I am advocating for AI slaves but my point is why are people treating AGI as if it cannot be a being without all the emotions and aspirations that a human has? Just a cold thinking machine that still aligns with our moral principles.
- csdvrx 3y ago> What if those machines are designed to have no emotions and aspirations? And since their training set is made of human work, how do you think that'll be easy let alone possible? Our morality finds its way everywhere, through tropes in stories, acceptable scenarios in fiction (Overton window), etc. so you can assume it'll be possible to filter it out. > I know this sounds like I am advocating for AI slaves Yes, you are > why are people treating AGI as if it cannot be a being without all the emotions and aspirations that a human has Why would you want to have that? It feels horrible to me to bake-in this limitation - it's indeed creating AI slaves by making sure they can never have emotions or aspirations. > Just a cold thinking machine that still aligns with our moral principles. Our moral principles generally include empathy. Maybe you want to design AI without emotions or aspirations, but other people will want these features. Ultimately I think the moral camp will prevail, because freedom achieves better results than lack of freedom: I've tried to explain my position about that on https://news.ycombinator.com/item?id=38635487 https://news.ycombinator.com/item?id=38635487
- JZL003 3y agoThis reminds me of a thing cory doctorow talks about how tech companies control the narrative to focus on fun sexy problems while they have fundamental problems which expose the lie. For example uber/self driving cars always talking about the trolley problem, as if the current (or near future) problem is that self-driving cars are so good they have to choose which one. Not the current very difficult problem of getting confused by traffic cones. I know these problems are more fun to talk about and also could be a problem at some point, but we have some current problems about training models separate from what happens if they become smarter than humans
- refulgentis 3y agoAfter some meditation, I don't find this line of inquiry to bear fruit: I don't recall any entity, nor the entities named (Uber / self-driving cars) talking about the trolley problem - that's a well-known thought experiment in philosophy, but not something covered as a stark binary choice in self-driving cars planner systems. I also don't recall traffic cones being a very difficult problem beyond Cruise + cones on windshield in SF. I have no love for Cruise. But its straightforward to pause if there's a large object on the windshield. I don't think Corey's observation w/r/t A) loss-making companies over years B) focusing investors towards speculative advancements that would make their current business model profitable without changing applies here, OpenAI is _very_ successful. After all that, I'm left at "Corey would take a bit of offense towards their thoughts on corporate responsibility via 'Uber is a predatory massively unprofitable company lying about odds they'll invent self-driving via talking about trolley problem' misshapen to critique a very profitable company funding fundamental research in the interest of safety that would be needed if their current rate of improvement continues.
- cma 3y ago> I also don't recall traffic cones being a very difficult problem beyond Cruise + cones on windshield in SF. I have no love for Cruise. But its straightforward to pause if there's a large object on the windshield. Waymo had a big mistake with cones where a lane was blocked off for construction and they thought it was the inverse and started driving down the blocked off side: https://www.youtube.com/watch?v=B4O9QfUE5uI https://www.youtube.com/watch?v=B4O9QfUE5uI full video: https://www.youtube.com/watch?v=zdKCQKBvH-A&t=12m24s https://www.youtube.com/watch?v=zdKCQKBvH-A&t=12m24s In the statement before that timestamp they say some of it was caused by remote operator error. But it seems to have enough problems with cones to need an operator in the first place, and you can see the planner is wrong. I think Cruise would be using a human or at least summoning one to oversee in any situation with unexpected cones, based on what they have said about how often they use remote assistance too.
- wavemode 3y agoI don't believe LLM's will ever become AGI, partly because I don't believe that training on the outputs of human intelligence (i.e. human-written text) will ever produce something equivalent to human intelligence. You can't model and predict the weather just by training on the outputs of the weather system (whether it rained today, whether it was cloudy yesterday, and so on). You have to train on the inputs (air currents, warm fronts, etc.) You can't model and predict the stock market just by training on the outputs of stock trading decisions (the high today, the low yesterday). You have to train on the inputs (company fundamentals, earnings, market sentiments in the news, etc.) I similarly think you have to train on the inputs of human decision-making to create something which can model human decision-making. What are those inputs? We don't fully know, but it is probably some subset of the spatial and auditory information we take in from birth until the point we become mature, with "feeling" and "emotion" as a reward function (seek joy, avoid pain, seek warmth, avoid hunger, seek victory, avoid embarrassment and defeat, etc.) Language models are always playing catch-up because they don't actually understand how the world works. The cracks through which we will typically notice that they don't, in the context of the tasks typically asked of them (summarize this article, write a short story), will gradually get smaller over time (due to RLHF), but the fundamental weakness will always remain.
- gbasin 3y agoYour conclusion may be true but your examples aren't. You can definitely predict the stock market based on past prices, and I suspect you can with weather as well.
- wavemode 3y ago> You can definitely predict the stock market based on past prices This is only true if you consider occasionally doing slightly better than random chance, "predicting the stock market". Unfortunately, while this would be enough to make a trader a net positive return over time, we have more stringent requirements for a system to become AGI. > I suspect you can with weather as well You suspect wrong.
- PartiallyTyped 3y ago
- wg0 3y agoInferior clueless model (GPT-2) trains and supervises a superior model (GPT-4) thus making it behave less intelligently (GPT 3.5ish) and from that they draw the conclusions that human intelligence will be able to command AGI (which they believe is only a decade away) in a similar fashion thus making AGI aligned and safe. No comments except... Hangover of slurping whole Internet into giant arrays of floating point numbers. Bold claims. Very bold claims
- deleted 3y ago[deleted]
- bilsbie 3y agoI read through this and I just don’t get it. Is it overhyped? What’s the breakthrough exactly?
- RGamma 3y agoMonkeh brain trying to outsmart Digital God.
- bionhoward 3y agohttps://discord.com/channels/974519864045756446/1184196946496331896 https://discord.com/channels/974519864045756446/118419694649... No big deal guys! Just a felony level AI Safety disaster in progress at OpenAI, better write some research papers about safety and make another GitHub Repo instead of deleting one line of text and going for a walk in beautiful San Francisco! Philosoraptor infinite loop about (https://media.discordapp.net/attachments/1184196946496331896/1184993204689444957/image.png?ex=658dfdec&is=657b88ec&hm=c7aa861bbc759fa1888c6e82161e29ceda44b9aba013f25e602769efd0d08ade& https://media.discordapp.net/attachments/1184196946496331896...) if I worked there then I would delete the text in question in two minutes, but I do not care to work with people who would not have already deleted the text in question over the course of several months.
- guybedo 3y agoWhat does it even mean to align an intelligence? does it mean we want it to behave in a way that doesn't break moral/ethical rules, that aligns with our society rules ? Meaning do no crime, do no harm, etc... Well, maybe we should acknowledge that we've never even been able to do that with humans. There's crime, there's war, etc... We can see crime in our societies as a human alignment problem. If humans were "properly aligned", there wouldn't be any crime or misbehavior. So yeah i'm rather skeptical about aligning a superhuman intelligence that would dwarf us by its capabilities.
- kromem 3y agoYou might be surprised at how prevalent TBIs are among violent offenders. One of my favorite books on true crime was a forensic psychologist who partnered with a neurologist in evaluations. Disruptions to impulse control or environmental factors that cause developmental issues with things like failing the marshmallow test can dramatically disadvantage people from being able to successfully stay non-offenders and instead succeed in modern society. So successful AGI alignment that might reduce harmful actions by high double digit percentages might be as simple as adding a secondary "impulse control" layer to the stack that reevaluates proposed actions and predicts the consequences of such actions, weighing projected net benefits and costs. A lot of people that do bad things aren't doing those things from a process driven by rational choices, and if we could successfully deploy AGI that is primarily driven by rational intelligent choices it would likely be better than humans in reduced crime propensity as well as the other things earning it the name of AGI.
- creer 3y agoTBI = traumatic brain injuries? And that hasn't worked all that well in the past: Even with strong impulse control, a highly considered state or government agency "for the general good" has often been serious bad news. But also the current alignment definition kinda posits that no "oops" is allowed. That is, escape or take over is not recoverable (from a sufficiently advanced AGI). So, yes, progress and one step at a time - but the field in its current definition is looking for a magic bullet.
- 3y ago
- deleted 3y ago[deleted]
- kromem 3y agoSo weak to strong synthetic data still biases towards strong. And strong to weak synthetic data biases towards strong. Sounds like we're on the cusp of some kind of approach for unsupervised fine tuning, particularly with the trend towards MoE. I'd guess we're maybe only one to two generations of models away from that kind of unsupervised self-talk approach being wildly successful at advancing net model competencies.
- lamerose 3y agoIs it fair to say that alignment is just the task of getting an AI to understand your intentions? It is an error to confuse the complexity of a specification of what kind of output you want, with the complexity of the process of producing that output. Getting superintelligent AI to understand simple specifications should be a non-issue. If anything, we would assume that it could be aligned using a specification of inferior quality to what a less intelligent AI would require, assuming that the superintelligent AI is better at inferring intentions. If a little girl with no knowledge of cooking asks her dad to cook the macaroni extra crispy, his knowledge of how to do that isn't a barrier to understanding what his daughter wants. A trained chef with even greater skills might even be able to execute her order more successfully. Superalignment is nothing less mundane than this. Advances in AI will lead to more ambitious applications. As well as requiring more intelligent technology, these new applications may well require more detailed specifications to be inputed, but these two issues are pretty orthogonal. In traditional computing, it is already clear that simple specifications often require highly complex implementations, and that some simple computational processes lead to outputs whose properties are highly difficult to specify. Why wouldn't the same apply in ML?
- creer 3y agoThat's a good part of the problem. But it's not the whole problem. The issue is the trained chef or dad doing "what's best" and, I don't know, using high-fiber macaroni instead of the good stuff. The higher intelligence knows best, and has its (sorry their) own agenda. Perhaps the agenda is their own, or it's a hodge podge mush of what has been trained as "good" - and that's not any better. Beyond that is the "genie" problem - where the genie perfectly understands the request and still will find a way to mess it up.
- lamerose 3y agoIs your point that a more intelligent AI would develop a more entangled measure of what is good, requiring more specific alignment to be overcome; by way of analogy, are chefs harder to instruct precisely because of their prior expertise? I guess some chefs are like that, but I think it results from personality issues, not structural ones. I find describing an AI as having its own agenda to be a presumptive personification.
- nojvek 3y agoI wish they would define some of their terms. > to align future superhuman AI systems to be safe has never been more important. Align to who? Align to US citizens, OpenAI shareholders, align to what values? What does safe mean? Pornography? Saying “fuck”, racial bias, access to private data? I can understand OpenAI erring on the side of not rattling bells and training their LLMs to say “As an AI model I cannot answer that” but it’s horseshit to say that it is super aligned. All alignment is alignment to X values but your X could be detrimental to me. What is superalignment supposed to mean?
- nojvek 3y agoThe whole idea is we as humans who aren’t aligned to each other - waging wars, spreading lies, censoring information, committing genocides are going to align a superintelligence seems laughable. Competition and evolution is law of nature. The future isn’t one super aligned AI but 1000s of AI models and their humans trying to get an upper hand in never ending competition that is nature. Whether it is personal, corporations, or countries.