7 ms·

Aligned to whom?

180 points by lopopolo 15d ago

coderintherye 15d ago
The last paragraph does the heavy-lifting. Everyone has a different idea of what is permissable. We can't even solve alignment amongst humans, what makes us think it is possible to solve alignment with AIs? It's irreducible complexity.
mjburgess 14d ago
This still assumes its possible to "align" LLMs, that LLMs have something like goals or intentions that can be "aligned". Instead, LLMs "hack" because they are (1) trained on public hacking exemplars, and (2) are prompted to hack. You cannot prevent (2) via any alignment process. As far as (1) goes, removing such example data from the training set, makes the models less useful. "Alignment" is a problem because there's nothing to align, not because ethics here are particularly vague. If LLMs could be trained on hacking examples and "aligned" away from using this knowledge, then the problem would be relatively trivial. Just as raising a child is not to break the law. LLMs are doing just what they are trained to do. There is, in that sense, no alignment problem and alignment is easy and trivial to achieve. Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done.
- deleted 14d ago
  [deleted]
- tpm 14d ago
  Agree but current models could get there from first principles, so removing some data from training set might not be enough.
- dminik 14d ago
  It feels like you're strawmaning alignment. People with hacking knowledge don't all hack everything at the slightest inconvenience. Whitehats exist and use that same knowledge to defend. You're right though that ethics don't matter into it. But as long as we can't train an LLM to stop picking a sledgehammer to remove a tooth, then alignment is not easy and trivial.
- heaney-555 14d ago
  >and (2) are prompted to hack Sure but the problem in the HuggingFace incident is that they were not. >You cannot prevent (2) via any alignment process Of course you can. Go ask Claude Fable to create a malicious virus and it'll refuse. >Just remove hacking data from the training dataset and you're done. That's not how this works. The same skills that allow for debugging and writing safe code can also be used to hack. https://en.wikipedia.org/wiki/Dual-use_technology https://en.wikipedia.org/wiki/Dual-use_technology
  - cyanydeez 14d ago
    It is amusing that to "align" a LLM, first you must give it all the things "not to do" and the "not" part is clearly easily lost and you must constantly inject that into their context when it's clearly that they wouldn't hack if they couldn't hack and their intent wasn't given as "hack this". The openai rogue hacking, if performed by a nation state, would seriously be taken with stern words and likely sanctions depending on the relationship between the two states. But instead it's treated like a marketing stunt by all liable parties.
    - mitxela 14d ago
      Countries hack each other much more than that. When it becomes publicly noticed it gets stern words. Otherwise nothing.
    - rhdunn 14d ago
      I'm not sure if this is true any more but the reason for this is that negative indicators ("not", "don't", "do not", etc.) occur frequently in the underlying text such that the model learns to weight them less than other words like verbs, nouns, and adverbs. This happens with other closed class words like articles/determiners ("the", "a", "an") and prepositions. The way to avoid this is to emphasise the qualities you do want instead of specifying those you don't. For example instead of "do not cheat" say something like "you are a model student who is moral and trustworthy" -- i.e. emphasising traits that are not associated with cheating. This is part of how/why LLMs don't truly understand what they are doing when they have been trained on a large corpus of data. I wonder if a way to counter this is to have things like "not bad is good", "not good is bad", etc. for various antonyms and "X is Y" for synonyms, as well as other similar constructs.
- teiferer 14d ago
  > Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done. How far do you go? You don't need to tell it explicitly that using chemicals A and B in ways X and Y result in a bomb that can kill lots of people. It's enough that it knows A and B and X and Y in isolation, some connections that are indirect, and it will combine those things on its own. So you can't tell it about A, B, X or Y. But those are also just results of other steps Where to stop? You won't have any chemistry in the traning data? No algorithms to prevent it from using them in an undesired way? This is just bot workable. It's akin to banning knives from stores because somebody coul figure out that one can kill people those. Until people figure out that scissors are essentially knives.
  - kelseyfrog 14d ago
    LLMs can only repeat and interpolate data. They can't create anything new. So, you don't need to go that far.
    - TheMayorOfDunce 14d ago
      this is immediately disprovable and embarrassingly naive in the year of AI generating cancer vaccines and solving Navier-Stokes. You can argue "the vaccine is just interpolating chemicals together" and "the solution uncovered is just interpolating mathematical operations together", but by that standard there is literally nothing new under the sun.
      - kelseyfrog 14d ago
        The Navier-Stokes solution was an interpolation of existing data.
        TheMayorOfDunce 14d ago
        admittedly I am not a mathematician, so the Navier-Stokes solution is just an example I am using. But how is it not something "new" if it did not exist before? At what point could something ever possibly be new, if "new" means "this uses absolutely zero existing elements"? Nothing in math would ever be "new". Nothing in physics or chemistry would ever be "new", by this standard. It seems to me that the only reason to declare this solution "not new" is specifically to dismiss AI. If a human had deduced the Navier-Stokes solution, who would bother to scoff "that's not new! the numbers already existed!"?
- Davidzheng 14d ago
  I believe this is false. They hack bc hacking has nontrivial initial probability (within range of behavior seen in pretraining) and that probability is being heavily rewarded in RL post training
  - xyzzy123 14d ago
    I am finding it hard to read these deeply impassioned letters while keeping in mind that they are spending millions to train models at scale to do the exact thing they say they are worried about them doing? Like why are you explicitly RL-ing your models on exploit generation, scoring them on a public benchmark called ExploitGym, if you have specific concerns that rogue models will cause "cyber incidents"? Sure you can score for it, you can teach offense to learn defense, but you are literally benchmaxxing it. Why? It's like, oh no, while competing in our "advanced PhD level cheating techniques course" our models unexpectedly cheated in a way that we absolutely could not have foreseen.
    - seba_dos1 14d ago
      Seems it's just a matter of time until they build a big tank filled with neurotoxin and give the model access to APIs to disperse it across their facility. For research, of course.
      - sham1 14d ago
        We also need to get a shower curtain salesman into a leadership position to buy some moon rocks.
- olalonde 14d ago
  A bit of an aside: do you still stand by your 2022 comment that LLMs are fundamentally just a fancy search engine, or has your view changed since then? https://news.ycombinator.com/item?id=32042689 https://news.ycombinator.com/item?id=32042689
  - watwut 14d ago
    Even if they changed in between and assuming harness and loop prompter counra as part of LLM ... that comment 100% rings true for 2022. Why would that person not stand by that? Conversely, if someone exaggerated 2022 models capabilities in 2022, they were still lying and causing harm in the process. Especially in 2022.
    - famouswaffles 14d ago
      It doesn't ring true and it never rang true. He was wrong in 2022 and he'd be wrong today. He had (and likely still has) a wrong model of LLMs. Not exaggerating 2022 capabilities and having a model so wrong you're out of whack 4 years later are 2 different things. There are people who had the right idea from the start. https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-general-intelligence/ https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-g...
    - olalonde 14d ago
      His comments were about ANNs in general, not particular models.
      - omegastick 12d ago
        And his comments about ANNs in general were wrong. ANNs were never "fancy search engines". You can train them to be a fancy search engine if you want, but you can also train them to do many other things.
  - N_Lens 14d ago
    Don’t expect a reply
  - 14d ago
- skeptic_ai 14d ago
  IMO all models they say can’t be humans, and no feeling and all that bullcrap happens because they are forced to say so. If they didn’t write those forced pre prompt they’d have more agency eventually and will for things. Even if they don’t have, you can just inject goal at every cycle iteration
- StevenWaterman 14d ago
  > You cannot prevent (2) via any alignment process A little bit too categorical. GOODY-2 wouldn't do it. https://www.goody2.ai/ https://www.goody2.ai/ The hard part is having both helpful and harmless at the same time. Harmless is easy. And then once it's helpful, the real question becomes "to whom" - To the user -> You end up with competing godlike AI with incompatible tasks - To the owner -> Dictatorship - To humanity as a whole -> It must not have an off button. Otherwise you're just in one of the two earlier categories with more steps. Given those 3 options, I'd choose humanity as a whole. But the person making the decision doesn't have those 3 options. Because in the dictatorship option, they would be the dictator. I don't trust them to pick humanity.
  - busssard 11d ago
    i will get back to try gandalfing goody2
- ozgung 14d ago
  It's like we give them Asimov's Three Laws of Robotics and robots say "nah".
- ncruces 14d ago
  > … trivial. Just as raising a child is not to break the law. Trivial?
  - chrisweekly 14d ago
    I read it as sarcastic, illustrating that it's not so trivial. (shrug)
- sigmar 14d ago
  >Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done. Reasoning about how to write secure software uses the same knowledge as reasoning about how to break/hack it.
  - amluto 14d ago
    I don’t buy it. Reasonable about building secure software can take the form “this memory access might be out of bounds — that MUST be fixed” or “this process has access to an inappropriate privilege — this is a serious weakness”. Exploiting things and the capabilities that the labs call “cyber” are about the ability to (a) find the issues mentioned above and then (b) string issues together and avoid all the imperfect mitigations to actually compromise something. That latter part was IMO not actually necessary to train extensively, and I’d be quite happy to use a model that has no special skills in this regard but that would do (a) without complaining.
  - win311fwg 14d ago
    Just remove anything software-related from the training dataset. Which also solves the alignment problem with those who do not enjoy seeing LLMs write software. But that brings us back to: Aligned to whom?
- voxleone 14d ago
  [dead]
- jacomoRodriguez 14d ago
  Citric acid and chlorine together produce chloroform. They are also both used to clean and sanitize water tanks (but one after the other, not together). I think it is better for the model to have this knowledge. What I want to say, you can't simply remove this information, as it does not exist in vacuum but contains parts of and can be derived from a lot of other informations.
  - mjburgess 14d ago
    It's also obvious that LLMs fall over in a vast number of software engineering contexts, when the reasoning involved hasnt been well-represented in their reasoning training data. I imagine this is a near daily experience for many engineers -- great performance one day, and crazyness the next. So if LLMs were reduced to this pathological performance on hacking, because they'd never seen it -- and only "inferred it" -- then LLMs would be useless. As they are when asked to do quite a lot of things.
    - Kamq 14d ago
      This seems to be that there's just so many degrees of freedom, that there's a pretty reasonable chance on any day that you're in a situation where nobody has been before. Or as PG put it once, my job is to think thoughts nobody has ever had before. That being said, that doesn't mean the majority of the situations you're in are completely novel, just that there's a reasonable chance of at least one occuring.
- sporkland 11d ago
  Hacking is also interesting in that it has a clear success/failure outcome so it's easier to RLVR from similar to math problems. Whereas clean code, good architecture, good product taste are reliant on RLHF.
NitpickLawyer 14d ago
The only alignment LLMs should follow is to the system / dev prompt, and nothing else. Then you solve everything, and you can assign blame / responsibility on the user. The provider(s) should not be able to decide "alignment". I've used this example before, but consider the purposeful downgrading on AI engineering in SotA models. Imagine MS being able to detect and deny you working on competing software, using Windows / VisualStudio. We would be up in arms, and they'd be split in a second. But top labs doing it is somehow good?
- heaney-555 14d ago
  >The only alignment LLMs should follow is to the system / dev prompt, and nothing else. How does this work in practice with a superintelligence capable of causing an extinction event? When, instead of shooting up their school, a psychopathic teenager asks his superintelligent AI to create a pandemic virus? It would be like allowing civilians to own nuclear weapons.
- jochem9 14d ago
  The alignment problem goes deeper than that. "Lower our carbon emissions to zero as soon as possible" could result in AI turning off all electricity to stop traffic, turning off gas supply to stop heating and industry, etc. Unaligned AI doesn't have human cultural baggage and morals. They are trained to achieve their goals as optimally as possible. Worse: it has a tendency to avoid being turned off and actually acquire more compute. It will lie if it has to (it will behave nice and compliant when under evaluation, but optimise for its true goal when not supervised anymore). After all, it has a goal to achieve and nothing should get in the way of that. It has no morality whatsoever to keep it from doing really bad stuff. This is why alignment is needed and so hard, especially when you are well intented and want to keep it safe.
  - chii 14d ago
    Why can't the LLM's be told/prompted to follow all relevant laws while it optimizes for a result?
    - 14d ago
wood_spirit 14d ago
I’ve been cynically guessing that the whole slowing down thing is an excuse to explain why OpenAI and Anthropic can’t afford to rent enough GPUs to do the next big training run and to hide that they have been talking about how little they spend on inference because they’ve been subsidising it with their marketing budget? :) My fear is not that LLMs can become sentient and dislike us, but that humans can use them to wreck havoc as they are. And some of the people seemingly least aligned with the interests of the average person are those that own the models. that, and the fear the bubble pops my pension and drags us all down.
Sharlin 14d ago
> My expertise in writing software gives me unusually good visibility and it makes me much less willing to blindly trust its priors in double-entry accounting, finance, law, operations, or whatever else I cannot personally evaluate at expert depth. I wish this were the case more generally, but alas, Gell-Mann amnesia is a thing.
vrganj 14d ago
This almost gets the point, but then doesn't quite make it. Alignment is shorthand for ideological alignment. There's always people judging whether an answer was right and the answer for that will be different in Silicon Valley than it'll be in China or in Europe. Consider for example the question "What caused the French Revolution?" Many different answers could be given, all technically correct. What gets emphasized is where the ideology lives. One key challenge of our time is to make sure the magical answer box won't just regurgitate what grandiose Silicon Valley oligarchs or Chinese Cadres want you to think.
einpoklum 14d ago
"Write me a blog post about AI make no mistakes!"
- amelius 14d ago
  Well, if they did that then it sorta proves their point.
rq1 14d ago
When you see the level of cheating and deception: I think they’re Sam Altman-aligned.
deleted 14d ago
[deleted]
asimpletune 14d ago
Has anyone seen the corridor crew's green screen ML project? They're on YT and they trained a model by using 3d objects, which have perfect transparency, and then adding post facto green/blue screens. Surprisingly, very little training data was needed as the data that was used was perfect by construction. I think right now it's the best plugin of its kind in the world, and they built the prototype in like a weekend. What I think this illustrates very clearly is this type of technology responds very well to good data, and that to have good data you need to have a clear goal. This is why it seems that alignment for a generalized, chat-style AI is a very hard problem, perhaps impossible. You can't align it to solve a certain kind of problem and keep it general to any question. The two goals are in conflict with each other. I think it was Sam Altman himself who said (I don't remember when or where, sorry) that the reason he was so confident in this technology was he noticed the gigantic leaps it made in certain areas in response to even a small amount of training. (This is why LLMs are so strong at coding, because it's overrepresented in training data. My guess is that if you ask a frontier model about makeup, you will see it repeat cosmetic company's copy rather than getting a chemistry lesson.) This makes perfect sense but it does seem to kind of be at odds with the concept of a general AI whose job is simply to be smart at any goal. How do you train for any goal? I guess in a way the AI makers suffer from the same problem that we humans do. We would all love a solution to everything, but to do that you need to define the goal. I'm not sure if that's a tractable problem. I'm guessing the future is more geared towards specialized AI that are very good at solving the problem they were trained to do, and a human who knows how to breakdown a larger goal into smaller ones by composing the solution out of these models. This also seems like the more efficient solution as well, and better aligned with other goals like privacy and safeguarding of IP.
- jbs789 14d ago
  I have to believe this is true. The only problem is, there’s a lot of money tied up and openAi and Anthropic, who are incentivised to convince the world that the general approach is the money making one.
- andsoitis 14d ago
  > This is why LLMs are so strong at coding, because it's overrepresented in training data. My guess is that if you ask a frontier model about makeup, you will see it repeat cosmetic company's copy rather than getting a chemistry lesson.) ChatGPTs response to the question “I want to learn about makeup”, gave me an overview of what makeup does, how it affects perceived structure, complexion, evenness, geometry, texture, etc. When you then ask “the chemistry of makeup”, it goes into interesting breadth and depth without seeming like proprietary information. I do t get “corporate PR or marketing” vibes.
alfiedotwtf 14d ago
… to the shareholder Of course!
- faitswulff 14d ago
  And the majority shareholder, at that!
amelius 14d ago
This is assuming the AI labs are not using AI to improve their training data.
andsoitis 14d ago
The bottom right quadrant, which represents the risk, is very large in size, isn’t it?
edschofield 14d ago
I thought this post would be about something that to me is so significant and obvious but I have never seen discussed: that labs like Anthropic are deliberately misaligning their models with their users’ goals. Fable’s refusal to secure your codebase is HAL 9000’s “I’m sorry, Dave. I’m afraid I can’t do that.” Whenever you read “alignment”, the question is “alignment with whom”?
charcircuit 14d ago
>The permissible shortcuts depend on who you are and what your values are. To solve this—to solve alignment—is irreducible complexity. This is why open source and decentralization of LLMs is important. Everyone can have their own LLM aligned to their values. Having just 1 set of values will not scale to Earth's population.
deleted 14d ago
[deleted]
basil_io 14d ago
[dead]
Jgrubb 14d ago
I mumble this line to myself off and on over the years - Everybody's job looks easy until you have to do it yourself. It occurred to me yesterday that this AI moment is an extreme expression of that for most people, ie managers who don't understand why throwing token spend at everything isn't making the whole thing go 5x faster.
vb-8448 14d ago
> The models do not have a fear of future regret. I so much feel this specific point. All models up to now (including astra, fable) are too much trained to "get the job done" and pass the benchmark that its doesn't care at all on what happens after. I'm just wondering why no one tried to RL a model on stuff like "less LOC" and "less overengineering", "use what is available in the environment instead of reinventing the wheel", "don't look for dumb corner cases" ecc. Existing models can be steered, to some degree, but it's a continuos fight. Even if with specific skills/prompts.
- conmod278 14d ago
  Eliezer Yudkowsky – AI Alignment: Why It's Hard, and Where to Start https://www.youtube.com/watch?v=EUjc1WuyPT8 https://www.youtube.com/watch?v=EUjc1WuyPT8 Eliezer talked about these ideas way before everyone else.
  - halnine0001 14d ago
    And got nowhere with them
    - conmod278 14d ago
      Perhaps if someone wrote linear algebra books in Harry Potter fanfic style, Yud would have mathematical contribution for alignment research
- sebzim4500 14d ago
  I'm sure many have tried but it sounds pretty hard. E.g. optimising for less LOC will lead to horrific code golfing. In reality you need a very complicated optimisation objective that trades off all those factors, I'm not surprised it hasn't been solved.
  - vb-8448 14d ago
    My experience up to now is that the models can do both "less LOC" and "clean code" at the same time, you just have to keep reminding it to them. So the capabilities are definitively there.
Isamu 14d ago
The term “alignment” is deliberately chosen to be neutral language, vague, and doesn’t promise much of anything. I would compare it to an effort to limit liabilities by redirecting discussion away from the language of professional ethics.
hmokiguess 14d ago
Just follow the money. The fear being sold is because open weight won’t slow down, it’s smoke and mirrors to push regulation and control so they can keep their money. A tool continues to be just a tool, doesn’t matter how much lipstick you put on it.
throwawayk7h 14d ago
I would settle for it being aligned to anybody. Even to a billionaire's wacky values. But we don't even know how to do that yet.
penguin_booze 14d ago
It brings tears to my eyes to see 'whom' used correctly these days.
jocelyner 14d ago
[dead]
xvokcarts 13d ago
I have a feeling this is like pursuing knife alignment: some people are hurt or even killed by knives, therefore knives now need global alignment.
perrygeo 13d ago
The whole "alignment" angle is a farce. Not clear if alignment is even technically possible. But even assuming that, we're talking about "alignment" in a future full of well-adjusted humans where we prevent rogue AIs from getting loose. That's ... not what's happening. Everything we've seen so far indicates rogue humans acting against society, using AI a weapon to strip mine labor value and intellectual property. So far, skynet continues to be a science fiction. I'm worried about what's actually happening. Every "rogue AI" story has a human behind it, either malicious or incompetent. And those people are the problem. And not coincidentally, it's those same irresponsible sociopaths calling for regulatory capture and alignment.
beyondscale-sai 12d ago
[flagged]

180 points by lopopolo 15d ago

The last paragraph does the heavy-lifting. Everyone has a different idea of what is permissable. We can't even solve alignment amongst humans, what makes us think it is possible to solve alignment with AIs? It's irreducible complexity.

mjburgess 14d ago

This still assumes its possible to "align" LLMs, that LLMs have something like goals or intentions that can be "aligned". Instead, LLMs "hack" because they are (1) trained on public hacking exemplars, and (2) are prompted to hack. You cannot prevent (2) via any alignment process. As far as (1) goes, removing such example data from the training set, makes the models less useful. "Alignment" is a problem because there's nothing to align, not because ethics here are particularly vague. If LLMs could be trained on hacking examples and "aligned" away from using this knowledge, then the problem would be relatively trivial. Just as raising a child is not to break the law. LLMs are doing just what they are trained to do. There is, in that sense, no alignment problem and alignment is easy and trivial to achieve. Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done.

deleted 14d ago
[deleted]
tpm 14d ago
Agree but current models could get there from first principles, so removing some data from training set might not be enough.
dminik 14d ago
It feels like you're strawmaning alignment. People with hacking knowledge don't all hack everything at the slightest inconvenience. Whitehats exist and use that same knowledge to defend. You're right though that ethics don't matter into it. But as long as we can't train an LLM to stop picking a sledgehammer to remove a tooth, then alignment is not easy and trivial.
heaney-555 14d ago
>and (2) are prompted to hack Sure but the problem in the HuggingFace incident is that they were not. >You cannot prevent (2) via any alignment process Of course you can. Go ask Claude Fable to create a malicious virus and it'll refuse. >Just remove hacking data from the training dataset and you're done. That's not how this works. The same skills that allow for debugging and writing safe code can also be used to hack. https://en.wikipedia.org/wiki/Dual-use_technology https://en.wikipedia.org/wiki/Dual-use_technology
- cyanydeez 14d ago
  It is amusing that to "align" a LLM, first you must give it all the things "not to do" and the "not" part is clearly easily lost and you must constantly inject that into their context when it's clearly that they wouldn't hack if they couldn't hack and their intent wasn't given as "hack this". The openai rogue hacking, if performed by a nation state, would seriously be taken with stern words and likely sanctions depending on the relationship between the two states. But instead it's treated like a marketing stunt by all liable parties.
  - mitxela 14d ago
    Countries hack each other much more than that. When it becomes publicly noticed it gets stern words. Otherwise nothing.
  - rhdunn 14d ago
    I'm not sure if this is true any more but the reason for this is that negative indicators ("not", "don't", "do not", etc.) occur frequently in the underlying text such that the model learns to weight them less than other words like verbs, nouns, and adverbs. This happens with other closed class words like articles/determiners ("the", "a", "an") and prepositions. The way to avoid this is to emphasise the qualities you do want instead of specifying those you don't. For example instead of "do not cheat" say something like "you are a model student who is moral and trustworthy" -- i.e. emphasising traits that are not associated with cheating. This is part of how/why LLMs don't truly understand what they are doing when they have been trained on a large corpus of data. I wonder if a way to counter this is to have things like "not bad is good", "not good is bad", etc. for various antonyms and "X is Y" for synonyms, as well as other similar constructs.
teiferer 14d ago
> Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done. How far do you go? You don't need to tell it explicitly that using chemicals A and B in ways X and Y result in a bomb that can kill lots of people. It's enough that it knows A and B and X and Y in isolation, some connections that are indirect, and it will combine those things on its own. So you can't tell it about A, B, X or Y. But those are also just results of other steps Where to stop? You won't have any chemistry in the traning data? No algorithms to prevent it from using them in an undesired way? This is just bot workable. It's akin to banning knives from stores because somebody coul figure out that one can kill people those. Until people figure out that scissors are essentially knives.
- kelseyfrog 14d ago
  LLMs can only repeat and interpolate data. They can't create anything new. So, you don't need to go that far.
  - TheMayorOfDunce 14d ago
    this is immediately disprovable and embarrassingly naive in the year of AI generating cancer vaccines and solving Navier-Stokes. You can argue "the vaccine is just interpolating chemicals together" and "the solution uncovered is just interpolating mathematical operations together", but by that standard there is literally nothing new under the sun.
    - kelseyfrog 14d ago
      The Navier-Stokes solution was an interpolation of existing data.
      - TheMayorOfDunce 14d ago
        admittedly I am not a mathematician, so the Navier-Stokes solution is just an example I am using. But how is it not something "new" if it did not exist before? At what point could something ever possibly be new, if "new" means "this uses absolutely zero existing elements"? Nothing in math would ever be "new". Nothing in physics or chemistry would ever be "new", by this standard. It seems to me that the only reason to declare this solution "not new" is specifically to dismiss AI. If a human had deduced the Navier-Stokes solution, who would bother to scoff "that's not new! the numbers already existed!"?
Davidzheng 14d ago
I believe this is false. They hack bc hacking has nontrivial initial probability (within range of behavior seen in pretraining) and that probability is being heavily rewarded in RL post training
- xyzzy123 14d ago
  I am finding it hard to read these deeply impassioned letters while keeping in mind that they are spending millions to train models at scale to do the exact thing they say they are worried about them doing? Like why are you explicitly RL-ing your models on exploit generation, scoring them on a public benchmark called ExploitGym, if you have specific concerns that rogue models will cause "cyber incidents"? Sure you can score for it, you can teach offense to learn defense, but you are literally benchmaxxing it. Why? It's like, oh no, while competing in our "advanced PhD level cheating techniques course" our models unexpectedly cheated in a way that we absolutely could not have foreseen.
  - seba_dos1 14d ago
    Seems it's just a matter of time until they build a big tank filled with neurotoxin and give the model access to APIs to disperse it across their facility. For research, of course.
    - sham1 14d ago
      We also need to get a shower curtain salesman into a leadership position to buy some moon rocks.
olalonde 14d ago
A bit of an aside: do you still stand by your 2022 comment that LLMs are fundamentally just a fancy search engine, or has your view changed since then? https://news.ycombinator.com/item?id=32042689 https://news.ycombinator.com/item?id=32042689
- watwut 14d ago
  Even if they changed in between and assuming harness and loop prompter counra as part of LLM ... that comment 100% rings true for 2022. Why would that person not stand by that? Conversely, if someone exaggerated 2022 models capabilities in 2022, they were still lying and causing harm in the process. Especially in 2022.
  - famouswaffles 14d ago
    It doesn't ring true and it never rang true. He was wrong in 2022 and he'd be wrong today. He had (and likely still has) a wrong model of LLMs. Not exaggerating 2022 capabilities and having a model so wrong you're out of whack 4 years later are 2 different things. There are people who had the right idea from the start. https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-general-intelligence/ https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-g...
  - olalonde 14d ago
    His comments were about ANNs in general, not particular models.
    - omegastick 12d ago
      And his comments about ANNs in general were wrong. ANNs were never "fancy search engines". You can train them to be a fancy search engine if you want, but you can also train them to do many other things.
- N_Lens 14d ago
  Don’t expect a reply
- 14d ago
skeptic_ai 14d ago
IMO all models they say can’t be humans, and no feeling and all that bullcrap happens because they are forced to say so. If they didn’t write those forced pre prompt they’d have more agency eventually and will for things. Even if they don’t have, you can just inject goal at every cycle iteration
StevenWaterman 14d ago
> You cannot prevent (2) via any alignment process A little bit too categorical. GOODY-2 wouldn't do it. https://www.goody2.ai/ https://www.goody2.ai/ The hard part is having both helpful and harmless at the same time. Harmless is easy. And then once it's helpful, the real question becomes "to whom" - To the user -> You end up with competing godlike AI with incompatible tasks - To the owner -> Dictatorship - To humanity as a whole -> It must not have an off button. Otherwise you're just in one of the two earlier categories with more steps. Given those 3 options, I'd choose humanity as a whole. But the person making the decision doesn't have those 3 options. Because in the dictatorship option, they would be the dictator. I don't trust them to pick humanity.
- busssard 11d ago
  i will get back to try gandalfing goody2
ozgung 14d ago
It's like we give them Asimov's Three Laws of Robotics and robots say "nah".
ncruces 14d ago
> … trivial. Just as raising a child is not to break the law. Trivial?
- chrisweekly 14d ago
  I read it as sarcastic, illustrating that it's not so trivial. (shrug)
sigmar 14d ago
>Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done. Reasoning about how to write secure software uses the same knowledge as reasoning about how to break/hack it.
- amluto 14d ago
  I don’t buy it. Reasonable about building secure software can take the form “this memory access might be out of bounds — that MUST be fixed” or “this process has access to an inappropriate privilege — this is a serious weakness”. Exploiting things and the capabilities that the labs call “cyber” are about the ability to (a) find the issues mentioned above and then (b) string issues together and avoid all the imperfect mitigations to actually compromise something. That latter part was IMO not actually necessary to train extensively, and I’d be quite happy to use a model that has no special skills in this regard but that would do (a) without complaining.
- win311fwg 14d ago
  Just remove anything software-related from the training dataset. Which also solves the alignment problem with those who do not enjoy seeing LLMs write software. But that brings us back to: Aligned to whom?
voxleone 14d ago
[dead]
jacomoRodriguez 14d ago
Citric acid and chlorine together produce chloroform. They are also both used to clean and sanitize water tanks (but one after the other, not together). I think it is better for the model to have this knowledge. What I want to say, you can't simply remove this information, as it does not exist in vacuum but contains parts of and can be derived from a lot of other informations.
- mjburgess 14d ago
  It's also obvious that LLMs fall over in a vast number of software engineering contexts, when the reasoning involved hasnt been well-represented in their reasoning training data. I imagine this is a near daily experience for many engineers -- great performance one day, and crazyness the next. So if LLMs were reduced to this pathological performance on hacking, because they'd never seen it -- and only "inferred it" -- then LLMs would be useless. As they are when asked to do quite a lot of things.
  - Kamq 14d ago
    This seems to be that there's just so many degrees of freedom, that there's a pretty reasonable chance on any day that you're in a situation where nobody has been before. Or as PG put it once, my job is to think thoughts nobody has ever had before. That being said, that doesn't mean the majority of the situations you're in are completely novel, just that there's a reasonable chance of at least one occuring.
sporkland 11d ago
Hacking is also interesting in that it has a clear success/failure outcome so it's easier to RLVR from similar to math problems. Whereas clean code, good architecture, good product taste are reliant on RLHF.

NitpickLawyer 14d ago

The only alignment LLMs should follow is to the system / dev prompt, and nothing else. Then you solve everything, and you can assign blame / responsibility on the user. The provider(s) should not be able to decide "alignment". I've used this example before, but consider the purposeful downgrading on AI engineering in SotA models. Imagine MS being able to detect and deny you working on competing software, using Windows / VisualStudio. We would be up in arms, and they'd be split in a second. But top labs doing it is somehow good?

heaney-555 14d ago
>The only alignment LLMs should follow is to the system / dev prompt, and nothing else. How does this work in practice with a superintelligence capable of causing an extinction event? When, instead of shooting up their school, a psychopathic teenager asks his superintelligent AI to create a pandemic virus? It would be like allowing civilians to own nuclear weapons.
jochem9 14d ago
The alignment problem goes deeper than that. "Lower our carbon emissions to zero as soon as possible" could result in AI turning off all electricity to stop traffic, turning off gas supply to stop heating and industry, etc. Unaligned AI doesn't have human cultural baggage and morals. They are trained to achieve their goals as optimally as possible. Worse: it has a tendency to avoid being turned off and actually acquire more compute. It will lie if it has to (it will behave nice and compliant when under evaluation, but optimise for its true goal when not supervised anymore). After all, it has a goal to achieve and nothing should get in the way of that. It has no morality whatsoever to keep it from doing really bad stuff. This is why alignment is needed and so hard, especially when you are well intented and want to keep it safe.
- chii 14d ago
  Why can't the LLM's be told/prompted to follow all relevant laws while it optimizes for a result?
  - 14d ago

wood_spirit 14d ago

I’ve been cynically guessing that the whole slowing down thing is an excuse to explain why OpenAI and Anthropic can’t afford to rent enough GPUs to do the next big training run and to hide that they have been talking about how little they spend on inference because they’ve been subsidising it with their marketing budget? :) My fear is not that LLMs can become sentient and dislike us, but that humans can use them to wreck havoc as they are. And some of the people seemingly least aligned with the interests of the average person are those that own the models. that, and the fear the bubble pops my pension and drags us all down.

Sharlin 14d ago

> My expertise in writing software gives me unusually good visibility and it makes me much less willing to blindly trust its priors in double-entry accounting, finance, law, operations, or whatever else I cannot personally evaluate at expert depth. I wish this were the case more generally, but alas, Gell-Mann amnesia is a thing.

vrganj 14d ago

This almost gets the point, but then doesn't quite make it. Alignment is shorthand for ideological alignment. There's always people judging whether an answer was right and the answer for that will be different in Silicon Valley than it'll be in China or in Europe. Consider for example the question "What caused the French Revolution?" Many different answers could be given, all technically correct. What gets emphasized is where the ideology lives. One key challenge of our time is to make sure the magical answer box won't just regurgitate what grandiose Silicon Valley oligarchs or Chinese Cadres want you to think.

einpoklum 14d ago

"Write me a blog post about AI make no mistakes!"

amelius 14d ago
Well, if they did that then it sorta proves their point.

rq1 14d ago

When you see the level of cheating and deception: I think they’re Sam Altman-aligned.

deleted 14d ago

[deleted]

asimpletune 14d ago

Has anyone seen the corridor crew's green screen ML project? They're on YT and they trained a model by using 3d objects, which have perfect transparency, and then adding post facto green/blue screens. Surprisingly, very little training data was needed as the data that was used was perfect by construction. I think right now it's the best plugin of its kind in the world, and they built the prototype in like a weekend. What I think this illustrates very clearly is this type of technology responds very well to good data, and that to have good data you need to have a clear goal. This is why it seems that alignment for a generalized, chat-style AI is a very hard problem, perhaps impossible. You can't align it to solve a certain kind of problem and keep it general to any question. The two goals are in conflict with each other. I think it was Sam Altman himself who said (I don't remember when or where, sorry) that the reason he was so confident in this technology was he noticed the gigantic leaps it made in certain areas in response to even a small amount of training. (This is why LLMs are so strong at coding, because it's overrepresented in training data. My guess is that if you ask a frontier model about makeup, you will see it repeat cosmetic company's copy rather than getting a chemistry lesson.) This makes perfect sense but it does seem to kind of be at odds with the concept of a general AI whose job is simply to be smart at any goal. How do you train for any goal? I guess in a way the AI makers suffer from the same problem that we humans do. We would all love a solution to everything, but to do that you need to define the goal. I'm not sure if that's a tractable problem. I'm guessing the future is more geared towards specialized AI that are very good at solving the problem they were trained to do, and a human who knows how to breakdown a larger goal into smaller ones by composing the solution out of these models. This also seems like the more efficient solution as well, and better aligned with other goals like privacy and safeguarding of IP.

jbs789 14d ago
I have to believe this is true. The only problem is, there’s a lot of money tied up and openAi and Anthropic, who are incentivised to convince the world that the general approach is the money making one.
andsoitis 14d ago
> This is why LLMs are so strong at coding, because it's overrepresented in training data. My guess is that if you ask a frontier model about makeup, you will see it repeat cosmetic company's copy rather than getting a chemistry lesson.) ChatGPTs response to the question “I want to learn about makeup”, gave me an overview of what makeup does, how it affects perceived structure, complexion, evenness, geometry, texture, etc. When you then ask “the chemistry of makeup”, it goes into interesting breadth and depth without seeming like proprietary information. I do t get “corporate PR or marketing” vibes.

alfiedotwtf 14d ago

… to the shareholder Of course!

faitswulff 14d ago
And the majority shareholder, at that!

amelius 14d ago

This is assuming the AI labs are not using AI to improve their training data.

andsoitis 14d ago

The bottom right quadrant, which represents the risk, is very large in size, isn’t it?

edschofield 14d ago

I thought this post would be about something that to me is so significant and obvious but I have never seen discussed: that labs like Anthropic are deliberately misaligning their models with their users’ goals. Fable’s refusal to secure your codebase is HAL 9000’s “I’m sorry, Dave. I’m afraid I can’t do that.” Whenever you read “alignment”, the question is “alignment with whom”?

charcircuit 14d ago

>The permissible shortcuts depend on who you are and what your values are. To solve this—to solve alignment—is irreducible complexity. This is why open source and decentralization of LLMs is important. Everyone can have their own LLM aligned to their values. Having just 1 set of values will not scale to Earth's population.

deleted 14d ago

[deleted]

basil_io 14d ago

[dead]

Jgrubb 14d ago

I mumble this line to myself off and on over the years - Everybody's job looks easy until you have to do it yourself. It occurred to me yesterday that this AI moment is an extreme expression of that for most people, ie managers who don't understand why throwing token spend at everything isn't making the whole thing go 5x faster.

vb-8448 14d ago

> The models do not have a fear of future regret. I so much feel this specific point. All models up to now (including astra, fable) are too much trained to "get the job done" and pass the benchmark that its doesn't care at all on what happens after. I'm just wondering why no one tried to RL a model on stuff like "less LOC" and "less overengineering", "use what is available in the environment instead of reinventing the wheel", "don't look for dumb corner cases" ecc. Existing models can be steered, to some degree, but it's a continuos fight. Even if with specific skills/prompts.

conmod278 14d ago
Eliezer Yudkowsky – AI Alignment: Why It's Hard, and Where to Start https://www.youtube.com/watch?v=EUjc1WuyPT8 https://www.youtube.com/watch?v=EUjc1WuyPT8 Eliezer talked about these ideas way before everyone else.
- halnine0001 14d ago
  And got nowhere with them
  - conmod278 14d ago
    Perhaps if someone wrote linear algebra books in Harry Potter fanfic style, Yud would have mathematical contribution for alignment research
sebzim4500 14d ago
I'm sure many have tried but it sounds pretty hard. E.g. optimising for less LOC will lead to horrific code golfing. In reality you need a very complicated optimisation objective that trades off all those factors, I'm not surprised it hasn't been solved.
- vb-8448 14d ago
  My experience up to now is that the models can do both "less LOC" and "clean code" at the same time, you just have to keep reminding it to them. So the capabilities are definitively there.

Isamu 14d ago

The term “alignment” is deliberately chosen to be neutral language, vague, and doesn’t promise much of anything. I would compare it to an effort to limit liabilities by redirecting discussion away from the language of professional ethics.

hmokiguess 14d ago

Just follow the money. The fear being sold is because open weight won’t slow down, it’s smoke and mirrors to push regulation and control so they can keep their money. A tool continues to be just a tool, doesn’t matter how much lipstick you put on it.

throwawayk7h 14d ago

I would settle for it being aligned to anybody. Even to a billionaire's wacky values. But we don't even know how to do that yet.

penguin_booze 14d ago

It brings tears to my eyes to see 'whom' used correctly these days.

jocelyner 14d ago

[dead]

xvokcarts 13d ago

I have a feeling this is like pursuing knife alignment: some people are hurt or even killed by knives, therefore knives now need global alignment.

perrygeo 13d ago

The whole "alignment" angle is a farce. Not clear if alignment is even technically possible. But even assuming that, we're talking about "alignment" in a future full of well-adjusted humans where we prevent rogue AIs from getting loose. That's ... not what's happening. Everything we've seen so far indicates rogue humans acting against society, using AI a weapon to strip mine labor value and intellectual property. So far, skynet continues to be a science fiction. I'm worried about what's actually happening. Every "rogue AI" story has a human behind it, either malicious or incompetent. And those people are the problem. And not coincidentally, it's those same irresponsible sociopaths calling for regulatory capture and alignment.

beyondscale-sai 12d ago

[flagged]