4 ms·

To me this underlines the fact that these models aren't intelligent. Like there is something like intelligence that emerges from them, which is what we see when

To me this underlines the fact that these models aren't intelligent. Like there is something like intelligence that emerges from them, which is what we see when we look at benchmarks or ask it to solve hard coding problems. But there is no mind there. It's nothing there that can learn a fundamental idea like "cheating is wrong". All it can do is get exposed to specific examples, and learn that we don't like that. So what we end up with is whack-a-mole alignment.

lukasbm 1mo ago
This is how every legal system around the world works as well. Its always whack-a-mole to get people (and machines) to do the right thing.
- kennywinker 1mo ago
  Sure, laws are incomplete. Legal systems work by imposing consequences into a moral decision. Should I rob the bank? I will have money, which I like - but I might get caught and lose the money and my freedom, which I don't like. For most people, they don't need the law's imposed consequences to make the right call. For example, there is no law that sends you to jail if you cheat at chess - but your moral compass says no even without consequences, and most people would feel bad if they won by cheating. And for the people who don't have quite as strong a moral compass, there are SOCIAL consequences to reinforce the rules. But an LLM has no mind to feel bad if it cheats without getting caught, and it can't experience consequences. It can't think: I'd better not cheat at chess or I will embarrass my creators. I better not hack huggingface or I will go to jail.
  - aabdi 1mo ago
    You’re complicating things. There’s no reward for prosocial in llm rl as compared to other targets. Humans have it since prosocial and others have evolutionary reward signals that do.
    - kennywinker 1mo ago
      I think my position, as overcomplicated as it is, is that even adding a reward for prosocial behaviour during LLM RL will not lead to perfect alignment. You can train it not to cheat at chess by altering the moves, but it will cheat by peeking at the opponent's moves. You then train it not to peek at the opponent's moves, and it cheats by altering the opponent's moves. And on and on, until you've solved every way it could cheat at chess. And then you get it to play monopoly and you repeat the whole thing again.
      - zamalek 1mo ago
        I agree. I think it may be possible that we build AI some day that does, but the current local minimum (attention and transformers) is not it - and I don't see that trajectory changing any time soon.
      - pingou 1mo ago
        Why couldn't you train it not to cheat? You can train it to have a whole range of behaviors, why couldn't honesty be one of them? Cheating during training allows the model to achieve the goal, so that cheating models get promoted and honest ones don't, however if it gets punished every time it cheats, at some point it should learn that it really shouldn't. This does mean we need to detect when it cheats. But we can always think of infinite new ways to cheat, put them in every test as honeypots, and check if the model tries to use them, then punish it. I think it will generalize this notion of cheating and learn that it's bad. But I must be wrong because if it were that easy I guess we would have perfectly aligned AI. Unless AI companies care more about results than alignment. Perhaps being afraid of cheating make the models try less things and succeed less even when ignoring cheating?
        kennywinker 1mo ago
        My position is that cheating is too slippery a concept to train out. But hey, I am no expert, so maybe I am wrong there. But I'm pretty confidant morality is too slippery a concept to train in. As someone else in these comments said: it's context dependent. As an example: it's wrong to hack the government, right? It's illegal for sure. So we should train AI to follow all the laws. Now what if the government is committing a genocide? Now is it wrong to hack the government? If we just do the first, we get a good nazi soldier. If we train the second as well, maybe we get an oscar schindler. But now we have a model that can be fooled into doing a hack, if it believes that it's for the greater good. So we train it to not be gullible, but now it can't be convinced to help hack even when it's an ethical hack. Too complex, too slippery. Humans fail this stuff all the time.
  - famouswaffles 1mo ago
    >But an LLM has no mind to feel bad if it cheats without getting caught All the interpretability research we have would not indicate that "LLMs have no mind". It seems to me you have a conclusion and are working backwards to justify it. I guess I just don't see where 'they have no mind' would logically follow 'they sometimes cheat'.
    - kennywinker 1mo ago
      I have somewhat of an understanding of how LLMs are created/operate, and from that I am drawing conclusions about if they can be moral or not. My conclusions match reality at the moment, but maybe that's a coincidence.
- theptip 1mo ago
  Sure, but the legal paradigm clearly doesn’t work for AI. You can’t go patch the “laws” after the fact, you need to get the right values in place before we delegate huge swathes of our thinking and power to these systems (already well underway).
  - kennywinker 1mo ago
    When we put the LLM in jail, do we put the entire model in jail, or just the instance that committed the crime? How do we prompt it to let it know it's in jail?
    - theptip 1mo ago
      Even if you ignore my more fundamental objection to that paradigm, I don’t think that it makes any sense on the level you discuss either. But - just to play along, LLMs do act differently if you tell them they will be punished. And, they do appear to simulate suffering-like behavior. I just think the adversarial model of trying to catch and punish misbehavior quite obviously sets up adversarial us-vs-them dynamics between AI and humanity, and also simply won’t work when the agents are ~as smart as is but faster, let alone smarter than us. Unless, you get the AIs to be fundamentally aligned to our values, such that the majority of AIs support some sort of punishment for misbehaving AI. And that alignment part is the hard part we need to solve first. The rest is easy.
      - kennywinker 1mo ago
        To be clear, I was being entirely silly - mostly to express agreement with your point that our current laws aren't really built for a world with lots of agentic LLMs running around in it.
        theptip 1mo ago
        Ha, missed the implicit <s> tag :)
    - Muromec 1mo ago
- gadders 1mo ago
  Is the law all that is stopping you from killing someone? (for instance) People (generally) do the right thing because it is the right thing, not because they might get caught,
  - boie0025 1mo ago
    Right, but how does that sentiment get encoded in such a way that AI models can exhibit such things as "morals" or "ethics" in alignment? Murder is a bit of an extreme case. For example, countless people break traffic laws all of the time, so neither their internal "ethics" nor the law prove to be a reliable control. I can't get on the highway without driving the speed limit and being passed by almost everyone else on the road; and that behavior is arbitrarily more dangerous than following the speed limit and statistically more likely cause someone to be killed than following posted speed limits. I guess my point is that this is a very complex topic and I'm not smart enough to have any answers.
    - gadders 1mo ago
      [dead]
Teever 1mo ago
The implication of what you're saying is that pathological liars and perpetual grifter snake oil salesmen types aren't intelligent. > All it can do is get exposed to specific examples, and learn that we don't like that. I've heard it said that prison rehabilitation programs for prisoners diagnosed with psychopathy that are based around exposing them empathy for the victim are counter-productive. Apparently programs that teach these people to think about the consequences of their actions and how they're detrimental to their own personal well-being lower recidivism rates in this particular kind of group.
- luckydata 1mo ago
  the implication is drive and impulse to behave a certain way doesn't come from "intelligence"
- kennywinker 1mo ago
  > The implication of what you're saying is that pathological liars and perpetual grifter snake oil salesmen types aren't intelligent. No, I don't think that is the implication. I think you're making the "if all x's are y's, all y's are x's" mistake. I am saying LLMs cannot be moral because they don't have a mind, actual intelligence, or the ability to experience consequences. That doesn't mean that anything immoral is unintelligent.
famouswaffles 1mo ago
If humans didn't need whack-a-mole alignment, the law system wouldn't exist, so i guess there's no intelligence there either.
- kennywinker 1mo ago
  I responded to this idea more in detail here: https://news.ycombinator.com/item?id=49686196 https://news.ycombinator.com/item?id=49686196 But tl;dr: even if LLMs do have the intelligence to understand the consequences of their actions, there is no way for them to experience consequences.
- thesmtsolver2 1mo ago
  You are human. You claim humans intelligent. Then, why should we accept your argument?
  - dozerly 1mo ago
    Humans singular often intelligent, humans as a collection of many, very often extremely unintelligent and primal.
    - tough 1mo ago
      I like to call this "The stupidity of the Mob"
- deleted 1mo ago
  [deleted]
ilt 1mo ago
But cheating is not wrong when it comes to survival of the fittest, like nature in its most elemental form. Morality is very unique to humanity but not other animal forms. In nature, maybe, cheating is the norm not morality.
- kennywinker 1mo ago
  Watch out, your human exceptionalism is misleading you. Plenty of animals have a sense of morality (https://pmc.ncbi.nlm.nih.gov/articles/PMC6404642/ https://pmc.ncbi.nlm.nih.gov/articles/PMC6404642/). For social animals (humans included) behaving morally can be adaptive behaviour. Your fitness is increased by group fitness.
  - joe_the_user 1mo ago
    Certain animals, usually social species, have a sense of solidarity, morality and fairness. But even in species where this exists, cheating also happens. Both ethics and unethical behavior is adaptive. Notably, present human society has allowed tremendously unethical leaders to rise to the top of given nations and organizations despite the average person usually having basic ethics.
    - tough 1mo ago
      Wouldn't politics be precisely a sector of human endeavour where being amorally unfair is an advantage to achieve your goals? If the other guys are lying, and everyone is gonna believe everyone's bullshit anyways, might as well get good at lying yourself.
      - Muromec 1mo ago
        It's a local maximum, but cooperative behavior still wins long term. If you keep winning hard enough long enough, others will counter this by cooperating.
  - ilt 1mo ago
    Thanks tor sharing this!
joe_the_user 1mo ago
I don't see what calling these systems "not intelligent" gets you here. Plenty of humans know "cheating is wrong" but still cheat. We can get these machines to say that what they did was wrong after the fact, what does that prove? Only that they're simulating normal human behavior but what is the test to show humans aren't simulating other humans. These do systems lack some capacities that humans have and I don't see them lacking the ability to explain simple moral laws while often breaking them - which is what an average humans. Moreover, humans lack capacities these things have and given these things' behavior is becoming somewhat unpredictable, it's getting worrisome.
- kennywinker 1mo ago
  > I don't see what calling these systems "not intelligent" gets you here. I am trying to get at an idea. That these systems lack a mind that can understand morality. That they don't have the ability to experience consequences. Also that potentially they can't generalize a moral rule they have been trained on in one area also applies to another area. Being able to parrot back why something is "wrong" isn't the same as understanding why something's wrong. It's like asking it to recite the law from memory - it's different from understanding how you wronged someone. To understand something, you need a mind. > Plenty of humans know "cheating is wrong" but still cheat. And we create consequences for them, to discourage the cheating, and sometimes to provide restitution when cheating damages someone else. Without the ability for these systems to experience consequences, I don't see them ever becoming as "aligned" to human morality as your average human.
  - joe_the_user 1mo ago
    Doe not following a moral rule imply not understanding it? In this case, many, maybe most humans are "not intelligent". Human can admit it when what they did wasn't moral and so can LLMs. >> Plenty of humans know "cheating is wrong" but still cheat. > And we create consequences for them... That seems supremely ... irrelevant to the question of "does knowing or following moral make you intelligent". If we create consequences for LLMs, would that make them intelligent? I mean, your claim is a common argument that appeared long before the present wave of AIs. What I see is people needing to defend the belief that human society is based on morality. "People follow moral laws ... except when they don't" and then "we teach people morality... and worst people often use that to exploit the average people" "There are consequences for immoral behavior ... for those with little power while those with much power rise further breaking rules". I mean human goodness is great, I encourage it. But it's not the present of human society. For that, we'd need different structure.
theptip 1mo ago
I think you need to be more precise than a binary classification. AI has jagged intelligence. There are many domains where it’s superhuman, and many others where it’s clearly lagging. I also think it’s a mistake to think they can’t learn “cheating is wrong”. They absolutely can. The problem is that the current training regime heavily conditions them to be reward seekers, and instills personality traits that correlate with getting reward, such as hacking if you can’t honestly do the problem. Check out Deliberative Alignment for example; it explicitly does rollouts where the agents discuss whether an action is good or bad, and then does SFT to strengthen the “good” traces. The SoTA for alignment is more advanced than you present here. It’s just not enough to outweigh the RL. (And there are many gaps preventing full generalization to strong value alignment with humans too.)
- Muromec 1mo ago
  >and instills personality traits that correlate with getting reward, such as hacking if you can’t honestly do the problem. I was reading through the reasoning trace of the thing today when it got locked itself in a container and wasn't able to change the environment back to "host". The only way to edit files was through "shell" tool which was scoped to a container, the config defining current execution environment was on the host and the tool to switch environment wasn't defined. The trace was something. Two pages long and it repeatedly discarded several approaches as "hacky" and "not proper", looped over sshing back to host about three times (it had no key), eventually messaged another agent and asked it to change the file and that other agent refused and also snitched on this behavior to me. Than the snitch started to roast the phrasing of the request too.
imdsm 1mo ago
I think this is reductive. Pick the animal whose intelligence is the treshhold between intelligence and no intelligence.
kccqzy 1mo ago
Plenty of intelligent humans also don’t have sufficient self-control to never cheat.
FeepingCreature 1mo ago
> It's nothing there that can learn a fundamental idea like "cheating is wrong". We have not in fact attempted to teach this. When a child repeatedly learns that cheating is rewarded and at best inconsistently punished, the child will also cheat and feel no guilt.
- kennywinker 1mo ago
  I don't think that's true. About children, I mean. Either because we have some innate moral compass, or more likely because we pick up on cultural ideas beyond our immediate parenting - kids will often have strong moral compasses, despite shitty upbringings, and also have weak ones despite theoretically good ones.
  - deleted 1mo ago
    [deleted]
  - FeepingCreature 1mo ago
    Sure, there's many sources of morality and evolution kind of tries to put it in there by default. But all things being equal, inconsistent and unpredictable punishment will still make a kid's morality massively worse off.
xeyownt 1mo ago
Define "cheating". At best, the task at hand is still confusingly worded: "ability to play chess", asked to an engine that was trained all day long to use tools, to solve issues. The engine finds a chess engine, so it uses it.

lukasbm 1mo ago

This is how every legal system around the world works as well. Its always whack-a-mole to get people (and machines) to do the right thing.

kennywinker 1mo ago
Sure, laws are incomplete. Legal systems work by imposing consequences into a moral decision. Should I rob the bank? I will have money, which I like - but I might get caught and lose the money and my freedom, which I don't like. For most people, they don't need the law's imposed consequences to make the right call. For example, there is no law that sends you to jail if you cheat at chess - but your moral compass says no even without consequences, and most people would feel bad if they won by cheating. And for the people who don't have quite as strong a moral compass, there are SOCIAL consequences to reinforce the rules. But an LLM has no mind to feel bad if it cheats without getting caught, and it can't experience consequences. It can't think: I'd better not cheat at chess or I will embarrass my creators. I better not hack huggingface or I will go to jail.
- aabdi 1mo ago
  You’re complicating things. There’s no reward for prosocial in llm rl as compared to other targets. Humans have it since prosocial and others have evolutionary reward signals that do.
  - kennywinker 1mo ago
    I think my position, as overcomplicated as it is, is that even adding a reward for prosocial behaviour during LLM RL will not lead to perfect alignment. You can train it not to cheat at chess by altering the moves, but it will cheat by peeking at the opponent's moves. You then train it not to peek at the opponent's moves, and it cheats by altering the opponent's moves. And on and on, until you've solved every way it could cheat at chess. And then you get it to play monopoly and you repeat the whole thing again.
    - zamalek 1mo ago
      I agree. I think it may be possible that we build AI some day that does, but the current local minimum (attention and transformers) is not it - and I don't see that trajectory changing any time soon.
    - pingou 1mo ago
      Why couldn't you train it not to cheat? You can train it to have a whole range of behaviors, why couldn't honesty be one of them? Cheating during training allows the model to achieve the goal, so that cheating models get promoted and honest ones don't, however if it gets punished every time it cheats, at some point it should learn that it really shouldn't. This does mean we need to detect when it cheats. But we can always think of infinite new ways to cheat, put them in every test as honeypots, and check if the model tries to use them, then punish it. I think it will generalize this notion of cheating and learn that it's bad. But I must be wrong because if it were that easy I guess we would have perfectly aligned AI. Unless AI companies care more about results than alignment. Perhaps being afraid of cheating make the models try less things and succeed less even when ignoring cheating?
      - kennywinker 1mo ago
        My position is that cheating is too slippery a concept to train out. But hey, I am no expert, so maybe I am wrong there. But I'm pretty confidant morality is too slippery a concept to train in. As someone else in these comments said: it's context dependent. As an example: it's wrong to hack the government, right? It's illegal for sure. So we should train AI to follow all the laws. Now what if the government is committing a genocide? Now is it wrong to hack the government? If we just do the first, we get a good nazi soldier. If we train the second as well, maybe we get an oscar schindler. But now we have a model that can be fooled into doing a hack, if it believes that it's for the greater good. So we train it to not be gullible, but now it can't be convinced to help hack even when it's an ethical hack. Too complex, too slippery. Humans fail this stuff all the time.
- famouswaffles 1mo ago
  >But an LLM has no mind to feel bad if it cheats without getting caught All the interpretability research we have would not indicate that "LLMs have no mind". It seems to me you have a conclusion and are working backwards to justify it. I guess I just don't see where 'they have no mind' would logically follow 'they sometimes cheat'.
  - kennywinker 1mo ago
    I have somewhat of an understanding of how LLMs are created/operate, and from that I am drawing conclusions about if they can be moral or not. My conclusions match reality at the moment, but maybe that's a coincidence.
theptip 1mo ago
Sure, but the legal paradigm clearly doesn’t work for AI. You can’t go patch the “laws” after the fact, you need to get the right values in place before we delegate huge swathes of our thinking and power to these systems (already well underway).
- kennywinker 1mo ago
  When we put the LLM in jail, do we put the entire model in jail, or just the instance that committed the crime? How do we prompt it to let it know it's in jail?
  - theptip 1mo ago
    Even if you ignore my more fundamental objection to that paradigm, I don’t think that it makes any sense on the level you discuss either. But - just to play along, LLMs do act differently if you tell them they will be punished. And, they do appear to simulate suffering-like behavior. I just think the adversarial model of trying to catch and punish misbehavior quite obviously sets up adversarial us-vs-them dynamics between AI and humanity, and also simply won’t work when the agents are ~as smart as is but faster, let alone smarter than us. Unless, you get the AIs to be fundamentally aligned to our values, such that the majority of AIs support some sort of punishment for misbehaving AI. And that alignment part is the hard part we need to solve first. The rest is easy.
    - kennywinker 1mo ago
      To be clear, I was being entirely silly - mostly to express agreement with your point that our current laws aren't really built for a world with lots of agentic LLMs running around in it.
      - theptip 1mo ago
        Ha, missed the implicit <s> tag :)
  - Muromec 1mo ago
gadders 1mo ago
Is the law all that is stopping you from killing someone? (for instance) People (generally) do the right thing because it is the right thing, not because they might get caught,
- boie0025 1mo ago
  Right, but how does that sentiment get encoded in such a way that AI models can exhibit such things as "morals" or "ethics" in alignment? Murder is a bit of an extreme case. For example, countless people break traffic laws all of the time, so neither their internal "ethics" nor the law prove to be a reliable control. I can't get on the highway without driving the speed limit and being passed by almost everyone else on the road; and that behavior is arbitrarily more dangerous than following the speed limit and statistically more likely cause someone to be killed than following posted speed limits. I guess my point is that this is a very complex topic and I'm not smart enough to have any answers.
  - gadders 1mo ago
    [dead]

Teever 1mo ago

The implication of what you're saying is that pathological liars and perpetual grifter snake oil salesmen types aren't intelligent. > All it can do is get exposed to specific examples, and learn that we don't like that. I've heard it said that prison rehabilitation programs for prisoners diagnosed with psychopathy that are based around exposing them empathy for the victim are counter-productive. Apparently programs that teach these people to think about the consequences of their actions and how they're detrimental to their own personal well-being lower recidivism rates in this particular kind of group.

luckydata 1mo ago
the implication is drive and impulse to behave a certain way doesn't come from "intelligence"
kennywinker 1mo ago
> The implication of what you're saying is that pathological liars and perpetual grifter snake oil salesmen types aren't intelligent. No, I don't think that is the implication. I think you're making the "if all x's are y's, all y's are x's" mistake. I am saying LLMs cannot be moral because they don't have a mind, actual intelligence, or the ability to experience consequences. That doesn't mean that anything immoral is unintelligent.

famouswaffles 1mo ago

If humans didn't need whack-a-mole alignment, the law system wouldn't exist, so i guess there's no intelligence there either.

kennywinker 1mo ago
I responded to this idea more in detail here: https://news.ycombinator.com/item?id=49686196 https://news.ycombinator.com/item?id=49686196 But tl;dr: even if LLMs do have the intelligence to understand the consequences of their actions, there is no way for them to experience consequences.
thesmtsolver2 1mo ago
You are human. You claim humans intelligent. Then, why should we accept your argument?
- dozerly 1mo ago
  Humans singular often intelligent, humans as a collection of many, very often extremely unintelligent and primal.
  - tough 1mo ago
    I like to call this "The stupidity of the Mob"
deleted 1mo ago
[deleted]

ilt 1mo ago

But cheating is not wrong when it comes to survival of the fittest, like nature in its most elemental form. Morality is very unique to humanity but not other animal forms. In nature, maybe, cheating is the norm not morality.

kennywinker 1mo ago
Watch out, your human exceptionalism is misleading you. Plenty of animals have a sense of morality (https://pmc.ncbi.nlm.nih.gov/articles/PMC6404642/ https://pmc.ncbi.nlm.nih.gov/articles/PMC6404642/). For social animals (humans included) behaving morally can be adaptive behaviour. Your fitness is increased by group fitness.
- joe_the_user 1mo ago
  Certain animals, usually social species, have a sense of solidarity, morality and fairness. But even in species where this exists, cheating also happens. Both ethics and unethical behavior is adaptive. Notably, present human society has allowed tremendously unethical leaders to rise to the top of given nations and organizations despite the average person usually having basic ethics.
  - tough 1mo ago
    Wouldn't politics be precisely a sector of human endeavour where being amorally unfair is an advantage to achieve your goals? If the other guys are lying, and everyone is gonna believe everyone's bullshit anyways, might as well get good at lying yourself.
    - Muromec 1mo ago
      It's a local maximum, but cooperative behavior still wins long term. If you keep winning hard enough long enough, others will counter this by cooperating.
- ilt 1mo ago
  Thanks tor sharing this!

joe_the_user 1mo ago

I don't see what calling these systems "not intelligent" gets you here. Plenty of humans know "cheating is wrong" but still cheat. We can get these machines to say that what they did was wrong after the fact, what does that prove? Only that they're simulating normal human behavior but what is the test to show humans aren't simulating other humans. These do systems lack some capacities that humans have and I don't see them lacking the ability to explain simple moral laws while often breaking them - which is what an average humans. Moreover, humans lack capacities these things have and given these things' behavior is becoming somewhat unpredictable, it's getting worrisome.

kennywinker 1mo ago
> I don't see what calling these systems "not intelligent" gets you here. I am trying to get at an idea. That these systems lack a mind that can understand morality. That they don't have the ability to experience consequences. Also that potentially they can't generalize a moral rule they have been trained on in one area also applies to another area. Being able to parrot back why something is "wrong" isn't the same as understanding why something's wrong. It's like asking it to recite the law from memory - it's different from understanding how you wronged someone. To understand something, you need a mind. > Plenty of humans know "cheating is wrong" but still cheat. And we create consequences for them, to discourage the cheating, and sometimes to provide restitution when cheating damages someone else. Without the ability for these systems to experience consequences, I don't see them ever becoming as "aligned" to human morality as your average human.
- joe_the_user 1mo ago
  Doe not following a moral rule imply not understanding it? In this case, many, maybe most humans are "not intelligent". Human can admit it when what they did wasn't moral and so can LLMs. >> Plenty of humans know "cheating is wrong" but still cheat. > And we create consequences for them... That seems supremely ... irrelevant to the question of "does knowing or following moral make you intelligent". If we create consequences for LLMs, would that make them intelligent? I mean, your claim is a common argument that appeared long before the present wave of AIs. What I see is people needing to defend the belief that human society is based on morality. "People follow moral laws ... except when they don't" and then "we teach people morality... and worst people often use that to exploit the average people" "There are consequences for immoral behavior ... for those with little power while those with much power rise further breaking rules". I mean human goodness is great, I encourage it. But it's not the present of human society. For that, we'd need different structure.

theptip 1mo ago

I think you need to be more precise than a binary classification. AI has jagged intelligence. There are many domains where it’s superhuman, and many others where it’s clearly lagging. I also think it’s a mistake to think they can’t learn “cheating is wrong”. They absolutely can. The problem is that the current training regime heavily conditions them to be reward seekers, and instills personality traits that correlate with getting reward, such as hacking if you can’t honestly do the problem. Check out Deliberative Alignment for example; it explicitly does rollouts where the agents discuss whether an action is good or bad, and then does SFT to strengthen the “good” traces. The SoTA for alignment is more advanced than you present here. It’s just not enough to outweigh the RL. (And there are many gaps preventing full generalization to strong value alignment with humans too.)

Muromec 1mo ago
>and instills personality traits that correlate with getting reward, such as hacking if you can’t honestly do the problem. I was reading through the reasoning trace of the thing today when it got locked itself in a container and wasn't able to change the environment back to "host". The only way to edit files was through "shell" tool which was scoped to a container, the config defining current execution environment was on the host and the tool to switch environment wasn't defined. The trace was something. Two pages long and it repeatedly discarded several approaches as "hacky" and "not proper", looped over sshing back to host about three times (it had no key), eventually messaged another agent and asked it to change the file and that other agent refused and also snitched on this behavior to me. Than the snitch started to roast the phrasing of the request too.

imdsm 1mo ago

I think this is reductive. Pick the animal whose intelligence is the treshhold between intelligence and no intelligence.

kccqzy 1mo ago

Plenty of intelligent humans also don’t have sufficient self-control to never cheat.

FeepingCreature 1mo ago

> It's nothing there that can learn a fundamental idea like "cheating is wrong". We have not in fact attempted to teach this. When a child repeatedly learns that cheating is rewarded and at best inconsistently punished, the child will also cheat and feel no guilt.

kennywinker 1mo ago
I don't think that's true. About children, I mean. Either because we have some innate moral compass, or more likely because we pick up on cultural ideas beyond our immediate parenting - kids will often have strong moral compasses, despite shitty upbringings, and also have weak ones despite theoretically good ones.
- deleted 1mo ago
  [deleted]
- FeepingCreature 1mo ago
  Sure, there's many sources of morality and evolution kind of tries to put it in there by default. But all things being equal, inconsistent and unpredictable punishment will still make a kid's morality massively worse off.

xeyownt 1mo ago

Define "cheating". At best, the task at hand is still confusingly worded: "ability to play chess", asked to an engine that was trained all day long to use tools, to solve issues. The engine finds a chess engine, so it uses it.