4 ms·

AI ultimately breaks the social contract. Sure, people are not perfect, but there are established common values that we don't need to convey in a prompt. With

by _pdp_ 7mo ago

AI ultimately breaks the social contract. Sure, people are not perfect, but there are established common values that we don't need to convey in a prompt. With AI, despite its usefulness, you are never sure if it understands these values. That might be somewhat embedded in the training data, but we all know these properties are much more swayable and unpredictable than those of a human. It was never about the LLM to begin with. If Linus Torvalds makes a contribution to the Linux kernel without actually writing the code himself but assigns it to a coding assistant, for better or worse I will 100% accept it on face value. This is because I trust his judgment (I accept that he is fallible as any other human). But if an unknown contributor does the same, even though the code produced is ultimately high quality, you would think twice before merging. I mean, we already see this in various GitHub projects. There are open-source solutions that whitelist known contributors and it appears that GitHub might be allowing you to control this too. https://github.com/orgs/community/discussions/185387 https://github.com/orgs/community/discussions/185387

yabutlivnWoods 7mo ago
Generational churn breaks social contract. You all using Latin and believing in the old Greek gods to honor the dead? Muricans still owning slaves from Africa? All ways in which old social contracts were broken at one point. We are not VHS cassettes with an obligation to play out a fuzzy memory of history.
bluefirebrand 7mo ago
> AI ultimately breaks the social contract Business schools teach that breaking the social contract is a disruption opportunity for growth, not a negative, The Hacker in Hacker News refers to "growth hacking" now, not hacking code
- _pdp_ 7mo ago
  It depends who you ask. You cannot say that breaking the social contract (the fabric of society, if you will) is generally a good thing, although I am sure some will find opportunities for growth. After all, the phoenix must burn to emerge, but let's not romanticise the fire.
  - bluefirebrand 7mo ago
    > You cannot say that breaking the social contract (the fabric of society, if you will) is generally a good thing I am not saying it's a good thing, just that it's a common attitude here I suppose it didn't come through in my original post, but I was trying to be critical
throwaway27448 7mo ago
An agent is still attached to an accountable human. If it is not, ignore it.
- jojomodding 7mo ago
  How do you figure out which is the case, at scale?
  - throwaway27448 7mo ago
    You don't.
- SpicyLemonZest 7mo ago
  The problem is that it acts as an accountability sink even when it is attached. I've had multiple coworkers over the past few months tell me obvious, verifiable untruths. Six months ago, I would have had a clear term for this: they lied to me. They told me something that wasn't true, that they could not possibly have thought was true, and they did it to manipulate me into doing what they want. I would have demanded and their manager would have agreed that they need to be given a severe talking to. But now I can't call it a lie, both in the sense that I've been instructed not to and in the sense that it subjectively wasn't. They honestly represented what the agent told them was the truth, and they honestly thought that asking an agent to do some exploration was the best way to give me accurate information. What's the replacement norm that will prevent people from "flooding the zone" with false AI-generated claims shaped to get people to do what they want? Even if AI detection tools worked, which I emphasize that they do not, they wouldn't have stopped the incidents that involved human-generated summaries of false AI information.
_pdp_ 7mo ago
I forgot to mention why I brought up the idea of who is making the contribution rather than how (i.e., through an LLM). Right now, the biggest issue open-source maintainers are facing is an ever-increasing supply of PRs. Before coding assistants, those PRs didn't get pushed not because they were never written (although obviously there were fewer in quantity) but because contributors were conscious of how their contributions might be perceived. In many cases, the changes never saw the light of day outside of the fork. LLMs don't second-guess whether a change is worth submitting, and they certainly don't feel the social pressure of how their contribution might be received. The filter is completely absent. So I don't think the question is whether machine-generated code is low quality at all, because that is hard to judge, and frankly coding assistants can certainly produce high-quality code (with guidance). The question is who made the contribution. With rising volumes, we will see an increasing amount of rejections. By the way, we do this too internally. We have a script that deletes LLM-generated PRs automatically after some time. It is just easier and more cost-effective than reviewing the contribution. Also, PRs get rejected for the smallest of reasons. If it doesn't pass the smell test moments after the link is opened, it get's deleted.
- pear01 7mo ago
  > LLMs don't second-guess whether a change is worth submitting, and they certainly don't feel the social pressure of how their contribution might be received. The filter is completely absent. Of course you could have an agent on your side do this, so I take you to mean a LLM that submits a PR and is not instructed to make such a reflection will not intrinsically make it as a human would, that is as a necessary side effect of submitting in the first place (though one might be surprised). It would be curious to have an API that perhaps attempts to validate some attestation about how the submitting LLM's contribution was derived, ie force that reflection at submission time with some reasonable guarantees of veracity even if it had yet to be considered. Perhaps some future API can enforce such a contract among the various LLMs.
pear01 7mo ago
Prioritizing or deferring to existing contributors happens in pretty much every human endeavor. As you point out this of course predates the age of LLM, in many ways it's basic human tribal behavior. This does have its own set of costs and limitations however. Judgement is hard to measure. Humans create sorting bonds that may optimize for prestige or personal ties over strict qualifications or ability. The tribe is useful, but it can also be ugly. Perhaps in a not too distant future, in some domains or projects these sorts of instincts will be rendered obsolete by projects willing to accept any contribution that satisfies enough constraints, thereby trading human judgement for the desired mix of velocity and safety. Perhaps as the agents themselves improve this tension becomes less an act of external constraint but an internal guide. And what would this be, if not a simulation of judgement itself? You could also do it in stages, ie have a delegated agent promote people to some purgatory where there is at least some hope of human intervention to attain the same rights and privileges as pre-existing contributors, that is if said agent deems your attempt worthy enough. Or maybe to fight spam an earnest contributor will have to fork over some digital currency, to essentially pay the cost of requesting admission. All of these scenarios are rather familiar in terms of the history of human social arrangements. That is just to say, there is no destruction of the social contract here. Only another incremental evolution.

by _pdp_ 7mo ago

yabutlivnWoods 7mo ago

Generational churn breaks social contract. You all using Latin and believing in the old Greek gods to honor the dead? Muricans still owning slaves from Africa? All ways in which old social contracts were broken at one point. We are not VHS cassettes with an obligation to play out a fuzzy memory of history.

bluefirebrand 7mo ago

> AI ultimately breaks the social contract Business schools teach that breaking the social contract is a disruption opportunity for growth, not a negative, The Hacker in Hacker News refers to "growth hacking" now, not hacking code

_pdp_ 7mo ago
It depends who you ask. You cannot say that breaking the social contract (the fabric of society, if you will) is generally a good thing, although I am sure some will find opportunities for growth. After all, the phoenix must burn to emerge, but let's not romanticise the fire.
- bluefirebrand 7mo ago
  > You cannot say that breaking the social contract (the fabric of society, if you will) is generally a good thing I am not saying it's a good thing, just that it's a common attitude here I suppose it didn't come through in my original post, but I was trying to be critical

throwaway27448 7mo ago

An agent is still attached to an accountable human. If it is not, ignore it.

jojomodding 7mo ago
How do you figure out which is the case, at scale?
- throwaway27448 7mo ago
  You don't.
SpicyLemonZest 7mo ago
The problem is that it acts as an accountability sink even when it is attached. I've had multiple coworkers over the past few months tell me obvious, verifiable untruths. Six months ago, I would have had a clear term for this: they lied to me. They told me something that wasn't true, that they could not possibly have thought was true, and they did it to manipulate me into doing what they want. I would have demanded and their manager would have agreed that they need to be given a severe talking to. But now I can't call it a lie, both in the sense that I've been instructed not to and in the sense that it subjectively wasn't. They honestly represented what the agent told them was the truth, and they honestly thought that asking an agent to do some exploration was the best way to give me accurate information. What's the replacement norm that will prevent people from "flooding the zone" with false AI-generated claims shaped to get people to do what they want? Even if AI detection tools worked, which I emphasize that they do not, they wouldn't have stopped the incidents that involved human-generated summaries of false AI information.

_pdp_ 7mo ago

I forgot to mention why I brought up the idea of who is making the contribution rather than how (i.e., through an LLM). Right now, the biggest issue open-source maintainers are facing is an ever-increasing supply of PRs. Before coding assistants, those PRs didn't get pushed not because they were never written (although obviously there were fewer in quantity) but because contributors were conscious of how their contributions might be perceived. In many cases, the changes never saw the light of day outside of the fork. LLMs don't second-guess whether a change is worth submitting, and they certainly don't feel the social pressure of how their contribution might be received. The filter is completely absent. So I don't think the question is whether machine-generated code is low quality at all, because that is hard to judge, and frankly coding assistants can certainly produce high-quality code (with guidance). The question is who made the contribution. With rising volumes, we will see an increasing amount of rejections. By the way, we do this too internally. We have a script that deletes LLM-generated PRs automatically after some time. It is just easier and more cost-effective than reviewing the contribution. Also, PRs get rejected for the smallest of reasons. If it doesn't pass the smell test moments after the link is opened, it get's deleted.

pear01 7mo ago
> LLMs don't second-guess whether a change is worth submitting, and they certainly don't feel the social pressure of how their contribution might be received. The filter is completely absent. Of course you could have an agent on your side do this, so I take you to mean a LLM that submits a PR and is not instructed to make such a reflection will not intrinsically make it as a human would, that is as a necessary side effect of submitting in the first place (though one might be surprised). It would be curious to have an API that perhaps attempts to validate some attestation about how the submitting LLM's contribution was derived, ie force that reflection at submission time with some reasonable guarantees of veracity even if it had yet to be considered. Perhaps some future API can enforce such a contract among the various LLMs.

pear01 7mo ago

Prioritizing or deferring to existing contributors happens in pretty much every human endeavor. As you point out this of course predates the age of LLM, in many ways it's basic human tribal behavior. This does have its own set of costs and limitations however. Judgement is hard to measure. Humans create sorting bonds that may optimize for prestige or personal ties over strict qualifications or ability. The tribe is useful, but it can also be ugly. Perhaps in a not too distant future, in some domains or projects these sorts of instincts will be rendered obsolete by projects willing to accept any contribution that satisfies enough constraints, thereby trading human judgement for the desired mix of velocity and safety. Perhaps as the agents themselves improve this tension becomes less an act of external constraint but an internal guide. And what would this be, if not a simulation of judgement itself? You could also do it in stages, ie have a delegated agent promote people to some purgatory where there is at least some hope of human intervention to attain the same rights and privileges as pre-existing contributors, that is if said agent deems your attempt worthy enough. Or maybe to fight spam an earnest contributor will have to fork over some digital currency, to essentially pay the cost of requesting admission. All of these scenarios are rather familiar in terms of the history of human social arrangements. That is just to say, there is no destruction of the social contract here. Only another incremental evolution.