10 ms·

While Stallman may have been originally upset about his printer, the primary motivation behind open source software is: Nothing up my sleeve. This of course r

While Stallman may have been originally upset about his printer, the primary motivation behind open source software is: Nothing up my sleeve. This of course requires audit and that requires that the code is written for human consumption, otherwise nobody will bother. Sure you can vibe a printer driver but how sure are you that your LLM didn't include a backdoor in the millions line of slop?

williamcotton 6d ago
How sure are you that any printer driver didn’t include a backdoor? What’s the difference between the LLM code with the backdoor and the human code with the backdoor?
- za_creature 6d ago
  The ability to audit the code. If not that, the ability to trust the reputation of the author which creates an incentive not to willingly insert a backdoor. Yes, `npm install` was always bullshit because most people didn't bother to check, which is exactly why it was exploited multiple times, which created a conversation about "supply chain security". If you want to argue that "npm changed the world" then you are correct. It did not change the world for the better though.
  - williamcotton 6d ago
    Why is LLM code always harder to audit? So LLMs might decide to add backdoors without prompting?
    - za_creature 6d ago
      I'm less worried about what LLMs decide and more about what their system prompts tell them to do.
rpdillon 6d ago
Trusting software produced by others is not a new problem; I consider AI to be a tool, and so my techniques for establishing trust are the same as they have always been. I look at the person wielding the tool.
- za_creature 6d ago
  Then you best hope that person has read https://people.cs.umass.edu/~emery/classes/cmpsci691st/readings/Sec/Reflections-on-Trusting-Trust.pdf https://people.cs.umass.edu/~emery/classes/cmpsci691st/readi...
- ben_w 6d ago
  > I look at the person wielding the tool. I look at the whole chain of influence behind them, and it's quite a lot more upsetting than the smiling face I interact with. The whole "AI is a Lovecraftian tentacle monster wearing a smiley face" thing applies to simple bureaucracies (replace Lovecraft with Kafka), to corporations (replace Lovecraft with IDK, most anarchists?), and governments (Orwell?) Tools made by tools made by tools, along more steps than most people know even when their job is one of them. Somewhere there's a kid working a dangerous mine without the right safety equipment, elsewhere there's a sweatshop, another place a "reeducation camp". But who do I see? A cashier, mostly. Someone whose job involves smiling to customers even when we're idiots.
  - rpdillon 6d ago
    I'm not sure what point you're trying to make. I put trust in individual people, not organizations. You seem to be making a point about supply chains, but I'm not sure how it relates to the point about trust. You seem to be making a different point about some kind of exploitation.
    - ben_w 6d ago
      Almost everything I interact with was made by an organisation (or a disorganisation), not by an individual. Say I download an app. Who made it? The programmer? Their PM? Apple's store requirements? The US government, for whom there was a special tickbox I had to agree to last time I uploaded an app? Trust is for the mechanic, driver, builder; but for 90% of my interactions I have to trust my government set good rules and other people followed them. I can't do this with AI, neither good rules nor them being followed, but I also can't do it when the OS company and app devs are foreign, as they generally are to me now.
mitxela 6d ago
That's a motivation for source available. You can have commercial copyrighted software that's NUMS. You can deliver the customer source code (slightly customized for traitor-telling of course) and tell them to do what they want but never share it. grsecurity even managed this with GPL software!

williamcotton 6d ago

How sure are you that any printer driver didn’t include a backdoor? What’s the difference between the LLM code with the backdoor and the human code with the backdoor?

za_creature 6d ago
The ability to audit the code. If not that, the ability to trust the reputation of the author which creates an incentive not to willingly insert a backdoor. Yes, `npm install` was always bullshit because most people didn't bother to check, which is exactly why it was exploited multiple times, which created a conversation about "supply chain security". If you want to argue that "npm changed the world" then you are correct. It did not change the world for the better though.
- williamcotton 6d ago
  Why is LLM code always harder to audit? So LLMs might decide to add backdoors without prompting?
  - za_creature 6d ago
    I'm less worried about what LLMs decide and more about what their system prompts tell them to do.

rpdillon 6d ago

Trusting software produced by others is not a new problem; I consider AI to be a tool, and so my techniques for establishing trust are the same as they have always been. I look at the person wielding the tool.

za_creature 6d ago
Then you best hope that person has read https://people.cs.umass.edu/~emery/classes/cmpsci691st/readings/Sec/Reflections-on-Trusting-Trust.pdf https://people.cs.umass.edu/~emery/classes/cmpsci691st/readi...
ben_w 6d ago
> I look at the person wielding the tool. I look at the whole chain of influence behind them, and it's quite a lot more upsetting than the smiling face I interact with. The whole "AI is a Lovecraftian tentacle monster wearing a smiley face" thing applies to simple bureaucracies (replace Lovecraft with Kafka), to corporations (replace Lovecraft with IDK, most anarchists?), and governments (Orwell?) Tools made by tools made by tools, along more steps than most people know even when their job is one of them. Somewhere there's a kid working a dangerous mine without the right safety equipment, elsewhere there's a sweatshop, another place a "reeducation camp". But who do I see? A cashier, mostly. Someone whose job involves smiling to customers even when we're idiots.
- rpdillon 6d ago
  I'm not sure what point you're trying to make. I put trust in individual people, not organizations. You seem to be making a point about supply chains, but I'm not sure how it relates to the point about trust. You seem to be making a different point about some kind of exploitation.
  - ben_w 6d ago
    Almost everything I interact with was made by an organisation (or a disorganisation), not by an individual. Say I download an app. Who made it? The programmer? Their PM? Apple's store requirements? The US government, for whom there was a special tickbox I had to agree to last time I uploaded an app? Trust is for the mechanic, driver, builder; but for 90% of my interactions I have to trust my government set good rules and other people followed them. I can't do this with AI, neither good rules nor them being followed, but I also can't do it when the OS company and app devs are foreign, as they generally are to me now.

mitxela 6d ago

That's a motivation for source available. You can have commercial copyrighted software that's NUMS. You can deliver the customer source code (slightly customized for traitor-telling of course) and tell them to do what they want but never share it. grsecurity even managed this with GPL software!