5 ms·

I honestly believe the best to make AI responsibly is to make it open source. That way no single entity has total control over it, and researchers can study the

I honestly believe the best to make AI responsibly is to make it open source. That way no single entity has total control over it, and researchers can study them to better understand how they can be used nefariously as well as in a good way—doing that allows us to build defenses to minimize the risks, and reap the benefits. Meta is already doing that, but other companies and organizations should do that as well.

Ericson2314 3y ago
Getting the results is nice but that's "shareware" not "free software" (or, for a more modern example, that is like companies submitting firmware binary blobs into mainline Linux). Free software means you have to be able to build the final binary from source. Having 10 TB of text is no problem, but having a data center of GPUs is. Until the training cost comes down there is no way to make it free software.
- l33t7332273 3y ago
  If I publish a massive quantity of source code — to the point that it’s very expensive to compile — it’s still open source. If the training data and model training code is available then it should be considered open, even if it’s hard to train.
  - Ericson2314 3y ago
    If it was only feasible for a giant corporation to compile the code, I would consider it less than open source.
  - nextaccountic 3y ago
    > the training data This will never be fully open
    - l33t7332273 3y ago
      Maybe not for some closed models. That doesn’t mean truly open models can’t exist.
  - earthnail 3y ago
    I doubt you’d say that if one run of compiling the code would cost you $400M.
- PeterisP 3y ago
  Free software means that you have the ability - both legal and practical - to customize the tool for your needs. For software, that means you have to be able to build the final binary from source (so you can adapt the source and rebuild), for ML models that means you need the code and the model weights, which does allow you to fine-tune that model and adapt it to different purposes even without spending the compute cost for a full re-train.
aaaaaadddaz2 3y ago
[flagged]
gjsman-1000 3y ago
Great, Russia and China get the ability to use it or adapt it for any reason they want without any oversight.
- bcherny 3y ago
  One could argue that open source won’t change much with regard to China and Russia. Both countries have access to LLMs already. And if they didn’t, they would have built their own or gotten access through corporate espionage. What open source does is it helps us better understand & control the tech these countries use. And it helps level up our own homegrown tech. Both of these are good advantages to have.
  - mdhb 3y ago
    That last paragraph is an opinion you seem to have just formed as you typed it stated as a fact that doesn’t seem to hold up to even the lightest scrutiny.
- malwrar 3y ago
  There is no obvious reason they couldn't just train one themselves, or merely steal existing weights given enough time.
  - esafak 3y ago
    That is precious time that can be used to work on alignment.
    - bigfudge 3y ago
      But alignment is always going to rely on cooperation of users though? What benefit does the delay offer other than the direct one of a delay?
      - esafak 3y ago
        Why is it going to rely on co-operation if users don't have the means to change the model enough to misalign it?
deanCommie 3y ago
That's not necessarily true. It's entirely conceivable that even if AGI (or something comparably significant in terms of how impactful it would be to changing society or nation states) was achievable in our lifetime, it might be that: 1) Achieving it requires a critical mass of research talent in one place that perhaps currently exists at fewer than 5 companies - anecdotally only Google, Meta, and OpenAI. And a comparable number of world governments (At least in the US the best researchers in this field are at these companies, not in academia or government. China may be different.) This makes it sound like a "security by obscurity" situation, and on a long enough timeline it may be. Without World War 2, without the Manhattan Project, and without the looming Cold War how long would it have taken for Humanity to construct a nuclear bomb? An extra 10 years? 20? 50? Hard to know. Regardless, there is a possibility that for things like AI, with extra time comes the ability to better understand and build those defenses before they're needed. 2) It might also require an amount of computing capacity that only a dozen companies/governments have. If you open source all the work you remove the guard rails for the growth or what people focus investments on. It also means that hostile nations like Iran or North Korea who may not have the research talent but could acquire the raw compute could utilize it for unknown goals. Not to mention that what nefarious parties on the internet would use it for. We only know about deep fake porn and generated vocal audio of family members for extortion. Things can get much much worse.
- airgapstopgap 3y ago
  > there is a possibility that for things like AI, with extra time comes the ability to better understand and build those defenses before they're needed. Or not, and damaging wrongheaded ideas will become a self-reinforcing (because safety! humanity is at stake!) orthodoxy, leaving us completely butt-naked before actual risks once somebody makes a sudden clandestine breakthrough. https://bounded-regret.ghost.io/ai-pause-will-likely-backfire-by-nora/ https://bounded-regret.ghost.io/ai-pause-will-likely-backfir... > We don’t need to speculate about what would happen to AI alignment research during a pause—we can look at the historical record. Before the launch of GPT-3 in 2020, the alignment community had nothing even remotely like a general intelligence to empirically study, and spent its time doing theoretical research, engaging in philosophical arguments on LessWrong, and occasionally performing toy experiments in reinforcement learning. > The Machine Intelligence Research Institute (MIRI), which was at the forefront of theoretical AI safety research during this period, has since admitted that its efforts have utterly failed. Other agendas, such as “assistance games”, are still being actively pursued but have not been significantly integrated into modern deep learning systems— see Rohin Shah’s review here, as well as Alex Turner’s comments here. Finally, Nick Bostrom’s argument in Superintelligence, that value specification is the fundamental challenge to safety, seems dubious in light of LLM's ability to perform commonsense reasoning.[2] > At best, these theory-first efforts did very little to improve our understanding of how to align powerful AI. And they may have been net negative, insofar as they propagated a variety of actively misleading ways of thinking both among alignment researchers and the broader public. Some examples include the now-debunked analogy from evolution, the false distinction between “inner” and “outer” alignment, and the idea that AIs will be rigid utility maximizing consequentialists (here, here, and here). > During an AI pause, I expect alignment research would enter another “winter” in which progress stalls, and plausible-sounding-but-false speculations become entrenched as orthodoxy without empirical evidence to falsify them. While some good work would of course get done, it’s not clear that the field would be better off as a whole. And even if a pause would be net positive for alignment research, it would likely be net negative for humanity’s future all things considered, due to the pause’s various unintended consequences. We’ll look at that in detail in the final section of the essay.
kazinator 3y ago
GNU/Linux is open source. Is it being used responsibly? What is the "it" that no single entity has control over? You have absolutely no control of what your next door neighbor is doing with open source. Hey, if we want alcohol to be made responsibly, everyone should have their own still, made from freely redistributed blueprints. That way no single entity has control.
- JoshuaDavid 3y ago
  > Hey, if we want alcohol to be made responsibly, everyone should have their own still, made from freely redistributed blueprints. Anyone who wants to can, in fact, find blueprints for making their own still. For example, https://moonshinestillplans.com/ https://moonshinestillplans.com/ contains plans for a variety of different types of stills and guidance on which type to build based on how you want to use it. And in fact I think it's good that this site exists, because it's very easy to build a still that appears to work but actually leaves you with a high-methanol end product.
  - code_biologist 3y ago
    it's very easy to build a still that appears to work but actually leaves you with a high-methanol end product. Is it? I've always seen concern about methanol in moonshine but I presume it came from intentional contamination from evil bootleggers. It's difficult to get a wash containing enough methanol to meaningfully concentrate in the first place if you're making whiskey or rum. Maybe with fruit wine and hard cider there's a bit more. The physics of distillation kind of have your back here too. The lower temperature fractions with acetone and methanol always come out first during distillation (the "heads") and every resource and distiller will tell you to learn the taste and smell, then throw them out. The taste and smell of heads are really distinctive. A slow distillation to more effectively concentrate methanol also makes it easier to separate out. But even if you don't separate the heads from the hearts, the methanol in any traditional wash is dilute enough that it'll only give you a headache. I think it's extremely hard to build a still that appears to work but creates a high methanol end product.
    - 3y ago
absrec 3y ago
Exactly. The biggest question is why you would trust the single authority controlling the AI to be responsible. If there are enough random variables the good and the bad sort of cancel each other out to reach a happy neutral. But if an authority goes rogue what are you gonna do? Making it open is the only way AI fulfills a power to the people goal. Without open source and locally trainable models AI is just more power to the big-tech industry's authorities.
systemvoltage 3y ago
Is it just the model that needs to be open source? I thought the big secret sauce is the sources of data that is used to train the models. Without this, the model itself is useless quite literally.
- dragonwriter 3y ago
  No, the model is useful without the dataset, but its not functionally "open source", because while you can tune it if you have the training code, you can't replicate it or, more important, train it from scratch with a modified, but not completely new, dataset. (And, also, understanding the existing training data helps understand how to structure data to train that particular model, whether its with a new or modified data set from scratch, or for finetuning.) At least, that's my understanding.
- PeterisP 3y ago
  For various industry-specific or specialized task models (e.g. recognizing dangerous events in self-driving car scenario) having appropriate data is often the big secret sauce, however, for the specific case of LLMs there are reasonable sets of sufficiently large data available to the public, and even the specific RLHF adaptations aren't a limiting secret sauce because there are techniques to extract them from the available commercial models.
jeffreygoesto 3y ago
If it really is A"I", shouldn't it figure out for itself and do it?
reocha 3y ago
https://j3s.sh/thought/drones-run-linux-free-software-isnt-enough.html https://j3s.sh/thought/drones-run-linux-free-software-isnt-e...
martindbp 3y ago
I'm not a doomer but I honestly don't understand this argument. If releasing model as open source helps researchers determine if it's safe, what about when it's not deemed safe? Then it's already out there, on the hard drives of half of 4chan. It's much easier and cheaper to fine-tune a model, distil and quantize it and put it on a killer drone, than it is to train it from scratch. On the other hand I totally relate with the idea that it could be preferable that everyday has access to advanced AI and not just large companies and nation states.
- 123yawaworht456 3y ago
  what purpose does a LLM serve on a killing drone, exactly?
  - martindbp 3y ago
    Open source models in general. Meta has for instance released DINO which is a self supervised transformer model. LLMs are also going multi modal (see LLaVA for instance). The name "LLM" has stuck but they should really be called Large Transformer Models. LeCun is working on self supervised visual world models (I-JEPA) which if successful and released could form the basis for killer drones. It's still a lot of engineering work to fine tune and put a model like this on embedded hardware on a drone, but at some point it might be easy enough for small groups of determined people to pull it off.
    - Vetch 3y ago
      For a drone, an LLM derived solution is far too slow, unreliable, heavy and not fit for purpose. Developments in areas like optical flow, better small CNNs for vision, adaptive control and sensor fusion are what's needed. When neural networks are used, they are small, fast, specialized and cheap to train. A multimodal or segmentation algorithm is not the solution for bee-level path planning, obstacle avoidance or autonomous navigation. Getting LLMs to power a robot for household tasks with low latency to action and in an energy efficient manner is challenging enough, before talking about high-speed, highly maneuverable drones.

Ericson2314 3y ago

Getting the results is nice but that's "shareware" not "free software" (or, for a more modern example, that is like companies submitting firmware binary blobs into mainline Linux). Free software means you have to be able to build the final binary from source. Having 10 TB of text is no problem, but having a data center of GPUs is. Until the training cost comes down there is no way to make it free software.

l33t7332273 3y ago
If I publish a massive quantity of source code — to the point that it’s very expensive to compile — it’s still open source. If the training data and model training code is available then it should be considered open, even if it’s hard to train.
- Ericson2314 3y ago
  If it was only feasible for a giant corporation to compile the code, I would consider it less than open source.
- nextaccountic 3y ago
  > the training data This will never be fully open
  - l33t7332273 3y ago
    Maybe not for some closed models. That doesn’t mean truly open models can’t exist.
- earthnail 3y ago
  I doubt you’d say that if one run of compiling the code would cost you $400M.
PeterisP 3y ago
Free software means that you have the ability - both legal and practical - to customize the tool for your needs. For software, that means you have to be able to build the final binary from source (so you can adapt the source and rebuild), for ML models that means you need the code and the model weights, which does allow you to fine-tune that model and adapt it to different purposes even without spending the compute cost for a full re-train.

aaaaaadddaz2 3y ago

[flagged]

gjsman-1000 3y ago

Great, Russia and China get the ability to use it or adapt it for any reason they want without any oversight.

bcherny 3y ago
One could argue that open source won’t change much with regard to China and Russia. Both countries have access to LLMs already. And if they didn’t, they would have built their own or gotten access through corporate espionage. What open source does is it helps us better understand & control the tech these countries use. And it helps level up our own homegrown tech. Both of these are good advantages to have.
- mdhb 3y ago
  That last paragraph is an opinion you seem to have just formed as you typed it stated as a fact that doesn’t seem to hold up to even the lightest scrutiny.
malwrar 3y ago
There is no obvious reason they couldn't just train one themselves, or merely steal existing weights given enough time.
- esafak 3y ago
  That is precious time that can be used to work on alignment.
  - bigfudge 3y ago
    But alignment is always going to rely on cooperation of users though? What benefit does the delay offer other than the direct one of a delay?
    - esafak 3y ago
      Why is it going to rely on co-operation if users don't have the means to change the model enough to misalign it?

deanCommie 3y ago

That's not necessarily true. It's entirely conceivable that even if AGI (or something comparably significant in terms of how impactful it would be to changing society or nation states) was achievable in our lifetime, it might be that: 1) Achieving it requires a critical mass of research talent in one place that perhaps currently exists at fewer than 5 companies - anecdotally only Google, Meta, and OpenAI. And a comparable number of world governments (At least in the US the best researchers in this field are at these companies, not in academia or government. China may be different.) This makes it sound like a "security by obscurity" situation, and on a long enough timeline it may be. Without World War 2, without the Manhattan Project, and without the looming Cold War how long would it have taken for Humanity to construct a nuclear bomb? An extra 10 years? 20? 50? Hard to know. Regardless, there is a possibility that for things like AI, with extra time comes the ability to better understand and build those defenses before they're needed. 2) It might also require an amount of computing capacity that only a dozen companies/governments have. If you open source all the work you remove the guard rails for the growth or what people focus investments on. It also means that hostile nations like Iran or North Korea who may not have the research talent but could acquire the raw compute could utilize it for unknown goals. Not to mention that what nefarious parties on the internet would use it for. We only know about deep fake porn and generated vocal audio of family members for extortion. Things can get much much worse.

airgapstopgap 3y ago
> there is a possibility that for things like AI, with extra time comes the ability to better understand and build those defenses before they're needed. Or not, and damaging wrongheaded ideas will become a self-reinforcing (because safety! humanity is at stake!) orthodoxy, leaving us completely butt-naked before actual risks once somebody makes a sudden clandestine breakthrough. https://bounded-regret.ghost.io/ai-pause-will-likely-backfire-by-nora/ https://bounded-regret.ghost.io/ai-pause-will-likely-backfir... > We don’t need to speculate about what would happen to AI alignment research during a pause—we can look at the historical record. Before the launch of GPT-3 in 2020, the alignment community had nothing even remotely like a general intelligence to empirically study, and spent its time doing theoretical research, engaging in philosophical arguments on LessWrong, and occasionally performing toy experiments in reinforcement learning. > The Machine Intelligence Research Institute (MIRI), which was at the forefront of theoretical AI safety research during this period, has since admitted that its efforts have utterly failed. Other agendas, such as “assistance games”, are still being actively pursued but have not been significantly integrated into modern deep learning systems— see Rohin Shah’s review here, as well as Alex Turner’s comments here. Finally, Nick Bostrom’s argument in Superintelligence, that value specification is the fundamental challenge to safety, seems dubious in light of LLM's ability to perform commonsense reasoning.[2] > At best, these theory-first efforts did very little to improve our understanding of how to align powerful AI. And they may have been net negative, insofar as they propagated a variety of actively misleading ways of thinking both among alignment researchers and the broader public. Some examples include the now-debunked analogy from evolution, the false distinction between “inner” and “outer” alignment, and the idea that AIs will be rigid utility maximizing consequentialists (here, here, and here). > During an AI pause, I expect alignment research would enter another “winter” in which progress stalls, and plausible-sounding-but-false speculations become entrenched as orthodoxy without empirical evidence to falsify them. While some good work would of course get done, it’s not clear that the field would be better off as a whole. And even if a pause would be net positive for alignment research, it would likely be net negative for humanity’s future all things considered, due to the pause’s various unintended consequences. We’ll look at that in detail in the final section of the essay.

kazinator 3y ago

GNU/Linux is open source. Is it being used responsibly? What is the "it" that no single entity has control over? You have absolutely no control of what your next door neighbor is doing with open source. Hey, if we want alcohol to be made responsibly, everyone should have their own still, made from freely redistributed blueprints. That way no single entity has control.

JoshuaDavid 3y ago
> Hey, if we want alcohol to be made responsibly, everyone should have their own still, made from freely redistributed blueprints. Anyone who wants to can, in fact, find blueprints for making their own still. For example, https://moonshinestillplans.com/ https://moonshinestillplans.com/ contains plans for a variety of different types of stills and guidance on which type to build based on how you want to use it. And in fact I think it's good that this site exists, because it's very easy to build a still that appears to work but actually leaves you with a high-methanol end product.
- code_biologist 3y ago
  it's very easy to build a still that appears to work but actually leaves you with a high-methanol end product. Is it? I've always seen concern about methanol in moonshine but I presume it came from intentional contamination from evil bootleggers. It's difficult to get a wash containing enough methanol to meaningfully concentrate in the first place if you're making whiskey or rum. Maybe with fruit wine and hard cider there's a bit more. The physics of distillation kind of have your back here too. The lower temperature fractions with acetone and methanol always come out first during distillation (the "heads") and every resource and distiller will tell you to learn the taste and smell, then throw them out. The taste and smell of heads are really distinctive. A slow distillation to more effectively concentrate methanol also makes it easier to separate out. But even if you don't separate the heads from the hearts, the methanol in any traditional wash is dilute enough that it'll only give you a headache. I think it's extremely hard to build a still that appears to work but creates a high methanol end product.
  - 3y ago

absrec 3y ago

Exactly. The biggest question is why you would trust the single authority controlling the AI to be responsible. If there are enough random variables the good and the bad sort of cancel each other out to reach a happy neutral. But if an authority goes rogue what are you gonna do? Making it open is the only way AI fulfills a power to the people goal. Without open source and locally trainable models AI is just more power to the big-tech industry's authorities.

systemvoltage 3y ago

Is it just the model that needs to be open source? I thought the big secret sauce is the sources of data that is used to train the models. Without this, the model itself is useless quite literally.

dragonwriter 3y ago
No, the model is useful without the dataset, but its not functionally "open source", because while you can tune it if you have the training code, you can't replicate it or, more important, train it from scratch with a modified, but not completely new, dataset. (And, also, understanding the existing training data helps understand how to structure data to train that particular model, whether its with a new or modified data set from scratch, or for finetuning.) At least, that's my understanding.
PeterisP 3y ago
For various industry-specific or specialized task models (e.g. recognizing dangerous events in self-driving car scenario) having appropriate data is often the big secret sauce, however, for the specific case of LLMs there are reasonable sets of sufficiently large data available to the public, and even the specific RLHF adaptations aren't a limiting secret sauce because there are techniques to extract them from the available commercial models.

jeffreygoesto 3y ago

If it really is A"I", shouldn't it figure out for itself and do it?

reocha 3y ago

https://j3s.sh/thought/drones-run-linux-free-software-isnt-enough.html https://j3s.sh/thought/drones-run-linux-free-software-isnt-e...

martindbp 3y ago

I'm not a doomer but I honestly don't understand this argument. If releasing model as open source helps researchers determine if it's safe, what about when it's not deemed safe? Then it's already out there, on the hard drives of half of 4chan. It's much easier and cheaper to fine-tune a model, distil and quantize it and put it on a killer drone, than it is to train it from scratch. On the other hand I totally relate with the idea that it could be preferable that everyday has access to advanced AI and not just large companies and nation states.

123yawaworht456 3y ago
what purpose does a LLM serve on a killing drone, exactly?
- martindbp 3y ago
  Open source models in general. Meta has for instance released DINO which is a self supervised transformer model. LLMs are also going multi modal (see LLaVA for instance). The name "LLM" has stuck but they should really be called Large Transformer Models. LeCun is working on self supervised visual world models (I-JEPA) which if successful and released could form the basis for killer drones. It's still a lot of engineering work to fine tune and put a model like this on embedded hardware on a drone, but at some point it might be easy enough for small groups of determined people to pull it off.
  - Vetch 3y ago
    For a drone, an LLM derived solution is far too slow, unreliable, heavy and not fit for purpose. Developments in areas like optical flow, better small CNNs for vision, adaptive control and sensor fusion are what's needed. When neural networks are used, they are small, fast, specialized and cheap to train. A multimodal or segmentation algorithm is not the solution for bee-level path planning, obstacle avoidance or autonomous navigation. Getting LLMs to power a robot for household tasks with low latency to action and in an energy efficient manner is challenging enough, before talking about high-speed, highly maneuverable drones.