4 ms·

I used LLM because the people shouting the loudest come from a LLM company which claimed their newest language model can be used to create bioweapons in their w

I used LLM because the people shouting the loudest come from a LLM company which claimed their newest language model can be used to create bioweapons in their whitepaper. Semantics aside the recent interest in AI risk was clearly stimulated by LLMs and the camp that believes this is the path to AGI which may or may not be true depending who you ask.

ben_w 3y ago
I can only imagine Eleizer Yudkowsky and Rob Miles looking on this conversation with a depressed scream and a facepalm respectively. They've both been loudly concerned about optimisers doing over-optimisation, and society having a Nash equilibrium where everyone's using them as hard as possible regardless of errors, since before it was cool.
- haldujai 3y ago
  While true the ones doing media tours and speaking the most vocally in May 2023 are the LLM crowd. I don’t think it’s a mischaracterization to say OpenAI has sparked public debate on this topic.
  - ben_w 3y ago
    To the extent that this may be true (I've not exactly studied which public thinkpiece writers care about which AI so it might easily be the image generators that get this crown for all I know), that's because ChatGPT actually does something that a normal person understands. A paper titled "Dual use of artificial-intelligence-powered drug discovery" (published last year) got a few angst pieces and is mostly forgotten by the general public and media so far as I can tell; but the people behind it both talked directly to regulators and other labs to help advise them how many other precursor chemicals were now potential threats, and also went onto the usual podcasts and other public forums to raise awareness of the risk to other AI researchers. The story behind that was "ugh, they want us to think about risks… what if we ask it to find dangerous chemicals instead of safe ones? *overnight* oh no!"
- bnralt 3y ago
  > I can only imagine Eleizer Yudkowsky and Rob Miles looking on this conversation with a depressed scream and a facepalm respectively. Whenever Yudkowsky comes up on my Twitter feed I'm left with an impression that I'm not going to have any more luck conversing AI with those in his orbit than I am discussing the rapture with a fundamentalist Christian. For example, the following Tweet[1]. If a person believes this is from a deep thinker that should be taken very seriously rather than an unhinged nutcase, our worldviews are probably too far apart to ever reach a common understanding: > Fools often misrepresent me as saying that superintelligence can do anything because magic. To clearly show this false, here's a concrete list of stuff I expect superintelligence can or can't do: > - FTL (faster than light) travel: DEFINITE NO > - Find some hack for going >50 OOM past the amount of computation that naive calculations of available negentropy would suggest is possible within our local volume: PROBABLE NO > - Validly prove in first-order arithmetic that 1 + 1 = 5: DEFINITE NO > - Prove a contradiction from Zermelo-Frankel set theory: PROBABLE NO > - Using current human technology, synthesize a normal virus (meaning it has to reproduce itself inside human cells and is built of conventional bio materials) that infects over 50% of the world population within a month: YES > (note, this is not meant as an argument, this is meant as a concrete counterexample to people who claim 'lol doomers think AI can do anything just because its smart' showing that I rather have some particular model of what I roughly wildly guess to be a superintelligence's capability level) > - Using current human technology, synthesize a normal virus that infects 90% of Earth within an hour: NO > - Write a secure operating system on the first try, zero errors, no debugging phase, assuming away Meltdown-style hardware vulnerabilities in the chips: DEFINITE YES > - Write a secure operating system for actual modern hardware, on the first pass: YES > - Train an AI system with capability at least equivalent to GPT-4, from the same dataset GPT-4 used, starting from at most 50K of Python code, using 1000x less compute than was used to train GPT-4: YES > - Starting from current human tech, bootstrap to nanotechnology in a week: YES > - Starting from current human tech, bootstrap to nanotechnology in an hour: GOSH WOW IDK, I DON'T ACTUALLY KNOW HOW, BUT DON'T WANT TO CLAIM I CAN SEE ALL PATHWAYS, THIS ONE IS REALLY HARD FOR ME TO CALL, BRAIN LEGIT DOESN'T FEEL GOOD BETTING EITHER WAY, CALL IT 50:50?? > - Starting from current human tech and from the inside of a computer, bootstrap to nanotechnology in a minute: PROBABLE NO, EVEN IF A MINUTE IS LIKE 20 SUBJECTIVE YEARS TO THE SI > - Bootstrap to nanotechnology via a clean called shot: all the molecular interactions go as predicted the first time, no error-correction rounds needed: PROBABLY YES but please note this is not any kind of necessary assumption because It could just build Its own fucking lab, get back the observations, and do a debugging round; and none of the processes there intrinsically need to run at the speed of humans taking hourly bathroom breaks, it can happen at the speed of protein chemistry and electronics. Please consider asking for 6 seconds how a superintelligence might possibly overcome such incredible obstacles of 'I think you need a positive nonzero number of observations', for example, by doing a few observations, and then further asking yourself if those observations absolutely have to be slow like a sloth > - Bootstrap to nanotechnology by any means including a non-called shot where the SI designs more possible proteins than It needs to handle some of the less certain cases, and gets back some preliminary observations about how they interacted in a liquid medium, before it actually puts together the wetware lab on round 2: YES (The Tweet goes on, you can read the rest of it at the link below, but that should give you the gist.) [1] https://twitter.com/ESYudkowsky/status/1658616828741160960 https://twitter.com/ESYudkowsky/status/1658616828741160960

ben_w 3y ago

I can only imagine Eleizer Yudkowsky and Rob Miles looking on this conversation with a depressed scream and a facepalm respectively. They've both been loudly concerned about optimisers doing over-optimisation, and society having a Nash equilibrium where everyone's using them as hard as possible regardless of errors, since before it was cool.

haldujai 3y ago
While true the ones doing media tours and speaking the most vocally in May 2023 are the LLM crowd. I don’t think it’s a mischaracterization to say OpenAI has sparked public debate on this topic.
- ben_w 3y ago
  To the extent that this may be true (I've not exactly studied which public thinkpiece writers care about which AI so it might easily be the image generators that get this crown for all I know), that's because ChatGPT actually does something that a normal person understands. A paper titled "Dual use of artificial-intelligence-powered drug discovery" (published last year) got a few angst pieces and is mostly forgotten by the general public and media so far as I can tell; but the people behind it both talked directly to regulators and other labs to help advise them how many other precursor chemicals were now potential threats, and also went onto the usual podcasts and other public forums to raise awareness of the risk to other AI researchers. The story behind that was "ugh, they want us to think about risks… what if we ask it to find dangerous chemicals instead of safe ones? *overnight* oh no!"
bnralt 3y ago
> I can only imagine Eleizer Yudkowsky and Rob Miles looking on this conversation with a depressed scream and a facepalm respectively. Whenever Yudkowsky comes up on my Twitter feed I'm left with an impression that I'm not going to have any more luck conversing AI with those in his orbit than I am discussing the rapture with a fundamentalist Christian. For example, the following Tweet[1]. If a person believes this is from a deep thinker that should be taken very seriously rather than an unhinged nutcase, our worldviews are probably too far apart to ever reach a common understanding: > Fools often misrepresent me as saying that superintelligence can do anything because magic. To clearly show this false, here's a concrete list of stuff I expect superintelligence can or can't do: > - FTL (faster than light) travel: DEFINITE NO > - Find some hack for going >50 OOM past the amount of computation that naive calculations of available negentropy would suggest is possible within our local volume: PROBABLE NO > - Validly prove in first-order arithmetic that 1 + 1 = 5: DEFINITE NO > - Prove a contradiction from Zermelo-Frankel set theory: PROBABLE NO > - Using current human technology, synthesize a normal virus (meaning it has to reproduce itself inside human cells and is built of conventional bio materials) that infects over 50% of the world population within a month: YES > (note, this is not meant as an argument, this is meant as a concrete counterexample to people who claim 'lol doomers think AI can do anything just because its smart' showing that I rather have some particular model of what I roughly wildly guess to be a superintelligence's capability level) > - Using current human technology, synthesize a normal virus that infects 90% of Earth within an hour: NO > - Write a secure operating system on the first try, zero errors, no debugging phase, assuming away Meltdown-style hardware vulnerabilities in the chips: DEFINITE YES > - Write a secure operating system for actual modern hardware, on the first pass: YES > - Train an AI system with capability at least equivalent to GPT-4, from the same dataset GPT-4 used, starting from at most 50K of Python code, using 1000x less compute than was used to train GPT-4: YES > - Starting from current human tech, bootstrap to nanotechnology in a week: YES > - Starting from current human tech, bootstrap to nanotechnology in an hour: GOSH WOW IDK, I DON'T ACTUALLY KNOW HOW, BUT DON'T WANT TO CLAIM I CAN SEE ALL PATHWAYS, THIS ONE IS REALLY HARD FOR ME TO CALL, BRAIN LEGIT DOESN'T FEEL GOOD BETTING EITHER WAY, CALL IT 50:50?? > - Starting from current human tech and from the inside of a computer, bootstrap to nanotechnology in a minute: PROBABLE NO, EVEN IF A MINUTE IS LIKE 20 SUBJECTIVE YEARS TO THE SI > - Bootstrap to nanotechnology via a clean called shot: all the molecular interactions go as predicted the first time, no error-correction rounds needed: PROBABLY YES but please note this is not any kind of necessary assumption because It could just build Its own fucking lab, get back the observations, and do a debugging round; and none of the processes there intrinsically need to run at the speed of humans taking hourly bathroom breaks, it can happen at the speed of protein chemistry and electronics. Please consider asking for 6 seconds how a superintelligence might possibly overcome such incredible obstacles of 'I think you need a positive nonzero number of observations', for example, by doing a few observations, and then further asking yourself if those observations absolutely have to be slow like a sloth > - Bootstrap to nanotechnology by any means including a non-called shot where the SI designs more possible proteins than It needs to handle some of the less certain cases, and gets back some preliminary observations about how they interacted in a liquid medium, before it actually puts together the wetware lab on round 2: YES (The Tweet goes on, you can read the rest of it at the link below, but that should give you the gist.) [1] https://twitter.com/ESYudkowsky/status/1658616828741160960 https://twitter.com/ESYudkowsky/status/1658616828741160960