5 ms·

Great work as usual. I was pretty upset seeing the superalignment team dissolve at OpenAI, but as is typical for the AI space, the news of one day was quickly

by kromem 2y ago

Great work as usual. I was pretty upset seeing the superalignment team dissolve at OpenAI, but as is typical for the AI space, the news of one day was quickly eclipsed by the next day. Anthropic are really killing it right now, and it's very refreshing seeing their commitment to publishing novel findings. I hope this finally serves as the nail in the coffin on the "it's just fancy autocomplete" and "it doesn't understand what it's saying, bro" rhetoric.

Workaccount2 2y ago
> on the "it's just fancy autocomplete" and "it doesn't understand what it's saying, bro" rhetoric. No matter what, there will always be a group of people saying that. The power and drive of the brain to convince itself that it is weaved of magical energy on a divine substrate shouldn't be underestimated. Especially when media plays so hard into that idea (the robots that lose the war because they cannot overcome love, etc.) because brains really love being told they are right. I am almost certain that the first conscious silicon (or whatever material) will be subjected to immense suffering until a new generation that can accept the human brains banality can move things forward.
- ben_w 2y ago
  It tickles me somewhat to note that people using the phrase "stochastic parrot" are demonstrating in themselves the exact behaviour for which they are dismissive of the LLMs. > I am almost certain that the first conscious silicon (or whatever material) will be subjected to immense suffering until a new generation that can accept the human brains banality can move things forward. Indeed, though as we don't know what we're doing (and have 40 definitions of "consciousness" and no way to test for qualia), I would add that the first AI we make with these properties, will likely suffer from every permutation of severe and mild mental heath disorder that is logically possible, including many we have no word for because they would be incompatible with life if found in an organic brain.
astrange 2y ago
I think the research is good, but it's disappointing that they hype it by claiming it's going to help their basically entirely fictional "AI safety" project, as if the bits in their model are going to come alive and eat them.
- ben_w 2y ago
  We just had a pandemic made from a non-living virus that was basically trying to eat us. To riff off the quote: The virus does not hate you, nor does it love you, but you are made of atoms which it can use for something else.
  - astrange 2y ago
    Non-living isn't a great way to describe a virus because they certainly become part of a living system once they get in your cells. Models don't do that though, only if you run them in a loop with tools they can call, so mostly don't do that.
    - ben_w 2y ago
      > Models don't do that though, only if you run them in a loop with tools they can call, so mostly don't do that. That's also a description of DNA and RNA. They're chemicals, not magic. And there's loads of people all too eager to put any and every AI they find into such an environment[0], then connect it to a robot body[1], or connect it to the internet[2], just to see what happens. Or have an AI or algorithm design T-shirts[3] for them or trade stocks[4][5][6] for them because they don't stop and think about how this might go wrong. [0] https://community.openai.com/t/chaosgpt-an-ai-that-seeks-to-destroy-humanity/160028 https://community.openai.com/t/chaosgpt-an-ai-that-seeks-to-... [1] https://www.microsoft.com/en-us/research/group/autonomous-systems-group-robotics/articles/chatgpt-for-robotics/ https://www.microsoft.com/en-us/research/group/autonomous-sy... [2] https://platform.openai.com/docs/api-reference https://platform.openai.com/docs/api-reference [3] https://www.theguardian.com/technology/2013/mar/02/amazon-withdraws-rape-slogan-shirt https://www.theguardian.com/technology/2013/mar/02/amazon-wi... [4] https://intellectia.ai/blog/chatgpt-for-stock-trading https://intellectia.ai/blog/chatgpt-for-stock-trading [5] https://en.wikipedia.org/wiki/Algorithmic_trading https://en.wikipedia.org/wiki/Algorithmic_trading [6] https://en.wikipedia.org/wiki/2007–2008_financial_crisis https://en.wikipedia.org/wiki/2007–2008_financial_crisis
jwilber 2y ago
Love Anthropic research. Great visuals between Olah, Carter, and Pearce, as well. I don’t think this paper does much in the way of your final point, “it doesn’t understand what it’s saying”, though our understanding certainly has improved.
- kromem 2y ago
  They were able to demonstrate conceptual vectors that were consistent across different languages and different mediums (text vs images) and that when manipulated were able to represent the abstract concept in the output regardless of prompt. What kind of evidentiary threshold would you want if that's not sufficient?
  - jwilber 2y ago
    My point is that you claimed this is a rebuff against those claiming models don’t understand themselves. Your interpretation seems to assign intelligence to the algorithms. While this research allows us to interpret larger models in an amazing way, it doesn’t mean the models themselves ‘understand’ anything. You can use this on much smaller scale models as well, as they showed 8 months ago. Does that research tell us about how models understand themselves? Or does it help us understand how the models work?
    - kromem 2y ago
      "Understand themselves" is a very different thing than "understand what they are saying." Which exactly are we talking about here? Because no, the research doesn't say much about the former, but yes, it says a lot about the latter, especially on top of the many, many earlier papers working in smaller toy models demonstrating world modeling.

by kromem 2y ago

Workaccount2 2y ago

> on the "it's just fancy autocomplete" and "it doesn't understand what it's saying, bro" rhetoric. No matter what, there will always be a group of people saying that. The power and drive of the brain to convince itself that it is weaved of magical energy on a divine substrate shouldn't be underestimated. Especially when media plays so hard into that idea (the robots that lose the war because they cannot overcome love, etc.) because brains really love being told they are right. I am almost certain that the first conscious silicon (or whatever material) will be subjected to immense suffering until a new generation that can accept the human brains banality can move things forward.

ben_w 2y ago
It tickles me somewhat to note that people using the phrase "stochastic parrot" are demonstrating in themselves the exact behaviour for which they are dismissive of the LLMs. > I am almost certain that the first conscious silicon (or whatever material) will be subjected to immense suffering until a new generation that can accept the human brains banality can move things forward. Indeed, though as we don't know what we're doing (and have 40 definitions of "consciousness" and no way to test for qualia), I would add that the first AI we make with these properties, will likely suffer from every permutation of severe and mild mental heath disorder that is logically possible, including many we have no word for because they would be incompatible with life if found in an organic brain.

astrange 2y ago

I think the research is good, but it's disappointing that they hype it by claiming it's going to help their basically entirely fictional "AI safety" project, as if the bits in their model are going to come alive and eat them.

ben_w 2y ago
We just had a pandemic made from a non-living virus that was basically trying to eat us. To riff off the quote: The virus does not hate you, nor does it love you, but you are made of atoms which it can use for something else.
- astrange 2y ago
  Non-living isn't a great way to describe a virus because they certainly become part of a living system once they get in your cells. Models don't do that though, only if you run them in a loop with tools they can call, so mostly don't do that.
  - ben_w 2y ago
    > Models don't do that though, only if you run them in a loop with tools they can call, so mostly don't do that. That's also a description of DNA and RNA. They're chemicals, not magic. And there's loads of people all too eager to put any and every AI they find into such an environment[0], then connect it to a robot body[1], or connect it to the internet[2], just to see what happens. Or have an AI or algorithm design T-shirts[3] for them or trade stocks[4][5][6] for them because they don't stop and think about how this might go wrong. [0] https://community.openai.com/t/chaosgpt-an-ai-that-seeks-to-destroy-humanity/160028 https://community.openai.com/t/chaosgpt-an-ai-that-seeks-to-... [1] https://www.microsoft.com/en-us/research/group/autonomous-systems-group-robotics/articles/chatgpt-for-robotics/ https://www.microsoft.com/en-us/research/group/autonomous-sy... [2] https://platform.openai.com/docs/api-reference https://platform.openai.com/docs/api-reference [3] https://www.theguardian.com/technology/2013/mar/02/amazon-withdraws-rape-slogan-shirt https://www.theguardian.com/technology/2013/mar/02/amazon-wi... [4] https://intellectia.ai/blog/chatgpt-for-stock-trading https://intellectia.ai/blog/chatgpt-for-stock-trading [5] https://en.wikipedia.org/wiki/Algorithmic_trading https://en.wikipedia.org/wiki/Algorithmic_trading [6] https://en.wikipedia.org/wiki/2007–2008_financial_crisis https://en.wikipedia.org/wiki/2007–2008_financial_crisis

jwilber 2y ago

Love Anthropic research. Great visuals between Olah, Carter, and Pearce, as well. I don’t think this paper does much in the way of your final point, “it doesn’t understand what it’s saying”, though our understanding certainly has improved.

kromem 2y ago
They were able to demonstrate conceptual vectors that were consistent across different languages and different mediums (text vs images) and that when manipulated were able to represent the abstract concept in the output regardless of prompt. What kind of evidentiary threshold would you want if that's not sufficient?
- jwilber 2y ago
  My point is that you claimed this is a rebuff against those claiming models don’t understand themselves. Your interpretation seems to assign intelligence to the algorithms. While this research allows us to interpret larger models in an amazing way, it doesn’t mean the models themselves ‘understand’ anything. You can use this on much smaller scale models as well, as they showed 8 months ago. Does that research tell us about how models understand themselves? Or does it help us understand how the models work?
  - kromem 2y ago
    "Understand themselves" is a very different thing than "understand what they are saying." Which exactly are we talking about here? Because no, the research doesn't say much about the former, but yes, it says a lot about the latter, especially on top of the many, many earlier papers working in smaller toy models demonstrating world modeling.