3 ms·

Their position makes no sense to me. I don’t see how you can be a mainstream company selling your services worldwide (almost) if you also believe that you’re bu

by dgellow 3mo ago

Their position makes no sense to me. I don’t see how you can be a mainstream company selling your services worldwide (almost) if you also believe that you’re building an extremely dangerous AGI (supposedly based on the same technology you’re offering to everyone). If you actually believe that an AGI would be extremely dangerous that should 100% be a very strictly regulated area of research, similar to bio weapons. And we know that Chinese models are derived from OpenAI and Anthropic, they are at the same time talking about how dangerous models can be (even their aligned ones it seems), while being also responsible for the development of the whole industry and providing the basis for adversary countries to build their own. I don’t believe we would accept that for any other technology that is expected to be as risky for the world

ToValueFunfetti 3mo ago
The companies are begging to be regulated for this reason and have been doing so for years. HN's response is generally that this is performative for marketing or seeking regulatory capture or haha anthropic you get what you ask for. Maybe the cynics are right, but there's really nothing inconsistent about the naive view here, once you factor in race dynamics and obligations to investors.
- vasco 3mo ago
  > The companies are begging to be regulated for this reason and have been doing so for years Regulations are rules that you force on a market, but the actors in the market should not be assumed to be all operating against the regulations before they come into play. Said in other words, these companies don't need to wait for regulation to not destroy the world, if that's truly what they think will happen. > inb4 someone else will do it
  - deleted 3mo ago
    [deleted]
  - owebmaster 3mo ago
    > these companies don't need to wait for regulation to not destroy the world, if that's truly what they think will happen. They believe that if they don't destroy the world someone else will so better be them
    - vasco 3mo ago
      You might want to google what inb4 means, at least you could've put a bit more effort substantiating it.
      - ToValueFunfetti 3mo ago
        If your response to an argument is inb4, you don't get to tell somebody else they're not putting in enough effort to provide substance. Also, I brought up race dynamics before your inb4, so even if anticipation counted as more than a shallow dismissal, you didn't meet that bar. I really don't understand what's happening here lately such that 15-year-old accounts are behaving so poorly. This is the first time you've said 'inb4' in what I can only guess is thousands of comments over a decade and a half. If you don't care about the standards here anymore, why stick around and make things worse for the rest of us? Is there some other draw than quality of conversation? I wrote something earlier to the same effect and wound up deleting it because it let too much frustration through. I am frustrated, but you don't deserve the brunt of that. Sorry if that's still coming through.
        vasco 3mo ago
        Someone else will do it is the lowest form of justification for any bad behavior. Regarding the rest of your judgement of the quality of the comments that's why we have votes. Some of my comments for this 15 years are downvoted and others upvoted and some are even flagged and it lets me learn what the crowd agrees with and not and I reflect from it and you can see the average to judge for yourself what the crowd thinks of my contributions. I appreciate your answer but it's one person's opinion, as is mine.
        3mo ago
- watwut 3mo ago
  > The companies are begging to be regulated for this reason and have been doing so for years They can stop doing a thing they claim should be regulated. You dont need to be regulated and forced to do the thing you consider right, especially when you are the primary one collecting the money to do the bad thing. They could train ai for pro-social purposes, they dont here. They could make it useful for worker, they intentionally try to harm workers. And then pretend "it just happened".
  - simianwords 3mo ago
    what a naive comment. these companies have world class alignment researchers. a math Fields medalist is also joining OpenAI as one [1]. > They can stop doing a thing they claim should be regulated. That's not how the world works. there are tradeoffs and we need to learn how to navigate it. not just dismiss it straight up. [1] https://en.wikipedia.org/wiki/Jacob_Tsimerman https://en.wikipedia.org/wiki/Jacob_Tsimerman
- zmmmmm 3mo ago
  Tobacco company says "we are launching a new product that will cause cancer and kill people. It's highly addictive so we expect widespread uptake. We think it is crucial that regulation be introduced for mandatory regular cancer tests so that people can be streamlined into treatment faster when they get sick"
  - fwipsy 3mo ago
    I think Anthropic employees think that their product is more like opiates -- highly dangerous, but with a large potential benefit when applied correctly. I don't know what OpenAI employees were thinking, but thankfully it looks like they're thinking again.
simoncion 3mo ago
> Their position makes no sense to me. If one assumes that they don't actually care about security, and care very deeply about getting sensational press, their position makes a lot of sense. For all their chatter about how incredibly important "alignment" is, they still haven't bothered to remember the 30->50 year old computer security principle of "Don't blindly do what some random stranger tells you to do." and ensure that system instructions, user instructions, and instructions from untrusted sources are indelibly marked with their category and treated according to those markings. Every single time one of these systems fails to distinguish between these three classes of instructions -or confuses its internal chatter with user instructions-, that's proof that the major LLM companies cannot be bothered to follow one of the most basic computer security principles. "But it's all vectors, not language! The LLM can't tell where the instructions came from", one might retort. I'd reply: "Neither can a CPU, but somehow we managed to make it work way back in the day. Amazing, isn't it?".
- simonw 3mo ago
  I get the impression that every AI lab is desperately trying to figure out how to unambiguously separate instructions from data in their token streams. The fact that they haven't managed to yet suggests to me that it's a very, very difficult problem.
  - simoncion 3mo ago
    > I get the impression that every AI lab is desperately trying... Of course. I wonder how we managed way back in the day to produce systems that can handle untrusted inputs and reliably instruct a dumb-as-bricks CPU what to do based on those inputs. Must have been black magic lost to the mists of time.
    - simonw 3mo ago
      If you can figure out how to separate instructions from data in LLMs you should ship the first agent system that's guaranteed protected against prompt injection. You'll make millions.
andai 3mo ago
> If you actually believe that an AGI would be extremely dangerous that should 100% be a very strictly regulated area of research, similar to bio weapons. Yeah. They do believe that, and they have been pushing for regulations for years. And every time one of their models does something horrible, it helps them achieve that goal.
- mtrovo 3mo ago
  Considering their current valuation and the prospects of getting any of this money back, that's a genius exit strategy.
btown 3mo ago
If you are a company selling Red Team cybersecurity services, it’s in your interest to make your services indispensable. Your unwilling customers must subscribe to frontier cybersecurity scans and fixes to ensure they’re immune to just-behind-frontier attackers, who are training on those very same frontier models. And of course this also satisfies those who think the best prospect of aligning superintelligence is to be in The Room Where It Happens. Arms races are what make that room exist, after all. It’s the Yelp protection playbook too. If you don’t play ball, somebody else will control your reputation and livelihood. We live in a dark forest.
uselessTA 3mo ago
I know some people who are worried at Anthropic, and their position seems to be "if we don't do it, someone even less responsible will. Unilateral disarmament didn't work and real oversight seems unlikely to happen in time, so we'll just try to be as safe as we can be (while still winning the race)" Not that they're happy about it, they just see no other realistic choice
- dweinus 3mo ago
  https://theonion.com/sam-altman-if-i-dont-end-the-world-someone-far-more-dangerous-will/ https://theonion.com/sam-altman-if-i-dont-end-the-world-some...
  - wyrdcurt 3mo ago
    "AI will probably, most likely, sort of lead to the end of the world. But in the meantime, there will be great companies..." - actual Sam Altman quote, the man is so unhinged he's beyond satire
- dgellow 3mo ago
  I know, that’s the position Dario Amodei argues for in his essays. I did pass their cultural interview and had to consume a lot of their content to prepare, I think I have a good idea of their stated values. But what the company does and what the leadership states their vision is is pretty contradictory. They are providing everything bad guys need to develop their unaligned frontier models. Chinese models that Dario considers to be dangerous are distilled from Claude, and they know this. They are creating the FOMO around AI which pushes adversary countries to invest so much into unaligned models. They offer models as a service they know are jailbreakable and can be used by bad actors. They are running internal red-team experiments without adequate isolation. If I take their statements seriously, AGI research should really be seen as bioweapon, or cloning, or nuclear research. Something strictly regulated worldwide, with export controls for HBM and other hardware used for AI training. What they are trying is instead to boost their position by becoming too big to fail and too powerful to ban, but then want the industry to be regulated to pull the ladder behind them. It really doesn’t feel they are serious about their values, otherwise they wouldn’t be offering Mythos (a model that is unsafe from their own admission) as a service to their close partners

by dgellow 3mo ago

ToValueFunfetti 3mo ago

The companies are begging to be regulated for this reason and have been doing so for years. HN's response is generally that this is performative for marketing or seeking regulatory capture or haha anthropic you get what you ask for. Maybe the cynics are right, but there's really nothing inconsistent about the naive view here, once you factor in race dynamics and obligations to investors.

vasco 3mo ago
> The companies are begging to be regulated for this reason and have been doing so for years Regulations are rules that you force on a market, but the actors in the market should not be assumed to be all operating against the regulations before they come into play. Said in other words, these companies don't need to wait for regulation to not destroy the world, if that's truly what they think will happen. > inb4 someone else will do it
- deleted 3mo ago
  [deleted]
- owebmaster 3mo ago
  > these companies don't need to wait for regulation to not destroy the world, if that's truly what they think will happen. They believe that if they don't destroy the world someone else will so better be them
  - vasco 3mo ago
    You might want to google what inb4 means, at least you could've put a bit more effort substantiating it.
    - ToValueFunfetti 3mo ago
      If your response to an argument is inb4, you don't get to tell somebody else they're not putting in enough effort to provide substance. Also, I brought up race dynamics before your inb4, so even if anticipation counted as more than a shallow dismissal, you didn't meet that bar. I really don't understand what's happening here lately such that 15-year-old accounts are behaving so poorly. This is the first time you've said 'inb4' in what I can only guess is thousands of comments over a decade and a half. If you don't care about the standards here anymore, why stick around and make things worse for the rest of us? Is there some other draw than quality of conversation? I wrote something earlier to the same effect and wound up deleting it because it let too much frustration through. I am frustrated, but you don't deserve the brunt of that. Sorry if that's still coming through.
      - vasco 3mo ago
        Someone else will do it is the lowest form of justification for any bad behavior. Regarding the rest of your judgement of the quality of the comments that's why we have votes. Some of my comments for this 15 years are downvoted and others upvoted and some are even flagged and it lets me learn what the crowd agrees with and not and I reflect from it and you can see the average to judge for yourself what the crowd thinks of my contributions. I appreciate your answer but it's one person's opinion, as is mine.
        3mo ago
watwut 3mo ago
> The companies are begging to be regulated for this reason and have been doing so for years They can stop doing a thing they claim should be regulated. You dont need to be regulated and forced to do the thing you consider right, especially when you are the primary one collecting the money to do the bad thing. They could train ai for pro-social purposes, they dont here. They could make it useful for worker, they intentionally try to harm workers. And then pretend "it just happened".
- simianwords 3mo ago
  what a naive comment. these companies have world class alignment researchers. a math Fields medalist is also joining OpenAI as one [1]. > They can stop doing a thing they claim should be regulated. That's not how the world works. there are tradeoffs and we need to learn how to navigate it. not just dismiss it straight up. [1] https://en.wikipedia.org/wiki/Jacob_Tsimerman https://en.wikipedia.org/wiki/Jacob_Tsimerman
zmmmmm 3mo ago
Tobacco company says "we are launching a new product that will cause cancer and kill people. It's highly addictive so we expect widespread uptake. We think it is crucial that regulation be introduced for mandatory regular cancer tests so that people can be streamlined into treatment faster when they get sick"
- fwipsy 3mo ago
  I think Anthropic employees think that their product is more like opiates -- highly dangerous, but with a large potential benefit when applied correctly. I don't know what OpenAI employees were thinking, but thankfully it looks like they're thinking again.

simoncion 3mo ago

> Their position makes no sense to me. If one assumes that they don't actually care about security, and care very deeply about getting sensational press, their position makes a lot of sense. For all their chatter about how incredibly important "alignment" is, they still haven't bothered to remember the 30->50 year old computer security principle of "Don't blindly do what some random stranger tells you to do." and ensure that system instructions, user instructions, and instructions from untrusted sources are indelibly marked with their category and treated according to those markings. Every single time one of these systems fails to distinguish between these three classes of instructions -or confuses its internal chatter with user instructions-, that's proof that the major LLM companies cannot be bothered to follow one of the most basic computer security principles. "But it's all vectors, not language! The LLM can't tell where the instructions came from", one might retort. I'd reply: "Neither can a CPU, but somehow we managed to make it work way back in the day. Amazing, isn't it?".

simonw 3mo ago
I get the impression that every AI lab is desperately trying to figure out how to unambiguously separate instructions from data in their token streams. The fact that they haven't managed to yet suggests to me that it's a very, very difficult problem.
- simoncion 3mo ago
  > I get the impression that every AI lab is desperately trying... Of course. I wonder how we managed way back in the day to produce systems that can handle untrusted inputs and reliably instruct a dumb-as-bricks CPU what to do based on those inputs. Must have been black magic lost to the mists of time.
  - simonw 3mo ago
    If you can figure out how to separate instructions from data in LLMs you should ship the first agent system that's guaranteed protected against prompt injection. You'll make millions.

andai 3mo ago

> If you actually believe that an AGI would be extremely dangerous that should 100% be a very strictly regulated area of research, similar to bio weapons. Yeah. They do believe that, and they have been pushing for regulations for years. And every time one of their models does something horrible, it helps them achieve that goal.

mtrovo 3mo ago
Considering their current valuation and the prospects of getting any of this money back, that's a genius exit strategy.

btown 3mo ago

If you are a company selling Red Team cybersecurity services, it’s in your interest to make your services indispensable. Your unwilling customers must subscribe to frontier cybersecurity scans and fixes to ensure they’re immune to just-behind-frontier attackers, who are training on those very same frontier models. And of course this also satisfies those who think the best prospect of aligning superintelligence is to be in The Room Where It Happens. Arms races are what make that room exist, after all. It’s the Yelp protection playbook too. If you don’t play ball, somebody else will control your reputation and livelihood. We live in a dark forest.

uselessTA 3mo ago

I know some people who are worried at Anthropic, and their position seems to be "if we don't do it, someone even less responsible will. Unilateral disarmament didn't work and real oversight seems unlikely to happen in time, so we'll just try to be as safe as we can be (while still winning the race)" Not that they're happy about it, they just see no other realistic choice

dweinus 3mo ago
https://theonion.com/sam-altman-if-i-dont-end-the-world-someone-far-more-dangerous-will/ https://theonion.com/sam-altman-if-i-dont-end-the-world-some...
- wyrdcurt 3mo ago
  "AI will probably, most likely, sort of lead to the end of the world. But in the meantime, there will be great companies..." - actual Sam Altman quote, the man is so unhinged he's beyond satire
dgellow 3mo ago
I know, that’s the position Dario Amodei argues for in his essays. I did pass their cultural interview and had to consume a lot of their content to prepare, I think I have a good idea of their stated values. But what the company does and what the leadership states their vision is is pretty contradictory. They are providing everything bad guys need to develop their unaligned frontier models. Chinese models that Dario considers to be dangerous are distilled from Claude, and they know this. They are creating the FOMO around AI which pushes adversary countries to invest so much into unaligned models. They offer models as a service they know are jailbreakable and can be used by bad actors. They are running internal red-team experiments without adequate isolation. If I take their statements seriously, AGI research should really be seen as bioweapon, or cloning, or nuclear research. Something strictly regulated worldwide, with export controls for HBM and other hardware used for AI training. What they are trying is instead to boost their position by becoming too big to fail and too powerful to ban, but then want the industry to be regulated to pull the ladder behind them. It really doesn’t feel they are serious about their values, otherwise they wouldn’t be offering Mythos (a model that is unsafe from their own admission) as a service to their close partners