5 ms·

All this "AI safety" is purely moat-building for the likes of OpenAI et. al. to prevent upstarts like DeepSeek. LLMs will not get us to AGI. Not even close. A

by r00fus 2y ago

All this "AI safety" is purely moat-building for the likes of OpenAI et. al. to prevent upstarts like DeepSeek. LLMs will not get us to AGI. Not even close. Altman talking about this danger is like Musk talking about driverless taxis.

deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
ryaniscool 2y ago
Half moat-building, half marketing. The need for "safety" implies some awesome power. Don't get me wrong, they are impressive. I can see LLM's eventually enabling people to be 10x more productive in jobs that interact with a computer all day.
- bombcar 2y ago
  > The need for "safety" implies some awesome power. This is a big part of it, and you can get others to do it for you. It's like the drain cleaner sold in an extra bag. Obviously it must be the best, it's so scary they have to put it in a bag!
- r00fus 2y ago
  So it's a tool like the internal combustion engine, or the moveable typeset. Game-changing technology that may alter society but not dangerous like nukes.
- timewizard 2y ago
  > eventually enabling people to be 10x more productive in jobs that interact with a computer all day. I doubt this. Productivity is gained through experience and expertise. If you don't know what you don't know than the LLM is perfectly useless to you.
- ksynwa 2y ago
  Are you not alarmed by the startling discoveries made by the hard-at-work researchers where LLMs lie (when explicitly told to) and copy their source files (when explicitly told to)?
amelius 2y ago
I wouldn't be surprised if EU has their own competitor within a year or so.
- IshKebab 2y ago
  To OpenAI? The closest was DeepMind but that's owned by Google now.
  - amelius 2y ago
    Well, deepseek open sourced their model and published their algorithm. It may take a while before it is reproduced but if they start an initiative and get the funding in place it'll probably be sooner rather than later.
  - mattlondon 2y ago
    Owned by Google yes, but it is head quartered in London, with the majority of the staff there. So the skills, knowledge, and expertise are in the UK. Google can close the UK office tomorrow if they wanted to sure, but are 100% of those staff going to move to California? Doubt it. Some will, but a lot have lives in the UK (not least the CEO and founder etc) so even if Google pulls the rug I will bet there will be a new company founded and funded within days that will vacuum up all the staff.
    - tfsh 2y ago
      But will this company be British or European? I'd love to think so, but somehow I doubt that. There just isn't the money in UK tech, the highest paid tech jobs (other than big tech) are elite hedgefunds but they get by with minimal headcount.
- tucnak 2y ago
  Mistral exists
worik 2y ago
> LLMs will not get us to AGI Yes. And there is no reason to think that AGI would have desire. I think people are reading themselves into their fears.
- realce 2y ago
  > And there is no reason to think that AGI would have desire. The entire point of utilizing this tool is to feed it a desire and have it produce an appropriate output based upon that desire. Not only that, it's entire training corpus is filled with examples of our human desires. So either humans give it desire or it trains itself to function based on the inertia of "goal-seeking" which are effectively the same thing.
- Tossrock 2y ago
  There is evidence that as LLMs increase in scale, their preferences become more coherent, see Hendrycks et al 2025, summarizer at https://www.emergent-values.ai/ https://www.emergent-values.ai/
  - anon291 2y ago
    A preference is meaningless without consciousness and qualia.
    - int_19h 2y ago
      Consider a philosophical doomsday AI: it behaves exactly as if it has a desire to harm you (meaning that it does!), but it doesn't actually want to do that. Or we can just drop all this sophistry nonsense.
      - anon291 2y ago
        Look, i understand being concerned about robot and self-driving safety. AI is a nebulous term and we don't need to restrict anyone training AIs.
z7 2y ago
Waymo's driverless taxis are currently operating in San Francisco, Los Angeles and Phoenix.
- ceejayoz 2y ago
  Notably, not Musk's, and very different promised functionality.
  - hector126 2y ago
    What did Musk's promised driverless taxis provide that existing driverless taxis don't? The tech has arrived; it's a car that drives itself while the passenger sits in the back. Is the "gotcha" that the car isn't a Tesla? Seems like we're splitting hairs a bit here.
    - Mali- 2y ago
      He promised that you'd be able to turn your own Tesla into an autonomous taxi that would earn you money. That is a massive lie, not splitting hairs. Obviously, we're very desensitized to lying rats - but that's what he did.
      - hector126 2y ago
        https://www.reuters.com/technology/tesla-robotaxis-by-june-musk-turns-texas-hands-off-regulation-2025-02-10/ https://www.reuters.com/technology/tesla-robotaxis-by-june-m... Looks like it's still in the works. Sometimes when technologists promise something their timelines end up being overly optimistic. I'm sure this isn't news to you. By your language and past commentary though this seems like the kind of thing which elicits a pretty emotional response from you, so I'm not sure this would be a productive conversation either way.
- raincole 2y ago
  I am willing to bet that even when driverless taxis are operating in at least 50% of big cities around the world, you will still see comments like "auto driving is a pipe dream like NFT" on HN every other day.
yodsanklai 2y ago
I'd say AGI is like Musk talking about interstellar traveling.
edanm 2y ago
> All this "AI safety" is purely moat-building for the likes of OpenAI et. al. to prevent upstarts like DeepSeek. Modern AI safety originated with people like Eliezer Yudkowsky, Nick Bostrom, the LessWrong/rationality movement etc. They very much were not just talking about it only to build moats for OpenAI. For one thing, OpenAI didn't exist at the time, AI was not anywhere close to where it is today, and almost everyone thought their arguments were ridiculous. You might not agree with them, but you can't simply dismiss their arguments as only being there to prop up the existing AI players, that's wrong and disingenuous.
- z3n0n 2y ago
  entertaining podcast about all this "rationalist" weirdness: https://www.youtube.com/watch?v=e57oo7AgrJY https://www.youtube.com/watch?v=e57oo7AgrJY
anon291 2y ago
> LLMs will not get us to AGI. Not even close. Altman talking about this danger is like Musk talking about driverless taxis. AGI is a meaningless term. The LLM architecture has shown promise in every single domain once used for perceptron neural networks. By all accounts on those things that fit its 'senses' the LLMs are significantly smarter than the average human being.
wand3r 2y ago
Driverless taxis already exists?
KeplerBoy 2y ago
It's absolutely not? Deepseek and efforts by other non-aligned powers wouldn't care about any declarations signed by the EU, the US and other western powers anyways.
- piker 2y ago
  Agree or disagree with the premise, no doubt this sort of declaration grounds export controls and other regulations which do inhibit development extra-jurisdictionally.
jstummbillig 2y ago
How does your theory account for the Eliezer Yudkowsky type person, who clearly shows no love for any of the labs or the current progress, and yet is very much pro-"AI safety"?
elric 2y ago
The way I see "safety" isn't really about what AI "can do", but about how we allow it to be used. E.g. an AI that's used to assess an insurance claim should be fully explainable so we know it isn't using racist biases to deny claims based on skin colour. If the AI can't give that guarantee, it isn't fit for purpose and its use shouldn't be allowed. Same with killer robots (or whatever it is people are afraid of when they talk about "AI safety"). As long as we can control who they kill, when, and why, there's no real difference with any other weapon system. If that control is even slightly in doubt: it's not fit for service. Does this mean that bullshit generating LLMs aren't fit for service in many areas: it probably does. But maybe there steps can be taken to mitigate risks. I'm sure this will involve some bureaucratic overhead. But it seems worth the hassle to me. Being against AI Safety is a stupid hill to die on. Being against some concrete declaration or a part thereof, sure, that might make sense. But this smells a lot like the tabacco industry being against warnings/filters/low-tar, or the car industry being anti-seatbelt.
MichaelDickens 2y ago
If AI safety is moat-building for American companies, why doesn't JD Vance support it, since it would allegedly benefit American interests?

by r00fus 2y ago

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

ryaniscool 2y ago

Half moat-building, half marketing. The need for "safety" implies some awesome power. Don't get me wrong, they are impressive. I can see LLM's eventually enabling people to be 10x more productive in jobs that interact with a computer all day.

bombcar 2y ago
> The need for "safety" implies some awesome power. This is a big part of it, and you can get others to do it for you. It's like the drain cleaner sold in an extra bag. Obviously it must be the best, it's so scary they have to put it in a bag!
r00fus 2y ago
So it's a tool like the internal combustion engine, or the moveable typeset. Game-changing technology that may alter society but not dangerous like nukes.
timewizard 2y ago
> eventually enabling people to be 10x more productive in jobs that interact with a computer all day. I doubt this. Productivity is gained through experience and expertise. If you don't know what you don't know than the LLM is perfectly useless to you.
ksynwa 2y ago
Are you not alarmed by the startling discoveries made by the hard-at-work researchers where LLMs lie (when explicitly told to) and copy their source files (when explicitly told to)?

amelius 2y ago

I wouldn't be surprised if EU has their own competitor within a year or so.

IshKebab 2y ago
To OpenAI? The closest was DeepMind but that's owned by Google now.
- amelius 2y ago
  Well, deepseek open sourced their model and published their algorithm. It may take a while before it is reproduced but if they start an initiative and get the funding in place it'll probably be sooner rather than later.
- mattlondon 2y ago
  Owned by Google yes, but it is head quartered in London, with the majority of the staff there. So the skills, knowledge, and expertise are in the UK. Google can close the UK office tomorrow if they wanted to sure, but are 100% of those staff going to move to California? Doubt it. Some will, but a lot have lives in the UK (not least the CEO and founder etc) so even if Google pulls the rug I will bet there will be a new company founded and funded within days that will vacuum up all the staff.
  - tfsh 2y ago
    But will this company be British or European? I'd love to think so, but somehow I doubt that. There just isn't the money in UK tech, the highest paid tech jobs (other than big tech) are elite hedgefunds but they get by with minimal headcount.
tucnak 2y ago
Mistral exists

worik 2y ago

> LLMs will not get us to AGI Yes. And there is no reason to think that AGI would have desire. I think people are reading themselves into their fears.

realce 2y ago
> And there is no reason to think that AGI would have desire. The entire point of utilizing this tool is to feed it a desire and have it produce an appropriate output based upon that desire. Not only that, it's entire training corpus is filled with examples of our human desires. So either humans give it desire or it trains itself to function based on the inertia of "goal-seeking" which are effectively the same thing.
Tossrock 2y ago
There is evidence that as LLMs increase in scale, their preferences become more coherent, see Hendrycks et al 2025, summarizer at https://www.emergent-values.ai/ https://www.emergent-values.ai/
- anon291 2y ago
  A preference is meaningless without consciousness and qualia.
  - int_19h 2y ago
    Consider a philosophical doomsday AI: it behaves exactly as if it has a desire to harm you (meaning that it does!), but it doesn't actually want to do that. Or we can just drop all this sophistry nonsense.
    - anon291 2y ago
      Look, i understand being concerned about robot and self-driving safety. AI is a nebulous term and we don't need to restrict anyone training AIs.

z7 2y ago

Waymo's driverless taxis are currently operating in San Francisco, Los Angeles and Phoenix.

ceejayoz 2y ago
Notably, not Musk's, and very different promised functionality.
- hector126 2y ago
  What did Musk's promised driverless taxis provide that existing driverless taxis don't? The tech has arrived; it's a car that drives itself while the passenger sits in the back. Is the "gotcha" that the car isn't a Tesla? Seems like we're splitting hairs a bit here.
  - Mali- 2y ago
    He promised that you'd be able to turn your own Tesla into an autonomous taxi that would earn you money. That is a massive lie, not splitting hairs. Obviously, we're very desensitized to lying rats - but that's what he did.
    - hector126 2y ago
      https://www.reuters.com/technology/tesla-robotaxis-by-june-musk-turns-texas-hands-off-regulation-2025-02-10/ https://www.reuters.com/technology/tesla-robotaxis-by-june-m... Looks like it's still in the works. Sometimes when technologists promise something their timelines end up being overly optimistic. I'm sure this isn't news to you. By your language and past commentary though this seems like the kind of thing which elicits a pretty emotional response from you, so I'm not sure this would be a productive conversation either way.
raincole 2y ago
I am willing to bet that even when driverless taxis are operating in at least 50% of big cities around the world, you will still see comments like "auto driving is a pipe dream like NFT" on HN every other day.

yodsanklai 2y ago

I'd say AGI is like Musk talking about interstellar traveling.

edanm 2y ago

> All this "AI safety" is purely moat-building for the likes of OpenAI et. al. to prevent upstarts like DeepSeek. Modern AI safety originated with people like Eliezer Yudkowsky, Nick Bostrom, the LessWrong/rationality movement etc. They very much were not just talking about it only to build moats for OpenAI. For one thing, OpenAI didn't exist at the time, AI was not anywhere close to where it is today, and almost everyone thought their arguments were ridiculous. You might not agree with them, but you can't simply dismiss their arguments as only being there to prop up the existing AI players, that's wrong and disingenuous.

z3n0n 2y ago
entertaining podcast about all this "rationalist" weirdness: https://www.youtube.com/watch?v=e57oo7AgrJY https://www.youtube.com/watch?v=e57oo7AgrJY

anon291 2y ago

> LLMs will not get us to AGI. Not even close. Altman talking about this danger is like Musk talking about driverless taxis. AGI is a meaningless term. The LLM architecture has shown promise in every single domain once used for perceptron neural networks. By all accounts on those things that fit its 'senses' the LLMs are significantly smarter than the average human being.

wand3r 2y ago

Driverless taxis already exists?

KeplerBoy 2y ago

It's absolutely not? Deepseek and efforts by other non-aligned powers wouldn't care about any declarations signed by the EU, the US and other western powers anyways.

piker 2y ago
Agree or disagree with the premise, no doubt this sort of declaration grounds export controls and other regulations which do inhibit development extra-jurisdictionally.

jstummbillig 2y ago

How does your theory account for the Eliezer Yudkowsky type person, who clearly shows no love for any of the labs or the current progress, and yet is very much pro-"AI safety"?

elric 2y ago

The way I see "safety" isn't really about what AI "can do", but about how we allow it to be used. E.g. an AI that's used to assess an insurance claim should be fully explainable so we know it isn't using racist biases to deny claims based on skin colour. If the AI can't give that guarantee, it isn't fit for purpose and its use shouldn't be allowed. Same with killer robots (or whatever it is people are afraid of when they talk about "AI safety"). As long as we can control who they kill, when, and why, there's no real difference with any other weapon system. If that control is even slightly in doubt: it's not fit for service. Does this mean that bullshit generating LLMs aren't fit for service in many areas: it probably does. But maybe there steps can be taken to mitigate risks. I'm sure this will involve some bureaucratic overhead. But it seems worth the hassle to me. Being against AI Safety is a stupid hill to die on. Being against some concrete declaration or a part thereof, sure, that might make sense. But this smells a lot like the tabacco industry being against warnings/filters/low-tar, or the car industry being anti-seatbelt.

MichaelDickens 2y ago

If AI safety is moat-building for American companies, why doesn't JD Vance support it, since it would allegedly benefit American interests?