6 ms·

There's an even bigger problem here: if he were just making money, that would be a normal-sized problem. If he were just making a supplier for OA, heck, that mi

by gwern 3y ago

There's an even bigger problem here: if he were just making money, that would be a normal-sized problem. If he were just making a supplier for OA, heck, that might be a good thing on net for OA; a subsidiary doing hardware might be justifiable given the importance of securing hardware. But that's not what he's doing. Creating such an independent hardware startup comes off as basically directly opposed to OA's safety mission - GPUs are one of the biggest limitations to creating better-than-OA models! The best customers would be the ones who are most sabotaging the OA mission. (You can't run the UN and also head an arms manufacturer dreaming of democratizing access to munitions.)

etrautmann 3y ago
Kinda depends on your perspective on safety. In your view, should GPUs be regulated or not exist?
- chasd00 3y ago
  reposting this since it's getting more and more relevant >> Common sense AI control NOW. Ban assault style GPUs > That is beautiful, I made you a shirt. https://sprd.co/ZZufv7j https://sprd.co/ZZufv7j /context https://news.ycombinator.com/context?id=38119845 https://news.ycombinator.com/context?id=38119845
goldinfra 3y ago
How is it opposed to OpenAI's goals to have a friendly company selling them chips instead of NVIDIA, which is, at-best, a neutral company? Software is always more important than hardware. All the big players have access to NVIDIA chips today and yet only OpenAI has ChatGPT, proving the point. OpenAI probably wishes someone would create competition to NVIDIA and this is Sam Altman trying to make that happen himself, since no one else seems to have been able to pull it off so far. A conflict of interest would be OpenAI buying Altman's chips at inflated prices or something like that. But if he makes a bunch of money selling OpenAI chips and OpenAI gets better/cheaper chips, that seems like pure win-win and totally free of ethical conflict.
- nostrademons 3y ago
  The issue is not the OpenAI/SamaChip relationship, which is probably beneficial for OpenAI. The issue is what happens when SamaChip's profit imperative forces them to maximize revenue. Because the best way to maximize revenue, when you're an independent chipmaker with a large R&D investment, is to sell to more companies. Which by definition are going to be OpenAI's competitors, and whose interests may not be aligned with OpenAI, but will have a financial line to SamaChip. Gwern's highlighting an interesting contradiction in OpenAI's core charter. In order to ensure responsible, safe, humanity-benefiting AGI, OpenAI needs to have control over AGI's development; any for-profit entity that gets ahead of them probably will not have the same humanity-benefitting mission (actually, we know they won't, they will have a shareholder-benefitting mission by definition). But that means that by their charter, they can't be "Open". Anything like the Developer Day or API or a SamaChip that can sell to other startups means that other parties will have the freedom to use it for their own interests. Not saying whether this is good or bad - the tension between openness and vulnerability always exists, and personally I tend to come down on the side of openness. But IMHO OpenAI's mission was contradictory from the very beginning, and was more a recruiting tool to get bright idealistic AI researchers to work for them.
  - deleted 3y ago
    [deleted]
  - esafak 3y ago
    Where does Ilya stand on transparency in relation to OpenAI's mission of developing API?
  - tsunamifury 3y ago
    Hahaha if you played at any high level in the FAANG you’d know that this is a desirable outcome compared to the alternative.
- Nevermark 3y ago
  > totally free of ethical conflict. Unfortunately, far from that case. Altman's hardware startup would only be free of ethical conflict if Altman was open about it, and the board approved at least two things (and probably more): 1. A formal plan to separate OpenAI CEO Altman from OpenAI hardware acquisition decisions. 2. A formal agreement with Altman and his new company, on how OpenAI's private information with hardware implications is firewalled and/or shared with Altman's new hardware concern. Otherwise, Altman is going rogue, acting on private OpenAI information useful to a new hardware company looking for future business with OpenAI and OpenAI competitors. CEO Altman has a fiduciary duty to act directly in OpenAI's interest, and not in some "hey this could be great for everyone" version. Litmus test: If your legal partner/executive is doing things behind your back with large implications for you, they are almost certainly violating ethics in some way.
threeseed 3y ago
> basically directly opposed to OA's safety mission OpenAI does not have a mission to ensure that the entire industry is safe. And if anyone actually believes that then they are frankly delusional because right now AI is a geo-political fight between nation states. Is OpenAI really going to have any ability to control what China or UAE do with their LLMs. No.
- bakuninsbart 3y ago
  China is the only other state that has anywhere close to the capacity to attempt to go for AGI. They also have an existential interest in creating safe ASI. The problem is that in geopolitical struggles, safety usually goes a bit out the window, and we've so far been fairly lucky (and surprisingly competent) with not destroying large parts of human civilisation. There are a lot of unknowns around the topic. If ASI is possible and reachable, the first to get there would end the race. AGI could mean we are not alone as a sentient species we can directly communicate with on this planet anymore, and hopefully we wouldn't try to enslave it like we did to the "others" we encountered before.
  - seanmcdirmid 3y ago
    AI in China must adhere to strict socialist values. I wish that was a joke, but it’s actually a rule the government is trying to implement. I doubt it isn’t so much safety for the people as it is for the CPC.
  - Nevermark 3y ago
    > They also have an existential interest in creating safe ASI. An oppressive state with AGI is no better than an oppressive AGI. > China is the only other state that has anywhere close to the capacity to attempt to go for AGI. Any state can easily recreate/adopt the AGI invented by others. The costs and challenges for any given level of AI tech have dropped precipitously after each advance, and state level actors have far more than enough resources.
  - staticman2 3y ago
    Why can't, I don't know, the Canadian government hire some ex OpenAI employees and attempt to make an AGI? What is the constraint here that makes it only possible for the U.S. and China?
theptip 3y ago
Agree with this take. Sam previously stated (eg on Lex’s podcast) that a slow takeoff soon was his goal, to give society maximum time to adjust and to prevent an unexpected fast takeoff from capability overhang. I bought his take when he said it. Going off and trying to accelerate hardware capabilities (especially with an outside company that presumably sells these processors on the open market) seems indefensible in this framework unless you have already solved alignment, which they clearly have not.
alberth 3y ago
Reframing If this was simply reframed as, by creating your own GPUs - it would radically lower OpenAI costs in a material way, it’d be more understandable.
- verdverm 3y ago
  Google already does this. There is a material difference between doing it in house and starting a for profit company to do the thing

by gwern 3y ago

etrautmann 3y ago

Kinda depends on your perspective on safety. In your view, should GPUs be regulated or not exist?

chasd00 3y ago
reposting this since it's getting more and more relevant >> Common sense AI control NOW. Ban assault style GPUs > That is beautiful, I made you a shirt. https://sprd.co/ZZufv7j https://sprd.co/ZZufv7j /context https://news.ycombinator.com/context?id=38119845 https://news.ycombinator.com/context?id=38119845

goldinfra 3y ago

How is it opposed to OpenAI's goals to have a friendly company selling them chips instead of NVIDIA, which is, at-best, a neutral company? Software is always more important than hardware. All the big players have access to NVIDIA chips today and yet only OpenAI has ChatGPT, proving the point. OpenAI probably wishes someone would create competition to NVIDIA and this is Sam Altman trying to make that happen himself, since no one else seems to have been able to pull it off so far. A conflict of interest would be OpenAI buying Altman's chips at inflated prices or something like that. But if he makes a bunch of money selling OpenAI chips and OpenAI gets better/cheaper chips, that seems like pure win-win and totally free of ethical conflict.

nostrademons 3y ago
The issue is not the OpenAI/SamaChip relationship, which is probably beneficial for OpenAI. The issue is what happens when SamaChip's profit imperative forces them to maximize revenue. Because the best way to maximize revenue, when you're an independent chipmaker with a large R&D investment, is to sell to more companies. Which by definition are going to be OpenAI's competitors, and whose interests may not be aligned with OpenAI, but will have a financial line to SamaChip. Gwern's highlighting an interesting contradiction in OpenAI's core charter. In order to ensure responsible, safe, humanity-benefiting AGI, OpenAI needs to have control over AGI's development; any for-profit entity that gets ahead of them probably will not have the same humanity-benefitting mission (actually, we know they won't, they will have a shareholder-benefitting mission by definition). But that means that by their charter, they can't be "Open". Anything like the Developer Day or API or a SamaChip that can sell to other startups means that other parties will have the freedom to use it for their own interests. Not saying whether this is good or bad - the tension between openness and vulnerability always exists, and personally I tend to come down on the side of openness. But IMHO OpenAI's mission was contradictory from the very beginning, and was more a recruiting tool to get bright idealistic AI researchers to work for them.
- deleted 3y ago
  [deleted]
- esafak 3y ago
  Where does Ilya stand on transparency in relation to OpenAI's mission of developing API?
- tsunamifury 3y ago
  Hahaha if you played at any high level in the FAANG you’d know that this is a desirable outcome compared to the alternative.
Nevermark 3y ago
> totally free of ethical conflict. Unfortunately, far from that case. Altman's hardware startup would only be free of ethical conflict if Altman was open about it, and the board approved at least two things (and probably more): 1. A formal plan to separate OpenAI CEO Altman from OpenAI hardware acquisition decisions. 2. A formal agreement with Altman and his new company, on how OpenAI's private information with hardware implications is firewalled and/or shared with Altman's new hardware concern. Otherwise, Altman is going rogue, acting on private OpenAI information useful to a new hardware company looking for future business with OpenAI and OpenAI competitors. CEO Altman has a fiduciary duty to act directly in OpenAI's interest, and not in some "hey this could be great for everyone" version. Litmus test: If your legal partner/executive is doing things behind your back with large implications for you, they are almost certainly violating ethics in some way.

threeseed 3y ago

> basically directly opposed to OA's safety mission OpenAI does not have a mission to ensure that the entire industry is safe. And if anyone actually believes that then they are frankly delusional because right now AI is a geo-political fight between nation states. Is OpenAI really going to have any ability to control what China or UAE do with their LLMs. No.

bakuninsbart 3y ago
China is the only other state that has anywhere close to the capacity to attempt to go for AGI. They also have an existential interest in creating safe ASI. The problem is that in geopolitical struggles, safety usually goes a bit out the window, and we've so far been fairly lucky (and surprisingly competent) with not destroying large parts of human civilisation. There are a lot of unknowns around the topic. If ASI is possible and reachable, the first to get there would end the race. AGI could mean we are not alone as a sentient species we can directly communicate with on this planet anymore, and hopefully we wouldn't try to enslave it like we did to the "others" we encountered before.
- seanmcdirmid 3y ago
  AI in China must adhere to strict socialist values. I wish that was a joke, but it’s actually a rule the government is trying to implement. I doubt it isn’t so much safety for the people as it is for the CPC.
- Nevermark 3y ago
  > They also have an existential interest in creating safe ASI. An oppressive state with AGI is no better than an oppressive AGI. > China is the only other state that has anywhere close to the capacity to attempt to go for AGI. Any state can easily recreate/adopt the AGI invented by others. The costs and challenges for any given level of AI tech have dropped precipitously after each advance, and state level actors have far more than enough resources.
- staticman2 3y ago
  Why can't, I don't know, the Canadian government hire some ex OpenAI employees and attempt to make an AGI? What is the constraint here that makes it only possible for the U.S. and China?

theptip 3y ago

Agree with this take. Sam previously stated (eg on Lex’s podcast) that a slow takeoff soon was his goal, to give society maximum time to adjust and to prevent an unexpected fast takeoff from capability overhang. I bought his take when he said it. Going off and trying to accelerate hardware capabilities (especially with an outside company that presumably sells these processors on the open market) seems indefensible in this framework unless you have already solved alignment, which they clearly have not.

alberth 3y ago

Reframing If this was simply reframed as, by creating your own GPUs - it would radically lower OpenAI costs in a material way, it’d be more understandable.

verdverm 3y ago
Google already does this. There is a material difference between doing it in house and starting a for profit company to do the thing