5 ms·

I've always seen the breathless Singularitarian worrying about AI Alignment as a smokescreen to distract people from thinking clearly about the more pedestrian

I've always seen the breathless Singularitarian worrying about AI Alignment as a smokescreen to distract people from thinking clearly about the more pedestrian hazards of AI that isn't self-improving or superhuman, from algorithmic bias, to policy-washing, to energy costs and acceleration of wealth concentration. It also leads to so-called longtermism - discounting the benefits of solving current real problems and focusing entirely on solving a hypothetical one that you think will someday make them all irrelevant.

philipov 1y ago
yep, the biggest threat posed by AI comes from the capitalists who want to own it.
- parpfish 1y ago
  Or the propagandists that use it
  - bilbo0s 1y ago
    They won't be allowed to use it unless they serve the capitalists who own it. It's not social media. It's a model the capitalists train and own. Best the rest of us will have access to are open source ones. It's like the difference between trying to go into court backed by google searches as opposed to Lexis/Nexis. You're gonna have a bad day with the judge. Here's hoping the open source stuff gets trained on quality data rather than reddit and 4chan. Given how the courts are leaning on copyright, and lack of vetted data outside copyright holder remit, I'm not sanguine about the chances of parity long term.
    - int_19h 1y ago
      Looking at Grok especially, it doesn't feel like a given that you can have a true SOTA model that is properly brainwashed.
      - thrance 1y ago
        To be fair, Musk is probably the least subtle megabillionaire of them all, and that reflects in the odd behavior of his silicon child. I don't doubt in the competency of the likes of Thiel to build their techo-monarchy.
        int_19h 1y ago
        The problem for them is that it might be a rather fundamental limitation. We already know that RLHF makes models dumber. It is entirely possible that, in order to make the model buy fully into what those people are peddling, the amount of forceful training required would crater the model's overall performance.
  - thrance 1y ago
    The propagandists serve the capitalists, so it's all the same.
- impossiblefork 1y ago
  I actually think the people developing AI might well not get rich off it. Instead, unless there's a single winner, we will probably see the knowledge on how to train big LLMs and make them perform well diffuse throughout a large pool of AI researchers, with the hardware to train models reasonably close to the SotA becoming more quite accessible. I think the people who will benefit will be the owners of ordinary but hard-to-dislodge software firms, maybe those that have a hardware component. Maybe firms like Apple, maybe car manufacturers. Pure software firms might end up having AI assisted programmers as competitors instead, pushing margins down. This is of course pretty speculative, and it's not reality yet, since firms like Cursor etc. have high valuations, but I think this is what you'd get from the probably pressure if it keeps getting better.
  - cogman10 1y ago
    It smacks of a goldrush. The winners will be the people selling shovels (nVidia) and housing (AWS). It may also be the guides showing people the mountains (Cursor, OpenAI, etc). I suspect you'll see a few people "win" or strike it rich with AI, the vast majority will simply be left with a big bill.
  - nradov 1y ago
    When railroads were first being built across the continental USA, those companies also had high valuations (for the time). Most of them ultimately went bankrupt or were purchased for a fraction of their peak valuation. But the tracks remained, and many of those routes are still in use today.
    - bilbo0s 1y ago
      Just checked. The problem is the railroads were purchased by the winners. Who turned out to be the existing winners. Who then went on to continue to win. On the one hand, I guess that's just life here in reality. On the other, man, reality sucks sometimes.
      - baq 1y ago
        That’s capitalism for you - loser’s margins were winner’s opportunities. Imagine if they were bought by losers.
- conception 1y ago
  Nah, it’s from state actors using the technology to control and wage war by far.
tuveson 1y ago
My feeling has been that it’s a lot of people that work on B2B SaaS that are sad they hadn’t gotten the chance to work on the Manhattan Project. Be around the smartest people in your field. Contribute something significant (but dangerous! And we need to talk about it!) to humanity. But yeah computer science in the 21st century has not turned out to be as interesting as that. Maybe just as important! But Jeff Bezos important, not Richard Feynman important.
- HPsquared 1y ago
  "Overproduction of elites" is the expression.
thom 1y ago
The Singularitarians were breathlessly worrying 20+ years ago, when AI was absolute dogshit - Eliezer once stated that Doug Lenat was incautious in launching Eurisko because it could've gone through a hard takeoff. I don't think it's just an act to launder their evil plans, none of which at the time worked.
- salynchnew 1y ago
  Yeah, people were generally terrified of this stuff back before you could make money off of it.
- notahacker 1y ago
  Fair. OpenAI totally use those arguments to launder their plans, but that saga has been more Silicon Valley exploiting longstanding rationalist beliefs for PR purposes than rationalists getting rich... Eliezer did once state his intentions to build "friendly AI", but seems to have been thwarted by his first order reasoning about how AI decision theory should work being more important to him than building something that actually did work, even when others figured out the latter bit.
NoMoreNicksLeft 1y ago
>s a smokescreen to distract people from thinking clearly about the more pedestrian hazards of AI that isn't self-improving or superhuman, Anything that can't be self-improving or superhuman almost certainly isn't worthy of the moniker "AI". A true AI will be born into a world that has already unlocked the principles of intelligence. Humans in that world would be capable themselves of improving AI (slowly), but the AI itself will (presumably) run on silicon and be a quick thinker. It will be able to self-improve, rapidly at first, and then more rapidly as its increased intelligence allows for even quicker rates of improvement. And if not superhuman initially, it would soon become so. We don't even have anything resembling real AI at the moment. Generative models are probably some blind alley.
- danans 1y ago
  > We don't even have anything resembling real AI at the moment. Generative models are probably some blind alley. I think that the OP's point was that it doesn't matter whether it's "real AI" or not. Even if it's just a glorified auto-correct system, it's one that has the clear potential to overturn our information/communication systems and our assumptions about individuals' economic value.
  - NoMoreNicksLeft 1y ago
    If that has the potential to ruin economies, then the economic rot is so much more profound than anyone (me included) ever realized.
    - jay_kyburz 1y ago
      I think when the GP says "our assumptions about individuals' economic value." they mean half the workforce becoming unemployed because the auto corrector can do it cheaper. That's going to be a swift kick to your economy, no matter how strong.
      - NoMoreNicksLeft 1y ago
        If the "autocorrector" can do anything that used to be a paid wage/salary "cheaper", this isn't the case of another job being automated away. Either something is grossly wrong in that some are willing to have a stupid machine do that poorly (writing Sports Illustrated articles), or something was always grossly wrong in that we were paying someone to do something that never did matter (paying someone to write SEO spam). This isn't a swift kick in the economy, this is a "your economy was so weak a soft breeze knocked it over".
iNic 1y ago
From my experience people who worry about existential AI risk tend to worry more about the more mundane problems than the wider population. In particular they have been very vocal about scamming, cube security and also wealth (& power!) concentration.

philipov 1y ago

yep, the biggest threat posed by AI comes from the capitalists who want to own it.

parpfish 1y ago
Or the propagandists that use it
- bilbo0s 1y ago
  They won't be allowed to use it unless they serve the capitalists who own it. It's not social media. It's a model the capitalists train and own. Best the rest of us will have access to are open source ones. It's like the difference between trying to go into court backed by google searches as opposed to Lexis/Nexis. You're gonna have a bad day with the judge. Here's hoping the open source stuff gets trained on quality data rather than reddit and 4chan. Given how the courts are leaning on copyright, and lack of vetted data outside copyright holder remit, I'm not sanguine about the chances of parity long term.
  - int_19h 1y ago
    Looking at Grok especially, it doesn't feel like a given that you can have a true SOTA model that is properly brainwashed.
    - thrance 1y ago
      To be fair, Musk is probably the least subtle megabillionaire of them all, and that reflects in the odd behavior of his silicon child. I don't doubt in the competency of the likes of Thiel to build their techo-monarchy.
      - int_19h 1y ago
        The problem for them is that it might be a rather fundamental limitation. We already know that RLHF makes models dumber. It is entirely possible that, in order to make the model buy fully into what those people are peddling, the amount of forceful training required would crater the model's overall performance.
- thrance 1y ago
  The propagandists serve the capitalists, so it's all the same.
impossiblefork 1y ago
I actually think the people developing AI might well not get rich off it. Instead, unless there's a single winner, we will probably see the knowledge on how to train big LLMs and make them perform well diffuse throughout a large pool of AI researchers, with the hardware to train models reasonably close to the SotA becoming more quite accessible. I think the people who will benefit will be the owners of ordinary but hard-to-dislodge software firms, maybe those that have a hardware component. Maybe firms like Apple, maybe car manufacturers. Pure software firms might end up having AI assisted programmers as competitors instead, pushing margins down. This is of course pretty speculative, and it's not reality yet, since firms like Cursor etc. have high valuations, but I think this is what you'd get from the probably pressure if it keeps getting better.
- cogman10 1y ago
  It smacks of a goldrush. The winners will be the people selling shovels (nVidia) and housing (AWS). It may also be the guides showing people the mountains (Cursor, OpenAI, etc). I suspect you'll see a few people "win" or strike it rich with AI, the vast majority will simply be left with a big bill.
- nradov 1y ago
  When railroads were first being built across the continental USA, those companies also had high valuations (for the time). Most of them ultimately went bankrupt or were purchased for a fraction of their peak valuation. But the tracks remained, and many of those routes are still in use today.
  - bilbo0s 1y ago
    Just checked. The problem is the railroads were purchased by the winners. Who turned out to be the existing winners. Who then went on to continue to win. On the one hand, I guess that's just life here in reality. On the other, man, reality sucks sometimes.
    - baq 1y ago
      That’s capitalism for you - loser’s margins were winner’s opportunities. Imagine if they were bought by losers.
conception 1y ago
Nah, it’s from state actors using the technology to control and wage war by far.

tuveson 1y ago

My feeling has been that it’s a lot of people that work on B2B SaaS that are sad they hadn’t gotten the chance to work on the Manhattan Project. Be around the smartest people in your field. Contribute something significant (but dangerous! And we need to talk about it!) to humanity. But yeah computer science in the 21st century has not turned out to be as interesting as that. Maybe just as important! But Jeff Bezos important, not Richard Feynman important.

HPsquared 1y ago
"Overproduction of elites" is the expression.

thom 1y ago

The Singularitarians were breathlessly worrying 20+ years ago, when AI was absolute dogshit - Eliezer once stated that Doug Lenat was incautious in launching Eurisko because it could've gone through a hard takeoff. I don't think it's just an act to launder their evil plans, none of which at the time worked.

salynchnew 1y ago
Yeah, people were generally terrified of this stuff back before you could make money off of it.
notahacker 1y ago
Fair. OpenAI totally use those arguments to launder their plans, but that saga has been more Silicon Valley exploiting longstanding rationalist beliefs for PR purposes than rationalists getting rich... Eliezer did once state his intentions to build "friendly AI", but seems to have been thwarted by his first order reasoning about how AI decision theory should work being more important to him than building something that actually did work, even when others figured out the latter bit.

NoMoreNicksLeft 1y ago

>s a smokescreen to distract people from thinking clearly about the more pedestrian hazards of AI that isn't self-improving or superhuman, Anything that can't be self-improving or superhuman almost certainly isn't worthy of the moniker "AI". A true AI will be born into a world that has already unlocked the principles of intelligence. Humans in that world would be capable themselves of improving AI (slowly), but the AI itself will (presumably) run on silicon and be a quick thinker. It will be able to self-improve, rapidly at first, and then more rapidly as its increased intelligence allows for even quicker rates of improvement. And if not superhuman initially, it would soon become so. We don't even have anything resembling real AI at the moment. Generative models are probably some blind alley.

danans 1y ago
> We don't even have anything resembling real AI at the moment. Generative models are probably some blind alley. I think that the OP's point was that it doesn't matter whether it's "real AI" or not. Even if it's just a glorified auto-correct system, it's one that has the clear potential to overturn our information/communication systems and our assumptions about individuals' economic value.
- NoMoreNicksLeft 1y ago
  If that has the potential to ruin economies, then the economic rot is so much more profound than anyone (me included) ever realized.
  - jay_kyburz 1y ago
    I think when the GP says "our assumptions about individuals' economic value." they mean half the workforce becoming unemployed because the auto corrector can do it cheaper. That's going to be a swift kick to your economy, no matter how strong.
    - NoMoreNicksLeft 1y ago
      If the "autocorrector" can do anything that used to be a paid wage/salary "cheaper", this isn't the case of another job being automated away. Either something is grossly wrong in that some are willing to have a stupid machine do that poorly (writing Sports Illustrated articles), or something was always grossly wrong in that we were paying someone to do something that never did matter (paying someone to write SEO spam). This isn't a swift kick in the economy, this is a "your economy was so weak a soft breeze knocked it over".

iNic 1y ago

From my experience people who worry about existential AI risk tend to worry more about the more mundane problems than the wider population. In particular they have been very vocal about scamming, cube security and also wealth (& power!) concentration.