3 ms·

To me this reads like an old guy scared of new technology he does not understand, with a track record in astronomy so easily able to get papers published, pulli

To me this reads like an old guy scared of new technology he does not understand, with a track record in astronomy so easily able to get papers published, pulling numbers out of thin air with regards to the hypothetical emergence of artificial superintelligence within the next 15 years. Then plugging this random guess into the Drake equation and getting an answer close to one end of the scale. I've said it before and I'll say it again: despite the impressive advancements in current LLMs, there is absolutely zero evidence supporting the notion that these AIs want to do anything at all. They simply respond to prompts. They don't even remember anything, the entire previous conversation needs to be included in every prompt. There is no technical mechanism by which they are able to possess or attain agency. My best bet for the explanation of the Fermi paradox is simply the intersection of economics and interstellar travel timescales. The payback time for any "interstellar colonisation" is measured in centuries or millenia, and will require tens of thousands of people to condemn both themselves and their children and grandchildren to never see anything else than the inside of a spaceship. The only factor extreme enough to motivate humans to such endeavours is impending planetary destruction, which does not result in anything like exponential growth.

llamaimperative 2y ago
How this argument reads to someone concerned about the trajectory of AI and the governance systems around it (current or future): There are various technical problems preventing these systems from being as scary as you imagine, but don't worry because we're working hard to solve them! Also "agency" isn't a requirement for almost any AI doom scenario I've heard of. Goals and the formation of instrumental goals is all it takes.
- romafirst3 2y ago
  But “goals” is doing an insane amount of heavy lifting here. In order to have goals you need to have a model of the world. LLMs do not have a model of the world. Your goal then is changing this world somehow so you need actuators. So you need a model and actuators. All we have is a stream of words appended in a probabilistic manner.
  - llamaimperative 2y ago
    If only there were some examples of streams of words being used to enlist people to build (or become) actuators for evil purposes…
- semi-extrinsic 2y ago
  It's not that there are "technical problems" preventing anything. It's that these systems (LLMs) do not possess the fundamental properties that are necessary for being scary, and we have no clue at all how the systems would get those properties. Current "hyperintelligent AI" fears are exactly like the "grey goo" fear when nanotechnology was a buzzword in the early 2000s. We can all agree that if you take the fundamental vague concept "AI" or "nanotechnology" and pile on a couple of wheelbarrows full of hypotheticals, you get to something scary. That doesn't mean it has any relevance for the universe we currently exist in.
  - llamaimperative 2y ago
    And yet you qualify your statement with “current.” Why? People anticipating future risks and discussing how and when to detect and mitigate them is actually exactly what we should be doing.
romafirst3 2y ago
This. Llms aren’t very good and are just predictive text. And I say that as someone who has a lot of experience using them. We don’t have anything to fear from AI for quite some time. As an aside, the AI generated search snippets in google are messing me up. They seem good for random things I don’t know about but are awful for programming things where I’m looking for a specific solution to a specific problem. They always just give me the “right” way no matter how much it doesn’t really work like that. An example would be trying to figure out how to solve a weird edge case or error, something that stack overflow is amazing at and it just feeds me generic instructions - which honestly fooled me for a while before I learned to ignore.
rsynnott 2y ago
> The payback time for any "interstellar colonisation" is measured in centuries or millenia Payback? I’m not sure that you’d expect any payback at all; it’s not like shipping stuff _back_ would ever make more economic sense than just making the stuff at the point of origin. The space opera concept of shipping raw materials from interstellar colonies is just absurd on the face of it (unless you have really magic tech, I suppose).
anon84873628 2y ago
Yeah, the article is just a trojan horse to say, "it's scary that modern militaries are starting to use AI."

llamaimperative 2y ago

How this argument reads to someone concerned about the trajectory of AI and the governance systems around it (current or future): There are various technical problems preventing these systems from being as scary as you imagine, but don't worry because we're working hard to solve them! Also "agency" isn't a requirement for almost any AI doom scenario I've heard of. Goals and the formation of instrumental goals is all it takes.

romafirst3 2y ago
But “goals” is doing an insane amount of heavy lifting here. In order to have goals you need to have a model of the world. LLMs do not have a model of the world. Your goal then is changing this world somehow so you need actuators. So you need a model and actuators. All we have is a stream of words appended in a probabilistic manner.
- llamaimperative 2y ago
  If only there were some examples of streams of words being used to enlist people to build (or become) actuators for evil purposes…
semi-extrinsic 2y ago
It's not that there are "technical problems" preventing anything. It's that these systems (LLMs) do not possess the fundamental properties that are necessary for being scary, and we have no clue at all how the systems would get those properties. Current "hyperintelligent AI" fears are exactly like the "grey goo" fear when nanotechnology was a buzzword in the early 2000s. We can all agree that if you take the fundamental vague concept "AI" or "nanotechnology" and pile on a couple of wheelbarrows full of hypotheticals, you get to something scary. That doesn't mean it has any relevance for the universe we currently exist in.
- llamaimperative 2y ago
  And yet you qualify your statement with “current.” Why? People anticipating future risks and discussing how and when to detect and mitigate them is actually exactly what we should be doing.

romafirst3 2y ago

This. Llms aren’t very good and are just predictive text. And I say that as someone who has a lot of experience using them. We don’t have anything to fear from AI for quite some time. As an aside, the AI generated search snippets in google are messing me up. They seem good for random things I don’t know about but are awful for programming things where I’m looking for a specific solution to a specific problem. They always just give me the “right” way no matter how much it doesn’t really work like that. An example would be trying to figure out how to solve a weird edge case or error, something that stack overflow is amazing at and it just feeds me generic instructions - which honestly fooled me for a while before I learned to ignore.

rsynnott 2y ago

> The payback time for any "interstellar colonisation" is measured in centuries or millenia Payback? I’m not sure that you’d expect any payback at all; it’s not like shipping stuff _back_ would ever make more economic sense than just making the stuff at the point of origin. The space opera concept of shipping raw materials from interstellar colonies is just absurd on the face of it (unless you have really magic tech, I suppose).

anon84873628 2y ago

Yeah, the article is just a trojan horse to say, "it's scary that modern militaries are starting to use AI."