6 ms·

Situational Awareness: The Decade Ahead

81 points by mellosouls 2y ago

deleted 2y ago
[deleted]
sandbags 2y ago
"We are building machines that can think and reason." I'm looking for the evidence of "thinking" or "reasoning".
echelon 2y ago
> AGI by 2027 is strikingly plausible. GPT-2 to GPT-4 took us from ~preschooler to ~smart high-schooler abilities in 4 years. No it's not. GPT-4 is not a "smart high-schooler". This is so hype-y, it's ridiculous. > Over the past year, the talk of the town has shifted from $10 billion compute clusters to $100 billion clusters to trillion-dollar clusters. Talk amongst the memesters. This is a meme. I work in AI, and I'm building an AI animation company. But let's be realistic -- claims like these are not grounded and are going to result in distrust from stakeholders: investors, policy makers, and the general public.
- Treesrule14 2y ago
  Just to echo what you are saying, I've read the first chapter and I thought the thesis is interesting and the writing is good but I failed to be convinced becuase it makes a lot of classic mistakes you make in science. Even though logical arguments are being made there is no attempt not to overfit the data. The author brings out a lot of stats "smart high-schooler", "effective compute", "OOM", "Test Scores", "inference efficiency" but doesn't do a good job of explaining how the author predicted these things before hand (preregistering) and how they actual will result in new technologies or how we can extrapolate past the trend line. Also in the unhobbling section "Tools: Imagine if humans weren’t allowed to use calculators or computers. We’re only at the beginning here, but ChatGPT can now use a web browser, run some code, and so on. " This is so non-specific (because no one has really commercialized anything with this yet) that I worry that we don't actually know if we can make the kinds of effective tools the author is talking about. Would love some feedback on these critisms
  - Treesrule14 2y ago
    also one funny thing is that the author mentions power constraints, but then doesn't calculate how many terraflops for example the us grid can produce etc.
    - zooq_ai 2y ago
      That's where the Trillion $ cluster comes in. It also includes building power plants, not just data centers
    - ml-anon 2y ago
      That’s around the time he says we should build 1200 shale wells in Pennsylvania.
- golergka 2y ago
  > No it's not. GPT-4 is not a "smart high-schooler". This is so hype-y, it's ridiculous. I'm reading Feynman's lectures on electromagnetism right now, and GPT-4o can answer questions and help me with the math. I doubt that even a smart high school would be able to do it.
- johnthewise 2y ago
  >Talk amongst the memesters. This is a meme. Those memesters invented and deployed the current systems, are currently training the next, and raising funds for the next after. Some scientific breakthroughs were memes at first, but some people believed in them and proved everyone else wrong. Majority were memes and are still memes. I don't think whether this is a fad or not is that interesting, what's interesting is these 'memesters' who are optimistic about these systems have all the capital, determination and brain. Unlike in other times in history where public needed some convincing, whether through President on Manhattan or with Apollo project, memesters are using their own funds so don't need to convince anyone else.
elicksaur 2y ago
Important context, this appears to be a brief intro+table of contents for a 165-page pdf/html book.[0] It’ll probably be easy for HN to take potshots at snippets of the intro, but the writing intrigued me enough that I want to at least skim over the whole thing. [0] pdf link for reference, but the html navigation seems pretty clean in the OP link as well. https://situational-awareness.ai/wp-content/uploads/2024/06/situationalawareness.pdf https://situational-awareness.ai/wp-content/uploads/2024/06/...
- mellosouls 2y ago
  Yes, I (OP) came across the PDF first but sought out an alternative for readability from HN.
  - skilled 2y ago
    I think the website submission is the most optimal for readability. Nice and clean design too, and you can easily navigate the chapters. I don’t mind a PDF (and it is linked on the site), but in this case the web experience feels best.
  - elicksaur 2y ago
    Yeah, I agree! Knowing how people respond, I thought it was useful to add the full context of the breadth and depth of the author’s writing. Already there’s people pulling single line quotes to dismiss. But, that’s pretty typical of the internet.
skilled 2y ago
There are quite a few bold claims just in the introductory chapter, but we should not discount the authors work because of it. Those are his opinions. I personally don’t think LLMs can achieve reasoning, or any kind of innate awareness.
- xeckr 2y ago
  How do explain GPT-3's success with handling the prompt "make an HTML button that looks like a watermelon" without invoking reasoning?
  - elicksaur 2y ago
    Can you explain it with reasoning? The explanation without is that the LLM takes the prompt text, applies a math formula to produce the next token, iterates that function to get a set of tokens, and returns the response. Where is there reasoning? If that is sufficient for reasoning, is my calculator reasoning when I input 2+2 and it returns 4?
    - xeckr 2y ago
      >Can you explain it with reasoning? If you were asked to make an HTML button that looks like a watermelon, you would start by considering what a watermelon looks like: a green shell, a red interior, and a circular cross-section. You would then take this information and apply it when writing the CSS for your button. What word would you use to describe this process? >The explanation without is that the LLM takes the prompt text, applies a math formula to produce the next token, iterates that function to get a set of tokens, and returns the response. Where is there reasoning? The "formula" is dependent on prior knowledge about the world, which is used to successfully solve a problem that does not appear in the training data. Why assume that reasoning is somehow impervious to mathematical modelling?
      - elicksaur 2y ago
        Sure, that’s essentially what a person would do. Describing that process says nothing about what an LLM does, which we know by definition of the system is applying a set of weights to a text string input. If you want to reduce all reasoning to just math, then LLMs reason. But then saying something like the OP of “we’ve made computers that reason” is not very novel or useful. Under that framework, computers always reasoned. Does ChatGPT even “solve” this problem? How do you know it’s not in the training data? Example of a tutorial for making a watermelon in css from 2017: https://dev.to/munamohamed94/easy-css-watermelon-slice-animation-tutorial-3ngm https://dev.to/munamohamed94/easy-css-watermelon-slice-anima...
mayilian 2y ago
The author is yet a child. The hype inside him is normal.
- johnthewise 2y ago
  It's not just him though. His views are very common in AI labs, who actually build these systems and have better visibility than the rest of us on what's going on.
- gaws 2y ago
  For context: The author was 19 years old when he graduated from Columbia University in 2021. He's about 22 now.
reducesuffering 2y ago
If you prefer video format, the author, former OpenAI superalignment team, also just had an interview with Dwarkesh. https://www.youtube.com/watch?v=zdbVtZIn9IM https://www.youtube.com/watch?v=zdbVtZIn9IM
deleted 2y ago
[deleted]
z7 2y ago
His predictions: >AGI by 2027 is strikingly plausible. GPT-2 to GPT-4 took us from ~preschooler to ~smart high-schooler abilities in 4 years. Tracing trendlines in compute (~0.5 orders of magnitude or OOMs/year), algorithmic efficiencies (~0.5 OOMs/year), and “unhobbling” gains (from chatbot to agent), we should expect another preschooler-to-high-schooler-sized qualitative jump by 2027. >I make the following claim: it is strikingly plausible that by 2027, models will be able to do the work of an AI researcher/engineer. That doesn’t require believing in sci-fi; it just requires believing in straight lines on a graph.
- zurfer 2y ago
  I share the authors view that if things align and compound we are in for a very wild ride. However, I feel the bull case is over emphasized. Not all likely "walls" are sufficiently discussed, some that I think about: - will AI blow past human general intelligence or level off? (because of diminishing returns and "weak" human teachers) - can AI really achieve online learning (economically)? the current architecture can't (just shoving things into large context windows doesn't seem to be enough) - do we as a society tolerate this rapid change or will there be interventions? (protest because people don't want to lose their jobs)
- AnimalMuppet 2y ago
  > Tracing trendlines in compute (~0.5 orders of magnitude or OOMs/year) 0.5 orders of magnitude is about a factor of 3. We're growing by a factor of 3 in compute every year? I find that rather hard to believe. Oh, echelon points out the quote: > Over the past year, the talk of the town has shifted from $10 billion compute clusters to $100 billion clusters to trillion-dollar clusters. That's how they're getting a factor of 3 in compute every year - by spending more money every year. That's... probably not sustainable for four more years. If it is, it's not sustainable much past that. You're probably not going to see a ten-trillion-dollar cluster; you're definitely not going to see a one-hundred-trillion-dollar cluster, ever. > That doesn’t require believing in sci-fi; it just requires believing in straight lines on a graph. I don't believe in straight lines on a graph. Or rather, I believe that lines are straight over a short enough time span, but may not continue to be straight for, say, twice that time span. Or, they may stay straight for that long. But I don't rate the plausibility as high as he does. (Especially on the "spend more to increase compute" front.)
cdchn 2y ago
Might be AI Maximalism approaching Science Fiction but I still thought it was thought provoking.
fabiofzero 2y ago
AGI won't happen, full stop.
polotics 2y ago
This large piece of text starts kinda convincing but then there are nuggets, like I guess glitches in the matrix, that kind-of ruin the show. for example: """There’s some assumptions that flow into the exact numbers, including that humans “think” at 100 tokens/minute (just a rough order of magnitude estimate, e.g. consider your internal monologue)""" also for example quoting AlphaGo's move 37 against Lee Sedol, but failing to quote the fact that AlphaGo's domination against even non-grandmasters was soon enough destroyed (see https://arstechnica.com/information-technology/2023/02/man-beats-machine-at-go-in-human-victory-over-ai/ https://arstechnica.com/information-technology/2023/02/man-b...) This makes me think the whole thing is not really science. For example there is very little about energy costs and energy availability limits in there, just the amazing statement that the US only has to burn lots of its nat-gas (emitting lots of CO2 btw.) to get the power needed.
- ml-anon 2y ago
  It doesn’t start off convincing though. His premise is that since 2018-2024 follows exponential growth, 2024 onward will. Contrast with Yann’s oft repeated mantra that what looks like exponential growth is more likely the beginning of a sigmoid. That’s the difference between someone with 3+ decades in the field of AI research vs someone with barely 2 decades on earth.
  - nopromisessir 2y ago
    This is a fair critism. However, I don't think it matters much, to the overall argument he makes. Why? We don't need AGI to see massive geopolitical disruption. We are already seeing this. The US has put up a GPU wall around China. That is evidence enough. The capability we have already is enough to greatly upset the balance of power in a variety of spheres... Both sides lag in implementation. The capability is alresdy there. Capability will grow. AGI is not relevant to the concern for security. We've past that point already. The labs are a key NAT security asset. They just are privately owned/operated... For now. This is the way of things. History shows this.
ml-anon 2y ago
“You can see the future in San Francisco”, said unironically by a 25 year old who has worked at OpenAI for 1.5 years. It’s amazing how one can be so intelligent and yet sound so eyerollingly dumb at the same time.

81 points by mellosouls 2y ago

deleted 2y ago

[deleted]

sandbags 2y ago

"We are building machines that can think and reason." I'm looking for the evidence of "thinking" or "reasoning".

echelon 2y ago

> AGI by 2027 is strikingly plausible. GPT-2 to GPT-4 took us from ~preschooler to ~smart high-schooler abilities in 4 years. No it's not. GPT-4 is not a "smart high-schooler". This is so hype-y, it's ridiculous. > Over the past year, the talk of the town has shifted from $10 billion compute clusters to $100 billion clusters to trillion-dollar clusters. Talk amongst the memesters. This is a meme. I work in AI, and I'm building an AI animation company. But let's be realistic -- claims like these are not grounded and are going to result in distrust from stakeholders: investors, policy makers, and the general public.

Treesrule14 2y ago
Just to echo what you are saying, I've read the first chapter and I thought the thesis is interesting and the writing is good but I failed to be convinced becuase it makes a lot of classic mistakes you make in science. Even though logical arguments are being made there is no attempt not to overfit the data. The author brings out a lot of stats "smart high-schooler", "effective compute", "OOM", "Test Scores", "inference efficiency" but doesn't do a good job of explaining how the author predicted these things before hand (preregistering) and how they actual will result in new technologies or how we can extrapolate past the trend line. Also in the unhobbling section "Tools: Imagine if humans weren’t allowed to use calculators or computers. We’re only at the beginning here, but ChatGPT can now use a web browser, run some code, and so on. " This is so non-specific (because no one has really commercialized anything with this yet) that I worry that we don't actually know if we can make the kinds of effective tools the author is talking about. Would love some feedback on these critisms
- Treesrule14 2y ago
  also one funny thing is that the author mentions power constraints, but then doesn't calculate how many terraflops for example the us grid can produce etc.
  - zooq_ai 2y ago
    That's where the Trillion $ cluster comes in. It also includes building power plants, not just data centers
  - ml-anon 2y ago
    That’s around the time he says we should build 1200 shale wells in Pennsylvania.
golergka 2y ago
> No it's not. GPT-4 is not a "smart high-schooler". This is so hype-y, it's ridiculous. I'm reading Feynman's lectures on electromagnetism right now, and GPT-4o can answer questions and help me with the math. I doubt that even a smart high school would be able to do it.
johnthewise 2y ago
>Talk amongst the memesters. This is a meme. Those memesters invented and deployed the current systems, are currently training the next, and raising funds for the next after. Some scientific breakthroughs were memes at first, but some people believed in them and proved everyone else wrong. Majority were memes and are still memes. I don't think whether this is a fad or not is that interesting, what's interesting is these 'memesters' who are optimistic about these systems have all the capital, determination and brain. Unlike in other times in history where public needed some convincing, whether through President on Manhattan or with Apollo project, memesters are using their own funds so don't need to convince anyone else.

elicksaur 2y ago

Important context, this appears to be a brief intro+table of contents for a 165-page pdf/html book.[0] It’ll probably be easy for HN to take potshots at snippets of the intro, but the writing intrigued me enough that I want to at least skim over the whole thing. [0] pdf link for reference, but the html navigation seems pretty clean in the OP link as well. https://situational-awareness.ai/wp-content/uploads/2024/06/situationalawareness.pdf https://situational-awareness.ai/wp-content/uploads/2024/06/...

mellosouls 2y ago
Yes, I (OP) came across the PDF first but sought out an alternative for readability from HN.
- skilled 2y ago
  I think the website submission is the most optimal for readability. Nice and clean design too, and you can easily navigate the chapters. I don’t mind a PDF (and it is linked on the site), but in this case the web experience feels best.
- elicksaur 2y ago
  Yeah, I agree! Knowing how people respond, I thought it was useful to add the full context of the breadth and depth of the author’s writing. Already there’s people pulling single line quotes to dismiss. But, that’s pretty typical of the internet.

skilled 2y ago

There are quite a few bold claims just in the introductory chapter, but we should not discount the authors work because of it. Those are his opinions. I personally don’t think LLMs can achieve reasoning, or any kind of innate awareness.

xeckr 2y ago
How do explain GPT-3's success with handling the prompt "make an HTML button that looks like a watermelon" without invoking reasoning?
- elicksaur 2y ago
  Can you explain it with reasoning? The explanation without is that the LLM takes the prompt text, applies a math formula to produce the next token, iterates that function to get a set of tokens, and returns the response. Where is there reasoning? If that is sufficient for reasoning, is my calculator reasoning when I input 2+2 and it returns 4?
  - xeckr 2y ago
    >Can you explain it with reasoning? If you were asked to make an HTML button that looks like a watermelon, you would start by considering what a watermelon looks like: a green shell, a red interior, and a circular cross-section. You would then take this information and apply it when writing the CSS for your button. What word would you use to describe this process? >The explanation without is that the LLM takes the prompt text, applies a math formula to produce the next token, iterates that function to get a set of tokens, and returns the response. Where is there reasoning? The "formula" is dependent on prior knowledge about the world, which is used to successfully solve a problem that does not appear in the training data. Why assume that reasoning is somehow impervious to mathematical modelling?
    - elicksaur 2y ago
      Sure, that’s essentially what a person would do. Describing that process says nothing about what an LLM does, which we know by definition of the system is applying a set of weights to a text string input. If you want to reduce all reasoning to just math, then LLMs reason. But then saying something like the OP of “we’ve made computers that reason” is not very novel or useful. Under that framework, computers always reasoned. Does ChatGPT even “solve” this problem? How do you know it’s not in the training data? Example of a tutorial for making a watermelon in css from 2017: https://dev.to/munamohamed94/easy-css-watermelon-slice-animation-tutorial-3ngm https://dev.to/munamohamed94/easy-css-watermelon-slice-anima...

mayilian 2y ago

The author is yet a child. The hype inside him is normal.

johnthewise 2y ago
It's not just him though. His views are very common in AI labs, who actually build these systems and have better visibility than the rest of us on what's going on.
gaws 2y ago
For context: The author was 19 years old when he graduated from Columbia University in 2021. He's about 22 now.

reducesuffering 2y ago

If you prefer video format, the author, former OpenAI superalignment team, also just had an interview with Dwarkesh. https://www.youtube.com/watch?v=zdbVtZIn9IM https://www.youtube.com/watch?v=zdbVtZIn9IM

deleted 2y ago

[deleted]

z7 2y ago

His predictions: >AGI by 2027 is strikingly plausible. GPT-2 to GPT-4 took us from ~preschooler to ~smart high-schooler abilities in 4 years. Tracing trendlines in compute (~0.5 orders of magnitude or OOMs/year), algorithmic efficiencies (~0.5 OOMs/year), and “unhobbling” gains (from chatbot to agent), we should expect another preschooler-to-high-schooler-sized qualitative jump by 2027. >I make the following claim: it is strikingly plausible that by 2027, models will be able to do the work of an AI researcher/engineer. That doesn’t require believing in sci-fi; it just requires believing in straight lines on a graph.

zurfer 2y ago
I share the authors view that if things align and compound we are in for a very wild ride. However, I feel the bull case is over emphasized. Not all likely "walls" are sufficiently discussed, some that I think about: - will AI blow past human general intelligence or level off? (because of diminishing returns and "weak" human teachers) - can AI really achieve online learning (economically)? the current architecture can't (just shoving things into large context windows doesn't seem to be enough) - do we as a society tolerate this rapid change or will there be interventions? (protest because people don't want to lose their jobs)
AnimalMuppet 2y ago
> Tracing trendlines in compute (~0.5 orders of magnitude or OOMs/year) 0.5 orders of magnitude is about a factor of 3. We're growing by a factor of 3 in compute every year? I find that rather hard to believe. Oh, echelon points out the quote: > Over the past year, the talk of the town has shifted from $10 billion compute clusters to $100 billion clusters to trillion-dollar clusters. That's how they're getting a factor of 3 in compute every year - by spending more money every year. That's... probably not sustainable for four more years. If it is, it's not sustainable much past that. You're probably not going to see a ten-trillion-dollar cluster; you're definitely not going to see a one-hundred-trillion-dollar cluster, ever. > That doesn’t require believing in sci-fi; it just requires believing in straight lines on a graph. I don't believe in straight lines on a graph. Or rather, I believe that lines are straight over a short enough time span, but may not continue to be straight for, say, twice that time span. Or, they may stay straight for that long. But I don't rate the plausibility as high as he does. (Especially on the "spend more to increase compute" front.)

cdchn 2y ago

Might be AI Maximalism approaching Science Fiction but I still thought it was thought provoking.

fabiofzero 2y ago

AGI won't happen, full stop.

polotics 2y ago

This large piece of text starts kinda convincing but then there are nuggets, like I guess glitches in the matrix, that kind-of ruin the show. for example: """There’s some assumptions that flow into the exact numbers, including that humans “think” at 100 tokens/minute (just a rough order of magnitude estimate, e.g. consider your internal monologue)""" also for example quoting AlphaGo's move 37 against Lee Sedol, but failing to quote the fact that AlphaGo's domination against even non-grandmasters was soon enough destroyed (see https://arstechnica.com/information-technology/2023/02/man-beats-machine-at-go-in-human-victory-over-ai/ https://arstechnica.com/information-technology/2023/02/man-b...) This makes me think the whole thing is not really science. For example there is very little about energy costs and energy availability limits in there, just the amazing statement that the US only has to burn lots of its nat-gas (emitting lots of CO2 btw.) to get the power needed.

ml-anon 2y ago
It doesn’t start off convincing though. His premise is that since 2018-2024 follows exponential growth, 2024 onward will. Contrast with Yann’s oft repeated mantra that what looks like exponential growth is more likely the beginning of a sigmoid. That’s the difference between someone with 3+ decades in the field of AI research vs someone with barely 2 decades on earth.
- nopromisessir 2y ago
  This is a fair critism. However, I don't think it matters much, to the overall argument he makes. Why? We don't need AGI to see massive geopolitical disruption. We are already seeing this. The US has put up a GPU wall around China. That is evidence enough. The capability we have already is enough to greatly upset the balance of power in a variety of spheres... Both sides lag in implementation. The capability is alresdy there. Capability will grow. AGI is not relevant to the concern for security. We've past that point already. The labs are a key NAT security asset. They just are privately owned/operated... For now. This is the way of things. History shows this.

ml-anon 2y ago

“You can see the future in San Francisco”, said unironically by a 25 year old who has worked at OpenAI for 1.5 years. It’s amazing how one can be so intelligent and yet sound so eyerollingly dumb at the same time.