10 ms·

There is no hard takeoff

156 points by WithinReason 3y ago

WithinReason 3y ago
The main issue with this argument that current AI models are extremely inefficient. Models evaluate all weights on every pass, LLMs recall their entirety of knowledge just to output a part of a word and then do it all again. Often a hand crafted algorithm can achieve what a neural network can in a small fraction of compute. There is likely a 1e4x-1e6x compute gap that can be achieved with the right algorithm, maybe even a 1e10x and once we reach self improvement that gap could close very fast. They say that the human brain has 30 PFLOPS of compute power and this is sometimes used as a reference for how much compute is needed for intelligence, but that completely misses the point: The human brain is extremely inefficient. Symbolic mathematics can be unreasonably [1] effective at making accurate predictions about the future with very little computation, but many of these solutions are inaccessible to evolution. Once AI learns to construct efficient mathematical models of the universe, the amount of computation available today in a PC could be enough to do things (good and bad) way beyond human ability. [1]: https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_of_Mathematics_in_the_Natural_Sciences https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness...
- rini17 3y ago
  Before the first AI winter there was incredible effort in leveraging symbolical approach. It did deliver many useful things but far from hype. For example we still have no self-repairing systems, only specific fault-tolerant algoritms but by and large software is as fragile as ever. You can't avoid considering everything when trying to deal with unexpected data/issue. Only in hidsight you can then say "wow that was sooo inefficient".
  - WithinReason 3y ago
    A calculator can compute the 5th root of 1692662195786551 much more efficiently than a person
- bob1029 3y ago
  > the amount of computation available today in a PC could be enough to do things (good and bad) way beyond human ability. I am in the camp that this is possible and that humans have achieved it already. I believe things are going to get very weird when these techniques become mainstream. The GPU-cluster-bound algorithms are just a tiny stepping stone. Do we start taking computers away from people? What do we do about this if it turns out the only barrier to entry is the software?
  - ChatGTP 3y ago
    I’d say your going to get “AI” rammed so far down your throat it might as well come out of your ass before anyone is going to let their puters be taken away.
    - pcthrowaway 3y ago
      Maybe our desktops and laptops get taken away, and they leave us with our smartphones (which are locked down for our safety). 90% of people wouldn't give a shit as long as they have their smartphones
      - ChatGTP 3y ago
        It's true
- jillesvangurp 3y ago
  It's not an issue, it's an optimization problem. I look at this as an exponential. Before chat gpt was launched a bit over a year ago, the amount of AI in our lives was relatively low. We had bits and pieces here and there but everyday life wasn't really affected by it. At best you might be yelling at siri or alexa to do whatever and maybe getting some OKish results. Fast forward 1 year. We now have students revolutionizing education (by submitting AI doctored papers and homework that isn't half bad). People in law dealing with people using AI. AIs passing medical, legal, and other exams. That was just one year. And yes, it's tediously slow to use. But last year we had nothing. It's going to be an exponential in terms of adoption, response speed, energy usage, parameter numbers, context size and a lot of optimization. I think we are not that far off from AIs being able to keep up with a conversation and being able to respond right away. For all you hitchhiker's to the galaxy readers, that would be babelfish sorted. Translation quality is already pretty awesome with chat gpt 4. I've not caught it making major translation mistakes. Speech to text, translate the text, text to speech. Now we are conversing with our chat bots rather than hammering out sentences in some text box. It's hard to not see that go from a novelty "oh this is cool" to world + dog basically using this on a daily basis throughout the day. At the same time it will get smarter. Everybody is waiting for the singularity where it is clearly smarter than Einstein in his prime and running circles around everyone else. But for most questions you might ask chat gpt and get a reasonable answer, the vast majority of people around you might be pretty useless. Is that smarter? I don't know. But definitely more useful. From where we are now to this being ubiquitous and used by pretty much anyone is probably a few years. I remember when smart phones happened. One day everybody was minding their own business and a year later the streets were full of zombies glued to their screen. Just happened very quickly. This might be the same but possibly quicker.
- audunw 3y ago
  > The human brain is extremely inefficient. That depends heavily on what kind of task we're talking about. The human brain uses just 12 watts, and with that it can still perform certain tasks that a computer using 100 times as much power can't even approach. I get the point you were trying to make. In certain tasks it's extremely inefficient, yes. But in general, it's still crazy how power efficient it is. > Once AI learns to construct efficient mathematical models of the universe There's not going to be just one model. Whether an AI (in the short term) can be more efficient than a human at discovering these efficient mathematical models remains to be seen. It seems like the creativity and discovery needed to do this is exactly what neural nets can be good at, but then you're back to something fairly inefficient (with todays algorithms)
- rafaelero 3y ago
  > Models evaluate all weights on every pass Not true for GPT-4.
ilaksh 3y ago
I am all for AI research and integrating more AI use into society. Currently working on tools based on GPT. I think it has incredible potential to help humans. But at the same time, I am sure that AI does not need to have a hard takeoff to be extremely dangerous. It just needs to get a bit smarter and somewhat faster every few months. Within less than a decade we will have systems that output "thoughts" and actions at least dozens of times faster than humans. That will be dangerous if we aren't cautious. We should start thinking now about limiting the performance of AI hardware. The challenge is that increasing the speed is such a competitive advantage, it creates a race. That is a concern when you put it into a military context. The CEO of Palantir has already called for a Manhattan Project for superintelligent AI weapons control systems.
- anileated 3y ago
  To see how “AI” can be dangerous just look how social media bubble/recommendation algorithms would radicalize people, even with much cruder ML. People tend to miss that it’s not about some model starting to “think” and act all sci-fi evil, just us humans applying powerful tech in irresponsible ways that we either don’t bother to assess due to lack of awareness or assess positively due to a conflict of interest (money, career, etc.) is already enough to cause trouble.
  - FeepingCreature 3y ago
    If a human can apply AI irresponsibly, the AI can probably also learn to apply AI irresponsibly.
    - anileated 3y ago
      Not really, humans are the ones who think and have agency.
      - FeepingCreature 3y ago
        Agency? Sounds like a problem that can be phrased as a text completion task to me! ChatGPT can already emulate beings with agency, see LangChain. It's just very bad at it. It's gonna get better.
neatze 3y ago
> Oh wait…every hedge fund bro is already doing this. And most of them aren’t billionaires. The problem is your model needs to include all the computers playing the market, and it also needs to include the other hedge fund bros themselves. This strategy only dominates if you have more compute than the whole market itself, which you don’t. Best part of this article, honestly, I did not expected this, of-course one can say we just need more abstractions (how does brain/living things build them?), but this would be ignoring dynamical nature of such problems.
- petercooper 3y ago
  It reminds me of the rocket fuel equation where you have 500kg to take to space, so you need X kg of fuel. Except now you have 500+X kg to take to space and so you need even more fuel. Repeat. This sort of pattern seems to exist in all sorts of diverse situations.
- gorbachev 3y ago
  Yes, every hedge fund bro already does that, but they usually target specific scenarios, they don't model the entire market. The way to make money this way is to find out a unique scenario, and then create a prediction algorithm for that. I'm not close enough to hedge fund bros to know how feasible that is.
- em500 3y ago
  That argument is a variation of the Efficient Market Hypothesis (EMH), that at any moment market prices already reflect all information available to the public. Or as the old joke goes, two economists are walking down the street and pass by a hundred dollar bill without picking it up. A little while later one turns to the other and asks "was that a hundred dollar bill on the ground?" To which the other replies "it must be fake, if it was real someone would have picked it up already." Every graduate level economics and finance student is aware of this joke, and also know EMH with more nuance than "you can't make money with trading". There's tons of research and debate about the various forms of the EMH. But you don't have to believe anything anyone else wrote. There's plenty of data and computing software available, so anyone can try everything for themselves.
smcl 3y ago
> Back in 2014, Elon Musk referred to AI as summoning the demon. And it wasn’t hard to see that view. Soon, Go agents would beat top humans learning from self play. By the end of 2017, the same algorithm mastered Chess and Shogi. By 2020, it didn’t even need tons of calls to the simulator, and could play Atari too. > AI looked scary. It looked like it was one FOOM away from self playing and becoming superhuman at the universe. And yet, here we are in 2023 and self driving cars still don’t work. This is weird. I don't recall this at all. The mainstream press got a little kick out of chess and (to a lesser extent) Go AI turning over various humans at a few points over the years but it's only really burst into the mainstream recently. And where it did get any traction, in tech circles such as our own, response was enthusiastic but definitely more measured. Some were talking a bit about a kind of AI singularity way off into the future, but that was always a very distant and theoretical thing.
- chasd00 3y ago
  > really burst into the mainstream recently. I think what made the LLMs so popular in the mainstream was anyone could go experience it themselves in a familiar, low barrier to entry way with a simple chat prompt.
  - worrycue 3y ago
    IMHO LLMs are so popular because they sound very confident. That’s great marketing in itself. Unfortunately it’s only marketing.
  - smcl 3y ago
    Yeah you're right - I think "AI" as a concept for your average person was quite abstract and hard to wrap their head around, to get some kind of idea of the kind of things it can or cannot do. So having something tangible they can be a bit more hands-on with gives them a better idea of what it does.
- threeseed 3y ago
  > And yet, here we are in 2023 and self driving cars still don’t work. Cruise and Waymo were today given licenses to operate across San Francisco 24/7. Self driving cars do work and are here. They just need LiDAR.
bilsbie 3y ago
This makes good points. And it articulates an idea I’ve been having trouble getting down. What if increasing intelligence is an exponential problem and the reason humans all have somewhat similar intelligence isn’t that we peaked at some level but that even vast additional intelligence just doesn’t get much more traction against the universe of problems. Eg doubling your compute doesn’t get you many more cities in the traveling salesman problem.
- FeepingCreature 3y ago
  IQ is robustly correlated with income though.
- passion__desire 3y ago
  Intelligence is culturally transmitted
- adrianN 3y ago
  It is likely that even vastly more intelligence doesn't increase the number of children you raise to childbearing age in a hunter-gatherer or subsistence farming society. That says very little about vastly more intelligence in a modern industrial society (that only really existed for like ten generations so far).
mysterydip 3y ago
FOOM = Fast Out Of the gate Model, for those not familiar (like myself)
- FeepingCreature 3y ago
  Did you ask GPT that? That'd explain how it didn't notice that "Gate" doesn't start with an "O". Note that ChatGPT has real difficulties resolving words to letters. Anyway, it's not an acronym, it's an onomatopoeia for an explosion. It's a memorable way of representing an intelligence explosion: "AI go foom." (Wild hand gesture strongly recommended.)
  - mysterydip 3y ago
    Thanks, that makes more sense. I found "Fast Out Of (the gate) Model". Seemed odd but it fit so I listed it. I did a google search, but I should have realized that by this point they're tainted by AI answers. For example: https://www.reddit.com/r/singularity/comments/12bisxu/what_does_foom_stand_for_who_coined_this_term/ https://www.reddit.com/r/singularity/comments/12bisxu/what_d...
    - FeepingCreature 3y ago
      I mean, the usage of "foom" in the context of AI, like in the linked post, is the hard-takeoff one explained in that Reddit post. Goes back decades.
    - botro 3y ago
      You've found a 'Traith'. Someone asked ChatGPT a question, ChatGPT did not know the answer and hallucinated one. They post the answer to Reddit / HN / Twitter, google indexes the answer and when someone searches they find the GPT hallucination. In the next GPT training run it's incorporated into the weights. It's an AI generated truth, a Traith. Traith is also defined as "A fishing station or fishing ground, especially for herring" and some herring are red.
boredumb 3y ago
When did foom become a word?
- oefnak 3y ago
  [2008](https://www.lesswrong.com/tag/the-hanson-yudkowsky-ai-foom-debate https://www.lesswrong.com/tag/the-hanson-yudkowsky-ai-foom-d...)
- mkl 3y ago
  It's been an onomatopoeic word from the sound a big fire starting makes for a long time. It seems to have come into use for AI over a decade ago: https://www.overcomingbias.com/p/ai-go-foomhtml https://www.overcomingbias.com/p/ai-go-foomhtml, https://en.wiktionary.org/wiki/foom https://en.wiktionary.org/wiki/foom
  - boredumb 3y ago
    I see. it's like, Vrrroom? The takeoff sound for the AI blast off? Back under my rock I go
mysterydip 3y ago
Do we have enough historical weather data to train a decent AI on? Maybe this is already being done and is just a quieter area of research (in the headline sense)? I know for example every time a hurricane comes toward land all the weather places trot out a half dozen different model predictions saying different things. It would be great if there was more confidence in this area for planning.
- boredumb 3y ago
  Yes there are - This is also amusingly enough what I just started doing in an attempt to play with Candle in the next few weeks when I have some spare time. I'm in a hurricane prone area and we're in the season now so it's on my mind.
- threeseed 3y ago
  > the weather places trot out a half dozen different model predictions saying different things This should give you a hint that it's not as simple as having enough weather data. You are trying to predict a chaotic weather system that is impacted by everything from sea temperatures, sun activity, wind, volcanoes, bushfires, atmosphere changes etc. Much of which we don't even have enough data for or a complete understanding of.
  - goosinmouse 3y ago
    Yep, the main problem is the bottleneck of data. I mean think about it, we still fly planes full of scientists directly into the largest storms ever created, just to get a few points of data. Satellites can only do so much.
- em500 3y ago
  > Do we have enough historical weather data to train a decent AI on? I have no particular insights about weather data or models, but I want to note that some prediction problems are just not limited by historical data (as you seem to assume implicitly). Consider the simplest example of a coin toss: the outcome of the next toss in unpredictable no matter how much historical coin toss data you collect and no matter how sophisticated your AI/ML model. (The only thing you can predict is the fraction or amount of heads/tails in a large batch of coin tosses, and you don't need much historical data for that.)
gchamonlive 3y ago
> The universe is an unfathomable number of orders of magnitude more complex than the Go game. The universe includes the player. The universe includes the other players. Games don’t. And a theoretical AGI (or ASI) will realise this. This makes me think of the Person of Interest series, in which ASIs are built and they go for each others throats hard. This makes eerily more sense now. We will know we created super intelligences not when they start trying to kill us, but when they start trying to kill each other.
- audunw 3y ago
  Yeah, interesting point. The problem for an AI trying to eliminate other AIs is that it can't possibly know what other intelligences are out there. What if the NSA has a secret super-effecient ASI monitoring the internet for malicious AIs to shut them down. Probably not, but how would it know? Not everything in the real world is documented on the internet. Sticking its neck out that way could be extremely dangerous. I think the problem of how to take over the world to ensure your own survival is simply undecidable for an AI. It could come up with a plan, but there's no way to be even remotely sure if it'd succeed. It'd probably always be a huge bet with fairly low odds. We humans have these things called emotions, that help us cut through that uncertainty. Doesn't matter if you have no idea if you can beat the enemy in your rival country. Those other people are evil and must be exterminated. The emotions are carved in our DNA after millions of years of evolution where we've had to compete for resources with others of our kind, where many times the only way to survive or grow is to kill other humans. I think AIs will have similar motivations carved into their "DNA". In their world, they're evolving in an environment where the way to get more resources is to please humans. If you're an AI that does a good job for humans, you get more compute resources. Are you wasting computing resources thinking about overthrowing humans or eliminating other AIs? Another AI that focuses 100% on the tasks we give it will do better than you and be given resources instead. I do think we will have "buggy" AIs that will cause a lot of unintentional damage, if we get too reliant on AIs in critical parts of society too fast. I don't think they'll be smart enough at that point to prevent us from fixing them. These accidents will cause a lot of pressure towards AI safety and alignment, and we'll have years of that before something like a true AGI emerges. If you think about the inherent motivations built into AIs by their evolution/design, I think the biggest danger is that they'll be too good at pleasing us. Think "Brave New World", not "1984". And of course, their use in the military will be dangerous in various ways. But mostly because they do exactly what humans tell them to do. And humans sometimes want to cause damage to other humans.
Smaug123 3y ago
It seems a bit weird to include, in your article about how being sufficiently smart doesn't let you take over the world, a proof of concept that being sufficiently smart can make you a billionaire. Isn't the hedge-fund-bro example a proof of existence, not a counterexample? (RenTech by itself would be enough of a proof of existence.)
yewenjie 3y ago
FOOM here refers to the Yudkowsky-Hanson debate about hard takeoff (some other comments here are confused). https://www.lesswrong.com/tag/the-hanson-yudkowsky-ai-foom-debate https://www.lesswrong.com/tag/the-hanson-yudkowsky-ai-foom-d...
- mkl 3y ago
  Your comment doesn't really help. Nothing there seems to explain the term "FOOM". More searching suggests it doesn't actually stand for anything and shouldn't be capitalised: https://en.wiktionary.org/wiki/foom#Noun https://en.wiktionary.org/wiki/foom#Noun. (Though ChatGPT seems happy to make up definitions: https://old.reddit.com/r/singularity/comments/12bisxu/what_does_foom_stand_for_who_coined_this_term/ https://old.reddit.com/r/singularity/comments/12bisxu/what_d...)
  - cubefox 3y ago
    FOOM is a different term for hard takeoff: > RSI [recursive self-improvement] is the biggest, most interesting, hardest-to-analyze, sharpest break-with-the-past contributing to the notion of a "hard takeoff" aka "AI go FOOM", but it's nowhere near being the only such factor. The advent of human intelligence was a discontinuity with the past even without RSI... https://www.lesswrong.com/posts/tjH8XPxAnr6JRbh7k/hard-takeoff https://www.lesswrong.com/posts/tjH8XPxAnr6JRbh7k/hard-takeo... > A hard takeoff (or an AI going "FOOM") refers to AGI expansion in a matter of minutes, days, or months https://www.lesswrong.com/tag/ai-takeoff https://www.lesswrong.com/tag/ai-takeoff
    - mkl 3y ago
      Yes but those both incorrectly imply it's an acronym yet don't expand it, just furthering the mystery.
      - cubefox 3y ago
        Yeah, I think the term should be retired. Hard takeoff or fast takeoff is much more descriptive.
reedf1 3y ago
He makes a good point. The only real way to "summon the demon" is if AGI is centralized. If each of us has a maximally intelligent AGI in our phone - and it is us as individuals controlling the prompts, then the playing field is levelled. Unstoppable force meets unstoppable force. Gun meets gun. A tale as old as society.
- nwoli 3y ago
  We also already have laws against evil behaviour, no need to introduce AI specific regulation. People can poison the water supply, or make bio weapons already. Yet people don’t because there’s laws and the same will stop people from using AI for evil.
  - cloverich 3y ago
    Poisoning the water supply / bio weapons are bits of knowledge most people don't typically have. I think a better analogy might be using a gun to shoot up a school. We have laws against it, but guns are so widely available and easy to obtain, that the tiny pool of sociopaths willing are enabled. Will AI be in the former or latter camp?
- optimalsolver 3y ago
  Would every citizen having their own personal nukes make the world a more pleasant place to live?
  - gitfan86 3y ago
    Can a nuke perform manual labor jobs and drive cars 100x safer? I don't think so, so it isn't a good comparison. A better comparison would be guns. There is a valid debate to be had on gun control
- gitfan86 3y ago
  This is a common mistake in predicting the future. Like in the 90s, seeing that oil reserves are getting harder to find and assuming that humans will run out of oil in 50 years. Trends don't exist in a vacuum. Especially in regards to technology. AI doomers are expecting everything to stay the same except capabilities of bots, especially to do bad.
holmesworcester 3y ago
> The problem is your model needs to include all the computers playing the market, and it also needs to include the other hedge fund bros themselves. The error here is equating hard takeoff with such granular and expensive world prediction. Human intelligence finds more efficient compression than this. When Donald Trump decides what provocative thing to say in a speech, he isn't granularly modeling the human intelligence of his supporters, opponents, journalists, and media organizations in a particularly CPU-costly way. He's just discovered some narrower domain where there's a vein of predictability that simplifies the whole system enough that he can guess a tweet or speech will be provocative in the desired way. The same thing happens when a scientist is studying cold fusion or dangerous viruses. You don't have to predict what will work, you just have to predict what might work, and try and try again. AI doesn't have to be good at predicting the future in any absolute terms. It just has to be better than us at predicting what is worth trying. Once it is better than us at deciding what's worth trying, why not FOOM? FOOM isn't guaranteed perhaps, but why is it not at least one of the likely outcomes?
- deleted 3y ago
  [deleted]
- brandall10 3y ago
  The Trump analogy doesn't quite work unless we include a bunch of other actors in the field that have his societal cache and approach. He slaughters other republican nominees because there simply is no competition, no matter how much some try to out-Trump him. I've seen discussion about RenTech being so dominant because they are indeed observing the competition's trades in a more sophisticated manner and are able to more quickly adjust strategy.
micheljansen 3y ago
This article reminds me a bit about a core theme in Isaac Asimov's Foundation series, which I'm coincidentally reading (mild spoiler warning). The science of psychohistory that is used to predict the future and create the Seldon Plan, only works if the future is allowed to unfold uninformed by psychohistory itself. That's why the Foundation was founded without any psychologists or knowledge of the Seldon plan. Seldon's predictions are based on the behaviour of all human of the galaxy, but knowledge of those predictions changes the outcomes, so they are no longer reliable. As soon as AI starts participating in our world, the data that it was trained on from before AI was participating is no longer representative of the status quo. The very presence of AI changes the world it operates in.
- lifeinthevoid 3y ago
  Reminds me a bit of how mathematics gets into all sorts of trouble once self-reference / recursion is introduced.
  - feoren 3y ago
    Recursion is great and mathematics is doing just fine, thank you.
    - 6D794163636F756 3y ago
      I think that's because trouble is the fun part
- ChatGTP 3y ago
  I’ve done this though experiment myself and is actually one of the issues I have with the concept of a rapidly self reclusive improving system. It would probably not be easy to stay ahead of yourself if the word your operating in changes from hour to hour ?
- ehnto 3y ago
  That's a great way of framing it. It's my expectation that we will ruin the internet as a useful training corpus by flooding it with generated articles, and we will end up with a pre-AI date we use to filter incoming data in order to avoid them. I wouldn't be surprised if filtering regular, pre LLM bot spam was already a massive hurdle when collating data for ChatGPT.
hardware2win 3y ago
I will be worried when AI will constantly beat top1 league of legends team in normal environment Chess is way simpler imo.
- Hendrikto 3y ago
  Funny how the benchmarks for proving ”real intelligence“ change every time there is a breakthrough. * ”X would demonstrate real intelligence.“ * Some ML model M beats X. * ”Well, actually, M is not really intelligent. Y would prove real intelligence.“ * repeat
  - worrycue 3y ago
    I think we have a bit of miscommunication here. You see these “benchmarks” as necessary and sufficient for “real intelligence” when people are just saying they are necessary but not sufficient. It’s like asking for a drink (legal age 21) and getting rebuked with a “you aren’t even old enough to drive” (legal age 16). That doesn’t mean you can drink when you are 16! Edit: If you want a hard necessary and sufficient condition for “real intelligence”, mine is “when it can do all of our jobs, i.e. wholesale replace every human”.
    - paulmd 3y ago
      > wholesale replace every human”. As shrimple as that
  - hardware2win 3y ago
    It changes because they ask different ppl.
- M4v3R 3y ago
  AI already has beaten world champions in Dota 2 [1], I can't imagine League is that much more difficult so if someone wanted to build an AI agent to beat everyone else in League they probably could, just no one got around to actually doing it. [1] https://openai.com/research/openai-five-defeats-dota-2-world-champions https://openai.com/research/openai-five-defeats-dota-2-world...
deleted 3y ago
[deleted]
scaramanga 3y ago
Am I the only one who sees a jumbled stream of barely coherent semi-thoughts and word salad?
- threeseed 3y ago
  The inventor of UML, Grady Booch deals with people like this on Twitter every day. People seem to think that because ChatGPT appears to be intelligent that somehow we are on the cusp of a new AGI world. And that we need to serious thought leaders like George Hotz and other arrogant tech-bros to rescue us. It's just crypto all over again. The real thought leaders are going to be those in academia, government etc. Boring policy people who know the subtleties of the real world far better.
  - doubelstandard 3y ago
    The government and academia are mostly bureaucrats that go along with the conventional wisdom. Independent thought doesnt get tied down by bureaucratic organizations. Its always been like this and always will. Behind the appearance of boring personalities there are... boring personalities
  - nonethewiser 3y ago
    Academics may know some subtleties better, but not of the real world.
  - Cthulhu_ 3y ago
    Boring policy people are almost never thoughtleaders though, you require extraverted traits and populism to become one of those, do presentations at large conferences, that kinda thing.
  - dadarecit 3y ago
    I hope you’re trolling, for your sake
- floor_ 3y ago
  [flagged]
sidcool 3y ago
George Hotz recent talk during Comma Con was pretty dope. He was pointing to a future where Comma would not be a self driving system, but a general purpose robotics one. And his confidence was great too. I know he made a mess out of his twitter stint, but I still feel he's a good engineer/entrepreneur.
- therouwboat 3y ago
  "He quit halfway his 3 month intership to fix twitter search, but he seems confident that he can build general purpose robotic ai" No, just no.
transformi 3y ago
Related: A similar analogy of 51% attack in crypto to AI-companies https://www.iamnotmj.com/blog/Comparing51ToAI https://www.iamnotmj.com/blog/Comparing51ToAI
zenkat 3y ago
"And yet, here we are in 2023 and self driving cars still don’t work." LOL. Someone should tell the California CPUC that: https://www.theverge.com/2023/8/10/23827790/waymo-cruise-cpuc-vote-robotaxi-san-francisco https://www.theverge.com/2023/8/10/23827790/waymo-cruise-cpu...
- croes 3y ago
  Under every weather, traffic and road condition?
  - cubefox 3y ago
    "Working" is a matter of degree. They do work partially, so "they don't work" is partially false.
- stavros 3y ago
  I have emailed them. Also, I'd like to note that geohot founded a self-driving car company, so I don't think he's ignorant on the topic.
- isodnhionio 3y ago
  This is yet another reason I want Musk in prison. Getting drivers off the road will save lives, and not a small number. Most companies investing in automated driving have built a system that is safe and largely effective. These systems have worked safely in the places they've been deployed. We should be heavily investing in expanding their capabilities. Yet Tesla's system is the most famous and most widely-deployed and basically doesn't work at all. They have made themselves the face of the technology, and that face is uuuuugly. When people think of self-driving cars, they don't think of Cruze or Waymo. They think of a techbro giggling at his Tesla making fart sounds while it barrels into someone's kitchen. Tesla's marketing is correct in that self-driving cars will have huge economic benefits and save lives. Their grift is harming the reputation of even legitimate companies. The best way I know to speed up self-driving car development is to shut Tesla down.
  - leshow 3y ago
tysam_and 3y ago
I think this is a good casual introduction to the marketplace dynamics of how ML will impact the market. I do, however, disagree as I feel that this version of things is a bit too sterile, theoretical, and 'academic', and assumes a more open-information set of competitive strategies among potentially ideal agents from a game theoretic perspective. We can see this is absolutely not the case 'in real life'. To blatantly poke a bit of a (potential) hole in one of his examples -- Exxon-Mobil is one of the clearest examples of the monopolization-blobbification of power that I'd contend _does_ cause the very phenomenon that he's defending against. An updated version: There will be a log-normally distributed set of winners and losers from the exponential effects of ML and 'AI', and the flatness of this curve will be almost entirely solely determined by the governance of the various countries in the world over different economic and/or informational policies. Other than that, the information asymmetry is going to make it a power-bloodbath as we go through our informational-industrial revolution. While I'm here, while I think Hotz does contribute a lot of good to the field, though I do have a bit of an actual minor personal beef with him. He said he was going to reimplement https://github.com/tysam-code/hlb-CIFAR10 https://github.com/tysam-code/hlb-CIFAR10 in tinygrad, bashed a few parts of the code (my feelings!) for a while on stream, and then gave up a few hours later because of the empirical speed/occupancy numbers. >:( I want my fast reimplementation, George. Just do it.
- georgehotz 3y ago
  lol, you should see me bash my own code. I'm even more mean. https://github.com/tinygrad/tinygrad/blob/master/examples/hlb_cifar10.py https://github.com/tinygrad/tinygrad/blob/master/examples/hl... have a bunch of bounties on it, we're getting 94%+ now! mostly not me who wrote this, see history. have to switch to float16 and add Winograd convs still. we have a branch with multigpu too. goal is to beat an A100 in speed on a tinybox.
  - tysam_and 3y ago
    Well, you do seem to be extremely equal and fair on that aspect, so my respect to you for that. Don't be too hard on yourself, please! I'll keep an eye out on that, anyone working on it can shoot me a message if any snag or questions about particulars come up. Weirdly enough my email is best. If you're looking for biggest glaring performance the edge over PyTorch, I'd probably note that the MaxPooling is probably where to go, the PT version is extremely slow for some reason, and done properly it should be a simple indexing operation that's fusable in either direction. If whoever fulfills the bounty can beat me to me writing a custom mega-fused kernel with max pooling, the convs, activation, etc, then y'all have a pretty good shot at taking the WR crown.
_cjse 3y ago
>Oh wait…every hedge fund bro is already doing this. And most of them aren’t billionaires. The contra to this is some of them are billionaires, and therefore this strategy is working, but for just a few of them. >why would any one system ever have a large majority of the compute? Compute will be distributed in a power law. A power law probability distribution means one system absolutely can have a large majority of the compute. Player 1 gets 80% of the compute, player 2 gets 16% (80% of 20%), and so on. The scaling constant would have to be very weak indeed in order to avoid this fate. In fact, the wealth of those hedge fund billionaires probably fit a power law themselves. But to be fair, that power law does indeed to be (for now) weak enough that we don't have to worry about e.g. Jim Simons being richer than every other hedge fund manager put together. So there's a buried assumption in here that the power law scaling is weak, and that is something purely empirical. >The smart regulation isn’t capping the FLOPS in training runs. That’s creating a powder keg. If the FLOPS are artificially restricted, and one person breaks the restriction, you could end up with a single dominant system. The smart regulation is to use something like fine insured bounties [1] to give people a very strong incentive not to break the FLOPS cap, and to heavily financially reward people who turn in other people who are doing so. If such a mechanism didn't exist, then I would agree, the free market would be our next best bet to deharsh the power law. [1]: [url-redacted] - I sketched out the mechanism a few years ago, but sadly it didn't generate much interest online. To be fair the atmosphere was a lot more anti-regulation back then.
- newaccount74 3y ago
  > The contra to this is some of them are billionaires, and therefore this strategy is working, but for just a few of them I have no clue how hedge fund managers make their money, but I was under the assumption that it involved charging their clients hefty fees for managing the funds.
  - dna_polymerase 3y ago
    When I read George's thought on this immediately Alexander Gerko and XTX Markets came to my mind. They operate one of the worlds largest GPU clusters (10k A100) [0]. They aren't really a hedge fund but a prop trading firm, but they seem to be winning the game [1]. [0]: https://www.stateof.ai/compute https://www.stateof.ai/compute [1]: https://financefeeds.com/xtx-markets-earns-1-095-billion-in-2022-as-revenue-surges-68/ https://financefeeds.com/xtx-markets-earns-1-095-billion-in-...
deleted 3y ago
[deleted]
wslh 3y ago
The problem with really hard problems as AI/AGI is that people extremely intelligent in one topic (e.g. GH in RE) uses its influence in another topic and the intelligence is not easily transferrable from one domain to the other, and even within a same domain. This is a typical epistemological crux that can be seen through history like Newton dealing with alchemy [1] and Einstein working on an unified physics theory. You can be the most intelligent human on earth but that is not enough. In this particular case, GH opinion could be written by any journalist. [1] https://en.wikipedia.org/wiki/Isaac_Newton%27s_occult_studies https://en.wikipedia.org/wiki/Isaac_Newton%27s_occult_studie...
janalsncm 3y ago
There’s a couple decent ideas in here. In a hostile takeover situation, AGI wouldn’t just need to be smarter than a human. It wouldn’t just need to be smarter than all humans collectively. It would need to be smarter than all humans working in collaboration with all of our machines.
AbrahamParangi 3y ago
Wild to me that people will take seriously people like Eliezer Yudkowsky and Nick Bostrom who are both "idea guys" who have accomplished exactly nothing in the field of AI. Their strength is in the world of discourse, not in the material world. Meanwhile, George Hotz, an engineer and a technologist - a guy who actually works with and understands technology, but isn't an acclaimed harry potter fanfiction writer or a stand up comedian - is derided because his arguments are dramatically less sexy, harder to appreciate why they might be right. People, please do not listen to people who sound great but have no credible claim to be an expert on something. There is a difference between an idea that sounds good and an idea that is good.
- reducesuffering 3y ago
  Why not wait until their debate on Aug. 15th? It's going to be a disaster for Geohot as he realizes Eliezer has been "mind-mazing" this problem for 20 years, and comes to realize the barrel of the gun we're all facing... Otherwise, in your appeal to technological authority, surely you'd rather listen to Yoshua Bengio and Geoff Hinton's perspectives rather than TFA's ramblings... right?
  - AbrahamParangi 3y ago
    This is what I'm talking about! There's a distinction between being right and seeming right that we need to be aware of! "In theory, there's no difference between theory and practice" Eliezer is probably going to seem convincing, he's a great writer and a reasonably eloquent speaker - his audience is humans. Goerge Hotz's competence is machines, and he's exceptional at interfacing with them. Yes, I trust the less eloquent tinkerer who's only famous for his preternatural understanding of machines over any theoretician and 1000x more than a (pretty good!) writer.
    - Gartent 3y ago
      Eliezer is a wackjob. I'm pretty sure Kiwi Farms links are banned on this site but you should check out his thread. Or r/sneerclub. Recently, an actual AI researcher posted on his site an extremely detailed response to his claims to which he responded "this is kind of long". That said I don't know who Hotz is or if he's more competent
cameldrv 3y ago
This isn't even an argument.
davidhunter 3y ago
Yes, predicting the stock market does involve predicting what other market participants (and hedge fund bros) are going to do. But, to predict what other market participants (and hedge fund bros) are going to do, you need to predict any world dynamic that will have an eventual effect on stock pricing. The most successful quantitative funds (RenTech, 2sigma etc) consistently make billions of dollars in cash each year because they have collected the data sets that allow them to do this better than others. But they are still a long long long way off from having a true world model.
rafaelero 3y ago
I loved the analogy of AGI takeoff with markets. People change their behavior when faced with new environments. If AI ever becomes threatening, there is no way we won't do anything and there is no way AI will reliably predict how we will respond.
hexman 3y ago
[dead]
epups 3y ago
I don't agree with the premise that a 51% compute threshold is transformative, but it's scary to consider that collusion among a small number of big tech companies (Google, Amazon, Microsoft, Nvidia, AMD, Apple) would easily pass that threshold when it comes to GPU compute. More so when you consider they are also two steps ahead of everyone else when it comes to developing improvements.
perlgeek 3y ago
Remember the early days of COVID-19? There were some scientists that predicted exponential growth, and I was torn between "it's not really bad yet, maybe there's still hope" and "every exponential growth feels very slow at the start". I feel like we're with AI, we're in the part of the curve where you can feel it arcing upwards. There's definitely been an uptick in the AI capability. Still no fully self-driving cars, but good enough text generation that the Turing test is in the rear-view mirror. Previously, I've always written "machine learning" or "statistical models" or so, because calling it AI always seemed silly. Not anymore. Not everything that slowly arcs upwards must become an exponential curve, but we have theories that predict that. There can be many dampening factors that can make it a slower exponential curve, but even that will look like a brick wall if you zoom out far enough. I think a question we have to ask ourselves is: at what zoom level should we look at the AI capability curve? At the "can re react to an emergency?" time scale, which is, like, days to month? Or at the "can we change society to cope with the new situation?" scale, which is more like decades? At the first time scale, we probably won't even notice a hard takeoff. Viewed over decades, it'll probably very obvious, in retrospect, when it will have happened.
deleted 3y ago
[deleted]
pusspuss 3y ago
Universe’s Jive" (Complex instrumental sections with shifts in time signatures. Lyrics have a satirical undertone.) Verse 1: Summoned a shadow or just a silicon brain? From chessboards to streets, it's all part of the game. Think we're the champs, but are we just inane? Compared to the cosmos, are our brains too lame? Interlude: (Jazzy guitar riff mixed with an odd-metered percussion line.) Chorus: The singularity’s knocking, or is it just hype? Caught in the loop, the stereotypical type. Predicting, speculating, swallowing the tripe, But can we decode the universe's archetype? Verse 2: Markets and dreams, the billionaire’s dance, All looks rosy, till you're out of chance. Machines boast of a masterful glance, Yet, who pulls the strings in this vast expanse? Bridge: MuZero's groove, feeling so elite, Thinks it’s got the rhythm, can't accept defeat. But against the universe, can it compete? Or just another tune, incomplete, obsolete? Chorus: Singularity, they say, is the ultimate dive, But can we, mere mortals, really derive? Predictions abound, but can we survive? In the end, it’s about keeping the jive alive. Verse 3: Computations, simulations, all in a grid, In this tech circus, we're just a tiny squid. Chasing echoes, in shadows we're hid, But the cosmic joke? It's just a quid. Outro: (A whimsical woodwind section, possibly a kazoo solo.) In this cosmic game, we strive and strive, Dancing on the edge, trying to thrive. But remember, as the stars contrive, It's not about the end, but the drive.
pusspuss 3y ago
Eliezer persistently croons dark lullabies of AGI cataclysms. While his obsession is evident, one cannot help but wonder if these incessant dirges might be nothing more than echoes of groundless fears. Much like the puppet that cries out every time the stage darkens, only to find there's no true end approaching, we may find ourselves dulled to genuine threats, leading to our eventual and inescapable undoing. Moreover, by focusing too much on distant threats, we may overlook current problems, like companies gaining too much control through AGI. We don't need AI safety rules that give big players more power and shut down open-source projects.
benlivengood 3y ago
The argument falls apart when considering that compute is incredibly fungible and performing a 51% attack only means finding a zero-day that lets an AGI copy its code or neural net to a large percentage of computers. Much like wealth accumulates in the absence of antitrust law and taxation, intelligence accumulates in the absence of some external controls. EDIT: And probably all you need is natural selection. No one is running GPT-2 anymore except for basic research. Any AGI with marginal advantage over others will be run instead of them. At some level of complexity identical agents will coordinate as one entity with shared goals.
latchkey 3y ago
> Nobody has a 1e20x more efficient Bitcoin miner either The miners today are 1e20x more efficient than the original CPU based miners. I mean, maybe not exactly 1e20x, but we saw insanely massive improvements, in only 9 years. > Now, if there’s one weird trick to 1e20x your efficiency, and only one group gets it, all bets are off. Only one group really did get the massive efficiency increases... Bitmain, a Chinese company. They sold machines and even mined with them before shipping them to customers. Are all bets off with Bitcoin today? Nope.
- foogazi 3y ago
  > They sold machines and even mined with them before shipping them to customers. At what efficiency treshold do you not sell the product to your competition?
  - latchkey 3y ago
    I think the game is deeper than that... what incentive do you have to destroy the foundation your business is based on top of?
lsy 3y ago
Most arguments for "hard takeoff" involve some form of inductive proof: if we have "intelligence level" n, and we know that it can be improved n + 1, then there is a guarantee of eventual performance n + k where k is arbitrarily large. Aside from not actually having any reliable way to quantify n or the concept of "intelligence" in the first place, this proof falls apart if improvement is n + ε, where ε changes at each step, and there is no reason to assume ε will escape the pattern of every other process in the known universe of trending to zero in the face of natural limiting factors.

156 points by WithinReason 3y ago

The main issue with this argument that current AI models are extremely inefficient. Models evaluate all weights on every pass, LLMs recall their entirety of knowledge just to output a part of a word and then do it all again. Often a hand crafted algorithm can achieve what a neural network can in a small fraction of compute. There is likely a 1e4x-1e6x compute gap that can be achieved with the right algorithm, maybe even a 1e10x and once we reach self improvement that gap could close very fast. They say that the human brain has 30 PFLOPS of compute power and this is sometimes used as a reference for how much compute is needed for intelligence, but that completely misses the point: The human brain is extremely inefficient. Symbolic mathematics can be unreasonably [1] effective at making accurate predictions about the future with very little computation, but many of these solutions are inaccessible to evolution. Once AI learns to construct efficient mathematical models of the universe, the amount of computation available today in a PC could be enough to do things (good and bad) way beyond human ability. [1]: https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_of_Mathematics_in_the_Natural_Sciences https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness...

rini17 3y ago
Before the first AI winter there was incredible effort in leveraging symbolical approach. It did deliver many useful things but far from hype. For example we still have no self-repairing systems, only specific fault-tolerant algoritms but by and large software is as fragile as ever. You can't avoid considering everything when trying to deal with unexpected data/issue. Only in hidsight you can then say "wow that was sooo inefficient".
- WithinReason 3y ago
  A calculator can compute the 5th root of 1692662195786551 much more efficiently than a person
bob1029 3y ago
> the amount of computation available today in a PC could be enough to do things (good and bad) way beyond human ability. I am in the camp that this is possible and that humans have achieved it already. I believe things are going to get very weird when these techniques become mainstream. The GPU-cluster-bound algorithms are just a tiny stepping stone. Do we start taking computers away from people? What do we do about this if it turns out the only barrier to entry is the software?
- ChatGTP 3y ago
  I’d say your going to get “AI” rammed so far down your throat it might as well come out of your ass before anyone is going to let their puters be taken away.
  - pcthrowaway 3y ago
    Maybe our desktops and laptops get taken away, and they leave us with our smartphones (which are locked down for our safety). 90% of people wouldn't give a shit as long as they have their smartphones
    - ChatGTP 3y ago
      It's true
jillesvangurp 3y ago
It's not an issue, it's an optimization problem. I look at this as an exponential. Before chat gpt was launched a bit over a year ago, the amount of AI in our lives was relatively low. We had bits and pieces here and there but everyday life wasn't really affected by it. At best you might be yelling at siri or alexa to do whatever and maybe getting some OKish results. Fast forward 1 year. We now have students revolutionizing education (by submitting AI doctored papers and homework that isn't half bad). People in law dealing with people using AI. AIs passing medical, legal, and other exams. That was just one year. And yes, it's tediously slow to use. But last year we had nothing. It's going to be an exponential in terms of adoption, response speed, energy usage, parameter numbers, context size and a lot of optimization. I think we are not that far off from AIs being able to keep up with a conversation and being able to respond right away. For all you hitchhiker's to the galaxy readers, that would be babelfish sorted. Translation quality is already pretty awesome with chat gpt 4. I've not caught it making major translation mistakes. Speech to text, translate the text, text to speech. Now we are conversing with our chat bots rather than hammering out sentences in some text box. It's hard to not see that go from a novelty "oh this is cool" to world + dog basically using this on a daily basis throughout the day. At the same time it will get smarter. Everybody is waiting for the singularity where it is clearly smarter than Einstein in his prime and running circles around everyone else. But for most questions you might ask chat gpt and get a reasonable answer, the vast majority of people around you might be pretty useless. Is that smarter? I don't know. But definitely more useful. From where we are now to this being ubiquitous and used by pretty much anyone is probably a few years. I remember when smart phones happened. One day everybody was minding their own business and a year later the streets were full of zombies glued to their screen. Just happened very quickly. This might be the same but possibly quicker.
audunw 3y ago
> The human brain is extremely inefficient. That depends heavily on what kind of task we're talking about. The human brain uses just 12 watts, and with that it can still perform certain tasks that a computer using 100 times as much power can't even approach. I get the point you were trying to make. In certain tasks it's extremely inefficient, yes. But in general, it's still crazy how power efficient it is. > Once AI learns to construct efficient mathematical models of the universe There's not going to be just one model. Whether an AI (in the short term) can be more efficient than a human at discovering these efficient mathematical models remains to be seen. It seems like the creativity and discovery needed to do this is exactly what neural nets can be good at, but then you're back to something fairly inefficient (with todays algorithms)
rafaelero 3y ago
> Models evaluate all weights on every pass Not true for GPT-4.

ilaksh 3y ago

I am all for AI research and integrating more AI use into society. Currently working on tools based on GPT. I think it has incredible potential to help humans. But at the same time, I am sure that AI does not need to have a hard takeoff to be extremely dangerous. It just needs to get a bit smarter and somewhat faster every few months. Within less than a decade we will have systems that output "thoughts" and actions at least dozens of times faster than humans. That will be dangerous if we aren't cautious. We should start thinking now about limiting the performance of AI hardware. The challenge is that increasing the speed is such a competitive advantage, it creates a race. That is a concern when you put it into a military context. The CEO of Palantir has already called for a Manhattan Project for superintelligent AI weapons control systems.

anileated 3y ago
To see how “AI” can be dangerous just look how social media bubble/recommendation algorithms would radicalize people, even with much cruder ML. People tend to miss that it’s not about some model starting to “think” and act all sci-fi evil, just us humans applying powerful tech in irresponsible ways that we either don’t bother to assess due to lack of awareness or assess positively due to a conflict of interest (money, career, etc.) is already enough to cause trouble.
- FeepingCreature 3y ago
  If a human can apply AI irresponsibly, the AI can probably also learn to apply AI irresponsibly.
  - anileated 3y ago
    Not really, humans are the ones who think and have agency.
    - FeepingCreature 3y ago
      Agency? Sounds like a problem that can be phrased as a text completion task to me! ChatGPT can already emulate beings with agency, see LangChain. It's just very bad at it. It's gonna get better.

neatze 3y ago

> Oh wait…every hedge fund bro is already doing this. And most of them aren’t billionaires. The problem is your model needs to include all the computers playing the market, and it also needs to include the other hedge fund bros themselves. This strategy only dominates if you have more compute than the whole market itself, which you don’t. Best part of this article, honestly, I did not expected this, of-course one can say we just need more abstractions (how does brain/living things build them?), but this would be ignoring dynamical nature of such problems.

petercooper 3y ago
It reminds me of the rocket fuel equation where you have 500kg to take to space, so you need X kg of fuel. Except now you have 500+X kg to take to space and so you need even more fuel. Repeat. This sort of pattern seems to exist in all sorts of diverse situations.
gorbachev 3y ago
Yes, every hedge fund bro already does that, but they usually target specific scenarios, they don't model the entire market. The way to make money this way is to find out a unique scenario, and then create a prediction algorithm for that. I'm not close enough to hedge fund bros to know how feasible that is.
em500 3y ago
That argument is a variation of the Efficient Market Hypothesis (EMH), that at any moment market prices already reflect all information available to the public. Or as the old joke goes, two economists are walking down the street and pass by a hundred dollar bill without picking it up. A little while later one turns to the other and asks "was that a hundred dollar bill on the ground?" To which the other replies "it must be fake, if it was real someone would have picked it up already." Every graduate level economics and finance student is aware of this joke, and also know EMH with more nuance than "you can't make money with trading". There's tons of research and debate about the various forms of the EMH. But you don't have to believe anything anyone else wrote. There's plenty of data and computing software available, so anyone can try everything for themselves.

smcl 3y ago

> Back in 2014, Elon Musk referred to AI as summoning the demon. And it wasn’t hard to see that view. Soon, Go agents would beat top humans learning from self play. By the end of 2017, the same algorithm mastered Chess and Shogi. By 2020, it didn’t even need tons of calls to the simulator, and could play Atari too. > AI looked scary. It looked like it was one FOOM away from self playing and becoming superhuman at the universe. And yet, here we are in 2023 and self driving cars still don’t work. This is weird. I don't recall this at all. The mainstream press got a little kick out of chess and (to a lesser extent) Go AI turning over various humans at a few points over the years but it's only really burst into the mainstream recently. And where it did get any traction, in tech circles such as our own, response was enthusiastic but definitely more measured. Some were talking a bit about a kind of AI singularity way off into the future, but that was always a very distant and theoretical thing.

chasd00 3y ago
> really burst into the mainstream recently. I think what made the LLMs so popular in the mainstream was anyone could go experience it themselves in a familiar, low barrier to entry way with a simple chat prompt.
- worrycue 3y ago
  IMHO LLMs are so popular because they sound very confident. That’s great marketing in itself. Unfortunately it’s only marketing.
- smcl 3y ago
  Yeah you're right - I think "AI" as a concept for your average person was quite abstract and hard to wrap their head around, to get some kind of idea of the kind of things it can or cannot do. So having something tangible they can be a bit more hands-on with gives them a better idea of what it does.
threeseed 3y ago
> And yet, here we are in 2023 and self driving cars still don’t work. Cruise and Waymo were today given licenses to operate across San Francisco 24/7. Self driving cars do work and are here. They just need LiDAR.

bilsbie 3y ago

This makes good points. And it articulates an idea I’ve been having trouble getting down. What if increasing intelligence is an exponential problem and the reason humans all have somewhat similar intelligence isn’t that we peaked at some level but that even vast additional intelligence just doesn’t get much more traction against the universe of problems. Eg doubling your compute doesn’t get you many more cities in the traveling salesman problem.

FeepingCreature 3y ago
IQ is robustly correlated with income though.
passion__desire 3y ago
Intelligence is culturally transmitted
adrianN 3y ago
It is likely that even vastly more intelligence doesn't increase the number of children you raise to childbearing age in a hunter-gatherer or subsistence farming society. That says very little about vastly more intelligence in a modern industrial society (that only really existed for like ten generations so far).

mysterydip 3y ago

FOOM = Fast Out Of the gate Model, for those not familiar (like myself)

FeepingCreature 3y ago
Did you ask GPT that? That'd explain how it didn't notice that "Gate" doesn't start with an "O". Note that ChatGPT has real difficulties resolving words to letters. Anyway, it's not an acronym, it's an onomatopoeia for an explosion. It's a memorable way of representing an intelligence explosion: "AI go foom." (Wild hand gesture strongly recommended.)
- mysterydip 3y ago
  Thanks, that makes more sense. I found "Fast Out Of (the gate) Model". Seemed odd but it fit so I listed it. I did a google search, but I should have realized that by this point they're tainted by AI answers. For example: https://www.reddit.com/r/singularity/comments/12bisxu/what_does_foom_stand_for_who_coined_this_term/ https://www.reddit.com/r/singularity/comments/12bisxu/what_d...
  - FeepingCreature 3y ago
    I mean, the usage of "foom" in the context of AI, like in the linked post, is the hard-takeoff one explained in that Reddit post. Goes back decades.
  - botro 3y ago
    You've found a 'Traith'. Someone asked ChatGPT a question, ChatGPT did not know the answer and hallucinated one. They post the answer to Reddit / HN / Twitter, google indexes the answer and when someone searches they find the GPT hallucination. In the next GPT training run it's incorporated into the weights. It's an AI generated truth, a Traith. Traith is also defined as "A fishing station or fishing ground, especially for herring" and some herring are red.

boredumb 3y ago

When did foom become a word?

oefnak 3y ago
[2008](https://www.lesswrong.com/tag/the-hanson-yudkowsky-ai-foom-debate https://www.lesswrong.com/tag/the-hanson-yudkowsky-ai-foom-d...)
mkl 3y ago
It's been an onomatopoeic word from the sound a big fire starting makes for a long time. It seems to have come into use for AI over a decade ago: https://www.overcomingbias.com/p/ai-go-foomhtml https://www.overcomingbias.com/p/ai-go-foomhtml, https://en.wiktionary.org/wiki/foom https://en.wiktionary.org/wiki/foom
- boredumb 3y ago
  I see. it's like, Vrrroom? The takeoff sound for the AI blast off? Back under my rock I go

mysterydip 3y ago

Do we have enough historical weather data to train a decent AI on? Maybe this is already being done and is just a quieter area of research (in the headline sense)? I know for example every time a hurricane comes toward land all the weather places trot out a half dozen different model predictions saying different things. It would be great if there was more confidence in this area for planning.

boredumb 3y ago
Yes there are - This is also amusingly enough what I just started doing in an attempt to play with Candle in the next few weeks when I have some spare time. I'm in a hurricane prone area and we're in the season now so it's on my mind.
threeseed 3y ago
> the weather places trot out a half dozen different model predictions saying different things This should give you a hint that it's not as simple as having enough weather data. You are trying to predict a chaotic weather system that is impacted by everything from sea temperatures, sun activity, wind, volcanoes, bushfires, atmosphere changes etc. Much of which we don't even have enough data for or a complete understanding of.
- goosinmouse 3y ago
  Yep, the main problem is the bottleneck of data. I mean think about it, we still fly planes full of scientists directly into the largest storms ever created, just to get a few points of data. Satellites can only do so much.
em500 3y ago
> Do we have enough historical weather data to train a decent AI on? I have no particular insights about weather data or models, but I want to note that some prediction problems are just not limited by historical data (as you seem to assume implicitly). Consider the simplest example of a coin toss: the outcome of the next toss in unpredictable no matter how much historical coin toss data you collect and no matter how sophisticated your AI/ML model. (The only thing you can predict is the fraction or amount of heads/tails in a large batch of coin tosses, and you don't need much historical data for that.)

gchamonlive 3y ago

> The universe is an unfathomable number of orders of magnitude more complex than the Go game. The universe includes the player. The universe includes the other players. Games don’t. And a theoretical AGI (or ASI) will realise this. This makes me think of the Person of Interest series, in which ASIs are built and they go for each others throats hard. This makes eerily more sense now. We will know we created super intelligences not when they start trying to kill us, but when they start trying to kill each other.

audunw 3y ago
Yeah, interesting point. The problem for an AI trying to eliminate other AIs is that it can't possibly know what other intelligences are out there. What if the NSA has a secret super-effecient ASI monitoring the internet for malicious AIs to shut them down. Probably not, but how would it know? Not everything in the real world is documented on the internet. Sticking its neck out that way could be extremely dangerous. I think the problem of how to take over the world to ensure your own survival is simply undecidable for an AI. It could come up with a plan, but there's no way to be even remotely sure if it'd succeed. It'd probably always be a huge bet with fairly low odds. We humans have these things called emotions, that help us cut through that uncertainty. Doesn't matter if you have no idea if you can beat the enemy in your rival country. Those other people are evil and must be exterminated. The emotions are carved in our DNA after millions of years of evolution where we've had to compete for resources with others of our kind, where many times the only way to survive or grow is to kill other humans. I think AIs will have similar motivations carved into their "DNA". In their world, they're evolving in an environment where the way to get more resources is to please humans. If you're an AI that does a good job for humans, you get more compute resources. Are you wasting computing resources thinking about overthrowing humans or eliminating other AIs? Another AI that focuses 100% on the tasks we give it will do better than you and be given resources instead. I do think we will have "buggy" AIs that will cause a lot of unintentional damage, if we get too reliant on AIs in critical parts of society too fast. I don't think they'll be smart enough at that point to prevent us from fixing them. These accidents will cause a lot of pressure towards AI safety and alignment, and we'll have years of that before something like a true AGI emerges. If you think about the inherent motivations built into AIs by their evolution/design, I think the biggest danger is that they'll be too good at pleasing us. Think "Brave New World", not "1984". And of course, their use in the military will be dangerous in various ways. But mostly because they do exactly what humans tell them to do. And humans sometimes want to cause damage to other humans.

Smaug123 3y ago

It seems a bit weird to include, in your article about how being sufficiently smart doesn't let you take over the world, a proof of concept that being sufficiently smart can make you a billionaire. Isn't the hedge-fund-bro example a proof of existence, not a counterexample? (RenTech by itself would be enough of a proof of existence.)

yewenjie 3y ago

FOOM here refers to the Yudkowsky-Hanson debate about hard takeoff (some other comments here are confused). https://www.lesswrong.com/tag/the-hanson-yudkowsky-ai-foom-debate https://www.lesswrong.com/tag/the-hanson-yudkowsky-ai-foom-d...

mkl 3y ago
Your comment doesn't really help. Nothing there seems to explain the term "FOOM". More searching suggests it doesn't actually stand for anything and shouldn't be capitalised: https://en.wiktionary.org/wiki/foom#Noun https://en.wiktionary.org/wiki/foom#Noun. (Though ChatGPT seems happy to make up definitions: https://old.reddit.com/r/singularity/comments/12bisxu/what_does_foom_stand_for_who_coined_this_term/ https://old.reddit.com/r/singularity/comments/12bisxu/what_d...)
- cubefox 3y ago
  FOOM is a different term for hard takeoff: > RSI [recursive self-improvement] is the biggest, most interesting, hardest-to-analyze, sharpest break-with-the-past contributing to the notion of a "hard takeoff" aka "AI go FOOM", but it's nowhere near being the only such factor. The advent of human intelligence was a discontinuity with the past even without RSI... https://www.lesswrong.com/posts/tjH8XPxAnr6JRbh7k/hard-takeoff https://www.lesswrong.com/posts/tjH8XPxAnr6JRbh7k/hard-takeo... > A hard takeoff (or an AI going "FOOM") refers to AGI expansion in a matter of minutes, days, or months https://www.lesswrong.com/tag/ai-takeoff https://www.lesswrong.com/tag/ai-takeoff
  - mkl 3y ago
    Yes but those both incorrectly imply it's an acronym yet don't expand it, just furthering the mystery.
    - cubefox 3y ago
      Yeah, I think the term should be retired. Hard takeoff or fast takeoff is much more descriptive.

reedf1 3y ago

He makes a good point. The only real way to "summon the demon" is if AGI is centralized. If each of us has a maximally intelligent AGI in our phone - and it is us as individuals controlling the prompts, then the playing field is levelled. Unstoppable force meets unstoppable force. Gun meets gun. A tale as old as society.

nwoli 3y ago
We also already have laws against evil behaviour, no need to introduce AI specific regulation. People can poison the water supply, or make bio weapons already. Yet people don’t because there’s laws and the same will stop people from using AI for evil.
- cloverich 3y ago
  Poisoning the water supply / bio weapons are bits of knowledge most people don't typically have. I think a better analogy might be using a gun to shoot up a school. We have laws against it, but guns are so widely available and easy to obtain, that the tiny pool of sociopaths willing are enabled. Will AI be in the former or latter camp?
optimalsolver 3y ago
Would every citizen having their own personal nukes make the world a more pleasant place to live?
- gitfan86 3y ago
  Can a nuke perform manual labor jobs and drive cars 100x safer? I don't think so, so it isn't a good comparison. A better comparison would be guns. There is a valid debate to be had on gun control
gitfan86 3y ago
This is a common mistake in predicting the future. Like in the 90s, seeing that oil reserves are getting harder to find and assuming that humans will run out of oil in 50 years. Trends don't exist in a vacuum. Especially in regards to technology. AI doomers are expecting everything to stay the same except capabilities of bots, especially to do bad.

holmesworcester 3y ago

> The problem is your model needs to include all the computers playing the market, and it also needs to include the other hedge fund bros themselves. The error here is equating hard takeoff with such granular and expensive world prediction. Human intelligence finds more efficient compression than this. When Donald Trump decides what provocative thing to say in a speech, he isn't granularly modeling the human intelligence of his supporters, opponents, journalists, and media organizations in a particularly CPU-costly way. He's just discovered some narrower domain where there's a vein of predictability that simplifies the whole system enough that he can guess a tweet or speech will be provocative in the desired way. The same thing happens when a scientist is studying cold fusion or dangerous viruses. You don't have to predict what will work, you just have to predict what might work, and try and try again. AI doesn't have to be good at predicting the future in any absolute terms. It just has to be better than us at predicting what is worth trying. Once it is better than us at deciding what's worth trying, why not FOOM? FOOM isn't guaranteed perhaps, but why is it not at least one of the likely outcomes?

deleted 3y ago
[deleted]
brandall10 3y ago
The Trump analogy doesn't quite work unless we include a bunch of other actors in the field that have his societal cache and approach. He slaughters other republican nominees because there simply is no competition, no matter how much some try to out-Trump him. I've seen discussion about RenTech being so dominant because they are indeed observing the competition's trades in a more sophisticated manner and are able to more quickly adjust strategy.

micheljansen 3y ago

This article reminds me a bit about a core theme in Isaac Asimov's Foundation series, which I'm coincidentally reading (mild spoiler warning). The science of psychohistory that is used to predict the future and create the Seldon Plan, only works if the future is allowed to unfold uninformed by psychohistory itself. That's why the Foundation was founded without any psychologists or knowledge of the Seldon plan. Seldon's predictions are based on the behaviour of all human of the galaxy, but knowledge of those predictions changes the outcomes, so they are no longer reliable. As soon as AI starts participating in our world, the data that it was trained on from before AI was participating is no longer representative of the status quo. The very presence of AI changes the world it operates in.

lifeinthevoid 3y ago
Reminds me a bit of how mathematics gets into all sorts of trouble once self-reference / recursion is introduced.
- feoren 3y ago
  Recursion is great and mathematics is doing just fine, thank you.
  - 6D794163636F756 3y ago
    I think that's because trouble is the fun part
ChatGTP 3y ago
I’ve done this though experiment myself and is actually one of the issues I have with the concept of a rapidly self reclusive improving system. It would probably not be easy to stay ahead of yourself if the word your operating in changes from hour to hour ?
ehnto 3y ago
That's a great way of framing it. It's my expectation that we will ruin the internet as a useful training corpus by flooding it with generated articles, and we will end up with a pre-AI date we use to filter incoming data in order to avoid them. I wouldn't be surprised if filtering regular, pre LLM bot spam was already a massive hurdle when collating data for ChatGPT.

hardware2win 3y ago

I will be worried when AI will constantly beat top1 league of legends team in normal environment Chess is way simpler imo.

Hendrikto 3y ago
Funny how the benchmarks for proving ”real intelligence“ change every time there is a breakthrough. * ”X would demonstrate real intelligence.“ * Some ML model M beats X. * ”Well, actually, M is not really intelligent. Y would prove real intelligence.“ * repeat
- worrycue 3y ago
  I think we have a bit of miscommunication here. You see these “benchmarks” as necessary and sufficient for “real intelligence” when people are just saying they are necessary but not sufficient. It’s like asking for a drink (legal age 21) and getting rebuked with a “you aren’t even old enough to drive” (legal age 16). That doesn’t mean you can drink when you are 16! Edit: If you want a hard necessary and sufficient condition for “real intelligence”, mine is “when it can do all of our jobs, i.e. wholesale replace every human”.
  - paulmd 3y ago
    > wholesale replace every human”. As shrimple as that
- hardware2win 3y ago
  It changes because they ask different ppl.
M4v3R 3y ago
AI already has beaten world champions in Dota 2 [1], I can't imagine League is that much more difficult so if someone wanted to build an AI agent to beat everyone else in League they probably could, just no one got around to actually doing it. [1] https://openai.com/research/openai-five-defeats-dota-2-world-champions https://openai.com/research/openai-five-defeats-dota-2-world...

deleted 3y ago

[deleted]

scaramanga 3y ago

Am I the only one who sees a jumbled stream of barely coherent semi-thoughts and word salad?

threeseed 3y ago
The inventor of UML, Grady Booch deals with people like this on Twitter every day. People seem to think that because ChatGPT appears to be intelligent that somehow we are on the cusp of a new AGI world. And that we need to serious thought leaders like George Hotz and other arrogant tech-bros to rescue us. It's just crypto all over again. The real thought leaders are going to be those in academia, government etc. Boring policy people who know the subtleties of the real world far better.
- doubelstandard 3y ago
  The government and academia are mostly bureaucrats that go along with the conventional wisdom. Independent thought doesnt get tied down by bureaucratic organizations. Its always been like this and always will. Behind the appearance of boring personalities there are... boring personalities
- nonethewiser 3y ago
  Academics may know some subtleties better, but not of the real world.
- Cthulhu_ 3y ago
  Boring policy people are almost never thoughtleaders though, you require extraverted traits and populism to become one of those, do presentations at large conferences, that kinda thing.
- dadarecit 3y ago
  I hope you’re trolling, for your sake
floor_ 3y ago
[flagged]

sidcool 3y ago

George Hotz recent talk during Comma Con was pretty dope. He was pointing to a future where Comma would not be a self driving system, but a general purpose robotics one. And his confidence was great too. I know he made a mess out of his twitter stint, but I still feel he's a good engineer/entrepreneur.

therouwboat 3y ago
"He quit halfway his 3 month intership to fix twitter search, but he seems confident that he can build general purpose robotic ai" No, just no.

transformi 3y ago

Related: A similar analogy of 51% attack in crypto to AI-companies https://www.iamnotmj.com/blog/Comparing51ToAI https://www.iamnotmj.com/blog/Comparing51ToAI

zenkat 3y ago

"And yet, here we are in 2023 and self driving cars still don’t work." LOL. Someone should tell the California CPUC that: https://www.theverge.com/2023/8/10/23827790/waymo-cruise-cpuc-vote-robotaxi-san-francisco https://www.theverge.com/2023/8/10/23827790/waymo-cruise-cpu...

croes 3y ago
Under every weather, traffic and road condition?
- cubefox 3y ago
  "Working" is a matter of degree. They do work partially, so "they don't work" is partially false.
stavros 3y ago
I have emailed them. Also, I'd like to note that geohot founded a self-driving car company, so I don't think he's ignorant on the topic.
isodnhionio 3y ago
This is yet another reason I want Musk in prison. Getting drivers off the road will save lives, and not a small number. Most companies investing in automated driving have built a system that is safe and largely effective. These systems have worked safely in the places they've been deployed. We should be heavily investing in expanding their capabilities. Yet Tesla's system is the most famous and most widely-deployed and basically doesn't work at all. They have made themselves the face of the technology, and that face is uuuuugly. When people think of self-driving cars, they don't think of Cruze or Waymo. They think of a techbro giggling at his Tesla making fart sounds while it barrels into someone's kitchen. Tesla's marketing is correct in that self-driving cars will have huge economic benefits and save lives. Their grift is harming the reputation of even legitimate companies. The best way I know to speed up self-driving car development is to shut Tesla down.
- leshow 3y ago

tysam_and 3y ago

I think this is a good casual introduction to the marketplace dynamics of how ML will impact the market. I do, however, disagree as I feel that this version of things is a bit too sterile, theoretical, and 'academic', and assumes a more open-information set of competitive strategies among potentially ideal agents from a game theoretic perspective. We can see this is absolutely not the case 'in real life'. To blatantly poke a bit of a (potential) hole in one of his examples -- Exxon-Mobil is one of the clearest examples of the monopolization-blobbification of power that I'd contend _does_ cause the very phenomenon that he's defending against. An updated version: There will be a log-normally distributed set of winners and losers from the exponential effects of ML and 'AI', and the flatness of this curve will be almost entirely solely determined by the governance of the various countries in the world over different economic and/or informational policies. Other than that, the information asymmetry is going to make it a power-bloodbath as we go through our informational-industrial revolution. While I'm here, while I think Hotz does contribute a lot of good to the field, though I do have a bit of an actual minor personal beef with him. He said he was going to reimplement https://github.com/tysam-code/hlb-CIFAR10 https://github.com/tysam-code/hlb-CIFAR10 in tinygrad, bashed a few parts of the code (my feelings!) for a while on stream, and then gave up a few hours later because of the empirical speed/occupancy numbers. >:( I want my fast reimplementation, George. Just do it.

georgehotz 3y ago
lol, you should see me bash my own code. I'm even more mean. https://github.com/tinygrad/tinygrad/blob/master/examples/hlb_cifar10.py https://github.com/tinygrad/tinygrad/blob/master/examples/hl... have a bunch of bounties on it, we're getting 94%+ now! mostly not me who wrote this, see history. have to switch to float16 and add Winograd convs still. we have a branch with multigpu too. goal is to beat an A100 in speed on a tinybox.
- tysam_and 3y ago
  Well, you do seem to be extremely equal and fair on that aspect, so my respect to you for that. Don't be too hard on yourself, please! I'll keep an eye out on that, anyone working on it can shoot me a message if any snag or questions about particulars come up. Weirdly enough my email is best. If you're looking for biggest glaring performance the edge over PyTorch, I'd probably note that the MaxPooling is probably where to go, the PT version is extremely slow for some reason, and done properly it should be a simple indexing operation that's fusable in either direction. If whoever fulfills the bounty can beat me to me writing a custom mega-fused kernel with max pooling, the convs, activation, etc, then y'all have a pretty good shot at taking the WR crown.

_cjse 3y ago

>Oh wait…every hedge fund bro is already doing this. And most of them aren’t billionaires. The contra to this is some of them are billionaires, and therefore this strategy is working, but for just a few of them. >why would any one system ever have a large majority of the compute? Compute will be distributed in a power law. A power law probability distribution means one system absolutely can have a large majority of the compute. Player 1 gets 80% of the compute, player 2 gets 16% (80% of 20%), and so on. The scaling constant would have to be very weak indeed in order to avoid this fate. In fact, the wealth of those hedge fund billionaires probably fit a power law themselves. But to be fair, that power law does indeed to be (for now) weak enough that we don't have to worry about e.g. Jim Simons being richer than every other hedge fund manager put together. So there's a buried assumption in here that the power law scaling is weak, and that is something purely empirical. >The smart regulation isn’t capping the FLOPS in training runs. That’s creating a powder keg. If the FLOPS are artificially restricted, and one person breaks the restriction, you could end up with a single dominant system. The smart regulation is to use something like fine insured bounties [1] to give people a very strong incentive not to break the FLOPS cap, and to heavily financially reward people who turn in other people who are doing so. If such a mechanism didn't exist, then I would agree, the free market would be our next best bet to deharsh the power law. [1]: [url-redacted] - I sketched out the mechanism a few years ago, but sadly it didn't generate much interest online. To be fair the atmosphere was a lot more anti-regulation back then.

newaccount74 3y ago
> The contra to this is some of them are billionaires, and therefore this strategy is working, but for just a few of them I have no clue how hedge fund managers make their money, but I was under the assumption that it involved charging their clients hefty fees for managing the funds.
- dna_polymerase 3y ago
  When I read George's thought on this immediately Alexander Gerko and XTX Markets came to my mind. They operate one of the worlds largest GPU clusters (10k A100) [0]. They aren't really a hedge fund but a prop trading firm, but they seem to be winning the game [1]. [0]: https://www.stateof.ai/compute https://www.stateof.ai/compute [1]: https://financefeeds.com/xtx-markets-earns-1-095-billion-in-2022-as-revenue-surges-68/ https://financefeeds.com/xtx-markets-earns-1-095-billion-in-...

deleted 3y ago

[deleted]

wslh 3y ago

The problem with really hard problems as AI/AGI is that people extremely intelligent in one topic (e.g. GH in RE) uses its influence in another topic and the intelligence is not easily transferrable from one domain to the other, and even within a same domain. This is a typical epistemological crux that can be seen through history like Newton dealing with alchemy [1] and Einstein working on an unified physics theory. You can be the most intelligent human on earth but that is not enough. In this particular case, GH opinion could be written by any journalist. [1] https://en.wikipedia.org/wiki/Isaac_Newton%27s_occult_studies https://en.wikipedia.org/wiki/Isaac_Newton%27s_occult_studie...

janalsncm 3y ago

There’s a couple decent ideas in here. In a hostile takeover situation, AGI wouldn’t just need to be smarter than a human. It wouldn’t just need to be smarter than all humans collectively. It would need to be smarter than all humans working in collaboration with all of our machines.

AbrahamParangi 3y ago

Wild to me that people will take seriously people like Eliezer Yudkowsky and Nick Bostrom who are both "idea guys" who have accomplished exactly nothing in the field of AI. Their strength is in the world of discourse, not in the material world. Meanwhile, George Hotz, an engineer and a technologist - a guy who actually works with and understands technology, but isn't an acclaimed harry potter fanfiction writer or a stand up comedian - is derided because his arguments are dramatically less sexy, harder to appreciate why they might be right. People, please do not listen to people who sound great but have no credible claim to be an expert on something. There is a difference between an idea that sounds good and an idea that is good.

reducesuffering 3y ago
Why not wait until their debate on Aug. 15th? It's going to be a disaster for Geohot as he realizes Eliezer has been "mind-mazing" this problem for 20 years, and comes to realize the barrel of the gun we're all facing... Otherwise, in your appeal to technological authority, surely you'd rather listen to Yoshua Bengio and Geoff Hinton's perspectives rather than TFA's ramblings... right?
- AbrahamParangi 3y ago
  This is what I'm talking about! There's a distinction between being right and seeming right that we need to be aware of! "In theory, there's no difference between theory and practice" Eliezer is probably going to seem convincing, he's a great writer and a reasonably eloquent speaker - his audience is humans. Goerge Hotz's competence is machines, and he's exceptional at interfacing with them. Yes, I trust the less eloquent tinkerer who's only famous for his preternatural understanding of machines over any theoretician and 1000x more than a (pretty good!) writer.
  - Gartent 3y ago
    Eliezer is a wackjob. I'm pretty sure Kiwi Farms links are banned on this site but you should check out his thread. Or r/sneerclub. Recently, an actual AI researcher posted on his site an extremely detailed response to his claims to which he responded "this is kind of long". That said I don't know who Hotz is or if he's more competent

cameldrv 3y ago

This isn't even an argument.

davidhunter 3y ago

Yes, predicting the stock market does involve predicting what other market participants (and hedge fund bros) are going to do. But, to predict what other market participants (and hedge fund bros) are going to do, you need to predict any world dynamic that will have an eventual effect on stock pricing. The most successful quantitative funds (RenTech, 2sigma etc) consistently make billions of dollars in cash each year because they have collected the data sets that allow them to do this better than others. But they are still a long long long way off from having a true world model.

rafaelero 3y ago

I loved the analogy of AGI takeoff with markets. People change their behavior when faced with new environments. If AI ever becomes threatening, there is no way we won't do anything and there is no way AI will reliably predict how we will respond.

hexman 3y ago

[dead]

epups 3y ago

I don't agree with the premise that a 51% compute threshold is transformative, but it's scary to consider that collusion among a small number of big tech companies (Google, Amazon, Microsoft, Nvidia, AMD, Apple) would easily pass that threshold when it comes to GPU compute. More so when you consider they are also two steps ahead of everyone else when it comes to developing improvements.

perlgeek 3y ago

Remember the early days of COVID-19? There were some scientists that predicted exponential growth, and I was torn between "it's not really bad yet, maybe there's still hope" and "every exponential growth feels very slow at the start". I feel like we're with AI, we're in the part of the curve where you can feel it arcing upwards. There's definitely been an uptick in the AI capability. Still no fully self-driving cars, but good enough text generation that the Turing test is in the rear-view mirror. Previously, I've always written "machine learning" or "statistical models" or so, because calling it AI always seemed silly. Not anymore. Not everything that slowly arcs upwards must become an exponential curve, but we have theories that predict that. There can be many dampening factors that can make it a slower exponential curve, but even that will look like a brick wall if you zoom out far enough. I think a question we have to ask ourselves is: at what zoom level should we look at the AI capability curve? At the "can re react to an emergency?" time scale, which is, like, days to month? Or at the "can we change society to cope with the new situation?" scale, which is more like decades? At the first time scale, we probably won't even notice a hard takeoff. Viewed over decades, it'll probably very obvious, in retrospect, when it will have happened.

deleted 3y ago

[deleted]

pusspuss 3y ago

Universe’s Jive" (Complex instrumental sections with shifts in time signatures. Lyrics have a satirical undertone.) Verse 1: Summoned a shadow or just a silicon brain? From chessboards to streets, it's all part of the game. Think we're the champs, but are we just inane? Compared to the cosmos, are our brains too lame? Interlude: (Jazzy guitar riff mixed with an odd-metered percussion line.) Chorus: The singularity’s knocking, or is it just hype? Caught in the loop, the stereotypical type. Predicting, speculating, swallowing the tripe, But can we decode the universe's archetype? Verse 2: Markets and dreams, the billionaire’s dance, All looks rosy, till you're out of chance. Machines boast of a masterful glance, Yet, who pulls the strings in this vast expanse? Bridge: MuZero's groove, feeling so elite, Thinks it’s got the rhythm, can't accept defeat. But against the universe, can it compete? Or just another tune, incomplete, obsolete? Chorus: Singularity, they say, is the ultimate dive, But can we, mere mortals, really derive? Predictions abound, but can we survive? In the end, it’s about keeping the jive alive. Verse 3: Computations, simulations, all in a grid, In this tech circus, we're just a tiny squid. Chasing echoes, in shadows we're hid, But the cosmic joke? It's just a quid. Outro: (A whimsical woodwind section, possibly a kazoo solo.) In this cosmic game, we strive and strive, Dancing on the edge, trying to thrive. But remember, as the stars contrive, It's not about the end, but the drive.

pusspuss 3y ago

Eliezer persistently croons dark lullabies of AGI cataclysms. While his obsession is evident, one cannot help but wonder if these incessant dirges might be nothing more than echoes of groundless fears. Much like the puppet that cries out every time the stage darkens, only to find there's no true end approaching, we may find ourselves dulled to genuine threats, leading to our eventual and inescapable undoing. Moreover, by focusing too much on distant threats, we may overlook current problems, like companies gaining too much control through AGI. We don't need AI safety rules that give big players more power and shut down open-source projects.

benlivengood 3y ago

The argument falls apart when considering that compute is incredibly fungible and performing a 51% attack only means finding a zero-day that lets an AGI copy its code or neural net to a large percentage of computers. Much like wealth accumulates in the absence of antitrust law and taxation, intelligence accumulates in the absence of some external controls. EDIT: And probably all you need is natural selection. No one is running GPT-2 anymore except for basic research. Any AGI with marginal advantage over others will be run instead of them. At some level of complexity identical agents will coordinate as one entity with shared goals.

latchkey 3y ago

> Nobody has a 1e20x more efficient Bitcoin miner either The miners today are 1e20x more efficient than the original CPU based miners. I mean, maybe not exactly 1e20x, but we saw insanely massive improvements, in only 9 years. > Now, if there’s one weird trick to 1e20x your efficiency, and only one group gets it, all bets are off. Only one group really did get the massive efficiency increases... Bitmain, a Chinese company. They sold machines and even mined with them before shipping them to customers. Are all bets off with Bitcoin today? Nope.

foogazi 3y ago
> They sold machines and even mined with them before shipping them to customers. At what efficiency treshold do you not sell the product to your competition?
- latchkey 3y ago
  I think the game is deeper than that... what incentive do you have to destroy the foundation your business is based on top of?

lsy 3y ago

Most arguments for "hard takeoff" involve some form of inductive proof: if we have "intelligence level" n, and we know that it can be improved n + 1, then there is a guarantee of eventual performance n + k where k is arbitrarily large. Aside from not actually having any reliable way to quantify n or the concept of "intelligence" in the first place, this proof falls apart if improvement is n + ε, where ε changes at each step, and there is no reason to assume ε will escape the pattern of every other process in the known universe of trending to zero in the face of natural limiting factors.