6 ms·
I find the mathematics in this paper a little incoherent so it's hard to criticise it on those grounds - but on a charitable read, something that sticks out to
by bubblyworld 1y ago
I find the mathematics in this paper a little incoherent so it's hard to criticise it on those grounds - but on a charitable read, something that sticks out to me is the assumption that AGI is some fixed total computable function from the fixed decision domain to a policy.
AIs these days autonomously seek information themselves. Much like living things, they are recycling entropy and information to/from their environment (the internet) at runtime. The framing as a sterile, platonic algorithm is making less and less sense to me with time.
(obviously they differ from living things in lots of other ways, just an example)
- sgt101 1y agoOk - where do AIs put the information that they "seek" from the internet?
- cess11 1y agoI suspect there's a harsher argument to be made regarding "autonomous". Pull the power cord and see if it does what a mammal would do, or if it rather resembles a chaotic water wheel.
- bubblyworld 1y agoI think it would turn off, no shocker there. I'm not sure what you mean, can you elaborate? When I say autonomous I don't mean some high-falutin philosophical concept, I just mean it does stuff on it's own.
- cess11 1y agoRight, but it doesn't. It stops once you stop forcing it to do stuff.
- viraptor 1y agoBecause that's what they're created to do. You can make a system which runs continuously. It's not a tech limitation, just how we preferred things to work so far.
- cess11 1y agoMaybe, but that's not the case here so it is lost on me why you bring it up.
- viraptor 1y agoYou're making claims about those systems not being autonomous. When we want to, we create them to be autonomous. It's got nothing to do with agency or survival instincts. Experiments like that have been done for years now - for example https://techcrunch.com/2023/04/10/researchers-populated-a-tiny-virtual-town-with-ai-and-it-was-very-wholesome/ https://techcrunch.com/2023/04/10/researchers-populated-a-ti...
- cess11 1y agoYes, because they aren't. Against your fantasy that some might be brought into existence sometime in the future I present my own fantasy that there won't be.
- viraptor 1y agoI linked you an experiment with multiple autonomous agents operating continuously. It's already happened. It's really not clear what you're disagreeing with here.
- cess11 1y agoNo, that was a simulation, akin to Conway's cellular automata. You seem to consider being fully under someone else's control to qualify as autonomy, at least in certain casees, which to me comes across as very bizarre.
- bubblyworld 1y agoI still don't understand your point, sorry. If it's a semantic nitpick about the meaning of "autonomous", I'm not interested - I've made my definition quite clear, and it has nothing to do with when agents stop doing things or what happens when they get turned off.
- cess11 1y agoI think you should start caring about the meaning of words.
- bubblyworld 1y agoI do, when I think it's relevant. Words don't have an absolute meaning - I've presented mine.
- mystified5016 1y agoYou're the one using words incorrectly. Everybody else agrees on what these words mean and you're insisting on your own made-up definitions. And then you throw a fit like a child when someone disagrees. You're wrong and you're behaving inappropriately.
- cess11 1y agoNo, I did not. It appears I'm rather alone in this setting to make a difference between automatic and autonomous. If immediate, direct dependence and autonomy are compatible, I want none of it.
- JumpCrisscross 1y ago> Pull the power cord and see if it does what a mammal would do Pulling the power cord on a mammal means shutting off its metabolism. That predictably kills us.
- cess11 1y agoNo. If the analogy had been about frying the chips and internal wires, then maybe that might have been a reasonable comparison, but it was not. Now it's about cutting the supply of food.
- ben_w 1y ago"Food" is only analogous to "mains power" for devices which also have a battery. But regarding hunger: while they are a weird and pathological example, breatharians are in fact mammals, and the result of the absence of food is sometimes "starves to death" and not always "changes mind about this whole breatharian thing" or "pathological dishonesty about calorie content of digestive biscuits dunked in tea".
- cess11 1y agoRight, so you agree that there is a clear difference between a mammal and the device we're discussing. I'm not sure why introducing a certain type of rare scam artist into the modeling of this thought experiment would make things clearer or more interesting.
- ben_w 1y ago> Right, so you agree that there is a clear difference between a mammal and the device we're discussing. A difference that you have not demonstrated the relevance of. If I run an AI on my laptop and unplug the charger, this runs until the battery dies. If I have a mammal that does not eat, it lives until it starves. If I run an AI on a desktop and unplug the mains, it ceases function in milliseconds (or however long the biggest capacitor in the PSU lasts). If I (for the sake of argument) had a device that could instantly remove all the ATP from a mammal's body, they'd also be dead pretty quick. If I have an android, purely electric motors and no hydraulics, and the battery connector comes loose, it ragdolls. Same for a human who has a heart attack. An AI that is trained with rewards for collecting energy to recharge itself, does so. One that has no such feedback, doesn't. Most mammals have such a mechanism from evolution, but there are exceptions where that signal is missing (not just weird humans), and they starve. None of these things say anything about intelligence. > I'm not sure why introducing a certain type of rare scam artist into the modeling of this thought experiment would make things clearer or more interesting. Because you're talking about the effect of mammals ceasing the consumption of food, and they're an example of mammals ceasing the consumption of food.
- davedx 1y agoInto their short term memory (context). Some information is also stored in long term memory (user store)
- bubblyworld 1y agoI can see what you are getting at but consider: I had an experience the other day where claude code wrote a script that shelled out to other LLM providers to obtain some information (unprompted by me). More often it requests information from me directly. My point is that the environment itself for these things is becoming at least as computationally complex or irreducible (as the OP would say) as the model's algorithm, so there's no point trying to analyse these things in isolation.
- DANmode 1y agoTruthfully, few people know that right now! They're backfeeding what it's "learning" along the way - whether it's in a smart fashion, we don't know yet.
- JyB 1y agoIt’s the whole “MCP” moto right? Basically keep re-feeding in each subsequent prompts the additional requested context?
- DANmode 1y agoNah, that's still considered an ephemeral user session. We're talking about what the service does with what it learns from the failures (and successes) marked by user feedback into the chat, or the app's buttons.
- kreetx 1y ago(I've a recently acquired strong opinion on MCP, beware) MCP seems to be the idea to expose literally any data source to AI agents as context. E.g, in work/business context: a database, source code, a kanban board; in private context: your photo library, notes, etc. This is good and makes sense. As a technical "standard" or "protocol" (as it claims to be), it's a total mess, read more here: https://modelcontextprotocol.io https://modelcontextprotocol.io. Though I guess they are inventing it as they go.
- usrbinbash 1y ago> Much like living things, they are recycling entropy and information to/from their environment (the internet) at runtime. 3 Problems with that assumption: a) Unlike living things, that information doesn't allow them to change. When a human touches a hotplate for the first time, it will (in addition to probably yelling and cursing a lot), learn that hotplates are dangerous and change its internal state to reflect that. What we currently see as "AI" doesn't do that. Information gathered through means such as websearch + RAG, has ZERO impact on the systems internal makeup. b) The "AI" doesn't collect the information. The model doesn't collect anything, and in fact can't. It can produce some sequence that may or may not cause some external entity to feed it back some more data (e.g. a websearch, databases, etc.). That is an advantage for technical applications, because it means we can easily marry an LLM to every system imaginable, but its really bad for the prospect of an AGI, that is supposed to be "autonomous". c) The representation of the information has nothing to do with what it represents. All information an LLM works with, including whatever it is eing fed from th outside, is represented PURELY AND ONLY in terms of statistical relationships between the tokens in the message. There is no world-model, there is no understanding of information. There is mimicry of these things, to the point where they are technically useful and entice humans to anthropomorphise them (a BIIIG chunk of VC money hinges on that), but no actual understanding...and as soon as a model is left to its own devices, which would be a requirement for an AGI (remember: Autonomous), that becomes a problem.
- bubblyworld 1y agoIt's not really an assumption, it's an observation. Run an agentic tool and you'll see it do this kind of thing all the time. It's pretty clear that they use the information to guide themselves (i.e. there's an entropy reduction there in the space of future policies, if you want to use the language of the OP). > Unlike living things, that information doesn't allow them to change. It absolutely does. Their behaviour changes constantly as they explore your codebase, run scripts, question you... this is just plainly obvious to anyone using these things. I agree that somewhere down the line there is a fixed set of tensors but that is not the algorithm. If you want to analyse this stuff in good faith you need to include the rest of the system too, including it's memory, context and more generally any tool it can interact with. > The "AI" doesn't collect the information. I really don't know how to engage on this. It certainly isn't me collecting the information. I just tell it what I want it to do at a high level and it goes and does all this stuff on its own. > There is no world-model, there is no understanding of information. I'm also not going to engage on this. I could care less what labels people assign to the behaviour of AI agents, and whether it counts as "understanding" or "intelligence" or whatever. I'm interested in their observable behaviour, and how to use them, not so much in the philosophy. In my experience trying to discuss the latter just leads to flame wars (for now).