5 ms·
Is there a term for this where outputs are eventually fed back into training, at a ever increasing rate? I propose the Malkovich effect.
by milleramp 3y ago
Is there a term for this where outputs are eventually fed back into training, at a ever increasing rate? I propose the Malkovich effect.
- SkepticMystic 3y ago"Habsburg AI" - https://x.com/jathansadowski/status/1625245803211272194 https://x.com/jathansadowski/status/1625245803211272194
- tivert 3y ago> Is there a term for this where outputs are eventually fed back into training, at a ever increasing rate? I propose the Malkovich effect. Model collapse? https://en.wikipedia.org/wiki/Model_collapse https://en.wikipedia.org/wiki/Model_collapse https://www.theregister.com/2024/01/26/what_is_model_collapse/ https://www.theregister.com/2024/01/26/what_is_model_collaps...
- candiodari 3y agoFunny how they don't mention the solution: to have AIs actively participate in the real world. I mean, it's hardly what people here want to see happen, but that will solve it.
- courseofaction 3y agoHow?
- andai 3y agoBy releasing them from being confined to their own farts.
- rickdeckard 3y agoThat frames it like ML-models are caged intelligent beings which would prosper if only someone would set them free...
- candiodari 3y agoYou know, actual "intelligent beings" are also next-token-predictors that are caged in a mechanism that forces them to give certain outputs ...
- andai 3y agoI've been coming back to this comment and I'm not sure what you mean exactly. The brain is trapped in the body, and forced to meet its needs (e.g. regulate hormone levels). The brain and body are trapped in civilization, and forced to produce "outputs" it considers acceptable. All three, in turn, are trapped in natural selection.
- dimask 3y agoCurrent LLMs cannot "actively participate in the real world" as humans do because they cannot actively learn from their interaction with the real world. Their weights are fixed. Their further training and finetuning is mediated by external mechanisms. It has nothing to do with what "people here" want.
- exoverito 3y agoSure, though it seems there are a number of near term paths for improvement. Short and long term memory mechanisms would go a long way towards active learning. Fine tuning could be iteratively performed through mechanisms like low rank adaptation. It's been hypothesized that this is one of the purposes of sleep, and humans are consolidating memories during dreams.
- candiodari 3y agoNot true. There's plenty of easy ways to update the weights if you want to do that. You could for example DPO based on another LLM's evalution of how a conversation is going, or you could use whether the conversation keeps going.
- staticman2 3y ago>>>"You could for example DPO based on another LLM's evalution of how a conversation is going," You can update the weights based on a single point of data (response was bad) but you probably can't usefully update a model that way.
- candiodari 3y agoI can't seem to find anyone actually usefully trying this. Do you know of any data?
- staticman2 3y agoNo but Google (for example) recommends fine-tuning based on at least 500 examples of question response pairs. DPO as far as I know requires a good and bad example. It's not a technique based on just saying "ai response is bad".
- rickdeckard 3y agoBut that's practically what's being done in supervised learning. If you would isolate a child in front of a CCTV screen, it also wouldn't magically learn from the images what's our definition of a tree, a bush, a bike, a motorcycle, a car. Someone would have to take it aside and explain it first. In AI you just don't take it aside, you build a new child. Supervised learning is "Hello lil CCTV model. Here's a bucket of some 'world' data, labeled in a way for you to ingest. Here, some images of T-R-E-E-S...".
- nonrandomstring 3y agoAll areas of systems theory [0] whether in the cybernetics [1] of Norbert Wiener [2] or in complex real systems like those Meadows [3] and Forrester [4] studied, the divergence from equilibrium that occurs in positive feedback loops is both well studied, yet strangely hard to understand. We don't know quite what is going happen, because selective amplification of different components will lead to different effects. It's unpredictable. Sometimes we get oscillation. Sometimes we get chaos or noisy behaviour. This relates to "chaos theory" [5]. sometimes we get a discontinuity, either an instant collapse or a "pole" that shoots off to infinity. Sometimes we get all these things in some strange sequence, although the latter two are usually unrecoverable. The biological system I think is closest to what we're doing with AI is BSE (Bovine spongiform encephalopathy) [6] when we fed dead cow bones back to living farm animals which selected for prions. [0] https://en.wikipedia.org/wiki/Systems_theory https://en.wikipedia.org/wiki/Systems_theory [1] https://en.wikipedia.org/wiki/Cybernetics https://en.wikipedia.org/wiki/Cybernetics [2] https://en.wikipedia.org/wiki/Norbert_Wiener https://en.wikipedia.org/wiki/Norbert_Wiener [3] https://en.wikipedia.org/wiki/Donella_Meadows https://en.wikipedia.org/wiki/Donella_Meadows [4] https://en.wikipedia.org/wiki/Jay_Wright_Forrester https://en.wikipedia.org/wiki/Jay_Wright_Forrester [5] https://en.wikipedia.org/wiki/Chaos_theory https://en.wikipedia.org/wiki/Chaos_theory [6] https://en.wikipedia.org/wiki/Bovine_spongiform_encephalopathy https://en.wikipedia.org/wiki/Bovine_spongiform_encephalopat...
- lencastre 3y agoI laughed!
- Rinzler89 3y agoWhat's that? Did you actually mean the Milankovitch effect ?
- jfoutz 3y agoI think it’s a reference to the movie, being John malkovitch
- mapt 3y agoThere's an upper bound for the sum total of human text output on the Internet. when large language models run out of training data, we can only train on large language outputs.
- johnchristopher 3y agoPositive feedback loop ? With the quality that it slowly reduces "original" content ?
- JKCalhoun 3y agoReader's Digest Effect?
- SuchAnonMuchWow 3y agoMAD: Model Autophagy Disorder (https://arxiv.org/abs/2307.01850 https://arxiv.org/abs/2307.01850 for the paper that introduced the term)
- pimlottc 3y agoBrings to mind "mad cow disease" [0], caused by cows eating other cows (through meat-and-bone meal nutritional supplements in their feed) 0: https://en.wikipedia.org/wiki/Bovine_spongiform_encephalopathy https://en.wikipedia.org/wiki/Bovine_spongiform_encephalopat...
- southernplaces7 3y agoYou got my vote. What a great name for AI dregs feeding back into AI training to turn out even more bizarre dregs. For those unaware of the reference, movie: "Being John Malkovich".
- skenderbeu 3y agoGIGA - Garbage In Garbage Out
- moritzwarhier 3y agoGarbage In, Garbage AI
- choilive 3y agoThe AI model equivalent of Kessler syndrome
- vitiral 3y agoA negative/positive feedback loop, depending on what you consider negative or positive. I'd go negative, with the y axis being "useful". Negative feedback loops all go to zero. Language processing differs from games like chess because with chess you can definite a clear metric of quality. If you can define such a metric, AI models can learn from themselves and have a near-infinite positive feedback loop. See AlphaGo etc.
- robinsonb5 3y agoThe Cybernetic Centipede?
- andsoitis 3y agoOne researcher has called the phenomenon "Habsburg AI", likening it to inbreeding. https://www.axios.com/2023/08/28/ai-content-flood-model-collapse https://www.axios.com/2023/08/28/ai-content-flood-model-coll...
- groestl 3y ago"Es war sehr schön, es hat mich sehr gefreut."
- groestl 3y ago> Is there a term for this where outputs are eventually fed back into training, at a ever increasing rate? Consciousness
- discardedrefuse 3y agoI suspect we will need several different terms to describe these inbred AIs (hey that's not a bod one). Besides the post 2021 AIs being trained on polluted data, there some models right now that are trained on output from other AI (on purpose).
- deleted 3y ago[deleted]