2 ms·

This is a discussion in the art world that you'd hear a lot, before stable diffusion came out. The whole idea of "steal like an artist". It's one of those ques

This is a discussion in the art world that you'd hear a lot, before stable diffusion came out. The whole idea of "steal like an artist". It's one of those questions about the nature of us that's on the level of "is there a ghost in the machine"; "do we have actually novel ideas?". I'm kind of leaning the way you are; I don't think we do. What seems to be happening that we _call_ novelty is just an advanced form of recursive synthesis - we take information in and remix it. But we're capable of levels of abstraction, so we can remix it really well. I think that getting the abstraction part into these models might just be as simple as wiring them together with a meta-model. A network responsible for identifying similarity between outputs of different types, or one that serves as a connective layer, distributing tokens through other networks and then synthesizing the results. What I'm almost certain of, is that the important changes are going to be architectural.

BulgarianIdiot 4y ago
The important changes will be architectural, yes. Three things I'm noticing: 1. We're reinventing structured programming in AI in fast forward. First it's a plain Markov chain. Then it's an "attention" directed acyclic graph. Then we realize we need loops. Then we realize we need to jump to different points in the loop. Then we realize it's useful to recursively call yourself or parts of yourself as a subroutine, parametrized with specific input. Etc. 2. Even before we fully realize this framework of thought into a model, I'm almost sure the model EVOLVES some of these structures during training. In the form of crude unrolled loops etc. Simply because it's inevitable for processing certain types of input data. 3. In order to preserve pragmatic outcomes, I'd bet the future is not one giant monolithic model for AI, but many medium-sized models, communicating in a meta network, like meta neurons, sending meta (high-level) messages to each other. Essentially, we need to make neural networks more like a fractal. I have this rule of thumb that always works somehow: "no concept definition is complete, until it's made recursive". Neural networks will get there.

BulgarianIdiot 4y ago

The important changes will be architectural, yes. Three things I'm noticing: 1. We're reinventing structured programming in AI in fast forward. First it's a plain Markov chain. Then it's an "attention" directed acyclic graph. Then we realize we need loops. Then we realize we need to jump to different points in the loop. Then we realize it's useful to recursively call yourself or parts of yourself as a subroutine, parametrized with specific input. Etc. 2. Even before we fully realize this framework of thought into a model, I'm almost sure the model EVOLVES some of these structures during training. In the form of crude unrolled loops etc. Simply because it's inevitable for processing certain types of input data. 3. In order to preserve pragmatic outcomes, I'd bet the future is not one giant monolithic model for AI, but many medium-sized models, communicating in a meta network, like meta neurons, sending meta (high-level) messages to each other. Essentially, we need to make neural networks more like a fractal. I have this rule of thumb that always works somehow: "no concept definition is complete, until it's made recursive". Neural networks will get there.