It's not just loops; there is also a time axis: Our brains are highly concurrent/distributed systems where each neuron exists & acts independently of any speci

It's not just loops; there is also a time axis: Our brains are highly concurrent/distributed systems where each neuron exists & acts independently of any specific task you give the network. Inputs (both from external sources/senses as well as other neurons) might come in at different times, and neurons continuously compute their excitation levels (and then, with a bit of delay, might or might not go on to excite their neighbors). Put differently, there is no such thing as an inference "run", where you provide an input to the NN, follow its topology and do all your matrix computations, and then get an output. It's a highly cyclic machine that continuously reads inputs and continuously produces outputs. Heck, it seems that parts of the brain are active even when there's very little input and no visible output (like when you're dreaming). The feedback loops you mentioned certainly play a huge role here – excitations can take on a life of their own. > Is it because they would cost too much to train? I think it's both: Even a single run of, say, ChatGPT is already quite expensive in terms of compute and now you want to keep the network "alive" at all times – that would surely cost orders of magnitudes more. But on top of that, by introducing a time axis and continuously feeding inputs to the network and reading its outputs, you also lost that clear input—output relationship that training data exhibits these days. While it's certainly possible to introduce a time dimension to training data, too, that seems like a whole different ballgame to me.