3 ms·

If you look at the actual process for building these deep neural networks, it's actually far more emergent and bottom-up than you might think just reading the n

If you look at the actual process for building these deep neural networks, it's actually far more emergent and bottom-up than you might think just reading the news. I took a deep learning class before things really took off [0], and what we were taught was that a lot of deep learning research was just trying different structures to see what worked. Even the people who were good at it didn't base their architectures on strong theories for how and why things would work, they just had developed an intuition through lots of trial and error. All the explanations for what the various structures in a deep learning model are doing are post hoc—they're observations made of the behavior of the systems after we build them. [0] Edit: Not just a random deep learning class either. The TA for that class created AI Dungeon that semester. The professor came in one day with a story of how they accidentally racked up tens of thousands of dollars in Google Cloud bills in 24 hours when that went viral.

dundarious 3y ago
Early architecture/structural engineering (figuring out arches, up to building cathedrals) was done by "just trying different structures to see what worked". Calling this "emergent" is not what is typically meant by the word -- we don't consider cathedrals to be "emergent structures". It's fair to say that many people imprecisely use the term, but then you are contributing to its drift into becoming (or perhaps always having been) a floating signifier.
- wolframhempel 3y ago
  But Cathedrals still had central planning, blueprints, ropes stuck in the ground to outline what should be built. Thermites on the other hand use a process called "Stigmergy" in which work allocation and building structures emerge organically without central coordination.
  - dundarious 3y ago
    They justified their point by saying the research was "emergent". My point is that almost all pre-modern research was "emergent" by their usage, and a large amount of contemporaneous research is "emergent" as well. That would be slightly annoying, but they did this to then justify points about how the resulting model is "emergent". The research process, and the resulting output, can and (my main point) almost always do, differ as to whether they are truly emergent.
edgyquant 3y ago
There is nothing emergent about building a specific topology and hoping it works. Emergent would be providing N number of neurons with no topology and letting the topology arise by itself. No current ANN architectures do this, at least not any widespread ones, and trying several different topologies without knowing which one will work isn’t emergent either.

dundarious 3y ago

Early architecture/structural engineering (figuring out arches, up to building cathedrals) was done by "just trying different structures to see what worked". Calling this "emergent" is not what is typically meant by the word -- we don't consider cathedrals to be "emergent structures". It's fair to say that many people imprecisely use the term, but then you are contributing to its drift into becoming (or perhaps always having been) a floating signifier.

wolframhempel 3y ago
But Cathedrals still had central planning, blueprints, ropes stuck in the ground to outline what should be built. Thermites on the other hand use a process called "Stigmergy" in which work allocation and building structures emerge organically without central coordination.
- dundarious 3y ago
  They justified their point by saying the research was "emergent". My point is that almost all pre-modern research was "emergent" by their usage, and a large amount of contemporaneous research is "emergent" as well. That would be slightly annoying, but they did this to then justify points about how the resulting model is "emergent". The research process, and the resulting output, can and (my main point) almost always do, differ as to whether they are truly emergent.

edgyquant 3y ago

There is nothing emergent about building a specific topology and hoping it works. Emergent would be providing N number of neurons with no topology and letting the topology arise by itself. No current ANN architectures do this, at least not any widespread ones, and trying several different topologies without knowing which one will work isn’t emergent either.