4 ms·

What does neural mean nowadays anyway? I thought neural nets got their name from the perceptron's resemblance to interconnected neurons, but what about CNN's or

What does neural mean nowadays anyway? I thought neural nets got their name from the perceptron's resemblance to interconnected neurons, but what about CNN's or transformers? From what I can tell neural just means "plenty of parameters optimized using a learning process", in which case gaussian splats fits the description. I also think gaussian splats encode the same information as NeRFs, which have neural in the name.

throwup238 3y ago
I thought neural meant df/dx optimization applied to a giant function.
deleted 3y ago
[deleted]
bondarchuk 3y ago
>but what about CNN's or transformers? They also consist (in the abstract, at least) of interconnected neurons with activation functions.
numpad0 3y ago
Anything that require cuDNN installed in dev environment.
deleted 3y ago
[deleted]
danbruc 3y ago
[...] plenty of parameters optimized using a learning process [...] Would that not make fitting a degree 1,000,000 polynomial a neural method? I also think gaussian splats encode the same information as NeRFs, which have neural in the name. The collection of source images also encodes the same information but I don't think we should call them neural images. I would argue we should stick with the original meaning, inspired by biological neural networks, a network of artificial neurons. Calling everything neural because of the hype will benefit no one in the long run as the term will just lose any meaning.
hansvm 3y ago
Practically speaking, it's a history-dependent definition. Some set of things are considered "neural", somebody makes an innovation that's based on the existing set of "neural" tools, and now the set of "neural" things is expanded to include the new innovation. For this post, they have more traditional neural networks embedded in the pipeline (search TFA for "decoder"), and they're thus calling the composite technique a "neural" technique. As to the CNN/Transformer/... question, those are still very similar to a perceptron. A multi-layer perceptron has a sequence of alternating matmuls and activations. The CCN has a particular choice of sparse matmul and activation (and a couple choices of fast implementations to operate on that sparse matmul). A Transformer is one way to wire up many such MLP predictions (recurrent networks being another) to work with long/uneven inputs. There's more going on, but most of it is fluff to make the thing numerically stable and do something interesting. The core structure is still a dressed-up perceptron.

throwup238 3y ago

I thought neural meant df/dx optimization applied to a giant function.

deleted 3y ago

[deleted]

bondarchuk 3y ago

>but what about CNN's or transformers? They also consist (in the abstract, at least) of interconnected neurons with activation functions.

numpad0 3y ago

Anything that require cuDNN installed in dev environment.

deleted 3y ago

[deleted]

danbruc 3y ago

[...] plenty of parameters optimized using a learning process [...] Would that not make fitting a degree 1,000,000 polynomial a neural method? I also think gaussian splats encode the same information as NeRFs, which have neural in the name. The collection of source images also encodes the same information but I don't think we should call them neural images. I would argue we should stick with the original meaning, inspired by biological neural networks, a network of artificial neurons. Calling everything neural because of the hype will benefit no one in the long run as the term will just lose any meaning.

hansvm 3y ago

Practically speaking, it's a history-dependent definition. Some set of things are considered "neural", somebody makes an innovation that's based on the existing set of "neural" tools, and now the set of "neural" things is expanded to include the new innovation. For this post, they have more traditional neural networks embedded in the pipeline (search TFA for "decoder"), and they're thus calling the composite technique a "neural" technique. As to the CNN/Transformer/... question, those are still very similar to a perceptron. A multi-layer perceptron has a sequence of alternating matmuls and activations. The CCN has a particular choice of sparse matmul and activation (and a couple choices of fast implementations to operate on that sparse matmul). A Transformer is one way to wire up many such MLP predictions (recurrent networks being another) to work with long/uneven inputs. There's more going on, but most of it is fluff to make the thing numerically stable and do something interesting. The core structure is still a dressed-up perceptron.