Practically speaking, it's a history-dependent definition. Some set of things are considered "neural", somebody makes an innovation that's based on the existing

Practically speaking, it's a history-dependent definition. Some set of things are considered "neural", somebody makes an innovation that's based on the existing set of "neural" tools, and now the set of "neural" things is expanded to include the new innovation. For this post, they have more traditional neural networks embedded in the pipeline (search TFA for "decoder"), and they're thus calling the composite technique a "neural" technique. As to the CNN/Transformer/... question, those are still very similar to a perceptron. A multi-layer perceptron has a sequence of alternating matmuls and activations. The CCN has a particular choice of sparse matmul and activation (and a couple choices of fast implementations to operate on that sparse matmul). A Transformer is one way to wire up many such MLP predictions (recurrent networks being another) to work with long/uneven inputs. There's more going on, but most of it is fluff to make the thing numerically stable and do something interesting. The core structure is still a dressed-up perceptron.