4 ms·

NNs are not algorithms.

notfish 2y ago
An algorithm is “a process or set of rules to be followed in calculations or other problem-solving operations, especially by a computer” How does a giant pile of linear algebra not meet that definition?
- antirez 2y ago
  It's not made of "steps", it's an almost continuous function to its inputs. And a function is not an algorithm: it is not an object made of conditions, jumps, terminations, ... Obviously it has computation capabilities and is Turing-complete, but is the opposite of an algorithm.
  - raegis 2y ago
    > It's not made of "steps", it's an almost continuous function to its inputs. Can you define "almost continuous function"? Or explain what you mean by this, and how it is used in the A.I. stuff?
    - taneq 2y ago
      Well, it's a bunch of steps, but they're smaller. /s
  - janalsncm 2y ago
    If it wasn’t made of steps then Turing machines wouldn’t be able to execute them. Further, this is probably running an algorithm on top of an NN. Some kind of tree search. I get what you’re saying though. You’re trying to draw a distinction between statistical methods and symbolic methods. Someday we will have an algorithm which uses statistical methods that can match human performance on most cognitive tasks, and it won’t look or act like a brain. In some sense that’s disappointing. We can build supersonic jets without fully understanding how birds fly.
    - antirez 2y ago
      Let's see that Turing machines can approximate the execution of NN :) That's why there are issues related to numerical precision, but the contrary is also true indeed, NNs can discover and use similar techniques used by traditional algorithms. However: the two remain two different methods to do computations, and probably it's not just by chance that many things we can't do algorithmically, we can do with NNs, what I mean is that this is not just related to the fact that NNs discover complex algorithms via gradient descent, but also that the computational model of NNs is more adapt to solving certain tasks. So the inference algorithm of NNs (doing multiplications and other batch transformations) is just needed for standard computers to approximate the NN computational model. You can do this analogically, and nobody would claim much (maybe?) it's running an algorithm. Or that brains themselves are algorithms.
      - deleted 2y ago
        [deleted]
      - zeroonetwothree 2y ago
        We don’t have evidence that a TM can simulate a brain. But we know for a fact that it can execute a NN.
      - necovek 2y ago
        Computers can execute precise computations, it's just not efficient (and it's very much slow). NNs are exactly what "computers" are good for and we've been using since their inception: doing lots of computations quickly. "Analog neural networks" (brains) work much differently from what are "neural networks" in computing, and we have no understanding of their operation to claim they are or aren't algorithmic. But computing NNs are simply implementations of an algorithm. Edit: upon further rereading, it seems you equate "neural networks" with brain-like operation. But brain was an inspiration for NNs, they are not an "approximation" of it.
        antirez 2y ago
        But the inference itself is orthogonal to the computation the NN is going. Obviously the inference (and training) are algorithms.
  - deleted 2y ago
    [deleted]
  - mvkel 2y ago
    > continuous So, steps?
    - necovek 2y ago
      "Continuous" would imply infinitely small steps, and as such, would certainly be used as a differentiator (differential? ;) between larger discrete stepped approach. In essence, infinite calculus provides a link between "steps" and continuous, but those are different things indeed.
  - necovek 2y ago
    I would say you are right that function is not an algorithm, but it is an implementation of an algorithm. Is that your point? If so, I've long learned to accept imprecise language as long as the message can be reasonably extracted from it.
  - CooCooCaCha 2y ago
    Each layer of the network is like a step, and each token prediction is a repeat of those layers with the previous output fed back into it. So you have steps and a memory.
benlivengood 2y ago
Deterministic (ieee 754 floats), terminates on all inputs, correctness (produces loss < X on N training/test inputs) At most you can argue that there isn't a useful bounded loss on every possible input, but it turns out that humans don't achieve useful bounded loss on identifying arbitrary sets of pixels as a cat or whatever, either. Most problems NNs are aimed at are qualitative or probabilistic where provable bounds are less useful than Nth-percentile performance on real-world data.
KeplerBoy 2y ago
Running inference on a model certainly is a algorithm.
drdeca 2y ago
How do you define "algorithm"? I suspect it is a definition I would find somewhat unusual. Not to say that I strictly disagree, but only because to my mind "neural net" suggests something a bit more concrete than "algorithm", so I might instead say that an artificial neural net is an implementation of an algorithm, rather than or something like that. But, to my mind, something of the form "Train a neural network with an architecture generally like [blah], with a training method+data like [bleh], and save the result. Then, when inputs are received, run them through the NN in such-and-such way." would constitute an algorithm.
necovek 2y ago
NN is a very wide term applied in different contexts. When a NN is trained, it produces a set of parameters that basically define an algorithm to do inference with: it's a very big one though. We also call that a NN (the joy of natural language).

An algorithm is “a process or set of rules to be followed in calculations or other problem-solving operations, especially by a computer” How does a giant pile of linear algebra not meet that definition?

antirez 2y ago
It's not made of "steps", it's an almost continuous function to its inputs. And a function is not an algorithm: it is not an object made of conditions, jumps, terminations, ... Obviously it has computation capabilities and is Turing-complete, but is the opposite of an algorithm.
- raegis 2y ago
  > It's not made of "steps", it's an almost continuous function to its inputs. Can you define "almost continuous function"? Or explain what you mean by this, and how it is used in the A.I. stuff?
  - taneq 2y ago
    Well, it's a bunch of steps, but they're smaller. /s
- janalsncm 2y ago
  If it wasn’t made of steps then Turing machines wouldn’t be able to execute them. Further, this is probably running an algorithm on top of an NN. Some kind of tree search. I get what you’re saying though. You’re trying to draw a distinction between statistical methods and symbolic methods. Someday we will have an algorithm which uses statistical methods that can match human performance on most cognitive tasks, and it won’t look or act like a brain. In some sense that’s disappointing. We can build supersonic jets without fully understanding how birds fly.
  - antirez 2y ago
    Let's see that Turing machines can approximate the execution of NN :) That's why there are issues related to numerical precision, but the contrary is also true indeed, NNs can discover and use similar techniques used by traditional algorithms. However: the two remain two different methods to do computations, and probably it's not just by chance that many things we can't do algorithmically, we can do with NNs, what I mean is that this is not just related to the fact that NNs discover complex algorithms via gradient descent, but also that the computational model of NNs is more adapt to solving certain tasks. So the inference algorithm of NNs (doing multiplications and other batch transformations) is just needed for standard computers to approximate the NN computational model. You can do this analogically, and nobody would claim much (maybe?) it's running an algorithm. Or that brains themselves are algorithms.
    - deleted 2y ago
      [deleted]
    - zeroonetwothree 2y ago
      We don’t have evidence that a TM can simulate a brain. But we know for a fact that it can execute a NN.
    - necovek 2y ago
      Computers can execute precise computations, it's just not efficient (and it's very much slow). NNs are exactly what "computers" are good for and we've been using since their inception: doing lots of computations quickly. "Analog neural networks" (brains) work much differently from what are "neural networks" in computing, and we have no understanding of their operation to claim they are or aren't algorithmic. But computing NNs are simply implementations of an algorithm. Edit: upon further rereading, it seems you equate "neural networks" with brain-like operation. But brain was an inspiration for NNs, they are not an "approximation" of it.
      - antirez 2y ago
        But the inference itself is orthogonal to the computation the NN is going. Obviously the inference (and training) are algorithms.
- deleted 2y ago
  [deleted]
- mvkel 2y ago
  > continuous So, steps?
  - necovek 2y ago
    "Continuous" would imply infinitely small steps, and as such, would certainly be used as a differentiator (differential? ;) between larger discrete stepped approach. In essence, infinite calculus provides a link between "steps" and continuous, but those are different things indeed.
- necovek 2y ago
  I would say you are right that function is not an algorithm, but it is an implementation of an algorithm. Is that your point? If so, I've long learned to accept imprecise language as long as the message can be reasonably extracted from it.
- CooCooCaCha 2y ago
  Each layer of the network is like a step, and each token prediction is a repeat of those layers with the previous output fed back into it. So you have steps and a memory.

benlivengood 2y ago

Deterministic (ieee 754 floats), terminates on all inputs, correctness (produces loss < X on N training/test inputs) At most you can argue that there isn't a useful bounded loss on every possible input, but it turns out that humans don't achieve useful bounded loss on identifying arbitrary sets of pixels as a cat or whatever, either. Most problems NNs are aimed at are qualitative or probabilistic where provable bounds are less useful than Nth-percentile performance on real-world data.

KeplerBoy 2y ago

Running inference on a model certainly is a algorithm.

drdeca 2y ago

How do you define "algorithm"? I suspect it is a definition I would find somewhat unusual. Not to say that I strictly disagree, but only because to my mind "neural net" suggests something a bit more concrete than "algorithm", so I might instead say that an artificial neural net is an implementation of an algorithm, rather than or something like that. But, to my mind, something of the form "Train a neural network with an architecture generally like [blah], with a training method+data like [bleh], and save the result. Then, when inputs are received, run them through the NN in such-and-such way." would constitute an algorithm.

necovek 2y ago

NN is a very wide term applied in different contexts. When a NN is trained, it produces a set of parameters that basically define an algorithm to do inference with: it's a very big one though. We also call that a NN (the joy of natural language).