5 ms·

I have been writing optimization solvers of many forms to solve problems in engineering for about 20 years. from "make excel do linear regression on some data"

I have been writing optimization solvers of many forms to solve problems in engineering for about 20 years. from "make excel do linear regression on some data" to linear least squares to some nonlinear methods, simulated annealing, bayesian methods, deep neural networks -- none of this is "a new kind of programming", it's "do a bunch of data munging, throw matrix at a function, get matrix back, interpret/plot." there's really no other magic in it than that.

ipsum2 6y ago
How well does "linear least squares to some nonlinear methods, simulated annealing, bayesian methods" work when you're doing speech to text, text generation or object detection? Deep learning is different in that it's a huge leap closer to human capabilities compared to the other methods you've listed.
- f6v 6y ago
  Deep learning is closer to linear regression than it’s to human capabilities.
  - mr_toad 6y ago
    > Deep learning is closer to linear regression than it’s to human capabilities. Quantitatively perhaps, in that deep learning is equivalent to many neurons, whereas a least squares model is only equivalent to a single neuron. Other than that there is not much evidence that a human brain is qualitatively different.
    - f6v 6y ago
      Deep learning is much more than a "long" network nowadays. Most of DL success comes from clever architectures, like Inception Net. AFAIK there's no evidence human brain is anything like those fancy deep NNs.
- fock 6y ago
  well, your neural network is just a very complicated - ideally continously differentiable - function. In principle you could just fit this function to data with nonlinear least squares (e.g. Levenberg-Marquardt), we use backpropagation and SGD, because it's faster. Concerning simulated annealing: have a look at the old Alphafold and check how they optimize their neural networks... So, could you explain again how the concepts and foundations of deep learning differs from plain old regression techniques?
xapata 6y ago
Recurrent networks don't give you a matrix back. They return a state machine. Qualitatively different. Transitioning from a function that returns a scalar, to returning a polynomial, to returning an arbitrary function, and now to returning something stateful is a big difference. I suppose the next step is to write an algorithm that trains a Turing machine.
- _nhynes 6y ago
  RNNs provided by common frameworks really do give a matrix back (examples [0, 1]). They're about as stateful as any other object or generator function. [0]: https://pytorch.org/docs/stable/generated/torch.nn.RNN.html https://pytorch.org/docs/stable/generated/torch.nn.RNN.html [1]: https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cudnnRNNForwardTraining https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cu...
  - xapata 6y ago
    What other statistical machine learning algorithm gives you a state machine as an output? The particular encoding -- matrix or otherwise -- is just an implementation detail.
    - _nhynes 6y ago
      > What other statistical machine learning algorithm gives you a state machine as an output? * hidden Markov models * autoregressive models * learned LQR Sure, some of those have finite memory, but RNNs are practically limited–not as easily quantified. I see what you're saying about the single-step operation appearing unique, but I also think that RNNs can be viewed through a lens such that they look like "normal" programming concepts like generators and folding iterators.
      - xapata 6y ago
        I'm not familiar with LQRs. Versus HMMs and linear autoregressive models, I think avoiding the need to predefine the form of the autoregression is a bigger leap than it might appear right now. I'd always been a neural network skeptic, thinking they're excessively complex for no benefit. It seems to me that's no longer true. I'm not certain that'll hold, but it's exciting.

ipsum2 6y ago

How well does "linear least squares to some nonlinear methods, simulated annealing, bayesian methods" work when you're doing speech to text, text generation or object detection? Deep learning is different in that it's a huge leap closer to human capabilities compared to the other methods you've listed.

f6v 6y ago
Deep learning is closer to linear regression than it’s to human capabilities.
- mr_toad 6y ago
  > Deep learning is closer to linear regression than it’s to human capabilities. Quantitatively perhaps, in that deep learning is equivalent to many neurons, whereas a least squares model is only equivalent to a single neuron. Other than that there is not much evidence that a human brain is qualitatively different.
  - f6v 6y ago
    Deep learning is much more than a "long" network nowadays. Most of DL success comes from clever architectures, like Inception Net. AFAIK there's no evidence human brain is anything like those fancy deep NNs.
fock 6y ago
well, your neural network is just a very complicated - ideally continously differentiable - function. In principle you could just fit this function to data with nonlinear least squares (e.g. Levenberg-Marquardt), we use backpropagation and SGD, because it's faster. Concerning simulated annealing: have a look at the old Alphafold and check how they optimize their neural networks... So, could you explain again how the concepts and foundations of deep learning differs from plain old regression techniques?

xapata 6y ago

Recurrent networks don't give you a matrix back. They return a state machine. Qualitatively different. Transitioning from a function that returns a scalar, to returning a polynomial, to returning an arbitrary function, and now to returning something stateful is a big difference. I suppose the next step is to write an algorithm that trains a Turing machine.

_nhynes 6y ago
RNNs provided by common frameworks really do give a matrix back (examples [0, 1]). They're about as stateful as any other object or generator function. [0]: https://pytorch.org/docs/stable/generated/torch.nn.RNN.html https://pytorch.org/docs/stable/generated/torch.nn.RNN.html [1]: https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cudnnRNNForwardTraining https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cu...
- xapata 6y ago
  What other statistical machine learning algorithm gives you a state machine as an output? The particular encoding -- matrix or otherwise -- is just an implementation detail.
  - _nhynes 6y ago
    > What other statistical machine learning algorithm gives you a state machine as an output? * hidden Markov models * autoregressive models * learned LQR Sure, some of those have finite memory, but RNNs are practically limited–not as easily quantified. I see what you're saying about the single-step operation appearing unique, but I also think that RNNs can be viewed through a lens such that they look like "normal" programming concepts like generators and folding iterators.
    - xapata 6y ago
      I'm not familiar with LQRs. Versus HMMs and linear autoregressive models, I think avoiding the need to predefine the form of the autoregression is a bigger leap than it might appear right now. I'd always been a neural network skeptic, thinking they're excessively complex for no benefit. It seems to me that's no longer true. I'm not certain that'll hold, but it's exciting.