6 ms·

Tens of thousands of engineers (audio, vision, linguists etc.) spent millions of hours for billions of dollars in the past 30 years to invent algorithms that re

Tens of thousands of engineers (audio, vision, linguists etc.) spent millions of hours for billions of dollars in the past 30 years to invent algorithms that reliably tell us something about a bunch of data. For example, an corner feature algorithm (such as SIFT) can extract the locations of corners in an image and characterize them. This is essential to many kinds of information processing tasks because we want to apply the same algorithm to different data (generalization), so we kind of need an interface to the data. This interface is called a feature (or feature algorithm, feature extractor or feature-descriptor). All of this work (some of these papers have on the order of ten thousands of citations) is now obsolete because you can start with a random initialization of the weights of a neural network and iteratively improve the weights using backprop for any kind of task. All you need a measure of improvement that is relatively smooth and differentiable with respect to the network weights. What is surprising is that the circuits and programs within reach of backprop training of fully connected neural networks are actually astonishingly good at what they do. But ultimately, this is maybe not so surprising given that our brains do something similar all the time.

Kip9000 10y ago
>All of this work (some of these papers have on the order of ten thousands of citations) is now obsolete because you can start with a random initialization of the weights of a neural network and iteratively improve the weights using backprop for any kind of task Hardly correct. You can't magically learn any kind of task. You can't add arbitrary number of layers and hope for the back prop to do its magic. It is difficult. Deep learning techniques are what makes it somewhat feasible. SIFT is not obsolete because of NNs. They all have their pros and cons. You have to select the right tool for the job. BTW SIFT is not an edge detector (That's the Canny Transform). It describes images using salient features in scale invariant manner.
- hacker42 10y ago
  Typos fixed. "All" was hyperbole of course, but I think it definitely does not look good for the majority of the work done on features. SIFT was recently outperformed PN-Net for example.
  - nightski 10y ago
    SIFT is also quite old. It's amazing a single technique has retained so much value. Isn't it curious that modern convnets use convolution. On top that, they do convolutions at multiple scales (pooling). Starting to sound very familiar...
    - hacker42 10y ago
      Actually the neural net approaches are older than SIFT. Neural nets learn the distribution and even causal factors in the data. To me it seems that this distribution is often just too complex for it to be robustly captured by something that doesn't learn. Learning causal factors critically depends on learning along the depth of the network of latent variables which is a particularly opaque process, but this is what MLPs seem to do quite canonically (convnet being just a restricted special case of MLPs). I mean discerning causal factors is pretty much canonically the act of accumulating evidence with priors (weighted summation), deciding whether it is sufficient evidence and signaling how much it is (non-linearity).
      - nightski 10y ago
        Some of the approaches are, some aren't. SIFT itself builds upon knowledge that is much older than it. Either way it doesn't matter. The OP was arguing that the many years of man effort put into SIFT was a complete waste. I am saying that this is very shortsighted, as non-machine learning vision techniques have heavily influenced how we approach and think about vision problems even when using ML.
- deleted 10y ago
  [deleted]
- YeGoblynQueenne 10y ago
  >> SIFT is not obsolete because of NNs. They all have their pros and cons. Case in point- DNNs for image recogn. use Sobel edge detectors and other "obsolete" filters to do their magickal magic.
YeGoblynQueenne 10y ago
>> we kind of need an interface to the data. This interface is called a feature (or feature algorithm, feature extractor or feature-descriptor). Excellent in-a-nutshell explanation of features and thank you for a definition I really hadn't thought of. This though: >> this is maybe not so surprising given that our brains do something similar all the time. Is just so much fantasies, sorry to say. Neural nets (and machine learning in general) learn in ways that are completely unlike the human. They need huge, dense datasets, we can make do with scraps of sparse data. They need huge amounts of computational power, and time, we learn in the blink of an eye. They learn one thing at a time and can't generalise knowledge to even neighbouring domains, we can, oh yes indeed. An infant that can recognise images at the level of AlexNet, can at the same time tie its own shoelaces, speak rudimentary language and protect itself from danger etc. AlexNet can only map images to labels. It does that very well, but it's a one trick pony and so are all machine learning algorithms, fearsomely effective but heart-breakingly limited. Human minds are generalisation machines of the higest order and we are nowhere near figuring out how they (we) do it. Think of it this way: it took a few dozen researchers a few decades to come up with backprop. It took evolution billions of years to come up with a human mind. Which one do you think is the more optimised, and how much hubris does it take to convince oneself that they are pretty much the same in capabilities?

Kip9000 10y ago

>All of this work (some of these papers have on the order of ten thousands of citations) is now obsolete because you can start with a random initialization of the weights of a neural network and iteratively improve the weights using backprop for any kind of task Hardly correct. You can't magically learn any kind of task. You can't add arbitrary number of layers and hope for the back prop to do its magic. It is difficult. Deep learning techniques are what makes it somewhat feasible. SIFT is not obsolete because of NNs. They all have their pros and cons. You have to select the right tool for the job. BTW SIFT is not an edge detector (That's the Canny Transform). It describes images using salient features in scale invariant manner.

hacker42 10y ago
Typos fixed. "All" was hyperbole of course, but I think it definitely does not look good for the majority of the work done on features. SIFT was recently outperformed PN-Net for example.
- nightski 10y ago
  SIFT is also quite old. It's amazing a single technique has retained so much value. Isn't it curious that modern convnets use convolution. On top that, they do convolutions at multiple scales (pooling). Starting to sound very familiar...
  - hacker42 10y ago
    Actually the neural net approaches are older than SIFT. Neural nets learn the distribution and even causal factors in the data. To me it seems that this distribution is often just too complex for it to be robustly captured by something that doesn't learn. Learning causal factors critically depends on learning along the depth of the network of latent variables which is a particularly opaque process, but this is what MLPs seem to do quite canonically (convnet being just a restricted special case of MLPs). I mean discerning causal factors is pretty much canonically the act of accumulating evidence with priors (weighted summation), deciding whether it is sufficient evidence and signaling how much it is (non-linearity).
    - nightski 10y ago
      Some of the approaches are, some aren't. SIFT itself builds upon knowledge that is much older than it. Either way it doesn't matter. The OP was arguing that the many years of man effort put into SIFT was a complete waste. I am saying that this is very shortsighted, as non-machine learning vision techniques have heavily influenced how we approach and think about vision problems even when using ML.
deleted 10y ago
[deleted]
YeGoblynQueenne 10y ago
>> SIFT is not obsolete because of NNs. They all have their pros and cons. Case in point- DNNs for image recogn. use Sobel edge detectors and other "obsolete" filters to do their magickal magic.

YeGoblynQueenne 10y ago

>> we kind of need an interface to the data. This interface is called a feature (or feature algorithm, feature extractor or feature-descriptor). Excellent in-a-nutshell explanation of features and thank you for a definition I really hadn't thought of. This though: >> this is maybe not so surprising given that our brains do something similar all the time. Is just so much fantasies, sorry to say. Neural nets (and machine learning in general) learn in ways that are completely unlike the human. They need huge, dense datasets, we can make do with scraps of sparse data. They need huge amounts of computational power, and time, we learn in the blink of an eye. They learn one thing at a time and can't generalise knowledge to even neighbouring domains, we can, oh yes indeed. An infant that can recognise images at the level of AlexNet, can at the same time tie its own shoelaces, speak rudimentary language and protect itself from danger etc. AlexNet can only map images to labels. It does that very well, but it's a one trick pony and so are all machine learning algorithms, fearsomely effective but heart-breakingly limited. Human minds are generalisation machines of the higest order and we are nowhere near figuring out how they (we) do it. Think of it this way: it took a few dozen researchers a few decades to come up with backprop. It took evolution billions of years to come up with a human mind. Which one do you think is the more optimised, and how much hubris does it take to convince oneself that they are pretty much the same in capabilities?