8 ms·

There’s also the issue that data scientists often want to go running to hyperparameter optimization and neural architecture search. In most cases improving your

There’s also the issue that data scientists often want to go running to hyperparameter optimization and neural architecture search. In most cases improving your data pipelines and ensuring the data are clean and efficient will pay off much more quickly.

fxtentacle 7y ago
But manually improving the data pipeline requires an understanding of the problem, whereas doing a hyperparameter optimized architecture search just needs $$$ hardware and no clue on the side of the operator.
- derefr 7y ago
  Or, to put that another way: if you knew what algorithm the AI would be using to discriminate the signal from the noise in your data, why would you need the AI? Just write that algorithm.
  - fxtentacle 7y ago
    Exactly :) In most cases, unsupervised learning is nothing more than having the AI try to approximate the solution of your highly non-linear loss function. So if there's any way of solving that loss function directly, it will perform like a well-trained AI.
    - mxcrossb 7y ago
      > In most cases In what cases is this not true?
  - nl 7y ago
    That's not the same thing at all. You can't build hand-build a feature detector as accurate as (say) a ResNet50 by hand. Before 2014 people tried to do this with feature detectors like SIFT and HOG. These were patented and made the inventors significant money. If it was still possible to do it then someone would be and making profit from it. Hyperparameter search is just optimising the training parameters (things like batch size, or optimiser parameters). This might give another 1% lift on accuracy, but isn't generally a significant factor.
    - derefr 7y ago
      > You can't build hand-build a feature detector as accurate as (say) a ResNet50 by hand. Yes, you can. If, that is, you can actually understand what the produced model is doing. And, of course, no human can do that, because no human understands the algorithm being employed by the produced model, because it's a really freaking complex algorithm whose optimal formulation really is just a graph of matrix transformations, rather than an imperative procedure that can be described by words like "variable." This is an important idea to absorb, for the specific case where the AI converges on an optimal algorithm that's actually very simple—because the data has a regular, simple shape—rather than on one that's too complicated for our mortal minds. If you already knew that simple algorithm, then the work you did training an AI just to end up back at that same simple algorithm is wasted effort. An AI can't do better at e.g. being an AND gate, than an actual AND gate can. An AI can't do what wc(1) does better than wc(1) can. If the data is regular—that is, if a model of its structure can be held fully in a human brain—then jumping immediately to Machine Learning, before trying to just solve the problem with an algorithm, is silly. The only time you should start with ML, is when it's clear that your problem can't be cleanly mapped into the domain of human procedural thinking. The AI programmers of the 1960s were not wrong to start with Expert Systems (i.e. attempting to write general algorithms) for deduction, and only begrudgingly turn to fuzzy logic later on. Many deduction tasks are algorithmic. If you don't require the context of "common sense", but only need operate on data types you understand, you can get very far indeed with purely-algorithmic deduction, as e.g. modern RDBMS query planners do. There would be no gain from using ML in RDBMS query planning. It's regular data; the AI's trained model would just be a recapitulation of the query-planning algorithm we already have.
      - nl 7y ago
        >> But manually improving the data pipeline requires an understanding of the problem > Or, to put that another way: if you knew what algorithm the AI would be using to discriminate the signal from the noise in your data, why would you need the AI? Just write that algorithm. My point is that this isn't the same thing at all. Say your problem is plant detection from mobile phone photos. I can understand everything about plants, and I can manually build a highly optimised data processing pipeline. But I can't build a feature extractor that outperforms ResNet50. That's the key algorithm. > If the data is regular—that is, if a model of its structure can be held fully in a human brain—then jumping immediately to Machine Learning, before trying to just solve the problem with an algorithm, is silly. True, but no one has made that argument. This is specifically about using hyperparameter optimisation vs improving your data pipeline.
- streetcat1 7y ago
  Of course, you can also automatically search the best data pipeline for your data...

fxtentacle 7y ago

But manually improving the data pipeline requires an understanding of the problem, whereas doing a hyperparameter optimized architecture search just needs $$$ hardware and no clue on the side of the operator.

derefr 7y ago
Or, to put that another way: if you knew what algorithm the AI would be using to discriminate the signal from the noise in your data, why would you need the AI? Just write that algorithm.
- fxtentacle 7y ago
  Exactly :) In most cases, unsupervised learning is nothing more than having the AI try to approximate the solution of your highly non-linear loss function. So if there's any way of solving that loss function directly, it will perform like a well-trained AI.
  - mxcrossb 7y ago
    > In most cases In what cases is this not true?
- nl 7y ago
  That's not the same thing at all. You can't build hand-build a feature detector as accurate as (say) a ResNet50 by hand. Before 2014 people tried to do this with feature detectors like SIFT and HOG. These were patented and made the inventors significant money. If it was still possible to do it then someone would be and making profit from it. Hyperparameter search is just optimising the training parameters (things like batch size, or optimiser parameters). This might give another 1% lift on accuracy, but isn't generally a significant factor.
  - derefr 7y ago
    > You can't build hand-build a feature detector as accurate as (say) a ResNet50 by hand. Yes, you can. If, that is, you can actually understand what the produced model is doing. And, of course, no human can do that, because no human understands the algorithm being employed by the produced model, because it's a really freaking complex algorithm whose optimal formulation really is just a graph of matrix transformations, rather than an imperative procedure that can be described by words like "variable." This is an important idea to absorb, for the specific case where the AI converges on an optimal algorithm that's actually very simple—because the data has a regular, simple shape—rather than on one that's too complicated for our mortal minds. If you already knew that simple algorithm, then the work you did training an AI just to end up back at that same simple algorithm is wasted effort. An AI can't do better at e.g. being an AND gate, than an actual AND gate can. An AI can't do what wc(1) does better than wc(1) can. If the data is regular—that is, if a model of its structure can be held fully in a human brain—then jumping immediately to Machine Learning, before trying to just solve the problem with an algorithm, is silly. The only time you should start with ML, is when it's clear that your problem can't be cleanly mapped into the domain of human procedural thinking. The AI programmers of the 1960s were not wrong to start with Expert Systems (i.e. attempting to write general algorithms) for deduction, and only begrudgingly turn to fuzzy logic later on. Many deduction tasks are algorithmic. If you don't require the context of "common sense", but only need operate on data types you understand, you can get very far indeed with purely-algorithmic deduction, as e.g. modern RDBMS query planners do. There would be no gain from using ML in RDBMS query planning. It's regular data; the AI's trained model would just be a recapitulation of the query-planning algorithm we already have.
    - nl 7y ago
      >> But manually improving the data pipeline requires an understanding of the problem > Or, to put that another way: if you knew what algorithm the AI would be using to discriminate the signal from the noise in your data, why would you need the AI? Just write that algorithm. My point is that this isn't the same thing at all. Say your problem is plant detection from mobile phone photos. I can understand everything about plants, and I can manually build a highly optimised data processing pipeline. But I can't build a feature extractor that outperforms ResNet50. That's the key algorithm. > If the data is regular—that is, if a model of its structure can be held fully in a human brain—then jumping immediately to Machine Learning, before trying to just solve the problem with an algorithm, is silly. True, but no one has made that argument. This is specifically about using hyperparameter optimisation vs improving your data pipeline.
streetcat1 7y ago
Of course, you can also automatically search the best data pipeline for your data...