3 ms·
That's not the same thing at all. You can't build hand-build a feature detector as accurate as (say) a ResNet50 by hand. Before 2014 people tried to do this wi
by nl 7y ago
That's not the same thing at all.
You can't build hand-build a feature detector as accurate as (say) a ResNet50 by hand. Before 2014 people tried to do this with feature detectors like SIFT and HOG. These were patented and made the inventors significant money. If it was still possible to do it then someone would be and making profit from it.
Hyperparameter search is just optimising the training parameters (things like batch size, or optimiser parameters). This might give another 1% lift on accuracy, but isn't generally a significant factor.
- derefr 7y ago> You can't build hand-build a feature detector as accurate as (say) a ResNet50 by hand. Yes, you can. If, that is, you can actually understand what the produced model is doing. And, of course, no human can do that, because no human understands the algorithm being employed by the produced model, because it's a really freaking complex algorithm whose optimal formulation really is just a graph of matrix transformations, rather than an imperative procedure that can be described by words like "variable." This is an important idea to absorb, for the specific case where the AI converges on an optimal algorithm that's actually very simple—because the data has a regular, simple shape—rather than on one that's too complicated for our mortal minds. If you already knew that simple algorithm, then the work you did training an AI just to end up back at that same simple algorithm is wasted effort. An AI can't do better at e.g. being an AND gate, than an actual AND gate can. An AI can't do what wc(1) does better than wc(1) can. If the data is regular—that is, if a model of its structure can be held fully in a human brain—then jumping immediately to Machine Learning, before trying to just solve the problem with an algorithm, is silly. The only time you should start with ML, is when it's clear that your problem can't be cleanly mapped into the domain of human procedural thinking. The AI programmers of the 1960s were not wrong to start with Expert Systems (i.e. attempting to write general algorithms) for deduction, and only begrudgingly turn to fuzzy logic later on. Many deduction tasks are algorithmic. If you don't require the context of "common sense", but only need operate on data types you understand, you can get very far indeed with purely-algorithmic deduction, as e.g. modern RDBMS query planners do. There would be no gain from using ML in RDBMS query planning. It's regular data; the AI's trained model would just be a recapitulation of the query-planning algorithm we already have.
- nl 7y ago>> But manually improving the data pipeline requires an understanding of the problem > Or, to put that another way: if you knew what algorithm the AI would be using to discriminate the signal from the noise in your data, why would you need the AI? Just write that algorithm. My point is that this isn't the same thing at all. Say your problem is plant detection from mobile phone photos. I can understand everything about plants, and I can manually build a highly optimised data processing pipeline. But I can't build a feature extractor that outperforms ResNet50. That's the key algorithm. > If the data is regular—that is, if a model of its structure can be held fully in a human brain—then jumping immediately to Machine Learning, before trying to just solve the problem with an algorithm, is silly. True, but no one has made that argument. This is specifically about using hyperparameter optimisation vs improving your data pipeline.
- derefr 7y ago> True, but no one has made that argument. Er, yes, I did, in my original post. The form you quoted was me attempting to be more precise in rephrasing it. My point—my original point, this whole time—was that applying an advanced “feature extraction” algorithm to a data source whose features are explicitly encoded in a lossless, linearly-recoverable way in the data—what we usually call structured data—is silly. For example, there’s no point in using ResNet50 to extract the “features” of a formal grammar, like JSON. It’d just be badly simulating a JSON parser. In fact, there’s pretty much no data structure software engineers use, where ResNet50 would give you more information out than you’d get from just using the ADT interface of the data structure. What features are in a queue? Items and an ordering. What’s in a tree? Items, an ordering, and parent-child relationships. Etc. The only place where it might make sense to use ML when dealing with structured data, is with statistical data structures like Bloom filters. ResNet50 might be able to recover some of the original data out of a bloom filter, in est using it as a compressed-sensing tool, or (in the algorithmic CS domain) as a decompressor for a lossy, underdetermined compression codec. —— My second point was that, often, it turns out that your data is structured data, even when you didn’t ask for structured data. Some natural-world datasets are structured! Example: the standard model of quantum chromodynamics describes a clean digraph of possible spin configurations. You don’t need feature detection when looking at LHC data. The dataset is pre-bucketed, the items pre-tagged, by nature itself. But more often, what happens is that your data turns out to not be “raw” / primary-source data, but rather a secondary source that was already structured, enriched, and feature-extracted by someone else before you got there. Scraping social network data? It’s already a graph, and it often already has annotation fields in the JSON graph endpoints describing the relationships between the members. If you don’t just stop and look at the dataset, you might think your feature-extractor is doing something very clever, when actually it’s just finding the explicit pre-chewed “relationship” field and spitting it back out at you. —— You might not see the relation to a kD-sample-matrix feature-extractor like ResNet50, so here’s some more tightly-analogous examples: • What if the images in your training dataset turn out to be in Fireworks PNG format, where the raster data contains an embedding of the original vector image it was rendered from? Specializing your feature-extractor to this data is just going to make it learn to find those vectors (and extract features from those), rather than depending on the features in the raster data; and then it’ll fail on images without embedded vector descriptions. And if that’s all you want, why not just use a PNG parser to pull out the vectors? • What if your audio files turn out to all have been MIDI files rendered out from a certain synthesizer using its default set of instrument patches? Will feature-extraction on this rendered data beat just writing a program to exact-match and decode the instruments back to a MIDI description? Certainly there might be MIDI-level features you want to extract, but will ResNet50 be better at extracting those MIDI-level features for having seen the rendering, as opposed to having been fed the decoded MIDI-level data directly?