4 ms·

If I use this, could I make something that is better than machine learning using neural networks?

derefr 13y ago
Not better in the general case; most machine-learning algorithms do worse than the naive solution ("throw a Bayes net at it") in most applications, but then have a niche where they excel. If I recall, Numenta's model is said to excel in the case where the data it's being trained on is similar to the kind the human brain tends to process: sensory data with temporally-local "features" (letters on a page, instruments in a song), where the features form a hierarchy (shapes->letters->words->sentences, frequencies->notes->chords->riffs->melodies, etc.)
- tensor 13y ago
  > most machine-learning algorithms do worse than the naive solution ("throw a Bayes net at it") in most applications This is an extremely misleading statement. There is nothing naive about probabilistic graphical models. It's a massive topic with a huge number of different model variations, training methods, and techniques to avoid overfitting or do feature selection. More many probably you probably only want a linear classifier, and a discriminative one at that. But if your data is not linearly separable, or has structure, then you'll want something else again.
  - derefr 13y ago
    Hmm, I suppose I did phrase that in a way that would mislead someone who didn't already know what I meant by it. :) I didn't mean that Bayes nets are themselves naive--they're extremely good models for many use-cases. In fact, they're eventually ideal models for all use-cases, given enough training, since all ML models are ways to approximate Bayes' law, while Bayes nets just solve for Bayes' law directly. However, for being so generic, they have no embedded structure that allows them to efficiently learn/represent/compress data in any particular domain.[1] When I have a Big Data problem, and I have a BNN library close to hand, I just throw it at a random subsampling and see what happens. This is what I mean by "naive": since it's a generic solution, you can try applying it to a problem before even studying the structure of the problem domain, and you might just detect a signal soon enough that you don't need to go any further.[2] The rest of the machine-learning toolbox, then, is basically a set of models that you replace your BNN with when you have a model that hews closely to the structure of your data (in est, a set of screwdrivers for different kinds of screws.) The approximate model will get quicker training and clearer signals for that data, but not for just any data. --- [1] Another way to say it: a Bayes-net implementation is to gzip as other ML models are to JPEG, MP3, etc. Bayes-nets can learn anything, but they can only learn it as well as the data can be used to model the data. [2] People do use linear regression as the same kind of general-purpose tool, as you say--and linear classifier libraries are much easier to write, so you'll find them as "batteries included" in more math packages--but if you don't know whether the data is linearly separable, linear models aren't gonna give you even the slightest hint. A BNN is an extremely-inefficient, but completely-general probe.
    - volokoumphetico 13y ago
      I have no clue about all that academic stuff, what's a good reading material for such things?
  - Eliezer 13y ago
    Yeah, "throw a Bayes net at it" doesn't make much sense given the current state of machine learning tech AFAIK. Throw a linear regression or a regularized linear regression, sure. Throw k-nearest-neighbors or naive Bayes EM clustering, maybe. "Throw a Bayes net", not so much.
- volokoumphetico 13y ago
  Your description of Numenta is very exciting for me as it appears to fit my niche. For example, can Numenta look at two similar pages from pdf, and be able to deduce and discover the location of title, paragraph, heading, subheading, etc? Those pdf files are different in layout but a human can immediately determine the hierarchy by categorizing entities on that temporally local feature.

Not better in the general case; most machine-learning algorithms do worse than the naive solution ("throw a Bayes net at it") in most applications, but then have a niche where they excel. If I recall, Numenta's model is said to excel in the case where the data it's being trained on is similar to the kind the human brain tends to process: sensory data with temporally-local "features" (letters on a page, instruments in a song), where the features form a hierarchy (shapes->letters->words->sentences, frequencies->notes->chords->riffs->melodies, etc.)

tensor 13y ago
> most machine-learning algorithms do worse than the naive solution ("throw a Bayes net at it") in most applications This is an extremely misleading statement. There is nothing naive about probabilistic graphical models. It's a massive topic with a huge number of different model variations, training methods, and techniques to avoid overfitting or do feature selection. More many probably you probably only want a linear classifier, and a discriminative one at that. But if your data is not linearly separable, or has structure, then you'll want something else again.
- derefr 13y ago
  Hmm, I suppose I did phrase that in a way that would mislead someone who didn't already know what I meant by it. :) I didn't mean that Bayes nets are themselves naive--they're extremely good models for many use-cases. In fact, they're eventually ideal models for all use-cases, given enough training, since all ML models are ways to approximate Bayes' law, while Bayes nets just solve for Bayes' law directly. However, for being so generic, they have no embedded structure that allows them to efficiently learn/represent/compress data in any particular domain.[1] When I have a Big Data problem, and I have a BNN library close to hand, I just throw it at a random subsampling and see what happens. This is what I mean by "naive": since it's a generic solution, you can try applying it to a problem before even studying the structure of the problem domain, and you might just detect a signal soon enough that you don't need to go any further.[2] The rest of the machine-learning toolbox, then, is basically a set of models that you replace your BNN with when you have a model that hews closely to the structure of your data (in est, a set of screwdrivers for different kinds of screws.) The approximate model will get quicker training and clearer signals for that data, but not for just any data. --- [1] Another way to say it: a Bayes-net implementation is to gzip as other ML models are to JPEG, MP3, etc. Bayes-nets can learn anything, but they can only learn it as well as the data can be used to model the data. [2] People do use linear regression as the same kind of general-purpose tool, as you say--and linear classifier libraries are much easier to write, so you'll find them as "batteries included" in more math packages--but if you don't know whether the data is linearly separable, linear models aren't gonna give you even the slightest hint. A BNN is an extremely-inefficient, but completely-general probe.
  - volokoumphetico 13y ago
    I have no clue about all that academic stuff, what's a good reading material for such things?
- Eliezer 13y ago
  Yeah, "throw a Bayes net at it" doesn't make much sense given the current state of machine learning tech AFAIK. Throw a linear regression or a regularized linear regression, sure. Throw k-nearest-neighbors or naive Bayes EM clustering, maybe. "Throw a Bayes net", not so much.
volokoumphetico 13y ago
Your description of Numenta is very exciting for me as it appears to fit my niche. For example, can Numenta look at two similar pages from pdf, and be able to deduce and discover the location of title, paragraph, heading, subheading, etc? Those pdf files are different in layout but a human can immediately determine the hierarchy by categorizing entities on that temporally local feature.