3 ms·
If something requires training, it constructs a model, the model can be represented as a probability distribution. I think of that as the curve. ILP (let's say
by pas 8y ago
If something requires training, it constructs a model, the model can be represented as a probability distribution. I think of that as the curve.
ILP (let's say simplex algorithm or other hill climbers) don't require training, but they nevertheless try to optimize the utility (or goal) function (which implicitly defines a metric on the space), which is a hypersurface, which is a curve in an even higher dimension.
Since the cardinality of the set of solutions is not one, and usually infinite, picking one solution over the other is optimization. (And - if I'm not mistaken - unless we know that the search space is convex, we don't know if there are better solutions or not.)
I don't think anything that doesn't build a model is machine learning. (Decision trees build a model, Bayes classifiers build a model, etc. And I'd say all are curve fitting.)
I'm not doing a PhD at all, so I'm not claiming that I'm an authority on each correct terminus technicus for AI/ML/optimization math/CompSci, but to me they fall under the model/curve fitting umbrella.
- deleted 8y ago[deleted]
- YeGoblynQueenne 8y agoThe Simplex algorithm is a linear programming algorithm. ILP is Inductive Logic Progamming, not linear programming. ILP algorithms do not perform linear programming, do not have a utility function and do not use optimisation. What they do is search a space of hypotheses for a hypothesis that is consistent with some background theory and that explains some set of examples. Wikipedia has an introduction: https://en.wikipedia.org/wiki/Inductive_logic_programming https://en.wikipedia.org/wiki/Inductive_logic_programming You will not be able to understand the operation of ILP algorithms if you try to look at them from the point of view of function optimisation, or, indeed "curve fitting". You need a different set of tools, that of logic programming and automated theorem proving- because the process is essentially that of logical deduction. Please let me know if you delve into the subject and have any questions (see my profile for my email address). I'm always happy to support attempts to understand my research subject. The same goes for other machine learning algorithms: if you use the wrong abstractions to understand them, you'll get the wrong idea about them. For instance- KNN doesn't "require training" as per your condition for machine learning. It just memorises all its examples. Is KNN not machine learning? It is- because it gets better the more examples it has. If you stick to thinking of everything as "curve fitting", either you'll misunderstand what KNN does, or you'll start thinking that KNN is not machine learning. Decision tree nodes are all unique - their probability is always 1/n, where n is the number of all nodes in the tree. In other words, their distribution is uniform. If you try to understand decision trees as "a model represented as a probability distribution" you'll find out that every decision tree learner builds the same model. Obviously that's not true, because each decision tree recognises different classes. And so on. When we talk of "curve fitting" what we mean is that we have the function of a line (not necessarily straight) like αx + βy + γz + ... where the variables are the attributes of a set of examples and the constants are unknown and we wish to learn them. These are the "parameters" of the function that are optimised by curve fitting algorithms. We say that a curve is being "fit" because we assume there is some function that we don't know that describes a line whose points are the entire class of entities from where our examples are drawn. And since we don't know what this function is, we instead try to approximate it with a function that "steps" on the few points that we have- or in other words, "fits" those points. In decision trees, KNN, Bayesian inference, etc, etc, there is no line function and therefore no curve that we are trying to fit. You won't find a line in higher dimensions either. Whereas there is probably some curve that can fit the data points given as training examples to the algorithm, the algorithm, lacking a line function, cannot represent that curve. And while you can sometimes visualise the end result as a curve, this is only a convenience and, more importantly, it is not always possible. For example- decision trees again (because they're easy): look around for visualisations of decision trees. You will find tree diagrams aplenty, but not a single line curve. Because you can't really fit a line on a bunch of decision tree nodes. It's a graph. Where's the line? Where are the parameters? You've obviously made an effort to organise what you know about machine learning, the better to understand it. But you're missing lots of details and the result is actually hindering your undersanding. In a sense, by looking at everything from the point of view of "curve fitting" you're missing the forest for the um, trees. My advice is: if you really want to understand machine learning (or at least to sound a bit knowledgeable about it) keep an open mind and don't rush to conclusions. There is much more to machine learning than what you can get from reading a few blogs, a couple of wikipedia pages and a few popular articles in the tech press. Even experts don't yet have a high-level theory of what machine learning is and how to treat different machine learning algorithms in a high-level, unifrom manner, as some well defined class of objects with common characteristics. That should give you a hint about how close your all-encompasing, "it's all curve fitting" definition is to explaining anything.
- pas 8y agoInterestingly I thought ILP is integer linear programming. (The first hit for ILP algorithm.) And then I read DeepMind's delta-ILP post you linked and was even more confused as that talked about gradient descent, and then looked at your Metagol example, and ... So, decision trees are a great example, as they were the first thing I've encountered ~15 years ago about machine learning, and I think it's a first class ML citizen. But they are in their pure form absolutely vulnerable to noise. And as soon as noise enters the picture it becomes optimization. And I think that's the model fitting that should be the primary concept. (I used curve-slash-model, but I should have used just model.) [And even if there's no "fitting", like with kNN, you still have data, and either you use a constant classification set of points, or you'll see bias, but then the question becomes what set to use.] And you're right, that treating everything as X is folly, after all, it gives you no information, decreases no entropy. But I'm looking at it from the perspective of what's intelligence. And it seems to be always on this spectrum of how to optimize for something. (Making hard things easy, by learning rules/algorithms.) After all that's why are we really interested in ML, no? > But you're missing lots of details and the result is actually hindering your undersanding. In a sense, by looking at everything from the point of view of "curve fitting" you're missing the forest for the um, trees. If you were so kind to elaborate on this, I'm very interested. And in general, anything that comes to your mind about my ramblings! Thanks for your detailed comments!