3 ms·
The Simplex algorithm is a linear programming algorithm. ILP is Inductive Logic Progamming, not linear programming. ILP algorithms do not perform linear program
by YeGoblynQueenne 8y ago
The Simplex algorithm is a linear programming algorithm. ILP is Inductive
Logic Progamming, not linear programming. ILP algorithms do not perform
linear programming, do not have a utility function and do not use
optimisation. What they do is search a space of hypotheses for a hypothesis
that is consistent with some background theory and that explains some set of
examples. Wikipedia has an introduction:
https://en.wikipedia.org/wiki/Inductive_logic_programming https://en.wikipedia.org/wiki/Inductive_logic_programming
You will not be able to understand the operation of ILP algorithms if you try
to look at them from the point of view of function optimisation, or, indeed
"curve fitting". You need a different set of tools, that of logic programming
and automated theorem proving- because the process is essentially that of
logical deduction. Please let me know if you delve into the subject and have
any questions (see my profile for my email address). I'm always happy to
support attempts to understand my research subject.
The same goes for other machine learning algorithms: if you use the wrong
abstractions to understand them, you'll get the wrong idea about them.
For instance- KNN doesn't "require training" as per your condition for machine
learning. It just memorises all its examples. Is KNN not machine learning? It
is- because it gets better the more examples it has. If you stick to thinking
of everything as "curve fitting", either you'll misunderstand what KNN does,
or you'll start thinking that KNN is not machine learning.
Decision tree nodes are all unique - their probability is always 1/n, where n
is the number of all nodes in the tree. In other words, their distribution is
uniform. If you try to understand decision trees as "a model represented as a probability distribution" you'll
find out that every decision tree learner builds the same model. Obviously that's not true, because each decision tree
recognises different classes.
And so on.
When we talk of "curve fitting" what we mean is that we have the function of
a line (not necessarily straight) like αx + βy + γz + ... where the variables are the attributes of a
set of examples and the constants are unknown and we wish to learn them. These
are the "parameters" of the function that are optimised by curve fitting
algorithms. We say that a curve is being "fit" because we assume there is some
function that we don't know that describes a line whose points are the entire
class of entities from where our examples are drawn. And since we don't know
what this function is, we instead try to approximate it with a function that
"steps" on the few points that we have- or in other words, "fits" those
points.
In decision trees, KNN, Bayesian inference, etc, etc, there is no line
function and therefore no curve that we are trying to fit. You won't find a
line in higher dimensions either. Whereas there is probably some curve that
can fit the data points given as training examples to the algorithm, the
algorithm, lacking a line function, cannot represent that curve. And while you can sometimes
visualise the end result as a curve, this is only a convenience and, more
importantly, it is not always possible. For example- decision trees again
(because they're easy): look around for visualisations of decision trees. You
will find tree diagrams aplenty, but not a single line curve. Because you
can't really fit a line on a bunch of decision tree nodes. It's a graph.
Where's the line? Where are the parameters?
You've obviously made an effort to organise what you know about machine
learning, the better to understand it. But you're missing lots of details and
the result is actually hindering your undersanding. In a sense, by looking at
everything from the point of view of "curve fitting" you're missing the forest
for the um, trees.
My advice is: if you really want to understand machine learning (or at least
to sound a bit knowledgeable about it) keep an open mind and don't rush to
conclusions. There is much more to machine learning than what you can get from
reading a few blogs, a couple of wikipedia pages and a few popular articles in
the tech press. Even experts don't yet have a high-level theory of what
machine learning is and how to treat different machine learning algorithms in a high-level, unifrom manner, as
some well defined class of objects with common characteristics. That should give you a hint about how close your all-encompasing, "it's all curve fitting" definition is to explaining anything.
- pas 8y agoInterestingly I thought ILP is integer linear programming. (The first hit for ILP algorithm.) And then I read DeepMind's delta-ILP post you linked and was even more confused as that talked about gradient descent, and then looked at your Metagol example, and ... So, decision trees are a great example, as they were the first thing I've encountered ~15 years ago about machine learning, and I think it's a first class ML citizen. But they are in their pure form absolutely vulnerable to noise. And as soon as noise enters the picture it becomes optimization. And I think that's the model fitting that should be the primary concept. (I used curve-slash-model, but I should have used just model.) [And even if there's no "fitting", like with kNN, you still have data, and either you use a constant classification set of points, or you'll see bias, but then the question becomes what set to use.] And you're right, that treating everything as X is folly, after all, it gives you no information, decreases no entropy. But I'm looking at it from the perspective of what's intelligence. And it seems to be always on this spectrum of how to optimize for something. (Making hard things easy, by learning rules/algorithms.) After all that's why are we really interested in ML, no? > But you're missing lots of details and the result is actually hindering your undersanding. In a sense, by looking at everything from the point of view of "curve fitting" you're missing the forest for the um, trees. If you were so kind to elaborate on this, I'm very interested. And in general, anything that comes to your mind about my ramblings! Thanks for your detailed comments!