17 ms·

Machine learning is easier than it looks

adammil 13y ago
It is nice to read about this in plain language. But, can someone explain what the X and Y axis are meant to represent in the graph?
- bunderbunder 13y ago
  They don't really represent anything. Think of those graphs as being something like the rubber sheet analogy that's often used to explain general relativity - it's not a great analogy in all respects, but humans can't really visualize 4 dimensions and it captures the gist of the idea well enough. Machine learning models have a similar problem: In reality the data points in k-means clustering exist in some very high-dimensinal space that's impossible to draw. So for the sake of visual aids it's common to draw 2D plots where the only thing they're meant to suggest is that points that appear close together in the visual aid would also be fairly close to each other in the actual problem space.
- davmre 13y ago
  They don't mean anything in particular. The actual analysis is being done in a high-dimensional space, in which each post is represented by a high-dimensional vector of the form [0,0,1,0,...., 0,1,0]. The length of the vector is the total number of distinct words used across all blog posts (maybe something like 30,000), and each entry is either 0 or 1 depending on whether the corresponding word occurs in this post. All the distances and cluster centers are actually being computed in this 30000-dimensional space; the two-dimensional visualization is just for intuition. If you're wondering how the author came up with the two-dimensional representation, the article doesn't say, but it's likely he used something like Principal Component Analysis (http://en.wikipedia.org/wiki/Principal_component_analysis http://en.wikipedia.org/wiki/Principal_component_analysis). This is a standard technique for dimensionality reduction, meaning that it finds the "best" two-dimensional representation of the original 30,000-dimensional points, where "best" in this case means something like "preserves distances", so that points that were nearby in the original space are still relatively close in the low-dimensional representation.
- nilkn 13y ago
  The graphs are just illustrations. The actual graphs would be high-dimensional (one dimension or axis for each word in the total word set), and each axis would have only two ticks on it: 0 and 1. A post is mapped to the binary vector in this space whose coordinates are determined by the presence or absence of words in the post.
- benofsky 13y ago
  Sure, author here. The point of the graphs are as an example more than anything — in any real world solution it's not trivial to plot your documents (as representing documents with thousands of words in 1d, 2d or 3d is not trivial). But if _you could_ represent your documents in 2d, you could plot them like I do here. And, the X-axis would represent 1 feature of your documents, and the Y-axis another. Tl;dr don't focus on the axes — focus on the idea of being able to compare the relative distances between documents.
- hughlomas 13y ago
  The illustration has 2 dimensions, x and y. If we have x, y, z, i, j, and n then we'd have six dimensions. If each word represents a dimension, any blog title with that word gets a 1 on that word dimension, or a 0 if does not. You can then calculate the distance between these points considering all of the dimensions, although clearly it's going to be a little limited on binary inputs, since the square of the differences is always going to be 1 or 0. Like for 4 dimensions, sqrt( ( 0 - 1 )^2 + ( 0 - 0 )^2 + ( 0 - 0 )^2 + ( 1 - 0 )^2 ), which just always gets simplified to something like sqrt( 1 + 0 + 0 + 1 ), which is a little boring. The binary values cause your data points to all be stacked directly on top of each other, which leads me to believe that using binary inputs is a less than ideal application for k-means. Just look at it in the 2d case, where you have either [0,1], [0,0], [1,0], or [1,1] for each data point. Not very hard to determine the clustering there... basically just doing an overly complex boolean expression.
Rickasaurus 13y ago
It may be easier to do than it looks, but it's also harder to do well.
gms 13y ago
The difficult aspects take centre stage when things go wrong.
sieisteinmodel 13y ago
Also: aerodynamics is not really hard, anyone can fold paper planes! Or: programming 3D games is easy, just build new levels for an old game! Or: I don't know what I am doing here, but look, this photoshop effect looks really cool on my holiday photos! etc. Seriously: The writer would not be able to write anything about K-Means if not for people looking at it from a mathematical view point. This angle is of tremendous importance if you want to know how your algorithm behaves in corner cases. This does not suffice, if you have an actual application (e.g. a recommendation or a hand tracking or an object recognition engine). These need to work as good as you can make it because every improvement of it will result in $$$.
- nikatwork 13y ago
  Author is not proposing "1 weird machine learning trick mathematicians HATE!" They are encouraging non-maths people to give the basics a try even though it seems intimidating. And it worked on me, like some others in this thread my maths is shaky beyond diff calculus and my eyes glaze over when I see notation - but now I'd like to give this a whirl. I have no illusions that I will suddenly be an expert.
eof 13y ago
I feel I'm in a somewhat unique position to talk about easy/hardness of machine learning; I've been working for several months on a project with a machine learning aspect with a well-cited, respected scientist in the field. But I effectively "can't do" machine learning myself. I'm a primarily 'self-trained' hacker; started programming by writing 'proggies' for AOL in middle school in like 1996. My math starts getting pretty shaky around Calculus; vector calculus is beyond me. I did about half the machine learning class from coursera, andrew ng's. Machine learning is conceptually much simpler than one would guess; both gradient descent and the shallow-neural network type; and in fact it is actually pretty simple to get basic things to work. I agree with the author that the notation, etc, can be quite intimidating vs what is "really going on". however, applied machine learning is still friggin' hard; at least to me; and I consider myself a pretty decent programmer. Naive solutions are just unusable in almost any real application, and the author's use of loops and maps are great for teaching machine learning; but everything needs to be transformed to higher level vector/matrix problems in order to be genuinely useful. That isn't unattainable by any means; but the fact remains (imho) that without the strong base in vector calculus and idiosyncratic techniques for transforming these problems into more efficient means of computations; usable machine learning is far from "easy".
- jasonwatkinspdx 13y ago
  Good point. However, the author does recommend using open source ML libraries, so applying the techniques may not require some of the implementation knowledge you suggest. It's likely more relevant to understand the semantic constraints and pitfalls (eg, am I over-fitting right now?) since the work of a small number of talented implementers can be easily reused.
- pmelendez 13y ago
  I was going to say something around those lines but you save me the effort. Although in my opinion using ML and have some decent results is not that hard. Use it and being close to the state in the art (i.e improving your recognition rate from 65% to 80%) is a completely different matter and it is hard and it requires a fully understanding of the details of the algorithm in use. Still it doesn't meant that you can't do interesting things with an overall understanding of machine learning, but making a good cat detector is hard [0] [1] [0] http://research.google.com/archive/unsupervised_icml2012.html http://research.google.com/archive/unsupervised_icml2012.htm... [1] http://techtalks.tv/talks/machine-learning-and-ai-via-brain-simulations/57862/ http://techtalks.tv/talks/machine-learning-and-ai-via-brain-...
Daishiman 13y ago
It's easy until you have to start adjusting parameters, understand the results meaningfully, and tune the algorithms for actual "Bit Data". Try doing most statistical analysis with dense matrices and watch your app go out of memory in two seconds. It's great that we can stand on the shoulders of giants, but having a certain understanding of what these algorithms are doing is critical for choosing them and the parameters in question. Also, K-means is relatively easy to understand untuitively. Try doing that with Latent Dirichlet Allocation, Pachinko Allocation, etc. Even Principal Component Analysis and Linear Least Squares have some nontrivial properties that need to be understood.
samspenc 13y ago
Upvoted this for an interesting read, but I agree with the sentiments in the comments that (1) ML is in general hard (2) some parts of ML are not that hard, but are likely the minority (3) we are standing on the shoulders of giants, who did the hard work.
pesenti 13y ago
Of the two methods described - search vs. clustering - the first one - simpler and not involving ML - is better for this use case. The only reason it seems to give worst results is because it's only used with the titles and not the full body (unlike the clustering approach). So I guess machine learning is easier to mis-use than it looks...
misiti3780 13y ago
I disagree with this article, although I did find it interesting. Replace k-means with a supervised learning algorithm like an SVM, and use some more complicated features other than binary and this article would be a lot different. Also - maybe "article recommendation" is "easy" in this context, but other areas such as computer vision, sentiment analysis are not. Some other questions I might ask How do you know how well this algorithm is performing? How are you going to compare this model to other models? Which metrics will you use? What statistical tests would you use and why? What assumptions are you making here ? How do you know you can make them and why? There are a lot of things that this article fails to address. Disclaimer: I realize more complex models + features don't always lead to better performance, but you need to know how to verify that to be sure.
- iskander 13y ago
  SVM prediction is pretty straight-forward, even if you're using a kernel/dual formulation: "Here are some training points from classes A and B, each with an importance weight. Compute the weighted average of your new point's similarity to all the training exemplars. If the total comes out positive then it's from class A, otherwise it's from class B." If you tried to explain training from a quadratic programming perspective it would get messy, but if you use stochastic gradient descent then even the training algorithm (for primal linear SVM) is pretty light on math. Of course, this is only possible after a decade or two of researchers have played with an digested the ideas for you. I think until Léon Bottou's SGD SVM paper, even cursory explanations of SVMs seemed to get derailed into convex duality, quadratic programming solvers, and reproducing kernel hilbert spaces.
  - misiti3780 13y ago
    I'm not sure I agree:i I'll say it differently: As far as I know, k-means (as the author described it) has two parameters: initial number of clusters K and max iterations (although I could be wrong), SVMS have: UPDATE: here is a gist: https://gist.github.com/josephmisiti/7572696 https://gist.github.com/josephmisiti/7572696 taken from here : http://www.csie.ntu.edu.tw/~cjlin/libsvm/ http://www.csie.ntu.edu.tw/~cjlin/libsvm/ I do not think that picking up LibSVM, compiling it, and running it is straight forward for everyone ...
mau 13y ago
tldr: the ML algorithms look hard reading the papers, while the code looks simpler and shorter, also you can get pretty decent results in a few lines of R/Python/Ruby so ML is not that complex. I disagree in so many ways: 1. complex algorithms are usually very short in practice (e.g. dijkstra's shortest path or edit distance are the firsts that come to mind) 2. ML is not just applying ML algorithms: you have to evaluate your results, experiment with features, visualize data, think about what you can exploit and discover patterns that can improve your models. 3. If you know the properties of the algorithms you are using then you can have some insights that might help you on improving your results drastically. It's very easy to apply the right algorithms with the wrong normalizations and still get decent results in some tests.
hooande 13y ago
Most machine learning concepts are very simple. I agree with the author that mathematical formulae can be unnecessarily confusing in many cases. A lot of the concepts can be expressed very clearly in code or plain english. For example, a matrix factorization could be explained with two arrays, a and b, that represent objects in the prediction: for each example for each weight w prediction += a[w] x b[w] err = (prediction - actual_value) for each weight w a[w] += err x small_nuumber b[w] += err x small_number It's that simple. Multiply the weights of a by the weights of b, calculate error and adjust weights, repeat. K-Nearest Neighbor/KMeans are based on an even simpler operation: dist = 0 for each weight w: dist += (a[w] - b[w])**2 Then make predictions/build clusters based on the smallest aggregate distance. There are more advanced concepts. There are some serious mathematics involved in some predictors. But the most basic elements of statistical prediction are dead simple for a trained programmer to understand. Given enough data, 80% solutions can easily be achieved with simple tools. We should be spreading the word about the simplicity of fundamental prediction algorithms, not telling people that it's hard and a lot of math background is required. Machine learning is very powerful and can improve all of our lives, but only if there is enough data available. Since information tends to be unevenly distributed we need to get the tools into the hands of as many people as possible. It would be much better to focus on the concepts that everyone can understand instead of keeping statistics secrets behind the ivy clad walls of academia.
- thearn4 13y ago
  > Most machine learning concepts are very simple. Adding to that, a lot of methods that fall under the "machine learning" label are concepts that are pretty well understood in statistical point estimation, testing of hypotheses, and numerical analysis. I'm not 100% that it really ought to be treated as a new discipline.
- ACow_Adonis 13y ago
  Spot on. I agree completely. I work at this stuff for a living, and it never ceases to amaze how the academic literature and math papers are discussing fundamentally simple ideas using styles and techniques programming evolved past in the last few decades. Comments, structure, not-greek-characters (but they make me look smart!), abstractions. When was the last time you saw a stat/math paper in machine learning with comments/structure? And guess what, its as undecipherable as code without comments/structure. On the other hand, I'm also learning that what I/others find easy, a great deal of people/programmers find hard. The number of programmers/hackers who can actually implement such techniques on new real-world problems if they don't have someone holding their hand I'm discovering is a very small minority. So maybe its harder than it looks after all, and we just think its easy because we've spent so much time with it? After all, programming isn't a walk in the park for most people, and machine learning isn't a walk in the park for most programmers.
fexl 13y ago
I like the simple explanation of K-Means, and I like the contrast with the dense set-theoretic language -- a prime example of "mathematosis" as W.V.O. Quine put it.
- fexl 13y ago
  I admit that was an unfair swipe at formalism, but I do think that formalism is far more effective in the light of a clear, intuitive exposition that conveys the concept straight into the mind. Formalism is good for battening things down tightly, after the fact.
myth_drannon 13y ago
On Kaggle "The top 21 performers all have an M.S. or higher: 9 have Ph.D.s and several have multiple degrees (including one member who has two Ph.D.s)." http://plotting-success.softwareadvice.com/who-are-the-kaggle-big-data-wizards-1013/ http://plotting-success.softwareadvice.com/who-are-the-kaggl...
- diab0lic 13y ago
  While this may be true that doesn't by any means imply that someone with "only" a B.Sc. in Computer Science can't understand and use machine learning. In fact the only thing this implies is that those who have focused the most in the field are performing the best in competitions. Fairly standard wouldn't you say? The result shouldn't discourage anyone.
- vikp 13y ago
  I've done pretty well at Kaggle with just a Bachelors in American History: http://www.kaggle.com/users/19518/vik-paruchuri http://www.kaggle.com/users/19518/vik-paruchuri, although I haven't been using it as much lately. A lot of competition winners have lacked advanced degrees. What I like most about Kaggle is that the only thing that matters is what you can show. I learned a lot, and I highly recommend it to anyone starting out in machine learning. I expanded on some of this stuff in a Quora answer if you are interested: http://www.quora.com/Kaggle/Does-everyone-have-the-ability-to-do-well-on-Kaggle-competitions-if-they-put-enough-time-and-effort-into-them http://www.quora.com/Kaggle/Does-everyone-have-the-ability-t....
  - nl 13y ago
    Man.. To other readers, follow that link at your own risk. Vikp has a blog[1] which has hours and hours of excellent writing about machine learning. [1] http://vikparuchuri.com/blog/ http://vikparuchuri.com/blog/
    - vikp 13y ago
      Thanks, nl! Now I know that at least one person reads it.
rdtsc 13y ago
A lot of concepts are easier when you know how they work. CPUs were magical for me before I took a computer architecture course. So was AI and machine learning. Once you see the "trick" so to speak you lose some of the initial awe.
pallandt 13y ago
It's actually incredibly hard, especially if you want to achieve better results than with a current 'gold standard' technique/algorithm, applied on your particular problem. While the article doesn't have this title (why would you even choose one with such a high bias?), I presume the submitter decided upon this title after being encouraged by this affirmation of the article's author: 'This data indicates that the skills necessary to be a data “wizard” can be learned in disciplines other than computer sciences and mathematics.'. This is a half-baked conclusion. I'd reason most Kaggle participants are first of all, machine learning fans, either professionals or 'amateurs' with no formal qualifications, having studied it as a hobby. I doubt people with a degree in cognitive sciences or otherwise in the 'other' categories as mentioned in the article learned enough just through their university studies to readily be able to jump into machine learning.
tptacek 13y ago
Is k-means really what people are doing in serious production machine-learning settings? In a previous job, we did k-means clustering to identify groups of similar hosts on networks; we didn't call it "machine learning", but rather just "statistical clustering". I had always assumed the anomaly models we worked with were far simpler than what machine learning systems do; they seemed unworthy even of the term "mathematical models".
- kephra 13y ago
  We often sell our machine learning as mathematical optimization, because ML & AI have bad connotations to industrial customers. See: http://en.wikipedia.org/wiki/AI_winter http://en.wikipedia.org/wiki/AI_winter
  - nl 13y ago
    You might want to retest that assumption. Call it a "Big Data" tool and see how you go. The AI Winter was a long time ago now.
- jebus989 13y ago
  Ha yeah welcome to the field, linear regression is now machine learning. Formerly known as pattern recognition > artificial intelligence > statistics.
- davmre 13y ago
  K-means is simple, easy to implement and it often works quite well. It's unlikely to be state-of-the-art for any particular problem, but if you just need to do some straightforward clustering as part of a larger data-processing pipeline, K-means is a reasonable default choice, at least until you figure out that the clustering step is your bottleneck in terms of overall performance. For example, this paper http://www.stanford.edu/~acoates/papers/coatesng_nntot2012.pdf http://www.stanford.edu/~acoates/papers/coatesng_nntot2012.p... demonstrates that using K-means to learn feature representations for images (i.e., use k-means to cluster small patches of pixels from natural images, then represent images in the basis of cluster centers; repeat this several times to form a "deep" network of feature representations) gives image classification results that are competitive with much more complex / intimidating deep neural network models.
agibsonccc 13y ago
Wait till you have to hand craft your algorithms because the off the shelf ones are too slow ;). In the end you can stand on the shoulders of giants all day, but until you actually sit down and write an SVM or even something more cutting edge like stacked deep autoencoders yourself, machine learning isn't "easy". In the end, libs are there for simpler use cases or educational purposes. Realistically, that's more than good enough for 90% of people. That being said, it's not impossible to learn. Oversimplifying the statistics, tuning, and work that goes in to these algorithms you're using though? Not a good idea.
ronaldx 13y ago
I'm cynical about how machine learning of this type might be used in practice and this is an illustration of why: the stated goal is a "you might also like" section. There is no reason to believe the results are any better than a random method in respect of the goal (and it's reasonable to believe they may be worse) - we would have to measure this separately by clickthrough rate or user satisfaction survey, perhaps. I believe you would get far better results by always posting the three most popular articles. If you want to personalise, post personally-unread articles. A lot less technical work, a lot less on-the-fly calculation, a lot more effective. The machine learning tools do not fit the goal. The most effective real example of a "you might also like" section is the Mail Online's Sidebar of Shame. As best as I can tell, they display their popular articles in a fixed order. Machine Learning seems to make it easy to answer the wrong question.
- andreasvc 13y ago
  This is really only an argument that the example in the article is not realistic (which it doesn't have to be, it might be expository). There are in fact countless applications of machine learning in actual daily use, such as detecting credit card fraud, where simpler manual methods would perform measurably worse in terms of money lost.
  - ronaldx 13y ago
    Sure, there are realistic applications of Machine Learning with great results. But the article has failed in its headline goal ("ML is easier than it looks") if it chooses an example that is mathematically more complicated and still less effective than a naive alternative. The first difficult task is to identify an effective method.
kephra 13y ago
The question "do I need hard math for ML" often comes up in #machinelearning at irc.freenode.net My point here is: You don't need hard math (most of the times) because most machine learning methods are already coded in half a dozen different languages. So its similar to fft. You do not need to understand why fft works, just when and how to apply it. The typical machine learning workflow is: Data mining -> feature extraction -> applying a ML method. I often joke that I'm using Weka as a hammer, to check, if I managed to shape the problem into a nail. Now the critical part is feature extraction. Once this is done right, most methods show more or less good results. Just pick the one that fits best in results, time and memory constrains. You might need to recode the method from Java to C to speedup, or to embed it. But this requires nearly no math skills, just code reading, writing and testing skills.
Ihmahr 13y ago
As a graduate in artificial intelligence and machine learning I can tell you that machine learning IS hard. Sure, the basic concepts are easy to understand. Sure, you can hack together a program that performs quite well on some tasks. But there are so much (interesting) problems that are not at all easy to solve or understand. Like structural engineering it is easy to understand the concepts, and it is even easy to build a pillow fort in the living room, but it is not easy to build an actual bridge that is light, strong, etc.
- deleted 13y ago
  [deleted]
danialtz 13y ago
I recently read a book called "Data Smart" [1], where the author does k-means and prediction algorithms literally in Excel. This was quite eye opening as the view to ML is not so enigmatic to enter. However, the translation of your data into a format/model to run ML is another challenge. [1] http://www.amazon.com/Data-Smart-Science-Transform-Information/dp/111866146X http://www.amazon.com/Data-Smart-Science-Transform-Informati...
hokkos 13y ago
Matrix multiplication, orthonormal basis, triangular matrix, gradient descent, integrals, Lebesgue mesure, convex, and the mathematical notation in the paper are not harder than the code shown here. It is better to have solid prof of what you are doing is sound and will converge before jumping into the code.
dweinus 13y ago
They should try using tf-idf to create the initial representation of the keywords per post...also, I find there are many cases where applying machine learning/statistics correctly is harder than it looks, this single case not withstanding.
panarky 13y ago
Sure, some ML concepts are intuitive and accessible without advanced math. But it would help to highlight some of the fundamental challenges of a simplistic approach. For example, how is the author computing the distance between points in n-dimensional space? And does this mean that a one-paragraph post and a ten-paragraph post on the same topic probably wouldn't be clustered together?
- dylandrop 13y ago
  Well I don't think any of the problems you mentioned are difficult to solve, even by a non-seasoned programmer. For your first question, just use Euclidean distance. (In other words, sqrt((x1 - y1)^2 + (x2 - y2)^2 + ... + (xn - yn)^2). IMO, anyone who can program can do that. Also for your second question - probably - but he's using a very simplistic model. To fix that, you simply fix the heuristic. For example, you could instead score groups by how frequently certain important words occur, like "team" and "support".
  - m_ke 13y ago
    You have binary features, why would you use euclidean distance?
    - dylandrop 13y ago
      Well in this case, you can also do the normal, in other words adding the differences. The poster asked about finding the differences between points in N-dimensional space in general (he didn't specify for the author's example). But the idea is that your means wouldn't be bucketed into just a couple of values, but have a larger distribution. In any case, it doesn't matter much for this problem, but if you scored it by word count rather than a simple 1 or 0, it would matter.
  - panarky 13y ago
    It's not that it's hard to code, it's that a aimplistic approach will lead to questions that might not be intuitive. If you use Euclidean distance, then you essentially count 1 for each word difference. That means long posts will have longer distances from shorter posts with the same topic. If you fix that by treating "important" words differently, then you have to figure out importance, and then normalize by number of terms, etc. Before you know it you're building a naive Bayes classifier. Again, the challenge isn't in the coding, it's in understanding the limitations of the more simplistic design.
Irishsteve 13y ago
The post does do a good job of showing how easy it is to implement knn. The post doesn't really go into centroid selection or evaluation, or the fact that clustering on text is going to be painful once you move to a larger dataset.
m_ke 13y ago
This is as valid as someone stating that computer science is easy because they know HTML.
munificent 13y ago
This was a great post because I've heard of "k-means" but assumed it required more math than my idle curiosity would be willing to handle. I love algorithms, though, and now I feel like I have a handle on this. That's awesome! However, the higher level point of the post "ML is easy!" seems more than a little disingenuous. Knowing next to nothing about machine learning, obvious questions still come to mind: Since you start with random points, are you guaranteed to reach a global maximum? Can it get stuck? How do you know how many clusters you want? How do I pick K? This assumes that distance in the vector space strongly correlates to "similarity" in the thing I'm trying to understand. How do I know my vector model actually does that? (For example, how does the author know "has some word" is a useful metric for measuring post similarity?) I like what I got out of the post a lot, but the "this is easy" part only seems easy because it swept the hard part under the rug.
- andreasvc 13y ago
  You are asking exactly the right questions. As far as I know k-means will work well when you can answer those questions for your problem, otherwise not so much. In other words there's no silver bullet. If you rephrase the "ML is easy" idea from the article to "it's easy to do some simple but cool things with ML" then it's true, but in pushing the envelope you can make it as complex as you like.
- j2kun 13y ago
  This is why the author of this post is wrong. The answer is that there is no known efficient algorithm to ensure you get to a maximum of the k-Means objective (and there is unlikely to be one), even if you fix the dimension of the vectors to two. Also, nobody knows how to pick K.
- IanCal 13y ago
  > This assumes that distance in the vector space strongly correlates to "similarity" in the thing I'm trying to understand. How do I know my vector model actually does that? (For example, how does the author know "has some word" is a useful metric for measuring post similarity?) There isn't a great answer to this I'm afraid. It's a combination of "Well it seems to make sense" and "we tried it and it worked". There can be a lot of trial and error. > (For example, how does the author know "has some word" is a useful metric for measuring post similarity?) Specifically about this one, that's mostly because it makes some sense and is used a lot elsewhere. It's simple and tends to work reasonably well, doesn't require trying to parse and understand sentences, etc. The idea is that the words you say are likely to be closely linked to the thing you're talking about. The next step up from this (usually called BoW, or Bag of Words) is fairly straightforward, it's how often each word appears as a fraction of the total document. Up from there is one of the most commonly used approaches called TF-IDF (term frequency inverse document frequency) which is almost the same but it modifies the counts based on how rare the words are. I go straight for TF-IDF as it's fairly sensible, simple and tends to work well in practice. > I like what I got out of the post a lot, but the "this is easy" part only seems easy because it swept the hard part under the rug. It's the difference between making something that works and making something that works well. A lot of domain knowledge and technical knowledge might be required for the second, but the first might be a lot more attainable than people think.
deleted 13y ago
[deleted]
pyduan 13y ago
As someone who works in machine learning, I have mixed feelings about this article. While encouraging people to start learning about ML by demystifying it is a great thing, this article comes off as slightly cocky and dangerous. Programmers who believe they understand ML while only having a simplistic view of it risk not only to create less-than-optimal algorithms, and might instead create downright dangerous models: http://static.squarespace.com/static/5150aec6e4b0e340ec52710a/t/51525c33e4b0b3e0d10f77ab/1364352052403/Data_Science_VD.png http://static.squarespace.com/static/5150aec6e4b0e340ec52710... In the context of fraud detection (one of the main areas I work in these days), a model that is right for the wrong reasons might lead to catastrophic losses when the underlying assumption that made the results valid suddenly ceases to be true. Aside from the fact the techniques he mentioned are some of the simplest in machine learning (and are hardly those that would immediately come to mind when I think "machine learning"), the top comment on the article is spot on: > "The academic papers are introducing new algorithms and proving properties about them, you’re applying the result. You’re standing on giants’ shoulders and thinking it’s easy to see as far as they do." While understanding how the algorithm works is of course important (and I do agree that they are often more readable when translated to code), understanding why (and when) they work is equally important. Does each K-Means iteration always reach a stable configuration? When can you expect it to converge fast? How do you choose the number of clusters, and how does this affect convergence speed? Does the way you initialize your centroids have a significant effect on the outcome? If yes, which initializations tend to work better in which situations? These are all questions I might ask in an interview, but more importantly, being able to answer these is often the difference between blindly applying a technique and applying it intelligently. Even for "simple" algorithms such as K-Means, implementing them is often only the tip of the iceberg.
- benofsky 13y ago
  Hey, author here. > Aside from the fact the techniques he mentioned are some of the simplest in machine learning (and are hardly those that would immediately come to mind when I think "machine learning") The primary point of this article was the contrast between something as simple as K-Means, and the literature that describes it. It wasn't meant as a full intro to ML, but rather something along the lines of "give it a try, you might be surprised by what you can achieve". > Even for "simple" algorithms such as K-Means, implementing them is often only the tip of the iceberg. Yup. But getting more people to explore the tip of the iceberg is, in my opinion, a good thing. We don't discourage people from programming because they don't instantly understand the runtime complexity of hash tables and binary trees. We encourage them to use what's already built knowing that smart people will eventually explore the rest of the iceberg.
amit_m 13y ago
tl;dr: (1) Author does not understand the role of research papers (2) Claims mathematical notation is more complicated than code and (3) Thinks ML is easy because you can code the wrong algorithm in 40 lines of code. I will reply to each of these points: 1. Research papers are meant to be read by researchers who are interested in advancing the state of the art. They are usually pretty bad introductory texts. In particular, mathematical details regarding whether or not the space is closed, complete, convex, etc. are usually both irrelevant and incomprehensible to a practitioner but are essential to the inner workings of the mathematical proofs. Practitioners who want to apply the classic algorithms should seek a good book, a wikipedia article, blog post or survey paper. Just about anything OTHER than a research paper would be more helpful. 2. Mathematical notation is difficult if you cannot read it, just like any programming language. Try learning to parse it! It's not that hard, really. In cases where there is an equivalent piece of code implementing some computation, the mathematical notation is usually much shorter. 3. k-means is very simple, but its the wrong approach to this type of problem. There's an entire field called "recommender systems" with algorithms that would do a much better job here. Some of them are pretty simple too!
- Aardwolf 13y ago
  I'm pretty good at logic, problem solving, etc..., but do find parsing mathematical notation quite hard. Is there actually a good way to learn it? What I have most difficulty with is: it's not always clear which symbols/letters are knowns, which are unknowns, and which are values you choose yourself. Not all symbols/letters are always introduced, you sometimes have to guess what they are. Sometimes axes of graphs are not labeled. Sometimes explanation or examples for border cases are missing. And sometimes when in slides or so, the parsing of the mathematical formulas takes too much time compared to the speed, or, the memory of what was on previous slides fades away so the formula on a later slide using something from a previous one can no longer be parsed. Also when you need to program the [whatever is explained mathematically in a paper], then you have to tell the computer exactly how it works, for every edge case, while in math notation people can and will be inexact. Maybe there should be a compiler for math notation that gives an error if it's incomplete. :)
  - 13y ago
BjoernKW 13y ago
The fundamentals of linear algebra and statistics are indeed quite easy to understand. Common concepts and algorithms such as cosine similarity and k-means are very straightforward. Seemingly arcane mathematical notation is what frightens off beginners in many cases, though. Once you've understood that - for instance - a sum symbol actually is nothing else but a loop many things become a lot easier. However, the devil's in the details. Many edge cases and advanced methods of machine learning are really hard to understand. Moreover, when 'good enough' isn't just good enough any more things tend to become very complex very quickly.
aidos 13y ago
Most of the comments on here are from people in the field of ML saying "this is a toy example, ML is hard." Maybe that's the case. And maybe the title of the submission ruffled some feathers but the thrust of it is that ML is approachable. I'm sure there's devil in the detail, but it's nice for people who are unfamiliar in a subject to see it presented in a way that's more familiar to them with their current background. I have a university background in Maths and Comp Sci so I'm not scared of code or mathematical notation. Maybe if I'd read the comments on here I'd get the sense that ML is too vast and difficult to pick up. I'm doing Andrew Ng's coursera course at the moment and so far it's all been very easy to understand. I'm sure it gets harder (I even hope so) and maybe I'll never get to the point where I'm expert at it, but it would be nicer to see more of a nurturing environment on here instead of the knee jerk reactions this seems to have inspired.
- m_ke 13y ago
  Ng's coursera class is really dumbed down and although he gives a nice intuitive explanation of the basics, it's nowhere near as hard as his actual Stanford class (or any other ml class from a good engineering school).
  - mrcactu5 13y ago
    many people still can't pass it or understand it...
j2kun 13y ago
The author clearly didn't read the page of the math paper he posted in trying to argue his point. It says, and I quote: Stated informally, the k-means procedure consists of simply starting with k groups each of which consists of a single random point, and thereafter adding each new point to the group whose mean the new point is nearest. Admittedly, it's not the prettiest English sentence over written, but it's just as plain and simply stated as the author of this article. The article itself is interested in proving asymptotic guarantees of the algorithm (which the author of the article seems to completely ignore, as if it were not part of machine learning at all). Of course you need mathematics for that. If you go down further in the paper, the author reverts to a simple English explanation of the various parameters of the algorithm and how they affect the quality of the output. So basically the author is cherry-picking his evidence and not even doing a very good job of it.
tlarkworthy 13y ago
I find newbs in ml don't appreciate cross validation. That's the one main trick. Keep some data out of the learning process to test an approaches ability on data it has not seen. With this one trick you can determine which algorithm is best, and the parameters. Advanced stuff like Bayes means you don't need it, but for your own sanity you should still always cross validate. Machine learning is about generalisation to unseen examples, cross validation is the metric to test this. Machine learning is cross validation.
- jcutrell 13y ago
  This was of utmost importance in my ERP class at GaTech - implementing a data segmenting algorithmic solution always requires that you hold out a sample set. This really is across all statistics, if I'm not mistaken - the validation set and the holdout.
- syllogism 13y ago
  First, you're always going to need to evaluate supervised models on some data you didn't train on. No method will ever relieve the need for that. Second, I never use cross-fold if I can help it. It's a last resort if the data's really small. Otherwise, use two held-out sets, one for development, and another for testing. Cross-fold actually really sucks! The small problem is that cross-validation has you repeating everything N times, when your experiment iteration cycle is going to be a big bottleneck. Second, it can introduce a lot of subtle bugs. Let's say you need to read over your data to build some sort of dictionary of possible labels for the instances, given one of the attributes. This can save you a lot of time. But if you're cross-validating, you can forget to do these pre-processes cross-validated too, and so perform a subtle type of cheating. Coding that stuff correctly often has you writing code into the run-time about the cross-fold validation. Nasty. With held-out data, you can ensure your run-time never has access to the labels. It writes out the predictions, and a separate evaluation script is run. This way, you know your system is "clean". Finally, when you're done developing, you can do a last run on a test set. Bonus example of cross-fold problems: Sometimes your method assumes independence between examples, even of this isn't strictly true. For instance, if you're doing part-of-speech tagging, you might assume that sentences in your documents are independent of each other (i.e. you don't have any cross-sentence features). But they're not _actually_ independent! If you cross-fold, and get to evaluate on sentences from the same documents you've trained on, you might do much better than in a realistic evaluation, where you have to evaluate on fresh documents.
apu 13y ago
For those wanting to get started (or further) in machine learning, I highly recommend the article, "A Few Useful Things to Know About Machine Learning," by Pedro Domingos (a well respected ML researcher): http://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf http://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf. It's written in a very accessible style (almost no math); contains a wealth of practical information that everyone in the field "knows", but no one ever bothered to write down in one place, until now; and suggests the best approaches to use for a variety of common problems. As someone who uses machine learning heavily in my own research, a lot of this seemed like "common sense" to me when I read it, but on reflection I realized that this is precisely the stuff that is most valuable and hardest to find in existing papers and blog posts.
outworlder 13y ago
I don't get all negative comments. From my limited text comprehension abilities, the author did not say that the whole field is trivial and that we should sack all academics. Instead, the argument is that basic Machine Learning techniques are easy and one shouldn't be afraid of applying them.
- freshhawk 13y ago
  Likely this part at the beginning that exposed the author's anti-intellectualism/ignorance: "The majority of literature on machine learning, however, is riddled with complex notation, formulae and superfluous language. It puts walls up around fundamentally simple ideas." That's what got me, it's such a flat out stupid thing to say. Especially the "puts walls up" phrasing to imply that math beyond what the author understands is a conspiracy. Especially from someone who writes code ... which is full of "complex notation", symbols and "superfluous classes/libraries/error checking/abstractions" by the same absurd reasoning.
  - benofsky 13y ago
    Hi, author here. I love reading academic papers and I love Math. But it's an unfortunate fact that academic papers make up the majority of the literature on ML — and that their notation, and writing style exclude a large population who would otherwise find them useful. That's all I'm trying to get at here.
    - freshhawk 13y ago
      They don't exclude people by using math any more than programmers exclude people by using subclasses or anonymous functions. I too would love to see more entry level ML resources, but we are talking about academic papers here. Imagine how stupid it would seem if some outsider came in to HN and started telling programmers to dumb down their code when it's open source because their advanced techniques are too hard for a novice to understand off the bat. Instead of optimizing for efficiency, elegance and readability for their peers - the people they are collaborating with to solve actual problems - they're told to cater to novices always. The language in your post is textbook anti intellectualism isn't it? And strangely entitled. You would certainly not apply these criteria to code but since it's not your field they must cater to your lack of experience? You know better than they how to communicate ML research to other experts?
mrcactu5 13y ago
The equations look fine to me - I was a math major in college. Honestly, I get so tired of humanities people -- or programmers, bragging about how much they hate math. Except: https://gist.github.com/benmcredmond/0dec520b6ab2ce7c59d5#file-kmeans-rb https://gist.github.com/benmcredmond/0dec520b6ab2ce7c59d5#fi... I didn't know k-means clustering was that simple. I am taking notes... * pick two centers at random run 15 times: * for each post, find the closest center * take the average point of your two clusters as your new center This is cool. It is 2-means clustering and we can extend it to 5 or 13... We don't need any more math, as long as we don't ask whether this algorithm converges or how quickly
xyzzyz 13y ago
I'd like to chime in here as a mathematician. Many people here express their feelings that math or computer science papers are very difficult to read. Some even suggest that they're deliberately written this way. The truth is that yes, they in fact are deliberately written this way, but the reason is actually opposite of many HNers impression: authors want to make the papers easier to understand, and not more difficult. Take for example a page from a paper that's linked in this article. Someone here on HN complains that the paper talks about "p being absolutely continuous with respect to the Lebesque measure on En", hundreds of subscripts and superscripts, and unintuitively named variables, and that it makes paper very difficult to understand, especially without doing multiple passes. For non-mathematicians, it's very easy to identify with this sentiment. After all, what does it even mean for a measure to be absolutely continuous with respect to Lebesgue measure. Some of these words, like "measure" or "continuous" make some intuitive sense, but how can "measure" be "continuous" with respect to some other measure, and what the hell is Lebesgue measure anyway? Now, if you're a mathematician, you know that Lebesgue measure in simple cases is just a natural notion of area or volume, but you also know that it's very useful to be able to measure much more complicated sets than just rectangles, polyhedrals, balls, and other similar regular shapes. You know Greeks successfully approximated areas of curved shapes (like a disk) by polygons, so you try to define such measure by inscribing or circumscribing a nice, regular shapes for which the measure is easy to define, but you see it only works for very simple and regular shapes, and is very hard to work with in practice. You learned that Henri Lebesgue constructed a measure that assigns a volume to most sensible sets you can think of (indeed, it's hard to even come up with an example of a non-Lebesgue-measurable set), you've seen the construction of that measure, and you know that it's indeed a cunning and nontrivial work. You also know that any measure on Euclidean space satisfying some natural conditions (like measure of rectangle with sides a, b is equal to product ab, and if you move a set around without changing its shape, its measure shouldn't change) must already be Lebesgue measure. You also worked a lot with Lebesgue measure, it being an arguably most important measure of them all. You have an intimate knowledge of Lebesgue measure. Thus, you see a reason to honor Lebesgue by naming measure constructed by him with his name. Because of all of this, whenever you read or hear about Lebesgue measure, you know precisely what you're dealing with. You know that a measure p is absolutely continuous with respect to q, if whenever q(S) is zero for some set S, p(S) is also zero. You also know that if you tried to express the concept defined in a previous sentence, but without using names for measures involved, and a notation for a value a measure assigns to some set, the sentence would come out awkward and complicated, because you would have to say that a measure is absolutely continuous with respect to some other measure, if whenever that other measure assigns a zero value to some set, the value assigned to that set by the first measure must be zero as well. You also know, that since you're not a native English speaker (and I am not), your chance of making grammatical error in a sentence riddled with prepositions and conjunctions are very high, and it would make this sentence even more awkward. Your programmer friend suggested that you should use more intuitive and expressive names for your objects, but p and q are just any measures, and apart from the property you're just now trying to define, they don't have any additional interesting properties that would help you find names more sensible than SomeMeasure and SomeOtherMeasure. But you not only know the definition of absolute continuity of measures: in fact, if that was the only thing you knew about it was the definition, you'd have forgotten it long ago. You know that absolute continuity is important because of a Radon-Nikodym theorem, which states that if p is absolutely continuous with respect to q, then p(A) is in fact integral over A of some function g with respect to measure q (that is, p(A) = int_A g dq). You know that it's important, because it can help you reduce many questions about measure p to the questions about behaviour of function g with respect to measure q (which in our machine learning case is a measure we know very, very well, the Lebesgue measure). You also know why the hell it's called absolutely continuous: if you think about it for a while, the function g we just mentioned is kind of like a derivative of a measure of measure p with respect to measure q, kind of like dp/dq. Now, if you write p(A) = int_A (dp/dq) dq = int_A p'(q) dq, even though none of the symbols dp/dq or p'(q) make sense, it seems to mean that p is an "integral of its derivative", and you recall that there's a class of real valued functions for which it is true as well, guess what, the class of absolutely continuous functions. If you think about these concepts even harder, you'll see that the latter concept is a special case of our absolutely continuous measures, so all of this makes perfectly sense. So anyway, you read that "p is absolutely continuous with respect to Lebesgue measure", and instantly tons of associations light up in your memory, you know what they are working with, you have some ideas why they might need it, because you remember doing similar assumption in some similar context to obtain some result (and as you're reading the paper further, you realize you were right). All of what you're reading makes perfect sense, because you are very familiar with the concepts author introduces, with methods of working with them, and with known results about them. Every sentence you read is a clear consequence of the previous one. You feel you're home. ... Now, in alternate reality, a nonmathematician-you also tries to read the same paper. As the alternate-you haven't spent months and years internalizing these concept to become vis second nature, ve has to look up every other word, digress into Wikipedia to use DFS to find a connected component containing a concept you just don't yet understand. You spend hours, and after them you feel you learned nothing. You wonder if the mathematicians deliberately try to make everything complicated. Then you read a blog post which expresses the idea behind this paper very clearly. Wow, you think, these assholes mathematicians are really trying to keep their knowledge in an ivory tower of obscurity. But, since you only made it through the few paragraphs of the paper, you missed an intuitive explanation that's right there on that page from an paper reproduced by that blog post: Stated informally, the k-means procedure consists of simply starting with k groups each of which consists of a single random point, and thereafter adding each new point to the group whose mean the new point is nearest. After a point is added to a group, the mean of that groups is adjusted in order to take account of that new point Hey, so there was an intuitive explanation in that paper after all! So, what was all that bullshit about measures and absolute continuity all about? You try to implement an algorithm from the blog post, and, as you finish, one sentence from blog post catches your attention: Repeat steps 3-4. Until documents’ assignments stop changing. You wonder, but when that actually happens? How can you be sure that they will stop at all at some point? The blog post doesn't mention that. So you grab that paper again...
upquark 13y ago
Math is essential for this field, anyone who tells you otherwise doesn't know what they are talking about. You can hack together something quick and dirty without understanding the underlying math, and you certainly can use existing libraries and tools to do some basic stuff, but you won't get very far. Machine learning is easy only if you know your linear algebra, calculus, probability and stats, etc. I think this classic paper is a good way to test if you have the right math background to dive deeper into the field: http://www.cs.princeton.edu/~blei/papers/BleiNgJordan2003.pdf http://www.cs.princeton.edu/~blei/papers/BleiNgJordan2003.pd...
deleted 13y ago
[deleted]
syllogism 13y ago
I write academic papers, and I've started writing blog posts about them, and I think this post doesn't cover one of the main reasons that academic papers are less accessible to non-specialists. When you write an academic paper, it's basically a diff on previous work. It's one of the most important considerations when the paper first comes out. The reviewers and the people up-to-the-minute with the literature need to see which bit is specifically new. But to understand your algorithm from scratch, someone needs to go back and read the previous four or five papers --- and probably follow false leads, along the way! It's another reason why academic code is often pretty bad. You really really should write your system to first replicate the previous result, and then write your changes in on top of it, with the a _bare minimum_ branching logic, controlled by a flag, so that the same runtime can provide both results. And you should be able to look at each point where you branch on that flag, and check that your improvements are only exactly what you say they are. When you start from scratch and implement a good bang-for-buck idea, yes, you can get a very simple implementation with very good results. I wrote a blog post explaining a 200-line POS tagger that's about as good as any around.[1] Non-experts would usually not predict that the code could be so simple, from the original paper, Collins (2002).[2] I've got a follow-up blog post coming that describes a pretty good parser that comes in at under 500 lines, and performs about as accurately as the Stanford parser. The paper I wrote this year, which adds 0.2% to its accuracy, barely covers the main algorithm --- that's all background. Neither does the paper before me, released late last year, which adds about 2%. Nor the paper before that, which describes the features...etc. When you put it together and chop out the false-starts, okay, it's simple. But it took a lot of people a lot of years to come up with those 500 lines of Python...And they're almost certainly on the way towards a local maximum! The way forward will probably involve one of the many other methods discussed along the way, which don't help this particular system. [1] http://honnibal.wordpress.com/2013/09/11/a-good-part-of-speechpos-tagger-in-about-200-lines-of-python/ http://honnibal.wordpress.com/2013/09/11/a-good-part-of-spee... [2] http://acl.ldc.upenn.edu/W/W02/W02-1001.pdf‎ http://acl.ldc.upenn.edu/W/W02/W02-1001.pdf‎
Toenex 13y ago
I think this is one of the reasons why it should become standard practise to provide code implementations of described algorithms. It not only provides an executable demonstration of the algorithm but as importantly an alternative description that may be more accessible to other audiences. It can also be used as conformation that was is intended is indeed what is being done.
kamilafsar 13y ago
Some while back I implemented k-means in JavaScript. It's a really simple, straight forward algorithm which makes sense to me, as a visual thinker and a non-mathematician. Check out the implementation here: https://github.com/kamilafsar/k-means-visualizer/blob/master/k-means.js https://github.com/kamilafsar/k-means-visualizer/blob/master...
sytelus 13y ago
Building a model on some training data is easy. Debugging it is hard. Th machine learned model is essentially equivalent of code that gives probabilistic answers. But there are no breakpoints to put or watch to add. When model doesn't give the answer you expect, you need to do statistical debugging. Is the problem in training data, do you have correct sampling, are you missing a feature, is your feature selection optimal, does you featurizer have a bug somewhere? Possibilities are endless. Debugging ML model is black art. Most ML "users" would simply give up if model doesn't work on some cases or if they added a cool feature but results are not that good. In my experience less than 10% of people who claims ML expertise are actually good at doing statistical debugging and able to identify issue when model doesn't work as expected.

It is nice to read about this in plain language. But, can someone explain what the X and Y axis are meant to represent in the graph?

bunderbunder 13y ago
They don't really represent anything. Think of those graphs as being something like the rubber sheet analogy that's often used to explain general relativity - it's not a great analogy in all respects, but humans can't really visualize 4 dimensions and it captures the gist of the idea well enough. Machine learning models have a similar problem: In reality the data points in k-means clustering exist in some very high-dimensinal space that's impossible to draw. So for the sake of visual aids it's common to draw 2D plots where the only thing they're meant to suggest is that points that appear close together in the visual aid would also be fairly close to each other in the actual problem space.
davmre 13y ago
They don't mean anything in particular. The actual analysis is being done in a high-dimensional space, in which each post is represented by a high-dimensional vector of the form [0,0,1,0,...., 0,1,0]. The length of the vector is the total number of distinct words used across all blog posts (maybe something like 30,000), and each entry is either 0 or 1 depending on whether the corresponding word occurs in this post. All the distances and cluster centers are actually being computed in this 30000-dimensional space; the two-dimensional visualization is just for intuition. If you're wondering how the author came up with the two-dimensional representation, the article doesn't say, but it's likely he used something like Principal Component Analysis (http://en.wikipedia.org/wiki/Principal_component_analysis http://en.wikipedia.org/wiki/Principal_component_analysis). This is a standard technique for dimensionality reduction, meaning that it finds the "best" two-dimensional representation of the original 30,000-dimensional points, where "best" in this case means something like "preserves distances", so that points that were nearby in the original space are still relatively close in the low-dimensional representation.
nilkn 13y ago
The graphs are just illustrations. The actual graphs would be high-dimensional (one dimension or axis for each word in the total word set), and each axis would have only two ticks on it: 0 and 1. A post is mapped to the binary vector in this space whose coordinates are determined by the presence or absence of words in the post.
benofsky 13y ago
Sure, author here. The point of the graphs are as an example more than anything — in any real world solution it's not trivial to plot your documents (as representing documents with thousands of words in 1d, 2d or 3d is not trivial). But if _you could_ represent your documents in 2d, you could plot them like I do here. And, the X-axis would represent 1 feature of your documents, and the Y-axis another. Tl;dr don't focus on the axes — focus on the idea of being able to compare the relative distances between documents.
hughlomas 13y ago
The illustration has 2 dimensions, x and y. If we have x, y, z, i, j, and n then we'd have six dimensions. If each word represents a dimension, any blog title with that word gets a 1 on that word dimension, or a 0 if does not. You can then calculate the distance between these points considering all of the dimensions, although clearly it's going to be a little limited on binary inputs, since the square of the differences is always going to be 1 or 0. Like for 4 dimensions, sqrt( ( 0 - 1 )^2 + ( 0 - 0 )^2 + ( 0 - 0 )^2 + ( 1 - 0 )^2 ), which just always gets simplified to something like sqrt( 1 + 0 + 0 + 1 ), which is a little boring. The binary values cause your data points to all be stacked directly on top of each other, which leads me to believe that using binary inputs is a less than ideal application for k-means. Just look at it in the 2d case, where you have either [0,1], [0,0], [1,0], or [1,1] for each data point. Not very hard to determine the clustering there... basically just doing an overly complex boolean expression.

Rickasaurus 13y ago

It may be easier to do than it looks, but it's also harder to do well.

gms 13y ago

The difficult aspects take centre stage when things go wrong.

sieisteinmodel 13y ago

Also: aerodynamics is not really hard, anyone can fold paper planes! Or: programming 3D games is easy, just build new levels for an old game! Or: I don't know what I am doing here, but look, this photoshop effect looks really cool on my holiday photos! etc. Seriously: The writer would not be able to write anything about K-Means if not for people looking at it from a mathematical view point. This angle is of tremendous importance if you want to know how your algorithm behaves in corner cases. This does not suffice, if you have an actual application (e.g. a recommendation or a hand tracking or an object recognition engine). These need to work as good as you can make it because every improvement of it will result in $$$.

nikatwork 13y ago
Author is not proposing "1 weird machine learning trick mathematicians HATE!" They are encouraging non-maths people to give the basics a try even though it seems intimidating. And it worked on me, like some others in this thread my maths is shaky beyond diff calculus and my eyes glaze over when I see notation - but now I'd like to give this a whirl. I have no illusions that I will suddenly be an expert.

eof 13y ago

I feel I'm in a somewhat unique position to talk about easy/hardness of machine learning; I've been working for several months on a project with a machine learning aspect with a well-cited, respected scientist in the field. But I effectively "can't do" machine learning myself. I'm a primarily 'self-trained' hacker; started programming by writing 'proggies' for AOL in middle school in like 1996. My math starts getting pretty shaky around Calculus; vector calculus is beyond me. I did about half the machine learning class from coursera, andrew ng's. Machine learning is conceptually much simpler than one would guess; both gradient descent and the shallow-neural network type; and in fact it is actually pretty simple to get basic things to work. I agree with the author that the notation, etc, can be quite intimidating vs what is "really going on". however, applied machine learning is still friggin' hard; at least to me; and I consider myself a pretty decent programmer. Naive solutions are just unusable in almost any real application, and the author's use of loops and maps are great for teaching machine learning; but everything needs to be transformed to higher level vector/matrix problems in order to be genuinely useful. That isn't unattainable by any means; but the fact remains (imho) that without the strong base in vector calculus and idiosyncratic techniques for transforming these problems into more efficient means of computations; usable machine learning is far from "easy".

jasonwatkinspdx 13y ago
Good point. However, the author does recommend using open source ML libraries, so applying the techniques may not require some of the implementation knowledge you suggest. It's likely more relevant to understand the semantic constraints and pitfalls (eg, am I over-fitting right now?) since the work of a small number of talented implementers can be easily reused.
pmelendez 13y ago
I was going to say something around those lines but you save me the effort. Although in my opinion using ML and have some decent results is not that hard. Use it and being close to the state in the art (i.e improving your recognition rate from 65% to 80%) is a completely different matter and it is hard and it requires a fully understanding of the details of the algorithm in use. Still it doesn't meant that you can't do interesting things with an overall understanding of machine learning, but making a good cat detector is hard [0] [1] [0] http://research.google.com/archive/unsupervised_icml2012.html http://research.google.com/archive/unsupervised_icml2012.htm... [1] http://techtalks.tv/talks/machine-learning-and-ai-via-brain-simulations/57862/ http://techtalks.tv/talks/machine-learning-and-ai-via-brain-...

Daishiman 13y ago

It's easy until you have to start adjusting parameters, understand the results meaningfully, and tune the algorithms for actual "Bit Data". Try doing most statistical analysis with dense matrices and watch your app go out of memory in two seconds. It's great that we can stand on the shoulders of giants, but having a certain understanding of what these algorithms are doing is critical for choosing them and the parameters in question. Also, K-means is relatively easy to understand untuitively. Try doing that with Latent Dirichlet Allocation, Pachinko Allocation, etc. Even Principal Component Analysis and Linear Least Squares have some nontrivial properties that need to be understood.

samspenc 13y ago

Upvoted this for an interesting read, but I agree with the sentiments in the comments that (1) ML is in general hard (2) some parts of ML are not that hard, but are likely the minority (3) we are standing on the shoulders of giants, who did the hard work.

pesenti 13y ago

Of the two methods described - search vs. clustering - the first one - simpler and not involving ML - is better for this use case. The only reason it seems to give worst results is because it's only used with the titles and not the full body (unlike the clustering approach). So I guess machine learning is easier to mis-use than it looks...

misiti3780 13y ago

I disagree with this article, although I did find it interesting. Replace k-means with a supervised learning algorithm like an SVM, and use some more complicated features other than binary and this article would be a lot different. Also - maybe "article recommendation" is "easy" in this context, but other areas such as computer vision, sentiment analysis are not. Some other questions I might ask How do you know how well this algorithm is performing? How are you going to compare this model to other models? Which metrics will you use? What statistical tests would you use and why? What assumptions are you making here ? How do you know you can make them and why? There are a lot of things that this article fails to address. Disclaimer: I realize more complex models + features don't always lead to better performance, but you need to know how to verify that to be sure.

iskander 13y ago
SVM prediction is pretty straight-forward, even if you're using a kernel/dual formulation: "Here are some training points from classes A and B, each with an importance weight. Compute the weighted average of your new point's similarity to all the training exemplars. If the total comes out positive then it's from class A, otherwise it's from class B." If you tried to explain training from a quadratic programming perspective it would get messy, but if you use stochastic gradient descent then even the training algorithm (for primal linear SVM) is pretty light on math. Of course, this is only possible after a decade or two of researchers have played with an digested the ideas for you. I think until Léon Bottou's SGD SVM paper, even cursory explanations of SVMs seemed to get derailed into convex duality, quadratic programming solvers, and reproducing kernel hilbert spaces.
- misiti3780 13y ago
  I'm not sure I agree:i I'll say it differently: As far as I know, k-means (as the author described it) has two parameters: initial number of clusters K and max iterations (although I could be wrong), SVMS have: UPDATE: here is a gist: https://gist.github.com/josephmisiti/7572696 https://gist.github.com/josephmisiti/7572696 taken from here : http://www.csie.ntu.edu.tw/~cjlin/libsvm/ http://www.csie.ntu.edu.tw/~cjlin/libsvm/ I do not think that picking up LibSVM, compiling it, and running it is straight forward for everyone ...

mau 13y ago

tldr: the ML algorithms look hard reading the papers, while the code looks simpler and shorter, also you can get pretty decent results in a few lines of R/Python/Ruby so ML is not that complex. I disagree in so many ways: 1. complex algorithms are usually very short in practice (e.g. dijkstra's shortest path or edit distance are the firsts that come to mind) 2. ML is not just applying ML algorithms: you have to evaluate your results, experiment with features, visualize data, think about what you can exploit and discover patterns that can improve your models. 3. If you know the properties of the algorithms you are using then you can have some insights that might help you on improving your results drastically. It's very easy to apply the right algorithms with the wrong normalizations and still get decent results in some tests.

hooande 13y ago

Most machine learning concepts are very simple. I agree with the author that mathematical formulae can be unnecessarily confusing in many cases. A lot of the concepts can be expressed very clearly in code or plain english. For example, a matrix factorization could be explained with two arrays, a and b, that represent objects in the prediction: for each example for each weight w prediction += a[w] x b[w] err = (prediction - actual_value) for each weight w a[w] += err x small_nuumber b[w] += err x small_number It's that simple. Multiply the weights of a by the weights of b, calculate error and adjust weights, repeat. K-Nearest Neighbor/KMeans are based on an even simpler operation: dist = 0 for each weight w: dist += (a[w] - b[w])**2 Then make predictions/build clusters based on the smallest aggregate distance. There are more advanced concepts. There are some serious mathematics involved in some predictors. But the most basic elements of statistical prediction are dead simple for a trained programmer to understand. Given enough data, 80% solutions can easily be achieved with simple tools. We should be spreading the word about the simplicity of fundamental prediction algorithms, not telling people that it's hard and a lot of math background is required. Machine learning is very powerful and can improve all of our lives, but only if there is enough data available. Since information tends to be unevenly distributed we need to get the tools into the hands of as many people as possible. It would be much better to focus on the concepts that everyone can understand instead of keeping statistics secrets behind the ivy clad walls of academia.

thearn4 13y ago
> Most machine learning concepts are very simple. Adding to that, a lot of methods that fall under the "machine learning" label are concepts that are pretty well understood in statistical point estimation, testing of hypotheses, and numerical analysis. I'm not 100% that it really ought to be treated as a new discipline.
ACow_Adonis 13y ago
Spot on. I agree completely. I work at this stuff for a living, and it never ceases to amaze how the academic literature and math papers are discussing fundamentally simple ideas using styles and techniques programming evolved past in the last few decades. Comments, structure, not-greek-characters (but they make me look smart!), abstractions. When was the last time you saw a stat/math paper in machine learning with comments/structure? And guess what, its as undecipherable as code without comments/structure. On the other hand, I'm also learning that what I/others find easy, a great deal of people/programmers find hard. The number of programmers/hackers who can actually implement such techniques on new real-world problems if they don't have someone holding their hand I'm discovering is a very small minority. So maybe its harder than it looks after all, and we just think its easy because we've spent so much time with it? After all, programming isn't a walk in the park for most people, and machine learning isn't a walk in the park for most programmers.

fexl 13y ago

I like the simple explanation of K-Means, and I like the contrast with the dense set-theoretic language -- a prime example of "mathematosis" as W.V.O. Quine put it.

fexl 13y ago
I admit that was an unfair swipe at formalism, but I do think that formalism is far more effective in the light of a clear, intuitive exposition that conveys the concept straight into the mind. Formalism is good for battening things down tightly, after the fact.

myth_drannon 13y ago

On Kaggle "The top 21 performers all have an M.S. or higher: 9 have Ph.D.s and several have multiple degrees (including one member who has two Ph.D.s)." http://plotting-success.softwareadvice.com/who-are-the-kaggle-big-data-wizards-1013/ http://plotting-success.softwareadvice.com/who-are-the-kaggl...

diab0lic 13y ago
While this may be true that doesn't by any means imply that someone with "only" a B.Sc. in Computer Science can't understand and use machine learning. In fact the only thing this implies is that those who have focused the most in the field are performing the best in competitions. Fairly standard wouldn't you say? The result shouldn't discourage anyone.
vikp 13y ago
I've done pretty well at Kaggle with just a Bachelors in American History: http://www.kaggle.com/users/19518/vik-paruchuri http://www.kaggle.com/users/19518/vik-paruchuri, although I haven't been using it as much lately. A lot of competition winners have lacked advanced degrees. What I like most about Kaggle is that the only thing that matters is what you can show. I learned a lot, and I highly recommend it to anyone starting out in machine learning. I expanded on some of this stuff in a Quora answer if you are interested: http://www.quora.com/Kaggle/Does-everyone-have-the-ability-to-do-well-on-Kaggle-competitions-if-they-put-enough-time-and-effort-into-them http://www.quora.com/Kaggle/Does-everyone-have-the-ability-t....
- nl 13y ago
  Man.. To other readers, follow that link at your own risk. Vikp has a blog[1] which has hours and hours of excellent writing about machine learning. [1] http://vikparuchuri.com/blog/ http://vikparuchuri.com/blog/
  - vikp 13y ago
    Thanks, nl! Now I know that at least one person reads it.

rdtsc 13y ago

A lot of concepts are easier when you know how they work. CPUs were magical for me before I took a computer architecture course. So was AI and machine learning. Once you see the "trick" so to speak you lose some of the initial awe.

pallandt 13y ago

It's actually incredibly hard, especially if you want to achieve better results than with a current 'gold standard' technique/algorithm, applied on your particular problem. While the article doesn't have this title (why would you even choose one with such a high bias?), I presume the submitter decided upon this title after being encouraged by this affirmation of the article's author: 'This data indicates that the skills necessary to be a data “wizard” can be learned in disciplines other than computer sciences and mathematics.'. This is a half-baked conclusion. I'd reason most Kaggle participants are first of all, machine learning fans, either professionals or 'amateurs' with no formal qualifications, having studied it as a hobby. I doubt people with a degree in cognitive sciences or otherwise in the 'other' categories as mentioned in the article learned enough just through their university studies to readily be able to jump into machine learning.

tptacek 13y ago

Is k-means really what people are doing in serious production machine-learning settings? In a previous job, we did k-means clustering to identify groups of similar hosts on networks; we didn't call it "machine learning", but rather just "statistical clustering". I had always assumed the anomaly models we worked with were far simpler than what machine learning systems do; they seemed unworthy even of the term "mathematical models".

kephra 13y ago
We often sell our machine learning as mathematical optimization, because ML & AI have bad connotations to industrial customers. See: http://en.wikipedia.org/wiki/AI_winter http://en.wikipedia.org/wiki/AI_winter
- nl 13y ago
  You might want to retest that assumption. Call it a "Big Data" tool and see how you go. The AI Winter was a long time ago now.
jebus989 13y ago
Ha yeah welcome to the field, linear regression is now machine learning. Formerly known as pattern recognition > artificial intelligence > statistics.
davmre 13y ago
K-means is simple, easy to implement and it often works quite well. It's unlikely to be state-of-the-art for any particular problem, but if you just need to do some straightforward clustering as part of a larger data-processing pipeline, K-means is a reasonable default choice, at least until you figure out that the clustering step is your bottleneck in terms of overall performance. For example, this paper http://www.stanford.edu/~acoates/papers/coatesng_nntot2012.pdf http://www.stanford.edu/~acoates/papers/coatesng_nntot2012.p... demonstrates that using K-means to learn feature representations for images (i.e., use k-means to cluster small patches of pixels from natural images, then represent images in the basis of cluster centers; repeat this several times to form a "deep" network of feature representations) gives image classification results that are competitive with much more complex / intimidating deep neural network models.

agibsonccc 13y ago

Wait till you have to hand craft your algorithms because the off the shelf ones are too slow ;). In the end you can stand on the shoulders of giants all day, but until you actually sit down and write an SVM or even something more cutting edge like stacked deep autoencoders yourself, machine learning isn't "easy". In the end, libs are there for simpler use cases or educational purposes. Realistically, that's more than good enough for 90% of people. That being said, it's not impossible to learn. Oversimplifying the statistics, tuning, and work that goes in to these algorithms you're using though? Not a good idea.

ronaldx 13y ago

I'm cynical about how machine learning of this type might be used in practice and this is an illustration of why: the stated goal is a "you might also like" section. There is no reason to believe the results are any better than a random method in respect of the goal (and it's reasonable to believe they may be worse) - we would have to measure this separately by clickthrough rate or user satisfaction survey, perhaps. I believe you would get far better results by always posting the three most popular articles. If you want to personalise, post personally-unread articles. A lot less technical work, a lot less on-the-fly calculation, a lot more effective. The machine learning tools do not fit the goal. The most effective real example of a "you might also like" section is the Mail Online's Sidebar of Shame. As best as I can tell, they display their popular articles in a fixed order. Machine Learning seems to make it easy to answer the wrong question.

andreasvc 13y ago
This is really only an argument that the example in the article is not realistic (which it doesn't have to be, it might be expository). There are in fact countless applications of machine learning in actual daily use, such as detecting credit card fraud, where simpler manual methods would perform measurably worse in terms of money lost.
- ronaldx 13y ago
  Sure, there are realistic applications of Machine Learning with great results. But the article has failed in its headline goal ("ML is easier than it looks") if it chooses an example that is mathematically more complicated and still less effective than a naive alternative. The first difficult task is to identify an effective method.

kephra 13y ago

The question "do I need hard math for ML" often comes up in #machinelearning at irc.freenode.net My point here is: You don't need hard math (most of the times) because most machine learning methods are already coded in half a dozen different languages. So its similar to fft. You do not need to understand why fft works, just when and how to apply it. The typical machine learning workflow is: Data mining -> feature extraction -> applying a ML method. I often joke that I'm using Weka as a hammer, to check, if I managed to shape the problem into a nail. Now the critical part is feature extraction. Once this is done right, most methods show more or less good results. Just pick the one that fits best in results, time and memory constrains. You might need to recode the method from Java to C to speedup, or to embed it. But this requires nearly no math skills, just code reading, writing and testing skills.

Ihmahr 13y ago

As a graduate in artificial intelligence and machine learning I can tell you that machine learning IS hard. Sure, the basic concepts are easy to understand. Sure, you can hack together a program that performs quite well on some tasks. But there are so much (interesting) problems that are not at all easy to solve or understand. Like structural engineering it is easy to understand the concepts, and it is even easy to build a pillow fort in the living room, but it is not easy to build an actual bridge that is light, strong, etc.

deleted 13y ago
[deleted]

danialtz 13y ago

I recently read a book called "Data Smart" [1], where the author does k-means and prediction algorithms literally in Excel. This was quite eye opening as the view to ML is not so enigmatic to enter. However, the translation of your data into a format/model to run ML is another challenge. [1] http://www.amazon.com/Data-Smart-Science-Transform-Information/dp/111866146X http://www.amazon.com/Data-Smart-Science-Transform-Informati...

hokkos 13y ago

Matrix multiplication, orthonormal basis, triangular matrix, gradient descent, integrals, Lebesgue mesure, convex, and the mathematical notation in the paper are not harder than the code shown here. It is better to have solid prof of what you are doing is sound and will converge before jumping into the code.

dweinus 13y ago

They should try using tf-idf to create the initial representation of the keywords per post...also, I find there are many cases where applying machine learning/statistics correctly is harder than it looks, this single case not withstanding.

panarky 13y ago

Sure, some ML concepts are intuitive and accessible without advanced math. But it would help to highlight some of the fundamental challenges of a simplistic approach. For example, how is the author computing the distance between points in n-dimensional space? And does this mean that a one-paragraph post and a ten-paragraph post on the same topic probably wouldn't be clustered together?

dylandrop 13y ago
Well I don't think any of the problems you mentioned are difficult to solve, even by a non-seasoned programmer. For your first question, just use Euclidean distance. (In other words, sqrt((x1 - y1)^2 + (x2 - y2)^2 + ... + (xn - yn)^2). IMO, anyone who can program can do that. Also for your second question - probably - but he's using a very simplistic model. To fix that, you simply fix the heuristic. For example, you could instead score groups by how frequently certain important words occur, like "team" and "support".
- m_ke 13y ago
  You have binary features, why would you use euclidean distance?
  - dylandrop 13y ago
    Well in this case, you can also do the normal, in other words adding the differences. The poster asked about finding the differences between points in N-dimensional space in general (he didn't specify for the author's example). But the idea is that your means wouldn't be bucketed into just a couple of values, but have a larger distribution. In any case, it doesn't matter much for this problem, but if you scored it by word count rather than a simple 1 or 0, it would matter.
- panarky 13y ago
  It's not that it's hard to code, it's that a aimplistic approach will lead to questions that might not be intuitive. If you use Euclidean distance, then you essentially count 1 for each word difference. That means long posts will have longer distances from shorter posts with the same topic. If you fix that by treating "important" words differently, then you have to figure out importance, and then normalize by number of terms, etc. Before you know it you're building a naive Bayes classifier. Again, the challenge isn't in the coding, it's in understanding the limitations of the more simplistic design.

Irishsteve 13y ago

The post does do a good job of showing how easy it is to implement knn. The post doesn't really go into centroid selection or evaluation, or the fact that clustering on text is going to be painful once you move to a larger dataset.

m_ke 13y ago

This is as valid as someone stating that computer science is easy because they know HTML.

munificent 13y ago

This was a great post because I've heard of "k-means" but assumed it required more math than my idle curiosity would be willing to handle. I love algorithms, though, and now I feel like I have a handle on this. That's awesome! However, the higher level point of the post "ML is easy!" seems more than a little disingenuous. Knowing next to nothing about machine learning, obvious questions still come to mind: Since you start with random points, are you guaranteed to reach a global maximum? Can it get stuck? How do you know how many clusters you want? How do I pick K? This assumes that distance in the vector space strongly correlates to "similarity" in the thing I'm trying to understand. How do I know my vector model actually does that? (For example, how does the author know "has some word" is a useful metric for measuring post similarity?) I like what I got out of the post a lot, but the "this is easy" part only seems easy because it swept the hard part under the rug.

andreasvc 13y ago
You are asking exactly the right questions. As far as I know k-means will work well when you can answer those questions for your problem, otherwise not so much. In other words there's no silver bullet. If you rephrase the "ML is easy" idea from the article to "it's easy to do some simple but cool things with ML" then it's true, but in pushing the envelope you can make it as complex as you like.
j2kun 13y ago
This is why the author of this post is wrong. The answer is that there is no known efficient algorithm to ensure you get to a maximum of the k-Means objective (and there is unlikely to be one), even if you fix the dimension of the vectors to two. Also, nobody knows how to pick K.
IanCal 13y ago
> This assumes that distance in the vector space strongly correlates to "similarity" in the thing I'm trying to understand. How do I know my vector model actually does that? (For example, how does the author know "has some word" is a useful metric for measuring post similarity?) There isn't a great answer to this I'm afraid. It's a combination of "Well it seems to make sense" and "we tried it and it worked". There can be a lot of trial and error. > (For example, how does the author know "has some word" is a useful metric for measuring post similarity?) Specifically about this one, that's mostly because it makes some sense and is used a lot elsewhere. It's simple and tends to work reasonably well, doesn't require trying to parse and understand sentences, etc. The idea is that the words you say are likely to be closely linked to the thing you're talking about. The next step up from this (usually called BoW, or Bag of Words) is fairly straightforward, it's how often each word appears as a fraction of the total document. Up from there is one of the most commonly used approaches called TF-IDF (term frequency inverse document frequency) which is almost the same but it modifies the counts based on how rare the words are. I go straight for TF-IDF as it's fairly sensible, simple and tends to work well in practice. > I like what I got out of the post a lot, but the "this is easy" part only seems easy because it swept the hard part under the rug. It's the difference between making something that works and making something that works well. A lot of domain knowledge and technical knowledge might be required for the second, but the first might be a lot more attainable than people think.

deleted 13y ago

[deleted]

pyduan 13y ago

As someone who works in machine learning, I have mixed feelings about this article. While encouraging people to start learning about ML by demystifying it is a great thing, this article comes off as slightly cocky and dangerous. Programmers who believe they understand ML while only having a simplistic view of it risk not only to create less-than-optimal algorithms, and might instead create downright dangerous models: http://static.squarespace.com/static/5150aec6e4b0e340ec52710a/t/51525c33e4b0b3e0d10f77ab/1364352052403/Data_Science_VD.png http://static.squarespace.com/static/5150aec6e4b0e340ec52710... In the context of fraud detection (one of the main areas I work in these days), a model that is right for the wrong reasons might lead to catastrophic losses when the underlying assumption that made the results valid suddenly ceases to be true. Aside from the fact the techniques he mentioned are some of the simplest in machine learning (and are hardly those that would immediately come to mind when I think "machine learning"), the top comment on the article is spot on: > "The academic papers are introducing new algorithms and proving properties about them, you’re applying the result. You’re standing on giants’ shoulders and thinking it’s easy to see as far as they do." While understanding how the algorithm works is of course important (and I do agree that they are often more readable when translated to code), understanding why (and when) they work is equally important. Does each K-Means iteration always reach a stable configuration? When can you expect it to converge fast? How do you choose the number of clusters, and how does this affect convergence speed? Does the way you initialize your centroids have a significant effect on the outcome? If yes, which initializations tend to work better in which situations? These are all questions I might ask in an interview, but more importantly, being able to answer these is often the difference between blindly applying a technique and applying it intelligently. Even for "simple" algorithms such as K-Means, implementing them is often only the tip of the iceberg.

benofsky 13y ago
Hey, author here. > Aside from the fact the techniques he mentioned are some of the simplest in machine learning (and are hardly those that would immediately come to mind when I think "machine learning") The primary point of this article was the contrast between something as simple as K-Means, and the literature that describes it. It wasn't meant as a full intro to ML, but rather something along the lines of "give it a try, you might be surprised by what you can achieve". > Even for "simple" algorithms such as K-Means, implementing them is often only the tip of the iceberg. Yup. But getting more people to explore the tip of the iceberg is, in my opinion, a good thing. We don't discourage people from programming because they don't instantly understand the runtime complexity of hash tables and binary trees. We encourage them to use what's already built knowing that smart people will eventually explore the rest of the iceberg.

amit_m 13y ago

tl;dr: (1) Author does not understand the role of research papers (2) Claims mathematical notation is more complicated than code and (3) Thinks ML is easy because you can code the wrong algorithm in 40 lines of code. I will reply to each of these points: 1. Research papers are meant to be read by researchers who are interested in advancing the state of the art. They are usually pretty bad introductory texts. In particular, mathematical details regarding whether or not the space is closed, complete, convex, etc. are usually both irrelevant and incomprehensible to a practitioner but are essential to the inner workings of the mathematical proofs. Practitioners who want to apply the classic algorithms should seek a good book, a wikipedia article, blog post or survey paper. Just about anything OTHER than a research paper would be more helpful. 2. Mathematical notation is difficult if you cannot read it, just like any programming language. Try learning to parse it! It's not that hard, really. In cases where there is an equivalent piece of code implementing some computation, the mathematical notation is usually much shorter. 3. k-means is very simple, but its the wrong approach to this type of problem. There's an entire field called "recommender systems" with algorithms that would do a much better job here. Some of them are pretty simple too!

Aardwolf 13y ago
I'm pretty good at logic, problem solving, etc..., but do find parsing mathematical notation quite hard. Is there actually a good way to learn it? What I have most difficulty with is: it's not always clear which symbols/letters are knowns, which are unknowns, and which are values you choose yourself. Not all symbols/letters are always introduced, you sometimes have to guess what they are. Sometimes axes of graphs are not labeled. Sometimes explanation or examples for border cases are missing. And sometimes when in slides or so, the parsing of the mathematical formulas takes too much time compared to the speed, or, the memory of what was on previous slides fades away so the formula on a later slide using something from a previous one can no longer be parsed. Also when you need to program the [whatever is explained mathematically in a paper], then you have to tell the computer exactly how it works, for every edge case, while in math notation people can and will be inexact. Maybe there should be a compiler for math notation that gives an error if it's incomplete. :)
- 13y ago

BjoernKW 13y ago

The fundamentals of linear algebra and statistics are indeed quite easy to understand. Common concepts and algorithms such as cosine similarity and k-means are very straightforward. Seemingly arcane mathematical notation is what frightens off beginners in many cases, though. Once you've understood that - for instance - a sum symbol actually is nothing else but a loop many things become a lot easier. However, the devil's in the details. Many edge cases and advanced methods of machine learning are really hard to understand. Moreover, when 'good enough' isn't just good enough any more things tend to become very complex very quickly.

aidos 13y ago

Most of the comments on here are from people in the field of ML saying "this is a toy example, ML is hard." Maybe that's the case. And maybe the title of the submission ruffled some feathers but the thrust of it is that ML is approachable. I'm sure there's devil in the detail, but it's nice for people who are unfamiliar in a subject to see it presented in a way that's more familiar to them with their current background. I have a university background in Maths and Comp Sci so I'm not scared of code or mathematical notation. Maybe if I'd read the comments on here I'd get the sense that ML is too vast and difficult to pick up. I'm doing Andrew Ng's coursera course at the moment and so far it's all been very easy to understand. I'm sure it gets harder (I even hope so) and maybe I'll never get to the point where I'm expert at it, but it would be nicer to see more of a nurturing environment on here instead of the knee jerk reactions this seems to have inspired.

m_ke 13y ago
Ng's coursera class is really dumbed down and although he gives a nice intuitive explanation of the basics, it's nowhere near as hard as his actual Stanford class (or any other ml class from a good engineering school).
- mrcactu5 13y ago
  many people still can't pass it or understand it...

j2kun 13y ago

The author clearly didn't read the page of the math paper he posted in trying to argue his point. It says, and I quote: Stated informally, the k-means procedure consists of simply starting with k groups each of which consists of a single random point, and thereafter adding each new point to the group whose mean the new point is nearest. Admittedly, it's not the prettiest English sentence over written, but it's just as plain and simply stated as the author of this article. The article itself is interested in proving asymptotic guarantees of the algorithm (which the author of the article seems to completely ignore, as if it were not part of machine learning at all). Of course you need mathematics for that. If you go down further in the paper, the author reverts to a simple English explanation of the various parameters of the algorithm and how they affect the quality of the output. So basically the author is cherry-picking his evidence and not even doing a very good job of it.

tlarkworthy 13y ago

I find newbs in ml don't appreciate cross validation. That's the one main trick. Keep some data out of the learning process to test an approaches ability on data it has not seen. With this one trick you can determine which algorithm is best, and the parameters. Advanced stuff like Bayes means you don't need it, but for your own sanity you should still always cross validate. Machine learning is about generalisation to unseen examples, cross validation is the metric to test this. Machine learning is cross validation.

jcutrell 13y ago
This was of utmost importance in my ERP class at GaTech - implementing a data segmenting algorithmic solution always requires that you hold out a sample set. This really is across all statistics, if I'm not mistaken - the validation set and the holdout.
syllogism 13y ago
First, you're always going to need to evaluate supervised models on some data you didn't train on. No method will ever relieve the need for that. Second, I never use cross-fold if I can help it. It's a last resort if the data's really small. Otherwise, use two held-out sets, one for development, and another for testing. Cross-fold actually really sucks! The small problem is that cross-validation has you repeating everything N times, when your experiment iteration cycle is going to be a big bottleneck. Second, it can introduce a lot of subtle bugs. Let's say you need to read over your data to build some sort of dictionary of possible labels for the instances, given one of the attributes. This can save you a lot of time. But if you're cross-validating, you can forget to do these pre-processes cross-validated too, and so perform a subtle type of cheating. Coding that stuff correctly often has you writing code into the run-time about the cross-fold validation. Nasty. With held-out data, you can ensure your run-time never has access to the labels. It writes out the predictions, and a separate evaluation script is run. This way, you know your system is "clean". Finally, when you're done developing, you can do a last run on a test set. Bonus example of cross-fold problems: Sometimes your method assumes independence between examples, even of this isn't strictly true. For instance, if you're doing part-of-speech tagging, you might assume that sentences in your documents are independent of each other (i.e. you don't have any cross-sentence features). But they're not _actually_ independent! If you cross-fold, and get to evaluate on sentences from the same documents you've trained on, you might do much better than in a realistic evaluation, where you have to evaluate on fresh documents.

apu 13y ago

For those wanting to get started (or further) in machine learning, I highly recommend the article, "A Few Useful Things to Know About Machine Learning," by Pedro Domingos (a well respected ML researcher): http://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf http://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf. It's written in a very accessible style (almost no math); contains a wealth of practical information that everyone in the field "knows", but no one ever bothered to write down in one place, until now; and suggests the best approaches to use for a variety of common problems. As someone who uses machine learning heavily in my own research, a lot of this seemed like "common sense" to me when I read it, but on reflection I realized that this is precisely the stuff that is most valuable and hardest to find in existing papers and blog posts.

outworlder 13y ago

I don't get all negative comments. From my limited text comprehension abilities, the author did not say that the whole field is trivial and that we should sack all academics. Instead, the argument is that basic Machine Learning techniques are easy and one shouldn't be afraid of applying them.

freshhawk 13y ago
Likely this part at the beginning that exposed the author's anti-intellectualism/ignorance: "The majority of literature on machine learning, however, is riddled with complex notation, formulae and superfluous language. It puts walls up around fundamentally simple ideas." That's what got me, it's such a flat out stupid thing to say. Especially the "puts walls up" phrasing to imply that math beyond what the author understands is a conspiracy. Especially from someone who writes code ... which is full of "complex notation", symbols and "superfluous classes/libraries/error checking/abstractions" by the same absurd reasoning.
- benofsky 13y ago
  Hi, author here. I love reading academic papers and I love Math. But it's an unfortunate fact that academic papers make up the majority of the literature on ML — and that their notation, and writing style exclude a large population who would otherwise find them useful. That's all I'm trying to get at here.
  - freshhawk 13y ago
    They don't exclude people by using math any more than programmers exclude people by using subclasses or anonymous functions. I too would love to see more entry level ML resources, but we are talking about academic papers here. Imagine how stupid it would seem if some outsider came in to HN and started telling programmers to dumb down their code when it's open source because their advanced techniques are too hard for a novice to understand off the bat. Instead of optimizing for efficiency, elegance and readability for their peers - the people they are collaborating with to solve actual problems - they're told to cater to novices always. The language in your post is textbook anti intellectualism isn't it? And strangely entitled. You would certainly not apply these criteria to code but since it's not your field they must cater to your lack of experience? You know better than they how to communicate ML research to other experts?

mrcactu5 13y ago

The equations look fine to me - I was a math major in college. Honestly, I get so tired of humanities people -- or programmers, bragging about how much they hate math. Except: https://gist.github.com/benmcredmond/0dec520b6ab2ce7c59d5#file-kmeans-rb https://gist.github.com/benmcredmond/0dec520b6ab2ce7c59d5#fi... I didn't know k-means clustering was that simple. I am taking notes... * pick two centers at random run 15 times: * for each post, find the closest center * take the average point of your two clusters as your new center This is cool. It is 2-means clustering and we can extend it to 5 or 13... We don't need any more math, as long as we don't ask whether this algorithm converges or how quickly

xyzzyz 13y ago

I'd like to chime in here as a mathematician. Many people here express their feelings that math or computer science papers are very difficult to read. Some even suggest that they're deliberately written this way. The truth is that yes, they in fact are deliberately written this way, but the reason is actually opposite of many HNers impression: authors want to make the papers easier to understand, and not more difficult. Take for example a page from a paper that's linked in this article. Someone here on HN complains that the paper talks about "p being absolutely continuous with respect to the Lebesque measure on En", hundreds of subscripts and superscripts, and unintuitively named variables, and that it makes paper very difficult to understand, especially without doing multiple passes. For non-mathematicians, it's very easy to identify with this sentiment. After all, what does it even mean for a measure to be absolutely continuous with respect to Lebesgue measure. Some of these words, like "measure" or "continuous" make some intuitive sense, but how can "measure" be "continuous" with respect to some other measure, and what the hell is Lebesgue measure anyway? Now, if you're a mathematician, you know that Lebesgue measure in simple cases is just a natural notion of area or volume, but you also know that it's very useful to be able to measure much more complicated sets than just rectangles, polyhedrals, balls, and other similar regular shapes. You know Greeks successfully approximated areas of curved shapes (like a disk) by polygons, so you try to define such measure by inscribing or circumscribing a nice, regular shapes for which the measure is easy to define, but you see it only works for very simple and regular shapes, and is very hard to work with in practice. You learned that Henri Lebesgue constructed a measure that assigns a volume to most sensible sets you can think of (indeed, it's hard to even come up with an example of a non-Lebesgue-measurable set), you've seen the construction of that measure, and you know that it's indeed a cunning and nontrivial work. You also know that any measure on Euclidean space satisfying some natural conditions (like measure of rectangle with sides a, b is equal to product ab, and if you move a set around without changing its shape, its measure shouldn't change) must already be Lebesgue measure. You also worked a lot with Lebesgue measure, it being an arguably most important measure of them all. You have an intimate knowledge of Lebesgue measure. Thus, you see a reason to honor Lebesgue by naming measure constructed by him with his name. Because of all of this, whenever you read or hear about Lebesgue measure, you know precisely what you're dealing with. You know that a measure p is absolutely continuous with respect to q, if whenever q(S) is zero for some set S, p(S) is also zero. You also know that if you tried to express the concept defined in a previous sentence, but without using names for measures involved, and a notation for a value a measure assigns to some set, the sentence would come out awkward and complicated, because you would have to say that a measure is absolutely continuous with respect to some other measure, if whenever that other measure assigns a zero value to some set, the value assigned to that set by the first measure must be zero as well. You also know, that since you're not a native English speaker (and I am not), your chance of making grammatical error in a sentence riddled with prepositions and conjunctions are very high, and it would make this sentence even more awkward. Your programmer friend suggested that you should use more intuitive and expressive names for your objects, but p and q are just any measures, and apart from the property you're just now trying to define, they don't have any additional interesting properties that would help you find names more sensible than SomeMeasure and SomeOtherMeasure. But you not only know the definition of absolute continuity of measures: in fact, if that was the only thing you knew about it was the definition, you'd have forgotten it long ago. You know that absolute continuity is important because of a Radon-Nikodym theorem, which states that if p is absolutely continuous with respect to q, then p(A) is in fact integral over A of some function g with respect to measure q (that is, p(A) = int_A g dq). You know that it's important, because it can help you reduce many questions about measure p to the questions about behaviour of function g with respect to measure q (which in our machine learning case is a measure we know very, very well, the Lebesgue measure). You also know why the hell it's called absolutely continuous: if you think about it for a while, the function g we just mentioned is kind of like a derivative of a measure of measure p with respect to measure q, kind of like dp/dq. Now, if you write p(A) = int_A (dp/dq) dq = int_A p'(q) dq, even though none of the symbols dp/dq or p'(q) make sense, it seems to mean that p is an "integral of its derivative", and you recall that there's a class of real valued functions for which it is true as well, guess what, the class of absolutely continuous functions. If you think about these concepts even harder, you'll see that the latter concept is a special case of our absolutely continuous measures, so all of this makes perfectly sense. So anyway, you read that "p is absolutely continuous with respect to Lebesgue measure", and instantly tons of associations light up in your memory, you know what they are working with, you have some ideas why they might need it, because you remember doing similar assumption in some similar context to obtain some result (and as you're reading the paper further, you realize you were right). All of what you're reading makes perfect sense, because you are very familiar with the concepts author introduces, with methods of working with them, and with known results about them. Every sentence you read is a clear consequence of the previous one. You feel you're home. ... Now, in alternate reality, a nonmathematician-you also tries to read the same paper. As the alternate-you haven't spent months and years internalizing these concept to become vis second nature, ve has to look up every other word, digress into Wikipedia to use DFS to find a connected component containing a concept you just don't yet understand. You spend hours, and after them you feel you learned nothing. You wonder if the mathematicians deliberately try to make everything complicated. Then you read a blog post which expresses the idea behind this paper very clearly. Wow, you think, these assholes mathematicians are really trying to keep their knowledge in an ivory tower of obscurity. But, since you only made it through the few paragraphs of the paper, you missed an intuitive explanation that's right there on that page from an paper reproduced by that blog post: Stated informally, the k-means procedure consists of simply starting with k groups each of which consists of a single random point, and thereafter adding each new point to the group whose mean the new point is nearest. After a point is added to a group, the mean of that groups is adjusted in order to take account of that new point Hey, so there was an intuitive explanation in that paper after all! So, what was all that bullshit about measures and absolute continuity all about? You try to implement an algorithm from the blog post, and, as you finish, one sentence from blog post catches your attention: Repeat steps 3-4. Until documents’ assignments stop changing. You wonder, but when that actually happens? How can you be sure that they will stop at all at some point? The blog post doesn't mention that. So you grab that paper again...

upquark 13y ago

Math is essential for this field, anyone who tells you otherwise doesn't know what they are talking about. You can hack together something quick and dirty without understanding the underlying math, and you certainly can use existing libraries and tools to do some basic stuff, but you won't get very far. Machine learning is easy only if you know your linear algebra, calculus, probability and stats, etc. I think this classic paper is a good way to test if you have the right math background to dive deeper into the field: http://www.cs.princeton.edu/~blei/papers/BleiNgJordan2003.pdf http://www.cs.princeton.edu/~blei/papers/BleiNgJordan2003.pd...

deleted 13y ago

[deleted]

syllogism 13y ago

I write academic papers, and I've started writing blog posts about them, and I think this post doesn't cover one of the main reasons that academic papers are less accessible to non-specialists. When you write an academic paper, it's basically a diff on previous work. It's one of the most important considerations when the paper first comes out. The reviewers and the people up-to-the-minute with the literature need to see which bit is specifically new. But to understand your algorithm from scratch, someone needs to go back and read the previous four or five papers --- and probably follow false leads, along the way! It's another reason why academic code is often pretty bad. You really really should write your system to first replicate the previous result, and then write your changes in on top of it, with the a _bare minimum_ branching logic, controlled by a flag, so that the same runtime can provide both results. And you should be able to look at each point where you branch on that flag, and check that your improvements are only exactly what you say they are. When you start from scratch and implement a good bang-for-buck idea, yes, you can get a very simple implementation with very good results. I wrote a blog post explaining a 200-line POS tagger that's about as good as any around.[1] Non-experts would usually not predict that the code could be so simple, from the original paper, Collins (2002).[2] I've got a follow-up blog post coming that describes a pretty good parser that comes in at under 500 lines, and performs about as accurately as the Stanford parser. The paper I wrote this year, which adds 0.2% to its accuracy, barely covers the main algorithm --- that's all background. Neither does the paper before me, released late last year, which adds about 2%. Nor the paper before that, which describes the features...etc. When you put it together and chop out the false-starts, okay, it's simple. But it took a lot of people a lot of years to come up with those 500 lines of Python...And they're almost certainly on the way towards a local maximum! The way forward will probably involve one of the many other methods discussed along the way, which don't help this particular system. [1] http://honnibal.wordpress.com/2013/09/11/a-good-part-of-speechpos-tagger-in-about-200-lines-of-python/ http://honnibal.wordpress.com/2013/09/11/a-good-part-of-spee... [2] http://acl.ldc.upenn.edu/W/W02/W02-1001.pdf‎ http://acl.ldc.upenn.edu/W/W02/W02-1001.pdf‎

Toenex 13y ago

I think this is one of the reasons why it should become standard practise to provide code implementations of described algorithms. It not only provides an executable demonstration of the algorithm but as importantly an alternative description that may be more accessible to other audiences. It can also be used as conformation that was is intended is indeed what is being done.

kamilafsar 13y ago

Some while back I implemented k-means in JavaScript. It's a really simple, straight forward algorithm which makes sense to me, as a visual thinker and a non-mathematician. Check out the implementation here: https://github.com/kamilafsar/k-means-visualizer/blob/master/k-means.js https://github.com/kamilafsar/k-means-visualizer/blob/master...

sytelus 13y ago

Building a model on some training data is easy. Debugging it is hard. Th machine learned model is essentially equivalent of code that gives probabilistic answers. But there are no breakpoints to put or watch to add. When model doesn't give the answer you expect, you need to do statistical debugging. Is the problem in training data, do you have correct sampling, are you missing a feature, is your feature selection optimal, does you featurizer have a bug somewhere? Possibilities are endless. Debugging ML model is black art. Most ML "users" would simply give up if model doesn't work on some cases or if they added a cool feature but results are not that good. In my experience less than 10% of people who claims ML expertise are actually good at doing statistical debugging and able to identify issue when model doesn't work as expected.