5 ms·

>Maybe someday some people in data science or artificial intelligence will exploit the KTC Climb out of under the rock will you :) [We have met enough over HN

• graycat 10y ago
  That rock you mention is where I work on my startup -- with money in mind. My main reason to do applied math is to apply it, to business, the money making kind. Yes you have told me about Vapnik before, but the rest I see, e.g., deep learning, smells like tweaks on Breiman's CART. I have a lot of respect for Breiman, maybe more for his fellow student Neveu as they were at Berkeley under Loeve. I have a tough time taking Silicon Valley seriously on anything serious about machine learning -- new words for statistical model building and estimation but done with much more data. In Trevor Hastie, Robert Tibshirani, and Jerome Friedman, The Elements of Statistical Learning Data: Mining, Inference, and Prediction, Second Edition, Springer, 2008. I find mention of Vapnik, mostly for separating hyperplanes, and references Vapnik, V. (1996). The Nature of Statistical Learning Theory, Springer, New York. Vapnik, V. (1998). Statistical Learning Theory, Wiley, New York. Looking at Hastie, et al., which seems relatively elementary, I wouldn't expect to find much on the Kuhn-Tucker conditions or Glivenko Cantelli. In Kevin P. Murphy, Machine Learning: A Probabilistic Perspective, ISBN 978-0-262-01802-9, MIT Press, 2012. I saw no mention of Vapnik. In Shai Shalev-Shwartz and Shai Ben-David, Understanding Machine Learning From Theory to Algorithms, 2014. I see Vapnik, V. (1992), Principles of risk minimization for learning theory, in J. E. Moody, S. J. Hanson & R. P. Lippmann, eds, `Advances in Neural Information Processing Systems 4', Morgan Kaufmann, pp. 831{838. Vapnik, V. (1995), The Nature of Statistical Learning Theory, Springer. Vapnik, V. N. (1982), Estimation of Dependences Based on Empirical Data, Springer- Verlag. Vapnik, V. N. (1998), Statistical Learning Theory, Wiley. Vapnik, V. N. & Chervonenkis, A. Y. (1971), `On the uniform convergence of relative frequencies of events to their probabilities', Theory of Probability and its applications XVI(2), 264{280. Vapnik, V. N. & Chervonenkis, A. Y. (1974), Theory of pattern recognition, Nauka, Moscow. (In Russian). Okay, that may be a first-cut list of what to read on Vapnik. Gee, machine learning and not from Silicon Valley -- the second part is a step up! Not from Pittsburgh either -- two steps up! Probabilistic and from Russia? More steps up! For more steps up, maybe some French or Japanese could contribute? For such things, I could take Bertsekas at MIT seriously. I'm deep into my startup; there I've done my applied math derivations and written the code. It is true that I have some suspicions that some old techniques tweaked a little and maybe some newer techniques might increase accuracy some, but I'm leaving that on the back burner for now, avoiding premature optimization. But I'll index this post and not forget about Vapnik.
  • srean 10y ago
    Indeed, I can imagine (and still grossly underestimate) how frantically busy you would be. All the best for your startup. As I said, it was just a friendly ribbing. Given our interactions on HN I have come to gauge your taste a bit, hence my recommendations. Vapnik, V. N. (1998), Statistical Learning Theory is a tome. You probably would not have time to read this. I would say go with the other two books of his. They sort of summarize his body of early work. Again a lot of water has flown below the Vapnik bridge, but you will get a non-hyped view of what ML is about. Kevin P. Murphy, Machine Learning: A Probabilistic Perspective is also a hefty read, but its a good source and quite complementary to Vapnik's treatise. Kevin Murphy's book is about how to tractably model joint distribution of a (mostly discrete) set of random variables and draw inferences from them. So this is about smart and efficient ways to marginalize and condition and compress. Shai Shalev-Shwartz and Shai Ben-David, Understanding Machine Learning From Theory to Algorithms, 2014 This is also a great book. You will see a lot of KTC here but mostly within the framework of convexity and duality.