6 ms·

For a comparison with others techniques: Linear SVC (best performance): 92 % SVC rbf (best performance): 96.4 % SVC poly (best performance): 94.5 % Logistic

• grandma_tea 3y ago
  While it's cool that this works at all, I wish we would stop using MNIST as a benchmark given how trivial it is.
  • unoti 3y ago
    MNIST is pretty good as a proof of concept. While I agree it's trivial, I see MNIST as being more like how when you're making a toy programming language, the first thing you do with it is writing a recursive Fibonacci function.
  • dmazzoni 3y ago
    It's still a great data set because it's real-world, useful, high-quality, and an excellent size (~60k examples), and the "correct" classification isn't very subjective at all - humans can easily get 99% accuracy.
  • antiquark 3y ago
    Take a look at the EMNIST - Extended MNIST dataset. It has both digits and letters of the alphabet.
  • IKantRead 3y ago
    It's a good benchmark because it's so trivial. Sure it's not great at differentiating between SotA techniques, but it's very useful for sanity checks like this one. Even for SotA models, it's still useful to verify that you can get greater than 98% accuracy on MNIST, before exploring larger, more complex bench marks. It certainly shouldn't be the only benchmark but it's a great place to start iterating on ideas.
    • TimPC 3y ago
      It's a bad benchmark because it's artificially clean. It's effectively a 2d dataset with no occlusions. So nearly everything you try on it will work, and many things you try on it won't scale to typical image problems. There are good 3d datasets with more realistic examples that are still fairly simplistic compared to the state of the art large datasets, but at least give you signal that your technique is robust to common problems in vision. MNIST is so simplistic that you encounter none of the typical problems in computer vision settings so it doesn't give you a good prediction of how good your technique is.
      • ghshephard 3y ago
        Can you imagine a model that performs very poorly on MNIST that would perform well in real-world computer visions problems? If you can't, then MNIST is a nice simple smoke test when assessing models.
      • d3w4s9 3y ago
        What's wrong with "artificially clean"? The goal of benchmarks is to compare and know whether one model is better than the other. There is never a "perfect" or "objective" benchmark. Different benchmarks may highlight advantages in certain models, which is a good thing, but there is absolutely nothing wrong with using MNIST as a dataset to give you a basic idea of how models perform.
        • TimPC 3y ago
          Artificially clean gives you too many false positives. Most researchers I know these days start on CIFAR10 which is much more real world and has far fewer false positive signals. A portion of the hype in CV is a paper showing a technique works on MNIST that then fails to go anywhere on any other dataset.
      • tysam_and 3y ago
        That's what makes it a good benchmark. It's a benchmark. Not a real world problem. That's why the traditional path has been MNIST -> CIFAR10 (optionally -> CIFAR100) -> ImageNet -> ????!?. Because it gets gradually more complicated. Your iteration time is the constraint to development progress. Keep that down, and the bugs from your initial implementation will be significantly less impactful.
• dmazzoni 3y ago
  That blog page is not showing state-of-the-art results, it's just taking relatively naive SVM implementations and comparing them. The original research paper that introduced the MNIST data set was getting around 98% accuracy, and today neural nets are getting 99.87% accuracy. https://paperswithcode.com/sota/image-classification-on-mnist?metric=Accuracy https://paperswithcode.com/sota/image-classification-on-mnis...
  • GaggiX 3y ago
    Yep I was showing others simple techniques, of couse a neural network would be much better at it.
• IKantRead 3y ago
  Thanks for posting this. Most people don't realize that Logistic regression can get ~90% accuracy on MNIST. As a big fan of starting with simple models first and adding complexity later, I've frequently been told that "logistic regression won't work!" for problems where it can in fact perform excellent. When faced with this resistance to logistic regression I'll often ask what they think the baseline performance of it would be on MNIST. The guesses I hear most often are 20-30%. People, even machine learning people, often don't realize the rapidly diminishing returns you get for adding a lot of complexity in your models. Sometimes it's worth it, but it's always good to start simple first. It's also been my experience that if you don't get good performance with a simple model, you are very unlikely to get great performance from a more complex one.
  • nextos 3y ago
    Came here to say the same thing, actually NCD can probably do much better than 78%. Li & Vitanyi's book about Kolmogorov complexity has some interesting unsupervised examples. A simple CNN as implemented in Keras tutorial can easily exceed 98%. 78% is very poor performance for MNIST even if model complexity is penalized.
  • vjerancrnjak 3y ago
    Extending existing features with their non-linear mappings would improve logistic regression too, probably to the svc level (rbf or poly kernel is that but implicit). Linear models are really well researched, and today with the compute and with proper training and data preparation they can easily get to satisfying levels of performance for a variety of tasks.
  • short_sells_poo 3y ago
    To a large degree this happens because logistic regression is not a sexy approach that one can add to their CV. Everyone wants to solve problems with big, complicated and buzzwordy models, because that sells (and is perhaps more interesting as well). It's really a tragedy, because so many classic models would work fine for real world applications.
  • KRAKRISMOTT 3y ago
• TimPC 3y ago
  The whole point isn't to do better. It's to show that enough information survives compression that you can still get very large signal. Compression is intended to make things harder and still does.
  • amelius 3y ago
    Why? I mean it's trivial to add a layer of encryption ... (Also the terminology seems off here, as 100% of information survives gzip.)
  • GaggiX 3y ago
    The article OP linked is inspired by Gzip+Knn paper that was released a few months ago, compression is not intended to make things harder but to extract useful features that can be used by simple algorithms like Knn or SVG etc... For example it would be almost impossible to distinguish cat and dog images by using SVG or Knn directly on the data but it would be much easier if the images were first passed into an image encoder and a small embeddings vector would be used for each one instead. In the article OP linked it doesn't seem that Gzip is very good at extracting useful patterns for the MNIST dataset.
  • minitech 3y ago
    No, that wasn’t the point at all. Compressibility was used to determine similarity.
    • vintermann 3y ago
      And what it accidentally showed, was that NCD between individual digits in the training set is a really terrible distance metric for classification. You can do classification with KNN, which is obvious. You can also do classification with compression, which is less obvious, and neat. This approach tries to combine them in a way which doesn't work.
• westurner 3y ago
  So there is a more optimal compression algorithm for the relation between the MNIST inputs and outputs! Other models tend to add noise somewhere in there; what about feature engineering before the gzip? Maybe a gaussian blur and a convolution first and then deep learning for feature selection