5 ms·

In fairness you can run MNIST through UMAP and get near perfect seperation. I'm of the belief that you have to try pretty hard not to do well on MNIST these day

In fairness you can run MNIST through UMAP and get near perfect seperation. I'm of the belief that you have to try pretty hard not to do well on MNIST these days. https://github.com/lmcinnes/umap_paper_notebooks/blob/master/UMAP%20MNIST.ipynb https://github.com/lmcinnes/umap_paper_notebooks/blob/master... EDIT: I should add, unless it isn't clear, that we really should retire the dataset. Something like the QuickDraw dataset makes a lot more sense to me.

sundarurfriend 3y ago
Y'all are making the (rude) person below complaining about acronyms look reasonable. The repository doesn't define UMAP either, but if you believe ChatGPT it is: > UMAP, which stands for Uniform Manifold Approximation and Projection, is a dimensionality reduction technique and data visualization method commonly used in machine learning and data analysis.
- tomrod 3y ago
  UMAP is as ubiquitous as T-SNE these days, though to be fair those on HN that focus on JVM, LLVM, NPM, or TLS may not know about UMAP/T-SNE.
- jszymborski 3y ago
  It's[0] a non-linear dimensionality reduction technique in the vein of t-SNE[1] that is very well known in the field. My two cents is that if your problem can be solved with something like UMAP + kNN[2], then you really shouldn't be using Deep Learning to solve it. [0] https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduction#Uniform_manifold_approximation_and_projection https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduc... [1] https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding https://en.wikipedia.org/wiki/T-distributed_stochastic_neigh... [2] https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm
  - selimthegrim 3y ago
    Doesn’t your choice of initialization really affect the results? Lior Pachter really seems to keep beating that drum
    - jszymborski 3y ago
      There's some work[0] that argues that UMAP's performance relative to t-SNE is not that great when you control for the initialization. I'm not sure where I sit on that, but regardless, my understanding is that getting UMAP to separate MNIST doesnt really require you to turn knobs. [0] https://www.biorxiv.org/content/10.1101/2019.12.19.877522v1 https://www.biorxiv.org/content/10.1101/2019.12.19.877522v1
- wenc 3y ago
  We work in our own ecological niches which sometimes entails having a specialized language that allows us to be precise when referring to things. I sometimes see HN comments complaining that acronyms and terms are dropped with no explanation. These comments come across as entitled to me — instead of complaining, why not be curious and ask “what does that mean in this context?” Imagine someone dropping in here and complaining, “the article sucks because it uses the word ‘compiler’ and assumes we all know what a ‘compiler’ is.” This is what it sounds like to those of us to work in specialized fields. Instead if someone said, “I’m new to this, could someone help me understand what a ‘compiler’ is?”, it demonstrates curiosity and people are more inclined to explain.
js98 3y ago
Author here: completely agree. I think it is not much of an achievement by itself, but it is interesting to see that it works! I will add a comment/edit to the post once I am home to clarify the relative ease of solving MNIST
- jszymborski 3y ago
  Sorry if my comment came off as dismissive of the post! I was just trying to frame the results for those unfamiliar with the properties of MNIST. I think it is indeed interesting as it shows that Gzip can capture the same things that e.g. UMAP et co. are capturing when they acheive good scores on MNIST. Also, I'll add, that even despite some of the suspicions people have cast on the Gzip results in other experiments, I'm bullish on the utility of reducing entropy for classification problems.
ekidd 3y ago
> I should add, unless it isn't clear, that we really should retire the dataset. From a research perspective, MNIST is basically solved. I think current performance on MNIST is better than humans, correct? But it's still valuable as a teaching tool or a "Hello world" data set, because it's so simple and because most reasonable algorithms will reach 97% accuracy. Even if you build your tools from scratch, it makes a nice homework-sized problem. And it's an obviously useful task that anyone can understand. "Oh, digit recognition for mail!"
- jszymborski 3y ago
  I was thinking about this just now... the real utility of MNIST is that, since it's basically impossible to get a bad result, it's a good sanity check.
krick 3y ago
Yeah, but the thing is, gzip isn't "these days". It was a thing long before UMAP and even MNIST itself. And the approach isn't perticulary novel too, this is a very simple idea, if you do understand compression. It could've been written the first day MNIST was published, and it would be still 78% accuracy. This is kinda amazing to me.

sundarurfriend 3y ago

Y'all are making the (rude) person below complaining about acronyms look reasonable. The repository doesn't define UMAP either, but if you believe ChatGPT it is: > UMAP, which stands for Uniform Manifold Approximation and Projection, is a dimensionality reduction technique and data visualization method commonly used in machine learning and data analysis.

tomrod 3y ago
UMAP is as ubiquitous as T-SNE these days, though to be fair those on HN that focus on JVM, LLVM, NPM, or TLS may not know about UMAP/T-SNE.
jszymborski 3y ago
It's[0] a non-linear dimensionality reduction technique in the vein of t-SNE[1] that is very well known in the field. My two cents is that if your problem can be solved with something like UMAP + kNN[2], then you really shouldn't be using Deep Learning to solve it. [0] https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduction#Uniform_manifold_approximation_and_projection https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduc... [1] https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding https://en.wikipedia.org/wiki/T-distributed_stochastic_neigh... [2] https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm
- selimthegrim 3y ago
  Doesn’t your choice of initialization really affect the results? Lior Pachter really seems to keep beating that drum
  - jszymborski 3y ago
    There's some work[0] that argues that UMAP's performance relative to t-SNE is not that great when you control for the initialization. I'm not sure where I sit on that, but regardless, my understanding is that getting UMAP to separate MNIST doesnt really require you to turn knobs. [0] https://www.biorxiv.org/content/10.1101/2019.12.19.877522v1 https://www.biorxiv.org/content/10.1101/2019.12.19.877522v1
wenc 3y ago
We work in our own ecological niches which sometimes entails having a specialized language that allows us to be precise when referring to things. I sometimes see HN comments complaining that acronyms and terms are dropped with no explanation. These comments come across as entitled to me — instead of complaining, why not be curious and ask “what does that mean in this context?” Imagine someone dropping in here and complaining, “the article sucks because it uses the word ‘compiler’ and assumes we all know what a ‘compiler’ is.” This is what it sounds like to those of us to work in specialized fields. Instead if someone said, “I’m new to this, could someone help me understand what a ‘compiler’ is?”, it demonstrates curiosity and people are more inclined to explain.

js98 3y ago

Author here: completely agree. I think it is not much of an achievement by itself, but it is interesting to see that it works! I will add a comment/edit to the post once I am home to clarify the relative ease of solving MNIST

jszymborski 3y ago
Sorry if my comment came off as dismissive of the post! I was just trying to frame the results for those unfamiliar with the properties of MNIST. I think it is indeed interesting as it shows that Gzip can capture the same things that e.g. UMAP et co. are capturing when they acheive good scores on MNIST. Also, I'll add, that even despite some of the suspicions people have cast on the Gzip results in other experiments, I'm bullish on the utility of reducing entropy for classification problems.

ekidd 3y ago

> I should add, unless it isn't clear, that we really should retire the dataset. From a research perspective, MNIST is basically solved. I think current performance on MNIST is better than humans, correct? But it's still valuable as a teaching tool or a "Hello world" data set, because it's so simple and because most reasonable algorithms will reach 97% accuracy. Even if you build your tools from scratch, it makes a nice homework-sized problem. And it's an obviously useful task that anyone can understand. "Oh, digit recognition for mail!"

jszymborski 3y ago
I was thinking about this just now... the real utility of MNIST is that, since it's basically impossible to get a bad result, it's a good sanity check.

krick 3y ago

Yeah, but the thing is, gzip isn't "these days". It was a thing long before UMAP and even MNIST itself. And the approach isn't perticulary novel too, this is a very simple idea, if you do understand compression. It could've been written the first day MNIST was published, and it would be still 78% accuracy. This is kinda amazing to me.