4 ms·

I didn't get beyond his argument that ML is compression while NLU is decompression but that doesn't seem to be right. ML is often used to decompress data such a

I didn't get beyond his argument that ML is compression while NLU is decompression but that doesn't seem to be right. ML is often used to decompress data such as increasing image resolution. Of course it needs more data than just the compressed form, but only for training. For inference, of course you can use ML to add assumed common knowledge information.

karpierz 5y ago
> For inference, of course you can use ML to add assumed common knowledge information. I think this is easier said than done, and hasn't yet been accomplished in any general sense.
ZephyrBlu 5y ago
How is ML not compression? You take a training set, and you end up with a model that is smaller than the size of the training set yet performs well on it (Making the assumption the model is good here). That is compression. It has nothing to do with what the model is being used for.
- slunk 5y ago
  > It has nothing to do with what the model is being used for. I may be misunderstanding this passage of the article, but I thought the author was claiming that machine learning (specifically training) was equivalent to compression, while language understanding is equivalent to decompression. Therefore, they can't be the same thing. Why does language understand have to be analogous to training an ML model rather than using an ML model for inference?
  - ZephyrBlu 5y ago
    > Why does language understand have to be analogous to training an ML model rather than using an ML model for inference? Why would you look at ML model inferences in particular? There is no compression or decompression going on during inferences, you're just running data through the existing weights. Creating an ML model on the other hand is lossy compression. You reduce the size of the data (Training set -> model) in exchange for reduced accuracy (100% -> 90-95% or whatever). NLU is decompression because you are extracting information that doesn't exist in the text. I see ML as ahead-of-time compression (Creating a model), whereas NLU is just-in-time decompression (Extracting information from current context). Looking specifically at inference-time doesn't make sense to me because all the work for ML is done during training, not inference.
slunk 5y ago
I was hoping there would be discussion of this point higher up in the thread, because I had essentially the same reaction as you while reading this passage. I'm no expert of machine learning, NLP, or linguistics, but this struck me as a pretty obvious flaw in the author's argument.

karpierz 5y ago

> For inference, of course you can use ML to add assumed common knowledge information. I think this is easier said than done, and hasn't yet been accomplished in any general sense.

ZephyrBlu 5y ago

How is ML not compression? You take a training set, and you end up with a model that is smaller than the size of the training set yet performs well on it (Making the assumption the model is good here). That is compression. It has nothing to do with what the model is being used for.

slunk 5y ago
> It has nothing to do with what the model is being used for. I may be misunderstanding this passage of the article, but I thought the author was claiming that machine learning (specifically training) was equivalent to compression, while language understanding is equivalent to decompression. Therefore, they can't be the same thing. Why does language understand have to be analogous to training an ML model rather than using an ML model for inference?
- ZephyrBlu 5y ago
  > Why does language understand have to be analogous to training an ML model rather than using an ML model for inference? Why would you look at ML model inferences in particular? There is no compression or decompression going on during inferences, you're just running data through the existing weights. Creating an ML model on the other hand is lossy compression. You reduce the size of the data (Training set -> model) in exchange for reduced accuracy (100% -> 90-95% or whatever). NLU is decompression because you are extracting information that doesn't exist in the text. I see ML as ahead-of-time compression (Creating a model), whereas NLU is just-in-time decompression (Extracting information from current context). Looking specifically at inference-time doesn't make sense to me because all the work for ML is done during training, not inference.

slunk 5y ago

I was hoping there would be discussion of this point higher up in the thread, because I had essentially the same reaction as you while reading this passage. I'm no expert of machine learning, NLP, or linguistics, but this struck me as a pretty obvious flaw in the author's argument.