4 ms·

I'm confused by what you're saying... by data that hasn't seen before, do you mean new variables which need to be hypothesis tested? Or just a larger sample? I

I'm confused by what you're saying... by data that hasn't seen before, do you mean new variables which need to be hypothesis tested? Or just a larger sample? I also don't see why you can't create a "function" hash or otherwise, that works for 1000s of dimensions. The model itself, if it was perfect would be an equation... the number of dimensions really doesn't matter that much. Sorry, I'm just real confused by your statement.

psv1 7y ago
By data it hasn't seen before I mean a sample which doesn't exist in the training set. Maybe I'm not understanding something in the parent's argument. For example, you train an image recognition model to tell cats and dogs apart using images from the internet, then you take out your phone, snap an image of your dog and give it to your algorithm to determine the species. When preprocessed this picture is equivalent to one row in a table with thousands of columns and this specific combination of pixel values doesn't exist anywhere else. Where is the hash function looking and for what in that case?
mkolodny 7y ago
Say you're building a self-driving car. You need to take an image of what the car sees, and figure out whether/where there is a pedestrian/car/bicycle/etc in the image. So your input/variable is the image itself. Almost every image the car sees is going to be different. I'd guess that's why GP said that it doesn't make sense to use a hashing function - there's little value in mapping inputs to results, because your input images are pretty much always going to be different. So pretty much every time you look up an image in your hashmap, you wouldn't find a match. That's the point of ML models - find patterns in data so that when you see a new example, you can predict what you're seeing based on what you've seen in the past.
- xelxebar 7y ago
  It seems like you might be deep in the ML rabbit hole. Zoom out a bit. A hash function is "just a function." Every image labeling black box can be though of as a hash from images to label vectors. OP's comment effective asks whether there is another, more grokkable function that maps/hashes inputs to the same labels. Granted, that question boils down to "can we create human-understandable models?" which is the whole point of this discussion. It's a good question, though. If we had black-box-like spaghetti code performing the same task, I predict that the comments here would be very different.
  - mkolodny 7y ago
    Thanks for explaining that perspective. I was seeing things a little too narrow-mindedly for sure. Based on your phrasing of the issue, it seems like we could think of the problem as: can we reduce the number of parameters in an ML model to the point where humans can understand all of the parameters? That's related to an active research area - minimizing model size. ML models can have billions of parameters. It's unfeasible for a human to evaluate all of them. Research shows that (sometimes) you can reduce the number of parameters in a model 97% without hurting it's accuracy [0]. But, 3% of a billion parameters is still way too many for people to evaluate. So I think the answer so far is no, we can't create human-understandable models that perform as effectively as black boxes. [0] https://openreview.net/forum?id=rJl-b3RcF7 https://openreview.net/forum?id=rJl-b3RcF7

psv1 7y ago

By data it hasn't seen before I mean a sample which doesn't exist in the training set. Maybe I'm not understanding something in the parent's argument. For example, you train an image recognition model to tell cats and dogs apart using images from the internet, then you take out your phone, snap an image of your dog and give it to your algorithm to determine the species. When preprocessed this picture is equivalent to one row in a table with thousands of columns and this specific combination of pixel values doesn't exist anywhere else. Where is the hash function looking and for what in that case?

mkolodny 7y ago

Say you're building a self-driving car. You need to take an image of what the car sees, and figure out whether/where there is a pedestrian/car/bicycle/etc in the image. So your input/variable is the image itself. Almost every image the car sees is going to be different. I'd guess that's why GP said that it doesn't make sense to use a hashing function - there's little value in mapping inputs to results, because your input images are pretty much always going to be different. So pretty much every time you look up an image in your hashmap, you wouldn't find a match. That's the point of ML models - find patterns in data so that when you see a new example, you can predict what you're seeing based on what you've seen in the past.

xelxebar 7y ago
It seems like you might be deep in the ML rabbit hole. Zoom out a bit. A hash function is "just a function." Every image labeling black box can be though of as a hash from images to label vectors. OP's comment effective asks whether there is another, more grokkable function that maps/hashes inputs to the same labels. Granted, that question boils down to "can we create human-understandable models?" which is the whole point of this discussion. It's a good question, though. If we had black-box-like spaghetti code performing the same task, I predict that the comments here would be very different.
- mkolodny 7y ago
  Thanks for explaining that perspective. I was seeing things a little too narrow-mindedly for sure. Based on your phrasing of the issue, it seems like we could think of the problem as: can we reduce the number of parameters in an ML model to the point where humans can understand all of the parameters? That's related to an active research area - minimizing model size. ML models can have billions of parameters. It's unfeasible for a human to evaluate all of them. Research shows that (sometimes) you can reduce the number of parameters in a model 97% without hurting it's accuracy [0]. But, 3% of a billion parameters is still way too many for people to evaluate. So I think the answer so far is no, we can't create human-understandable models that perform as effectively as black boxes. [0] https://openreview.net/forum?id=rJl-b3RcF7 https://openreview.net/forum?id=rJl-b3RcF7