5 ms·

The whole point is that your models should perform well on data that they haven't seen before. You can't create a hash function for that. Let alone one that wor

by psv1 7y ago

The whole point is that your models should perform well on data that they haven't seen before. You can't create a hash function for that. Let alone one that works for 100s or 1000s of dimensions.

otnobot 7y ago
Yes, the model should work with any "new or old" data and also any "type". Data from any products, services, commodities or measurable force / energy. The model will need "adjustments" from time to time but the core will stay intact indefinitly.
fallous 7y ago
Unless there is a self-modifying feedback loop you can indeed replace any ML model with a hash function. The problem is one of size rather than any fundamental limitation of a look-up table.
shakna 7y ago
Shouldn't this fall into the "trust by verify" category of things? Whilst your model may not be able to explain to you today why it can operate on future data, if it can explain to you why it works on any data you've already given it, it makes it easier to explain behaviour. You have to trust it on new data. You can always verify why it made the choices it did on old data. That seems like something incredibly helpful, to me.
- psv1 7y ago
  > You can always verify why it made the choices it did on old data. This makes no sense. Your model typically isn't asked to make predictions on "old data" because it was trained on it. And understanding why it made certain prediction isn't any different on "new" vs old data. You can look into bias-variance tradeoff, train-test splitting and cross validation to get a better picture of this.
  - shakna 7y ago
    > You can look into bias-variance tradeoff, train-test splitting and cross validation to get a better picture of this. Or, like in the article, you can make use of introspective models that can tell you what dimensions were fitted. This can more easily show you where overfitting is occurring. You can see the weights of the model and follow the decision network, which allows you to explain why something has occurred in a much clearer fashion. An introspective model gives you a much clearer picture of why certain choices were made, rather than the fuzzy picture you get running statistical analysis against a black-box'd algorithm.
jayd16 7y ago
Uh, yeah you can. Hash functions handle data its never seen before.
rubyn00bie 7y ago
I'm confused by what you're saying... by data that hasn't seen before, do you mean new variables which need to be hypothesis tested? Or just a larger sample? I also don't see why you can't create a "function" hash or otherwise, that works for 1000s of dimensions. The model itself, if it was perfect would be an equation... the number of dimensions really doesn't matter that much. Sorry, I'm just real confused by your statement.
- psv1 7y ago
  By data it hasn't seen before I mean a sample which doesn't exist in the training set. Maybe I'm not understanding something in the parent's argument. For example, you train an image recognition model to tell cats and dogs apart using images from the internet, then you take out your phone, snap an image of your dog and give it to your algorithm to determine the species. When preprocessed this picture is equivalent to one row in a table with thousands of columns and this specific combination of pixel values doesn't exist anywhere else. Where is the hash function looking and for what in that case?
- mkolodny 7y ago
  Say you're building a self-driving car. You need to take an image of what the car sees, and figure out whether/where there is a pedestrian/car/bicycle/etc in the image. So your input/variable is the image itself. Almost every image the car sees is going to be different. I'd guess that's why GP said that it doesn't make sense to use a hashing function - there's little value in mapping inputs to results, because your input images are pretty much always going to be different. So pretty much every time you look up an image in your hashmap, you wouldn't find a match. That's the point of ML models - find patterns in data so that when you see a new example, you can predict what you're seeing based on what you've seen in the past.
  - xelxebar 7y ago
    It seems like you might be deep in the ML rabbit hole. Zoom out a bit. A hash function is "just a function." Every image labeling black box can be though of as a hash from images to label vectors. OP's comment effective asks whether there is another, more grokkable function that maps/hashes inputs to the same labels. Granted, that question boils down to "can we create human-understandable models?" which is the whole point of this discussion. It's a good question, though. If we had black-box-like spaghetti code performing the same task, I predict that the comments here would be very different.

by psv1 7y ago

The whole point is that your models should perform well on data that they haven't seen before. You can't create a hash function for that. Let alone one that works for 100s or 1000s of dimensions.

otnobot 7y ago

Yes, the model should work with any "new or old" data and also any "type". Data from any products, services, commodities or measurable force / energy. The model will need "adjustments" from time to time but the core will stay intact indefinitly.

fallous 7y ago

Unless there is a self-modifying feedback loop you can indeed replace any ML model with a hash function. The problem is one of size rather than any fundamental limitation of a look-up table.

shakna 7y ago

Shouldn't this fall into the "trust by verify" category of things? Whilst your model may not be able to explain to you today why it can operate on future data, if it can explain to you why it works on any data you've already given it, it makes it easier to explain behaviour. You have to trust it on new data. You can always verify why it made the choices it did on old data. That seems like something incredibly helpful, to me.

psv1 7y ago
> You can always verify why it made the choices it did on old data. This makes no sense. Your model typically isn't asked to make predictions on "old data" because it was trained on it. And understanding why it made certain prediction isn't any different on "new" vs old data. You can look into bias-variance tradeoff, train-test splitting and cross validation to get a better picture of this.
- shakna 7y ago
  > You can look into bias-variance tradeoff, train-test splitting and cross validation to get a better picture of this. Or, like in the article, you can make use of introspective models that can tell you what dimensions were fitted. This can more easily show you where overfitting is occurring. You can see the weights of the model and follow the decision network, which allows you to explain why something has occurred in a much clearer fashion. An introspective model gives you a much clearer picture of why certain choices were made, rather than the fuzzy picture you get running statistical analysis against a black-box'd algorithm.

jayd16 7y ago

Uh, yeah you can. Hash functions handle data its never seen before.

rubyn00bie 7y ago

I'm confused by what you're saying... by data that hasn't seen before, do you mean new variables which need to be hypothesis tested? Or just a larger sample? I also don't see why you can't create a "function" hash or otherwise, that works for 1000s of dimensions. The model itself, if it was perfect would be an equation... the number of dimensions really doesn't matter that much. Sorry, I'm just real confused by your statement.

psv1 7y ago
By data it hasn't seen before I mean a sample which doesn't exist in the training set. Maybe I'm not understanding something in the parent's argument. For example, you train an image recognition model to tell cats and dogs apart using images from the internet, then you take out your phone, snap an image of your dog and give it to your algorithm to determine the species. When preprocessed this picture is equivalent to one row in a table with thousands of columns and this specific combination of pixel values doesn't exist anywhere else. Where is the hash function looking and for what in that case?
mkolodny 7y ago
Say you're building a self-driving car. You need to take an image of what the car sees, and figure out whether/where there is a pedestrian/car/bicycle/etc in the image. So your input/variable is the image itself. Almost every image the car sees is going to be different. I'd guess that's why GP said that it doesn't make sense to use a hashing function - there's little value in mapping inputs to results, because your input images are pretty much always going to be different. So pretty much every time you look up an image in your hashmap, you wouldn't find a match. That's the point of ML models - find patterns in data so that when you see a new example, you can predict what you're seeing based on what you've seen in the past.
- xelxebar 7y ago
  It seems like you might be deep in the ML rabbit hole. Zoom out a bit. A hash function is "just a function." Every image labeling black box can be though of as a hash from images to label vectors. OP's comment effective asks whether there is another, more grokkable function that maps/hashes inputs to the same labels. Granted, that question boils down to "can we create human-understandable models?" which is the whole point of this discussion. It's a good question, though. If we had black-box-like spaghetti code performing the same task, I predict that the comments here would be very different.