4 ms·

This is a great push by the CFPB! Transparent information is helpful for a functional market. Even if one is using a trained model to make a credit decision, i

This is a great push by the CFPB! Transparent information is helpful for a functional market. Even if one is using a trained model to make a credit decision, it should be possible to pull out the weighted factors and list them or at least list the most heavily weighted factor in the decision.

version_five 4y ago
This comment, and the law, reveal a misunderstanding of how "complex algorithms" work. They implicitly assume there is an understandable and separated mapping between some set of features and the decision. That would describe some subset of models, but there are models, including most neural networks, that don't work that way. So basically lenders are being asked to either use a particular set of models for credit scoring, or use some smoke and mirrors (like feature importance) to pretend models work in a certain way. If you could explain simply how AI / ML came to a decision, you wouldn't need it, you'd just write out that explanation as a rule for how you make decisions.
- toomuchtodo 4y ago
  Require divulging the model as a regulatory requirement if necessary. A model is simply an algorithmic representation of business rules (underwriting standards or guidelines), trained on data. One might even expect this information to be divulged in prospectus for debt if securitized and sold in the capital markets. If you can't provide evidence how the model decisioned an applicant, you don't get to use the model. These are rules government can make, as the CFPB is demonstrating. There is no reason it must be permitted or allowed for a black box to make credit issuance decisions if explanations of the output aren’t possible. https://www.youtube.com/watch?v=mRNX6XJOeGU https://www.youtube.com/watch?v=mRNX6XJOeGU (thoughts and opinions are my own, contributed in risk management at an org where models are trained and used in credit decisioning; for each decline it could explained in a straightforward plainspoken manner what criteria caused the decline to occur)
  - version_five 4y ago
    I can see arguments for transparency obviously, I'd just also add that if there is a "black box" model that better estimates risk, it could benefit borrowers just as much as lenders, and so everybody can miss out because of artificial limitations. I'd also argue that fair or trustworthy models do not necessitate feature based explanations (which for complicated models are illusory anyway) and there are other ways, including simply stress testing, flagging out of distribution inputs, instance based methods, or simply human review, to be confident that the model works and that it isn't doing something you don't want. Having laypeople or special interests mandate how your computer program works or how you have to pretend it works is not the solution imo
    - toomuchtodo 4y ago
      I can understand how the belief that improved risk pricing provided by a black box model could be arrived at to be the "better" methodology for pricing credit, but importantly, the observability and introspection of the model is a non functional requirement without a pricing/revenue component. Some profit will be given up to meet regulatory requirements; if you can't follow the rules, you don't get to operate, at which point your model value rapidly approaches zero except as a parlor trick (jurisdiction dependent of course). > Having laypeople or special interests mandate how your computer program works or how you have to pretend it works is not the solution imo Agree to disagree. Regulators exist to thwart predatory behavior, including black boxes making decisions no one understands, and companies will not regulate themselves (proven time and time again [1] [2] [3]). [1] https://www.consumerfinance.gov/enforcement/actions/ https://www.consumerfinance.gov/enforcement/actions/ [2] https://www.consumerfinance.gov/enforcement/enforcement-by-the-numbers/ https://www.consumerfinance.gov/enforcement/enforcement-by-t... [3] https://www.consumerfinance.gov/enforcement/payments-harmed-consumers/payments-by-case/ https://www.consumerfinance.gov/enforcement/payments-harmed-...
      - rahimnathwani 4y ago
        What makes the use of black box decision models 'predatory behavior'? I can see that, if you're going to do something predatory, you should probably hide stuff. But is an unexplained model inherently predatory? (I'm not expressing an opinion either way about whether unexplainable models are good for lenders' profits.)
- wmf 4y ago
  Requiring explainable ML seems fine to me. Black-box models are just asking for trouble.
- pyuser583 4y ago
  I know this is a dumb question, but I don’t know the answer: why can’t we use AI to figure out how AI models work?
  - version_five 4y ago
    There might be some methods that could be considered using AI to figure our how the model works, like distillation or I think I've heard of simultaneously training feature maps along with a classifier. Those don't get around the fact that placing constraints on how the model can make its decision (by requiring that they have human interpretable explanations) limits the kinds of model that are available, and can limit performance. So the rule is more a slightly more advanced analogy of telling people they need to make a rubric with a list of factors, and then a weighing for each, and use that to make credit decisions.
- coderintherye 4y ago
  This comment, seems to misunderstand how Machine Learning actually works. If you want to limit to "neural networks" specifically then your comment would be more in line, but there are almost no lenders requiring exclusively on neural networks for their credit decisions. If you believe the complexity can't be explained then you have no business working in lending. I'm glad the law is in place to prevent this sort of mentality from being able to exist in the marketplace.
- Teever 4y ago
  That's a whole bunch of words to say that these companies don't want to follow the law. What I'm hearing from you is that we shouldn't be using these kinds of algorithms for this kind of thing.

version_five 4y ago

This comment, and the law, reveal a misunderstanding of how "complex algorithms" work. They implicitly assume there is an understandable and separated mapping between some set of features and the decision. That would describe some subset of models, but there are models, including most neural networks, that don't work that way. So basically lenders are being asked to either use a particular set of models for credit scoring, or use some smoke and mirrors (like feature importance) to pretend models work in a certain way. If you could explain simply how AI / ML came to a decision, you wouldn't need it, you'd just write out that explanation as a rule for how you make decisions.

toomuchtodo 4y ago
Require divulging the model as a regulatory requirement if necessary. A model is simply an algorithmic representation of business rules (underwriting standards or guidelines), trained on data. One might even expect this information to be divulged in prospectus for debt if securitized and sold in the capital markets. If you can't provide evidence how the model decisioned an applicant, you don't get to use the model. These are rules government can make, as the CFPB is demonstrating. There is no reason it must be permitted or allowed for a black box to make credit issuance decisions if explanations of the output aren’t possible. https://www.youtube.com/watch?v=mRNX6XJOeGU https://www.youtube.com/watch?v=mRNX6XJOeGU (thoughts and opinions are my own, contributed in risk management at an org where models are trained and used in credit decisioning; for each decline it could explained in a straightforward plainspoken manner what criteria caused the decline to occur)
- version_five 4y ago
  I can see arguments for transparency obviously, I'd just also add that if there is a "black box" model that better estimates risk, it could benefit borrowers just as much as lenders, and so everybody can miss out because of artificial limitations. I'd also argue that fair or trustworthy models do not necessitate feature based explanations (which for complicated models are illusory anyway) and there are other ways, including simply stress testing, flagging out of distribution inputs, instance based methods, or simply human review, to be confident that the model works and that it isn't doing something you don't want. Having laypeople or special interests mandate how your computer program works or how you have to pretend it works is not the solution imo
  - toomuchtodo 4y ago
    I can understand how the belief that improved risk pricing provided by a black box model could be arrived at to be the "better" methodology for pricing credit, but importantly, the observability and introspection of the model is a non functional requirement without a pricing/revenue component. Some profit will be given up to meet regulatory requirements; if you can't follow the rules, you don't get to operate, at which point your model value rapidly approaches zero except as a parlor trick (jurisdiction dependent of course). > Having laypeople or special interests mandate how your computer program works or how you have to pretend it works is not the solution imo Agree to disagree. Regulators exist to thwart predatory behavior, including black boxes making decisions no one understands, and companies will not regulate themselves (proven time and time again [1] [2] [3]). [1] https://www.consumerfinance.gov/enforcement/actions/ https://www.consumerfinance.gov/enforcement/actions/ [2] https://www.consumerfinance.gov/enforcement/enforcement-by-the-numbers/ https://www.consumerfinance.gov/enforcement/enforcement-by-t... [3] https://www.consumerfinance.gov/enforcement/payments-harmed-consumers/payments-by-case/ https://www.consumerfinance.gov/enforcement/payments-harmed-...
    - rahimnathwani 4y ago
      What makes the use of black box decision models 'predatory behavior'? I can see that, if you're going to do something predatory, you should probably hide stuff. But is an unexplained model inherently predatory? (I'm not expressing an opinion either way about whether unexplainable models are good for lenders' profits.)
wmf 4y ago
Requiring explainable ML seems fine to me. Black-box models are just asking for trouble.
pyuser583 4y ago
I know this is a dumb question, but I don’t know the answer: why can’t we use AI to figure out how AI models work?
- version_five 4y ago
  There might be some methods that could be considered using AI to figure our how the model works, like distillation or I think I've heard of simultaneously training feature maps along with a classifier. Those don't get around the fact that placing constraints on how the model can make its decision (by requiring that they have human interpretable explanations) limits the kinds of model that are available, and can limit performance. So the rule is more a slightly more advanced analogy of telling people they need to make a rubric with a list of factors, and then a weighing for each, and use that to make credit decisions.
coderintherye 4y ago
This comment, seems to misunderstand how Machine Learning actually works. If you want to limit to "neural networks" specifically then your comment would be more in line, but there are almost no lenders requiring exclusively on neural networks for their credit decisions. If you believe the complexity can't be explained then you have no business working in lending. I'm glad the law is in place to prevent this sort of mentality from being able to exist in the marketplace.
Teever 4y ago
That's a whole bunch of words to say that these companies don't want to follow the law. What I'm hearing from you is that we shouldn't be using these kinds of algorithms for this kind of thing.