4 ms·

This comment, and the law, reveal a misunderstanding of how "complex algorithms" work. They implicitly assume there is an understandable and separated mapping b

This comment, and the law, reveal a misunderstanding of how "complex algorithms" work. They implicitly assume there is an understandable and separated mapping between some set of features and the decision. That would describe some subset of models, but there are models, including most neural networks, that don't work that way. So basically lenders are being asked to either use a particular set of models for credit scoring, or use some smoke and mirrors (like feature importance) to pretend models work in a certain way. If you could explain simply how AI / ML came to a decision, you wouldn't need it, you'd just write out that explanation as a rule for how you make decisions.

toomuchtodo 4y ago
Require divulging the model as a regulatory requirement if necessary. A model is simply an algorithmic representation of business rules (underwriting standards or guidelines), trained on data. One might even expect this information to be divulged in prospectus for debt if securitized and sold in the capital markets. If you can't provide evidence how the model decisioned an applicant, you don't get to use the model. These are rules government can make, as the CFPB is demonstrating. There is no reason it must be permitted or allowed for a black box to make credit issuance decisions if explanations of the output aren’t possible. https://www.youtube.com/watch?v=mRNX6XJOeGU https://www.youtube.com/watch?v=mRNX6XJOeGU (thoughts and opinions are my own, contributed in risk management at an org where models are trained and used in credit decisioning; for each decline it could explained in a straightforward plainspoken manner what criteria caused the decline to occur)
- version_five 4y ago
  I can see arguments for transparency obviously, I'd just also add that if there is a "black box" model that better estimates risk, it could benefit borrowers just as much as lenders, and so everybody can miss out because of artificial limitations. I'd also argue that fair or trustworthy models do not necessitate feature based explanations (which for complicated models are illusory anyway) and there are other ways, including simply stress testing, flagging out of distribution inputs, instance based methods, or simply human review, to be confident that the model works and that it isn't doing something you don't want. Having laypeople or special interests mandate how your computer program works or how you have to pretend it works is not the solution imo
  - toomuchtodo 4y ago
    I can understand how the belief that improved risk pricing provided by a black box model could be arrived at to be the "better" methodology for pricing credit, but importantly, the observability and introspection of the model is a non functional requirement without a pricing/revenue component. Some profit will be given up to meet regulatory requirements; if you can't follow the rules, you don't get to operate, at which point your model value rapidly approaches zero except as a parlor trick (jurisdiction dependent of course). > Having laypeople or special interests mandate how your computer program works or how you have to pretend it works is not the solution imo Agree to disagree. Regulators exist to thwart predatory behavior, including black boxes making decisions no one understands, and companies will not regulate themselves (proven time and time again [1] [2] [3]). [1] https://www.consumerfinance.gov/enforcement/actions/ https://www.consumerfinance.gov/enforcement/actions/ [2] https://www.consumerfinance.gov/enforcement/enforcement-by-the-numbers/ https://www.consumerfinance.gov/enforcement/enforcement-by-t... [3] https://www.consumerfinance.gov/enforcement/payments-harmed-consumers/payments-by-case/ https://www.consumerfinance.gov/enforcement/payments-harmed-...
    - rahimnathwani 4y ago
      What makes the use of black box decision models 'predatory behavior'? I can see that, if you're going to do something predatory, you should probably hide stuff. But is an unexplained model inherently predatory? (I'm not expressing an opinion either way about whether unexplainable models are good for lenders' profits.)
- shitlord 4y ago
  Isn't the model a trade secret? Correct me if I'm wrong but the model is what allows the lender to optimize profit by maximizing good loans and minimizing bad ones. If the model is public then the lender would lose its competitive advantage because any competitor could just use the same model.
  - toomuchtodo 4y ago
    Providing model access to regulators doesn’t necessarily make the model public.
  - everforward 4y ago
    Sure, but there's an argument to be made that credit is such an underpinning facet of society that it's unwise to allow the decisions to be made in darkness. The obvious example is housing, which requires a loan for the vast, vast majority of people. These decisions have concrete implications for who can buy a house, or what house, or where. Redlining could be enforced almost entirely via lending (and it was, iirc). I'm not claiming that happens currently, because I have no idea, but it's a possibility. There are also workarounds. IP law could be used to restrict competitors using the models. We could make the models be open and operating under a fee basis (i.e. the creator of the model gets 0.1% of the loan amount as a payout). There are likely other options. I see it as a parallel to utilities. They are necessary for society to function, but no one will build them out if they have to share. So we hand over a monopoly with a lot of restrictions on how they can operate. I think the same could be true here.
- dataangel 4y ago
  The best models may not have an easy to boil down explanation, like it may actually not exist, it's not just that it can't be found. The models are just elaborate empirical observations that certain things together are common in people who don't pay back and other certain things together are common in people who do. But there may be no feature that is purely good or purely bad, it may always depend on the whole combination. These kinds of requirements will just lead to technical theater.
  - TheCoelacanth 4y ago
    If you can't explain it, then the whole model is the explanation. Provide the trained model for people to make their own empirical observations about how different factors impact the decisions.
wmf 4y ago
Requiring explainable ML seems fine to me. Black-box models are just asking for trouble.
pyuser583 4y ago
I know this is a dumb question, but I don’t know the answer: why can’t we use AI to figure out how AI models work?
- version_five 4y ago
  There might be some methods that could be considered using AI to figure our how the model works, like distillation or I think I've heard of simultaneously training feature maps along with a classifier. Those don't get around the fact that placing constraints on how the model can make its decision (by requiring that they have human interpretable explanations) limits the kinds of model that are available, and can limit performance. So the rule is more a slightly more advanced analogy of telling people they need to make a rubric with a list of factors, and then a weighing for each, and use that to make credit decisions.
coderintherye 4y ago
This comment, seems to misunderstand how Machine Learning actually works. If you want to limit to "neural networks" specifically then your comment would be more in line, but there are almost no lenders requiring exclusively on neural networks for their credit decisions. If you believe the complexity can't be explained then you have no business working in lending. I'm glad the law is in place to prevent this sort of mentality from being able to exist in the marketplace.
Teever 4y ago
That's a whole bunch of words to say that these companies don't want to follow the law. What I'm hearing from you is that we shouldn't be using these kinds of algorithms for this kind of thing.

toomuchtodo 4y ago

Require divulging the model as a regulatory requirement if necessary. A model is simply an algorithmic representation of business rules (underwriting standards or guidelines), trained on data. One might even expect this information to be divulged in prospectus for debt if securitized and sold in the capital markets. If you can't provide evidence how the model decisioned an applicant, you don't get to use the model. These are rules government can make, as the CFPB is demonstrating. There is no reason it must be permitted or allowed for a black box to make credit issuance decisions if explanations of the output aren’t possible. https://www.youtube.com/watch?v=mRNX6XJOeGU https://www.youtube.com/watch?v=mRNX6XJOeGU (thoughts and opinions are my own, contributed in risk management at an org where models are trained and used in credit decisioning; for each decline it could explained in a straightforward plainspoken manner what criteria caused the decline to occur)

version_five 4y ago
I can see arguments for transparency obviously, I'd just also add that if there is a "black box" model that better estimates risk, it could benefit borrowers just as much as lenders, and so everybody can miss out because of artificial limitations. I'd also argue that fair or trustworthy models do not necessitate feature based explanations (which for complicated models are illusory anyway) and there are other ways, including simply stress testing, flagging out of distribution inputs, instance based methods, or simply human review, to be confident that the model works and that it isn't doing something you don't want. Having laypeople or special interests mandate how your computer program works or how you have to pretend it works is not the solution imo
- toomuchtodo 4y ago
  I can understand how the belief that improved risk pricing provided by a black box model could be arrived at to be the "better" methodology for pricing credit, but importantly, the observability and introspection of the model is a non functional requirement without a pricing/revenue component. Some profit will be given up to meet regulatory requirements; if you can't follow the rules, you don't get to operate, at which point your model value rapidly approaches zero except as a parlor trick (jurisdiction dependent of course). > Having laypeople or special interests mandate how your computer program works or how you have to pretend it works is not the solution imo Agree to disagree. Regulators exist to thwart predatory behavior, including black boxes making decisions no one understands, and companies will not regulate themselves (proven time and time again [1] [2] [3]). [1] https://www.consumerfinance.gov/enforcement/actions/ https://www.consumerfinance.gov/enforcement/actions/ [2] https://www.consumerfinance.gov/enforcement/enforcement-by-the-numbers/ https://www.consumerfinance.gov/enforcement/enforcement-by-t... [3] https://www.consumerfinance.gov/enforcement/payments-harmed-consumers/payments-by-case/ https://www.consumerfinance.gov/enforcement/payments-harmed-...
  - rahimnathwani 4y ago
    What makes the use of black box decision models 'predatory behavior'? I can see that, if you're going to do something predatory, you should probably hide stuff. But is an unexplained model inherently predatory? (I'm not expressing an opinion either way about whether unexplainable models are good for lenders' profits.)
shitlord 4y ago
Isn't the model a trade secret? Correct me if I'm wrong but the model is what allows the lender to optimize profit by maximizing good loans and minimizing bad ones. If the model is public then the lender would lose its competitive advantage because any competitor could just use the same model.
- toomuchtodo 4y ago
  Providing model access to regulators doesn’t necessarily make the model public.
- everforward 4y ago
  Sure, but there's an argument to be made that credit is such an underpinning facet of society that it's unwise to allow the decisions to be made in darkness. The obvious example is housing, which requires a loan for the vast, vast majority of people. These decisions have concrete implications for who can buy a house, or what house, or where. Redlining could be enforced almost entirely via lending (and it was, iirc). I'm not claiming that happens currently, because I have no idea, but it's a possibility. There are also workarounds. IP law could be used to restrict competitors using the models. We could make the models be open and operating under a fee basis (i.e. the creator of the model gets 0.1% of the loan amount as a payout). There are likely other options. I see it as a parallel to utilities. They are necessary for society to function, but no one will build them out if they have to share. So we hand over a monopoly with a lot of restrictions on how they can operate. I think the same could be true here.
dataangel 4y ago
The best models may not have an easy to boil down explanation, like it may actually not exist, it's not just that it can't be found. The models are just elaborate empirical observations that certain things together are common in people who don't pay back and other certain things together are common in people who do. But there may be no feature that is purely good or purely bad, it may always depend on the whole combination. These kinds of requirements will just lead to technical theater.
- TheCoelacanth 4y ago
  If you can't explain it, then the whole model is the explanation. Provide the trained model for people to make their own empirical observations about how different factors impact the decisions.

wmf 4y ago

Requiring explainable ML seems fine to me. Black-box models are just asking for trouble.

pyuser583 4y ago

I know this is a dumb question, but I don’t know the answer: why can’t we use AI to figure out how AI models work?

version_five 4y ago
There might be some methods that could be considered using AI to figure our how the model works, like distillation or I think I've heard of simultaneously training feature maps along with a classifier. Those don't get around the fact that placing constraints on how the model can make its decision (by requiring that they have human interpretable explanations) limits the kinds of model that are available, and can limit performance. So the rule is more a slightly more advanced analogy of telling people they need to make a rubric with a list of factors, and then a weighing for each, and use that to make credit decisions.

coderintherye 4y ago

This comment, seems to misunderstand how Machine Learning actually works. If you want to limit to "neural networks" specifically then your comment would be more in line, but there are almost no lenders requiring exclusively on neural networks for their credit decisions. If you believe the complexity can't be explained then you have no business working in lending. I'm glad the law is in place to prevent this sort of mentality from being able to exist in the marketplace.

Teever 4y ago

That's a whole bunch of words to say that these companies don't want to follow the law. What I'm hearing from you is that we shouldn't be using these kinds of algorithms for this kind of thing.