5 ms·

I think it will be very hard to implement/enforce this regulation. I attended the London AI summit last week, and they had a speaker from a German lender called

by dharma1 10y ago

I think it will be very hard to implement/enforce this regulation. I attended the London AI summit last week, and they had a speaker from a German lender called Kreditech. There was a question of black box credit scoring, and the speaker made a fair point - their models have 20,000 vectors in determining credit worthiness. How would you begin to break that down to something explainable? You can list the sources of the data, or offer manual review of the application if rejected (which is something they do) - but it would be very hard to show exact causal reasoning patterns in the automatic scoring.

dec0dedab0de 10y ago
why not just have an optional pdf download with the 20000 vectors, and how much they were weighted.
GFK_of_xmaspast 10y ago
Sounds like a limitation of their business model.
kaijush 10y ago
>> their models have 20,000 vectors in determining credit worthiness. How would you begin to break that down to something explainable? Well, somehow they decided that their 20k-parameter model is accurate. They should at least be able to explain why they took that decision, even if the model itself is too complex.
- JoeAltmaier 10y ago
  Because the training set was covered with x% accuracy. Ok, how does knowing that help?
  - kaijush 10y ago
    OK, so we're assuming supervised learning. In that case, how were the features of the training set chosen? And how was the training set labelled? Most of the time, outside of semi-supervised learning, those things are not in the data, someone has to (painstakingly) decide them. That can very well be explained I believe. More generally, what I mean to say is that you can learn any number of models from the same data, so to a great extent you choose what your model learns by manipulating your algorithm's (hyper)parameters, the training set's features, choosing what goes into the training set itself etc etc. That process leaves room for a lot of regulatory oversight. - in principle. In practice I expect since we're talking about people lending money or selling insurance we're only ever going to get those details out of their cold dead hands, and only with considerable effort at that.
- dharma1 10y ago
  They validate the accuracy of the model with real users, starting with short term/low value loans, to mitigate risk. The explanation for them is - it works.
  - tomaskafka 10y ago
    "How would you find out that it doesn't work? For a specific group of people for example?"
    - dharma1 10y ago
      How do you find out that it doesn't work? Some loans default, company loses money on those. The model gets updated continuously from the real user data, so it learns from the mistakes
kartan 10y ago
> I think it will be very hard to implement/enforce this regulation. The regulation is quite simple and easy to enforce. If they can't explain then it is illegal. What your example implies is that there are business that doesn't take discrimination laws into account when designing their software, even for sensitive decisions. And enforcing discrimination free decisions is exactly the goal of this law. I worked for the gambling industry. "changing our software to meet the law is too complicated" never was a valid reasoning, even when complex and extremely hard to fine tune algorithms were at stake.
- madgar 10y ago
  > I worked for the gambling industry. I never thought I'd say it, but relatively speaking, the gambling industry is grounded in reality. Most of the commenters here are coming from Silicon Valley where regulation is virtually non-existent and typically derided as impeding the glorious all-consuming will of the free market.
323454 10y ago
Maybe if you can't explain your model you should be forced to open source your method and a large enough data sample for people to independently check your results.
viraptor 10y ago
Do you know how many actually matter for a single person? I understand that 20k vectors could be created to cover the entire population, but realistically how many are non-zero? I expect there's some threshold below which everything is either completely ignored, or doesn't make a practical difference. An average person's model will contain: - not divorced - not home owner - not prosecuted - not on probation - not dying - not (19k of situations)
- JoeAltmaier 10y ago
  assuming the vectors are one variable each. Which 'vector' implies they are not. So its more like {particular value of home ownership, age range, medical cost history in a certain bracket, no pet supply purchase history}. How do you explain that? If its been created via data mining and not, for instance, fuzzy logic rule sets, then no human ever knew why each vector was chosen. Its just a pattern discovered in the population. It would likely be very hard to explain most of the vectors.
dilemma 10y ago
This is exactly why the legislation is created: to stop peoples' lives from being destroyed or negatively affected by computer programs that not even their creator know how they work.

by dharma1 10y ago

dec0dedab0de 10y ago

why not just have an optional pdf download with the 20000 vectors, and how much they were weighted.

GFK_of_xmaspast 10y ago

Sounds like a limitation of their business model.

kaijush 10y ago

>> their models have 20,000 vectors in determining credit worthiness. How would you begin to break that down to something explainable? Well, somehow they decided that their 20k-parameter model is accurate. They should at least be able to explain why they took that decision, even if the model itself is too complex.

JoeAltmaier 10y ago
Because the training set was covered with x% accuracy. Ok, how does knowing that help?
- kaijush 10y ago
  OK, so we're assuming supervised learning. In that case, how were the features of the training set chosen? And how was the training set labelled? Most of the time, outside of semi-supervised learning, those things are not in the data, someone has to (painstakingly) decide them. That can very well be explained I believe. More generally, what I mean to say is that you can learn any number of models from the same data, so to a great extent you choose what your model learns by manipulating your algorithm's (hyper)parameters, the training set's features, choosing what goes into the training set itself etc etc. That process leaves room for a lot of regulatory oversight. - in principle. In practice I expect since we're talking about people lending money or selling insurance we're only ever going to get those details out of their cold dead hands, and only with considerable effort at that.
dharma1 10y ago
They validate the accuracy of the model with real users, starting with short term/low value loans, to mitigate risk. The explanation for them is - it works.
- tomaskafka 10y ago
  "How would you find out that it doesn't work? For a specific group of people for example?"
  - dharma1 10y ago
    How do you find out that it doesn't work? Some loans default, company loses money on those. The model gets updated continuously from the real user data, so it learns from the mistakes

kartan 10y ago

> I think it will be very hard to implement/enforce this regulation. The regulation is quite simple and easy to enforce. If they can't explain then it is illegal. What your example implies is that there are business that doesn't take discrimination laws into account when designing their software, even for sensitive decisions. And enforcing discrimination free decisions is exactly the goal of this law. I worked for the gambling industry. "changing our software to meet the law is too complicated" never was a valid reasoning, even when complex and extremely hard to fine tune algorithms were at stake.

madgar 10y ago
> I worked for the gambling industry. I never thought I'd say it, but relatively speaking, the gambling industry is grounded in reality. Most of the commenters here are coming from Silicon Valley where regulation is virtually non-existent and typically derided as impeding the glorious all-consuming will of the free market.

323454 10y ago

Maybe if you can't explain your model you should be forced to open source your method and a large enough data sample for people to independently check your results.

viraptor 10y ago

Do you know how many actually matter for a single person? I understand that 20k vectors could be created to cover the entire population, but realistically how many are non-zero? I expect there's some threshold below which everything is either completely ignored, or doesn't make a practical difference. An average person's model will contain: - not divorced - not home owner - not prosecuted - not on probation - not dying - not (19k of situations)

JoeAltmaier 10y ago
assuming the vectors are one variable each. Which 'vector' implies they are not. So its more like {particular value of home ownership, age range, medical cost history in a certain bracket, no pet supply purchase history}. How do you explain that? If its been created via data mining and not, for instance, fuzzy logic rule sets, then no human ever knew why each vector was chosen. Its just a pattern discovered in the population. It would likely be very hard to explain most of the vectors.

dilemma 10y ago

This is exactly why the legislation is created: to stop peoples' lives from being destroyed or negatively affected by computer programs that not even their creator know how they work.