6 ms·

New York City Moves to Create Accountability for Algorithms

79 points by dthal 9y ago

lelandgaunt 9y ago
This is a good move. I hope other states follow suit. I highly recommend reading Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy
- an_cap 9y ago
  I recommend reading this review before deciding to read the book. https://thezvi.wordpress.com/2017/06/04/book-review-weapons-of-math-destruction/ https://thezvi.wordpress.com/2017/06/04/book-review-weapons-...
  - meebs 9y ago
    "Review," in which a privileged person mocks the idea of invisible power resulting in perpetuating inequality being worse than visible power.
  - afsina 9y ago
    This was a good read, thanks.
- afsina 9y ago
  I have not read the book, but just the title reminded me I probably belong to a minority group that do not believe neither "equality" nor "democracy".
mc32 9y ago
This is a good thing --an algorithm with up to a 30 error rate is unreliable. One hopes this scrutiny will lead to development of more dependable and trustworthy algorithms. Also hopeful politicians don't have the ability to tweak things in order to fit their agendas. For example an algorithm is too good at detecting which gov't employee is stealing from the public. Or it tells them they should decrease fines in some area to get better results --but that would affect their budget adversely, etc.
- candiodari 9y ago
  This is an unrealistic law that doesn't matter. It sounds good, but it's futile. If this taskforce takes the decision that an algorithm that is working is unacceptable (which we all know they'll do on the worst possible basis - publicity), then what happens ? They'll pause the algorithm ? Replace it ? They can't do that ... The issue is that once you automate something you can't unautomate it without providing the workforce necessary. Because in nearly all cases a bad automated algorithm will far outperform 1% of the required workforce in humans, that's how algorithms win. Not because they beat humans, they don't. They win because they actually take action in 100% of the cases, whatever that number is. A dumb algorithm taking action 100.000 times can easily beat a very hardworking human that takes 100 smart decisions in a lot of cases. So that will be tens to hundreds of people in easy cases, and thousands to tens of thousands in bad cases, and this is New York, easily the size of a decent country. So it'll always be "we need to pause this NOW !", "OK, no problem at all, that'll be $15 million per hour. Under what budget item do we fit this ?" "Erm .... How about you just make it look like it's paused ?" Keep in mind stopping automation doesn't even just cause damage directly, it will also cause overloading costs onto other departments and even onto private companies. Often in surprising ways. Algorithms respond quickly, under nearly all circumstances, at any time. You wouldn't believe how efficient this makes interacting with organizations. Pausing automation has an enormous and accumulating cost, making the decision impossible. Also algorithms don't solve corruption (they may make it easier to track though, although there are ways around that).
  - pjc50 9y ago
    Taking bad decisions faster is not an improvement. Especially when it's being used to wrongly convict people. > takes the decision that an algorithm that is working is unacceptable "Unacceptable" == "not working".
  - matt4077 9y ago
    > which we all know they'll do on the worst possible basis - publicity That's such a cynical, nihilistic way of looking at government. I can only guess people arrive at it after being exposed to a rather superficial look at governments' work over a long time. In reality, the vast bureaucracy that is government takes thousands of actions every single day, almost all of which are uncontroversial. They work hard to establish procedures minimising uncertainty. The work is far more transparent than any private organisations'. And all decisions are subject to judicial review–with the judiciary having its own, long tradition of thoughtful deliberation and even-handedness. As one example, the list at https://www.regulations.gov/searchResults?rpp=25&so=DESC&sb=postedDate&po=0&np=90&dct=N%2BFR%2BPR https://www.regulations.gov/searchResults?rpp=25&so=DESC&sb=... shows some recent (federal) actions. Note that this list is only the tip of the iceberg, with the most controversial administration in modern history. Yet it is dominated by "Class E Airspace; Revocations: Eaton Rapids, MI" and other items of rather low publicity value.
patkai 9y ago
One of those paradoxical situations when something _really_ needs to be regulated but can't be regulated. E.g. how do you know what data set is the result of an algorithm (a dubious consolidation step?), or how do you regulate algorithmic learning when we don't fully understand how learning works?
- majewsky 9y ago
  > needs to be regulated but can't be regulated There is no binary choice between "not regulated at all" and "completely regulated". For example, just implementing a process for public scrutiny of algorithms and datasets may result in those building the algorithms and collecting the data becoming more aware of their biases. (That's "may" with a capital M, though.)
  - candiodari 9y ago
    What you mean to say is that politicians currently take decisions on things they aren't even remotely qualified to even talk about, then force it through with heavy hand, and therefore this won't change a thing ? That's true, and despite that it will change things.
- mannykannot 9y ago
  If you look at the specific issue that provoked this law, you will see issues that can - and should - be regulated. the city was using DNA matching software that turns out to have demonstrable errors, yet the office of the medical examiner stonewalled all attempts to have it audited until, thankfully, the courts forced the issue. This sort of unjustifiable secrecy (the accused absolutely have a right to examine the premises of the accusation) can be regulated. Unfortunately, this law substitutes nebulous criteria which, no matter how worthy, are likely to turn a clear-cut situation into a tar-pit of legal wrangling that the victims cannot afford to enter. The chief medical examiner is still holding fast on the very dubious claim that these flaws raise no doubts about the convictions in other cases where it was used, another area where I think specific legislation is needed. > How do you regulate algorithmic learning when we don't fully understand how learning works? This was not one of those cases. There are, however, cases - and this would be one if it applied - where it is reasonable to say that you can't use it until you can explain how it works.
skate22 9y ago
I wonder how an algorithm with low explainability will be treated, like a large nueral network. Seems like it would be hard to pin it on the algorithm rather than the data.
- vnchr 9y ago
  If there's a way to objectively evaluate quality of output, that seems sufficient. It's not like a building inspection where it's a component-based evaluation. The results need to be reasonably and verifiably reliable to an appropriate degree, but the courts don't need to care how that is accomplished.
  - AnthonyMouse 9y ago
    > If there's a way to objectively evaluate quality of output, that seems sufficient. But that's the whole problem. If you had something that calculates the the thing algorithm is supposed to do in a better way than the algorithm actually does it, you could just use that as the algorithm.
deleted 9y ago
[deleted]
SauciestGNU 9y ago
I think it's a travesty that the source code won't be shared with the public, but I hope it would at least be provided for review to the agencies adopting the algorithms. In my field we are highly regulated and must provide the regulatory agencies with an audit trail for all data. I would like to see an audit trail for these algorithms that would allow someone to follow the decision tree and review the outcome of the algorithms.
bob_theslob646 9y ago
I find this incredibly off putting since the city cannot even account for its own wasteful budget. There is nothing wrong with accountability and am glad they are starting somewhere but until they start holding others like the ones who purposesly destroy on time metrics of transit in order to keep thier salary gravy train choo-chooing, I am not holding my breath. https://mobile.nytimes.com/2017/11/18/nyregion/new-york-subway-system-failure-delays.html?referer=https://www.google.com/ https://mobile.nytimes.com/2017/11/18/nyregion/new-york-subw...
- sologoub 9y ago
  > Signal problems and car equipment failures occur twice as frequently as a decade ago, but hundreds of mechanic positions have been cut because there is not enough money to pay them — even though the average total compensation for subway managers has grown to nearly $300,000 a year. That's a pretty incredible quote from the article above, since it's an average...

79 points by dthal 9y ago

This is a good move. I hope other states follow suit. I highly recommend reading Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

an_cap 9y ago
I recommend reading this review before deciding to read the book. https://thezvi.wordpress.com/2017/06/04/book-review-weapons-of-math-destruction/ https://thezvi.wordpress.com/2017/06/04/book-review-weapons-...
- meebs 9y ago
  "Review," in which a privileged person mocks the idea of invisible power resulting in perpetuating inequality being worse than visible power.
- afsina 9y ago
  This was a good read, thanks.
afsina 9y ago
I have not read the book, but just the title reminded me I probably belong to a minority group that do not believe neither "equality" nor "democracy".

mc32 9y ago

This is a good thing --an algorithm with up to a 30 error rate is unreliable. One hopes this scrutiny will lead to development of more dependable and trustworthy algorithms. Also hopeful politicians don't have the ability to tweak things in order to fit their agendas. For example an algorithm is too good at detecting which gov't employee is stealing from the public. Or it tells them they should decrease fines in some area to get better results --but that would affect their budget adversely, etc.

candiodari 9y ago
This is an unrealistic law that doesn't matter. It sounds good, but it's futile. If this taskforce takes the decision that an algorithm that is working is unacceptable (which we all know they'll do on the worst possible basis - publicity), then what happens ? They'll pause the algorithm ? Replace it ? They can't do that ... The issue is that once you automate something you can't unautomate it without providing the workforce necessary. Because in nearly all cases a bad automated algorithm will far outperform 1% of the required workforce in humans, that's how algorithms win. Not because they beat humans, they don't. They win because they actually take action in 100% of the cases, whatever that number is. A dumb algorithm taking action 100.000 times can easily beat a very hardworking human that takes 100 smart decisions in a lot of cases. So that will be tens to hundreds of people in easy cases, and thousands to tens of thousands in bad cases, and this is New York, easily the size of a decent country. So it'll always be "we need to pause this NOW !", "OK, no problem at all, that'll be $15 million per hour. Under what budget item do we fit this ?" "Erm .... How about you just make it look like it's paused ?" Keep in mind stopping automation doesn't even just cause damage directly, it will also cause overloading costs onto other departments and even onto private companies. Often in surprising ways. Algorithms respond quickly, under nearly all circumstances, at any time. You wouldn't believe how efficient this makes interacting with organizations. Pausing automation has an enormous and accumulating cost, making the decision impossible. Also algorithms don't solve corruption (they may make it easier to track though, although there are ways around that).
- pjc50 9y ago
  Taking bad decisions faster is not an improvement. Especially when it's being used to wrongly convict people. > takes the decision that an algorithm that is working is unacceptable "Unacceptable" == "not working".
- matt4077 9y ago
  > which we all know they'll do on the worst possible basis - publicity That's such a cynical, nihilistic way of looking at government. I can only guess people arrive at it after being exposed to a rather superficial look at governments' work over a long time. In reality, the vast bureaucracy that is government takes thousands of actions every single day, almost all of which are uncontroversial. They work hard to establish procedures minimising uncertainty. The work is far more transparent than any private organisations'. And all decisions are subject to judicial review–with the judiciary having its own, long tradition of thoughtful deliberation and even-handedness. As one example, the list at https://www.regulations.gov/searchResults?rpp=25&so=DESC&sb=postedDate&po=0&np=90&dct=N%2BFR%2BPR https://www.regulations.gov/searchResults?rpp=25&so=DESC&sb=... shows some recent (federal) actions. Note that this list is only the tip of the iceberg, with the most controversial administration in modern history. Yet it is dominated by "Class E Airspace; Revocations: Eaton Rapids, MI" and other items of rather low publicity value.

patkai 9y ago

One of those paradoxical situations when something _really_ needs to be regulated but can't be regulated. E.g. how do you know what data set is the result of an algorithm (a dubious consolidation step?), or how do you regulate algorithmic learning when we don't fully understand how learning works?

majewsky 9y ago
> needs to be regulated but can't be regulated There is no binary choice between "not regulated at all" and "completely regulated". For example, just implementing a process for public scrutiny of algorithms and datasets may result in those building the algorithms and collecting the data becoming more aware of their biases. (That's "may" with a capital M, though.)
- candiodari 9y ago
  What you mean to say is that politicians currently take decisions on things they aren't even remotely qualified to even talk about, then force it through with heavy hand, and therefore this won't change a thing ? That's true, and despite that it will change things.
mannykannot 9y ago
If you look at the specific issue that provoked this law, you will see issues that can - and should - be regulated. the city was using DNA matching software that turns out to have demonstrable errors, yet the office of the medical examiner stonewalled all attempts to have it audited until, thankfully, the courts forced the issue. This sort of unjustifiable secrecy (the accused absolutely have a right to examine the premises of the accusation) can be regulated. Unfortunately, this law substitutes nebulous criteria which, no matter how worthy, are likely to turn a clear-cut situation into a tar-pit of legal wrangling that the victims cannot afford to enter. The chief medical examiner is still holding fast on the very dubious claim that these flaws raise no doubts about the convictions in other cases where it was used, another area where I think specific legislation is needed. > How do you regulate algorithmic learning when we don't fully understand how learning works? This was not one of those cases. There are, however, cases - and this would be one if it applied - where it is reasonable to say that you can't use it until you can explain how it works.

skate22 9y ago

I wonder how an algorithm with low explainability will be treated, like a large nueral network. Seems like it would be hard to pin it on the algorithm rather than the data.

vnchr 9y ago
If there's a way to objectively evaluate quality of output, that seems sufficient. It's not like a building inspection where it's a component-based evaluation. The results need to be reasonably and verifiably reliable to an appropriate degree, but the courts don't need to care how that is accomplished.
- AnthonyMouse 9y ago
  > If there's a way to objectively evaluate quality of output, that seems sufficient. But that's the whole problem. If you had something that calculates the the thing algorithm is supposed to do in a better way than the algorithm actually does it, you could just use that as the algorithm.

deleted 9y ago

[deleted]

SauciestGNU 9y ago

I think it's a travesty that the source code won't be shared with the public, but I hope it would at least be provided for review to the agencies adopting the algorithms. In my field we are highly regulated and must provide the regulatory agencies with an audit trail for all data. I would like to see an audit trail for these algorithms that would allow someone to follow the decision tree and review the outcome of the algorithms.

bob_theslob646 9y ago

I find this incredibly off putting since the city cannot even account for its own wasteful budget. There is nothing wrong with accountability and am glad they are starting somewhere but until they start holding others like the ones who purposesly destroy on time metrics of transit in order to keep thier salary gravy train choo-chooing, I am not holding my breath. https://mobile.nytimes.com/2017/11/18/nyregion/new-york-subway-system-failure-delays.html?referer=https://www.google.com/ https://mobile.nytimes.com/2017/11/18/nyregion/new-york-subw...

sologoub 9y ago
> Signal problems and car equipment failures occur twice as frequently as a decade ago, but hundreds of mechanic positions have been cut because there is not enough money to pay them — even though the average total compensation for subway managers has grown to nearly $300,000 a year. That's a pretty incredible quote from the article above, since it's an average...