7 ms·

City of Amsterdam’s Algorithm Register

144 points by cpeterso 6y ago

technicolorwhat 6y ago
Wow cool to see my country is doing this. is actually pretty awesome. Do other governments also do this? There is even an architecture breakdown on the Api that does the scanning of the license plate for ticketing. https://algoritmeregister.amsterdam.nl/en/automated-parking-control/ https://algoritmeregister.amsterdam.nl/en/automated-parking-...
- tuukkah 6y ago
  Same service in Helsinki: https://ai.hel.fi/en https://ai.hel.fi/en
  - monkeydust 6y ago
    Interesting some good quality details into the service and Algo construction.
  - svdr 6y ago
    The sites look the same. Both are developed by https://www.saidot.ai/ https://www.saidot.ai/.
    - tuukkah 6y ago
      Yeah, happens to be a company from the Helsinki area.
    - anddt 6y ago
      I wonder what are the implications and liabilities to leave such a task to a private company. Let's say Sadiot overlooks some model behavior that leads to racial discrimination, who'd be liable? The government's data science team who set up the model or Saidot?
sho 6y ago
Quite interesting to see how the word "algorithm" seems to be changing over time from a straightforward mathematical or CS "series of steps to solve a problem" to a general byword for any sort of automated reasoning, especially ones with a more malevolent undertone. I'm actually kind of struggling to come up with a concise definition in the new popular usage.
- atoav 6y ago
  Isn't it still just "a series of steps to solve a problem", but just in a little broader sense? E.g. if your problem would be how to rate a person's need for social housing, you probably would apply some objective parameters to that. You are trying to get a binary answer that ideally selects all people who would need such a thing and leaves out all the others, without having much false negatives or false positives. Unlike in the traditional algorithm definition this involves bureaucratic and legal steps, some of which can be a bit handwavey and have their outcome depend on world image and daily form of the individual "processor" who was tasked to carry them out (better not have your task run right before the lunch break by grumpy Edwald). In other cases the step might be more of the nature "insert numbers in that one Excel sheet some wizard built aeons ago and now nobody understands". An algorithm however doesn't need to be a good algorithm or entirely deterministic to fit the definition of "algorithm". It is a series of steps carried out to solve a problem. A cooking recipe.
  - skrebbel 6y ago
    Modern media use of the word "algorithm" is almost exclusively about AI type of stuff. Recommenders, likelihood detectors, image recognizers etc. That's very much not a series of steps: it's primarily a function of the data that it had been fed. You can't summarize a training body of 50000 cat pictures as "a series of steps". And the steps are worthless without the training data.
    - bryanrasmussen 6y ago
      No but the algorithms used by AI to do recommendation etc. are generally referred to as algorithms, hence https://www.geeksforgeeks.org/top-10-algorithms-every-machine-learning-engineer-should-know/ https://www.geeksforgeeks.org/top-10-algorithms-every-machin...
jmats 6y ago
Here's a white paper on 'Public AI Registers' by the CEO of the company (Saidot) that runs the service for both Amsterdam and Helsinki (co-authored with representatives of both cities): https://uploads-ssl.webflow.com/5c8abedb10ed656ecfb65fd9/5f6f334b49d5444079726a79_AI%20Registers%20-%20White%20paper%201.0.pdf https://uploads-ssl.webflow.com/5c8abedb10ed656ecfb65fd9/5f6...
2rsf 6y ago
Really interesting and useful way of displaying information in a very non government-y way
anddt 6y ago
I tend to believe this is a valid approach for ML application in public institutions. Being ML a black box or a potential source of discrimination, setting up systems to ensure credibility within the public should be addressed early on in the design process (perhaps while designing the model itself). I wouldn't be surprised to see a future where all ML applications affecting citizens and the general public interest (excluding defense, military, etc.) will be obliged to be open-source and subject to public scrutiny.
- read_if_gay_ 6y ago
  > I wouldn't be surprised to see a future where all ML applications affecting citizens and the general public interest (excluding defense, military, etc.) will be obliged to be open-source and subject to public scrutiny. I would be positively surprised.
  - anddt 6y ago
    Sorry if it didn't come across in the post, it'd be no doubt a fair step towards more ethically sound governments.
  - bettem 6y ago
    In Australia, any Federal Government algorithm can be FOI’d by a member of the public. It’s one of the reasons the Aus Federal Government statisticians tend away from black box AI and towards more traditional methods.
    - anddt 6y ago
      > [...] Government statisticians tend away from black box AI and towards more traditional methods. I was indeed wondering how constraints on model interpretation forces governments to stay away from NN and more complex approaches. I have worked for a fraud-prevention company for a few years, and the financial nature of the decisions involved led to reward model interpretation (sometime at accuracy's expense) in order not to create friction with clients. Definitely not the best not being able to explain why your model performs poorly or what actions are needed to solve a problem.
jsumrall 6y ago
Since it's just copied (or bought) from Helsinki (https://ai.hel.fi/en/ai-register/ https://ai.hel.fi/en/ai-register/) / Saidot (Saidot.ai), I wonder if it's just something they bought but don't really care/understand what it's about or what they're putting there.
- iruoy 6y ago
  On the Saidot website it sais: > Saidot develops and operates the City of Amsterdam's Algorithm Register > Saidot develops and operates the City of Helsinki's AI Register I don't know how much they've poured into this, but I've worked on a project managed by Amsterdam Data [0] before and they're pretty serious about data. [0]: https://data.amsterdam.nl https://data.amsterdam.nl
  - LindaAms 6y ago
    Amsterdam and Helsinki BOTH have developed this register together with Saidot actually. We've build on our work on procurement conditions for AI, defining and operationalizing concepts like technical and procedural transparency (see https://www.amsterdam.nl/wonen-leefomgeving/innovatie/de-digitale-stad/grip-op-algoritmes/ https://www.amsterdam.nl/wonen-leefomgeving/innovatie/de-dig... at the bottom of the page) The development of these two registers has been a very deliberate process for both cities, to come up with a register that has technical detail as well as makes the algorithms understandable for the masses. If we can improve in any way, please leave your feedback in the survey, it's most welcome!
avar 6y ago
Regarding the automatic parking enforcement they claim, on the topic of non-discrimination: > The service works the same way for all license plates regardless of the car model, age, or the owner’s profile. This whole thing seems to be designed to give the appearance of government transparency without any of the transparency. I bet if they released some "raw" (but anonymized) data from their collection it would be easy to find evidence of discrimination. E.g. some of the richest parts of the city have some of the smallest parking spots. Surely cars bunched up against each other lowers the accuracy rate. Same with certain car models with recessed areas holding the number plate, which again will have socio-economic correlations. I don't see how you could plausibly design a system like this without any discrimination built into it. But instead they just say there's no discrimination, without providing any of the source code, models or other data for the system. If they actually found that it was discriminatory does anyone think Amsterdam's first task would be to change its website to say "we're discriminating!", or would they just quietly try to fix it? That's the issue with these sorts of systems. At least if one poor neighborhood is full of parking attendants with none in the adjacent rich neighborhood people will notice sooner than later. You have almost none of that basic sorts of transparency with automated scanning.
- bluesign 6y ago
  I don’t see if one neighborhood is full of parking attendants and rich one is less proving discrimination. This is an optimization problem, you have limited capability of enforcement, and you need to use it with most efficient way.
  - IkmoIkmo 6y ago
    > I don’t see if one neighborhood is full of parking attendants and rich one is less proving discrimination. Suppose only a poor neighbourhood has laws enforced, while in another there is no enforcement. Would that be discriminatory? I think in effect, absolutely. In terms of intentions, it remains to be seen what the reasons were, it could've been a mistake or a matter of optimising resources. But there are clearly limits to this argument, and it is also clearly discriminatory in practice. For example, racial profiling is very much a form of discrimination. Suppose two ethnicities live in a city, one (A) has a 0.5% serious crime rate, the other (B) has a 2% serious crime rate, on average, and say equal chance of minor offences. Profiling citizens of group B more often, e.g. stop & frisk, would be a more optimal use of police resources than random selection, to stop serious crimes. Yet, 98% of group B is innocent of serious crimes and is being subjected to more enforcement, which lead to a higher than proportionate conviction rate of minor offences, simply for belonging to a different ethnic group. It also leads to more run-ins with the police, which disturbs your day, risks escalation, makes you feel like a second-class citizen etc. That's discrimination, because the 98% innocent individuals from group B are being treated more harshly without cause, purely on the basis of their belonging to a group. Discrimination is when an individual is maltreated on the basis of belonging to a group, rather than as an individual. Yet also 'more optimal/efficient use of resources'. They're not mutually exclusive. And that's true not just within policing, by the way. If you know that 80% of ethnic group A have empirically been good hires, and 70% of ethnic group B, in the past, and suppose you had many people to choose from to invite for an interview, and then interview. Suppose you have limited resources and selection & interviewing is time-consuming. Does that allow you to 'optimize use of scarce resources and efficiently reject all applicants from group B'? No. It'd be perhaps efficient, but also very much discriminatory. Of course if you have two neighbourhoods and one has rampant parking abuse, the other not, then indeed it does make sense to allocate enforcement capacity accordingly. But only to a point. And that point must be evaluated. And whether you've reached that point, will be indicated by the data. That data isn't available, yet it's a priori stated that the parking enforcement cannot be discriminatory. OP noted a few possibilities that could indicate it is discriminatory, but we'd need to see the data to draw any conclusions. I think he's right in saying that.
mettamage 6y ago
What is the intent of an algorithm register? And why Amsterdam? Isn't this more something that should be on an international level?
- jiofih 6y ago
  This kind of question gives me strong flashbacks of corporate meetings (: Transparency. It’s Amsterdam because the city of Amsterdam decided to publish their algorithms in use. They do the same for all kinds of data and systems already, see data.amsterdam.nl
- LindaAms 6y ago
  Amsterdam is trying to set a precedent. By developing this register, they're showing national/international policy makers that transparency can be done. Aside from the fact that any level of gov should be open about how technology affects their citizens.
endymi0n 6y ago
Nice try, I like the gesture. Count me extremely skeptical though. You can get decent to really good tabular learning right off the shelf and it‘s so good and ubiquitous by now that interestingly enough, a few things start to happen that imho might make this effort close to impossible: - The differences between different algorithms get ever smaller, leaving more and more weight of any machine driven decision to the actual input data itself, including proper labeling, preparation, feature engineering, QA and debiasing. You can‘t properly separate the data from the algorithm itself anymore these days. - Even IF you added all this data together with the algorithm in a register (which brings its own privacy and practicality concerns), with the massive breadth and width of modern data sources, most data scientists don‘t even understand model decisions by themselves anymore. You‘ll need to dig deep into the explainability toolbox to be able to at least partially explain them. - There are dozens of documented cases where algorithms just mirror and regurgitate human biases through data. Algorithms by themselves are (mostly) innocuous, but can get to make racist, sexist and populist decisions anyway, purely based on what they have trained on. So if you can‘t even trust the people who build the algorithms to understand their reasoning, what benefit could a publication possibly bring to any outsider?
- david_draco 6y ago
  An outsider can throw test data points at the model and see how it behaves. For example, whether changing name, race, sex, or variables correlating with such critical variables, alter a decision. Pointing out these flaws could then lead to better systems, or not using machine learning on some case where input data is unsuitable.
jonkoops 6y ago
Nice to see my employer getting posted for this kind of stuff. In case you are interested we try to open-source as much as possible so check out our Github: https://github.com/amsterdam https://github.com/amsterdam
jurbaas 6y ago
Nice to see this picked up! We're hiring technical people at the municipality, reach out to me via my profile if you're interested.
meerih 6y ago
So great to see this discussion. When developing these registers, the target has been to enable citizen participation and public oversight of city algorithms. And to help cities and other government organizations to systematically govern their algorithmic systems. With that target in mind, conversations like this, anchored on proper data on the systems themselves, really count. Thank you! I'd love to invite everyone to give feedback also via our surveys at both sites. Meeri / Saidot

144 points by cpeterso 6y ago

Wow cool to see my country is doing this. is actually pretty awesome. Do other governments also do this? There is even an architecture breakdown on the Api that does the scanning of the license plate for ticketing. https://algoritmeregister.amsterdam.nl/en/automated-parking-control/ https://algoritmeregister.amsterdam.nl/en/automated-parking-...

tuukkah 6y ago
Same service in Helsinki: https://ai.hel.fi/en https://ai.hel.fi/en
- monkeydust 6y ago
  Interesting some good quality details into the service and Algo construction.
- svdr 6y ago
  The sites look the same. Both are developed by https://www.saidot.ai/ https://www.saidot.ai/.
  - tuukkah 6y ago
    Yeah, happens to be a company from the Helsinki area.
  - anddt 6y ago
    I wonder what are the implications and liabilities to leave such a task to a private company. Let's say Sadiot overlooks some model behavior that leads to racial discrimination, who'd be liable? The government's data science team who set up the model or Saidot?

sho 6y ago

Quite interesting to see how the word "algorithm" seems to be changing over time from a straightforward mathematical or CS "series of steps to solve a problem" to a general byword for any sort of automated reasoning, especially ones with a more malevolent undertone. I'm actually kind of struggling to come up with a concise definition in the new popular usage.

atoav 6y ago
Isn't it still just "a series of steps to solve a problem", but just in a little broader sense? E.g. if your problem would be how to rate a person's need for social housing, you probably would apply some objective parameters to that. You are trying to get a binary answer that ideally selects all people who would need such a thing and leaves out all the others, without having much false negatives or false positives. Unlike in the traditional algorithm definition this involves bureaucratic and legal steps, some of which can be a bit handwavey and have their outcome depend on world image and daily form of the individual "processor" who was tasked to carry them out (better not have your task run right before the lunch break by grumpy Edwald). In other cases the step might be more of the nature "insert numbers in that one Excel sheet some wizard built aeons ago and now nobody understands". An algorithm however doesn't need to be a good algorithm or entirely deterministic to fit the definition of "algorithm". It is a series of steps carried out to solve a problem. A cooking recipe.
- skrebbel 6y ago
  Modern media use of the word "algorithm" is almost exclusively about AI type of stuff. Recommenders, likelihood detectors, image recognizers etc. That's very much not a series of steps: it's primarily a function of the data that it had been fed. You can't summarize a training body of 50000 cat pictures as "a series of steps". And the steps are worthless without the training data.
  - bryanrasmussen 6y ago
    No but the algorithms used by AI to do recommendation etc. are generally referred to as algorithms, hence https://www.geeksforgeeks.org/top-10-algorithms-every-machine-learning-engineer-should-know/ https://www.geeksforgeeks.org/top-10-algorithms-every-machin...

jmats 6y ago

Here's a white paper on 'Public AI Registers' by the CEO of the company (Saidot) that runs the service for both Amsterdam and Helsinki (co-authored with representatives of both cities): https://uploads-ssl.webflow.com/5c8abedb10ed656ecfb65fd9/5f6f334b49d5444079726a79_AI%20Registers%20-%20White%20paper%201.0.pdf https://uploads-ssl.webflow.com/5c8abedb10ed656ecfb65fd9/5f6...

2rsf 6y ago

Really interesting and useful way of displaying information in a very non government-y way

anddt 6y ago

I tend to believe this is a valid approach for ML application in public institutions. Being ML a black box or a potential source of discrimination, setting up systems to ensure credibility within the public should be addressed early on in the design process (perhaps while designing the model itself). I wouldn't be surprised to see a future where all ML applications affecting citizens and the general public interest (excluding defense, military, etc.) will be obliged to be open-source and subject to public scrutiny.

read_if_gay_ 6y ago
> I wouldn't be surprised to see a future where all ML applications affecting citizens and the general public interest (excluding defense, military, etc.) will be obliged to be open-source and subject to public scrutiny. I would be positively surprised.
- anddt 6y ago
  Sorry if it didn't come across in the post, it'd be no doubt a fair step towards more ethically sound governments.
- bettem 6y ago
  In Australia, any Federal Government algorithm can be FOI’d by a member of the public. It’s one of the reasons the Aus Federal Government statisticians tend away from black box AI and towards more traditional methods.
  - anddt 6y ago
    > [...] Government statisticians tend away from black box AI and towards more traditional methods. I was indeed wondering how constraints on model interpretation forces governments to stay away from NN and more complex approaches. I have worked for a fraud-prevention company for a few years, and the financial nature of the decisions involved led to reward model interpretation (sometime at accuracy's expense) in order not to create friction with clients. Definitely not the best not being able to explain why your model performs poorly or what actions are needed to solve a problem.

jsumrall 6y ago

Since it's just copied (or bought) from Helsinki (https://ai.hel.fi/en/ai-register/ https://ai.hel.fi/en/ai-register/) / Saidot (Saidot.ai), I wonder if it's just something they bought but don't really care/understand what it's about or what they're putting there.

iruoy 6y ago
On the Saidot website it sais: > Saidot develops and operates the City of Amsterdam's Algorithm Register > Saidot develops and operates the City of Helsinki's AI Register I don't know how much they've poured into this, but I've worked on a project managed by Amsterdam Data [0] before and they're pretty serious about data. [0]: https://data.amsterdam.nl https://data.amsterdam.nl
- LindaAms 6y ago
  Amsterdam and Helsinki BOTH have developed this register together with Saidot actually. We've build on our work on procurement conditions for AI, defining and operationalizing concepts like technical and procedural transparency (see https://www.amsterdam.nl/wonen-leefomgeving/innovatie/de-digitale-stad/grip-op-algoritmes/ https://www.amsterdam.nl/wonen-leefomgeving/innovatie/de-dig... at the bottom of the page) The development of these two registers has been a very deliberate process for both cities, to come up with a register that has technical detail as well as makes the algorithms understandable for the masses. If we can improve in any way, please leave your feedback in the survey, it's most welcome!

avar 6y ago

Regarding the automatic parking enforcement they claim, on the topic of non-discrimination: > The service works the same way for all license plates regardless of the car model, age, or the owner’s profile. This whole thing seems to be designed to give the appearance of government transparency without any of the transparency. I bet if they released some "raw" (but anonymized) data from their collection it would be easy to find evidence of discrimination. E.g. some of the richest parts of the city have some of the smallest parking spots. Surely cars bunched up against each other lowers the accuracy rate. Same with certain car models with recessed areas holding the number plate, which again will have socio-economic correlations. I don't see how you could plausibly design a system like this without any discrimination built into it. But instead they just say there's no discrimination, without providing any of the source code, models or other data for the system. If they actually found that it was discriminatory does anyone think Amsterdam's first task would be to change its website to say "we're discriminating!", or would they just quietly try to fix it? That's the issue with these sorts of systems. At least if one poor neighborhood is full of parking attendants with none in the adjacent rich neighborhood people will notice sooner than later. You have almost none of that basic sorts of transparency with automated scanning.

bluesign 6y ago
I don’t see if one neighborhood is full of parking attendants and rich one is less proving discrimination. This is an optimization problem, you have limited capability of enforcement, and you need to use it with most efficient way.
- IkmoIkmo 6y ago
  > I don’t see if one neighborhood is full of parking attendants and rich one is less proving discrimination. Suppose only a poor neighbourhood has laws enforced, while in another there is no enforcement. Would that be discriminatory? I think in effect, absolutely. In terms of intentions, it remains to be seen what the reasons were, it could've been a mistake or a matter of optimising resources. But there are clearly limits to this argument, and it is also clearly discriminatory in practice. For example, racial profiling is very much a form of discrimination. Suppose two ethnicities live in a city, one (A) has a 0.5% serious crime rate, the other (B) has a 2% serious crime rate, on average, and say equal chance of minor offences. Profiling citizens of group B more often, e.g. stop & frisk, would be a more optimal use of police resources than random selection, to stop serious crimes. Yet, 98% of group B is innocent of serious crimes and is being subjected to more enforcement, which lead to a higher than proportionate conviction rate of minor offences, simply for belonging to a different ethnic group. It also leads to more run-ins with the police, which disturbs your day, risks escalation, makes you feel like a second-class citizen etc. That's discrimination, because the 98% innocent individuals from group B are being treated more harshly without cause, purely on the basis of their belonging to a group. Discrimination is when an individual is maltreated on the basis of belonging to a group, rather than as an individual. Yet also 'more optimal/efficient use of resources'. They're not mutually exclusive. And that's true not just within policing, by the way. If you know that 80% of ethnic group A have empirically been good hires, and 70% of ethnic group B, in the past, and suppose you had many people to choose from to invite for an interview, and then interview. Suppose you have limited resources and selection & interviewing is time-consuming. Does that allow you to 'optimize use of scarce resources and efficiently reject all applicants from group B'? No. It'd be perhaps efficient, but also very much discriminatory. Of course if you have two neighbourhoods and one has rampant parking abuse, the other not, then indeed it does make sense to allocate enforcement capacity accordingly. But only to a point. And that point must be evaluated. And whether you've reached that point, will be indicated by the data. That data isn't available, yet it's a priori stated that the parking enforcement cannot be discriminatory. OP noted a few possibilities that could indicate it is discriminatory, but we'd need to see the data to draw any conclusions. I think he's right in saying that.

mettamage 6y ago

What is the intent of an algorithm register? And why Amsterdam? Isn't this more something that should be on an international level?

jiofih 6y ago
This kind of question gives me strong flashbacks of corporate meetings (: Transparency. It’s Amsterdam because the city of Amsterdam decided to publish their algorithms in use. They do the same for all kinds of data and systems already, see data.amsterdam.nl
LindaAms 6y ago
Amsterdam is trying to set a precedent. By developing this register, they're showing national/international policy makers that transparency can be done. Aside from the fact that any level of gov should be open about how technology affects their citizens.

endymi0n 6y ago

Nice try, I like the gesture. Count me extremely skeptical though. You can get decent to really good tabular learning right off the shelf and it‘s so good and ubiquitous by now that interestingly enough, a few things start to happen that imho might make this effort close to impossible: - The differences between different algorithms get ever smaller, leaving more and more weight of any machine driven decision to the actual input data itself, including proper labeling, preparation, feature engineering, QA and debiasing. You can‘t properly separate the data from the algorithm itself anymore these days. - Even IF you added all this data together with the algorithm in a register (which brings its own privacy and practicality concerns), with the massive breadth and width of modern data sources, most data scientists don‘t even understand model decisions by themselves anymore. You‘ll need to dig deep into the explainability toolbox to be able to at least partially explain them. - There are dozens of documented cases where algorithms just mirror and regurgitate human biases through data. Algorithms by themselves are (mostly) innocuous, but can get to make racist, sexist and populist decisions anyway, purely based on what they have trained on. So if you can‘t even trust the people who build the algorithms to understand their reasoning, what benefit could a publication possibly bring to any outsider?

david_draco 6y ago
An outsider can throw test data points at the model and see how it behaves. For example, whether changing name, race, sex, or variables correlating with such critical variables, alter a decision. Pointing out these flaws could then lead to better systems, or not using machine learning on some case where input data is unsuitable.

jonkoops 6y ago

Nice to see my employer getting posted for this kind of stuff. In case you are interested we try to open-source as much as possible so check out our Github: https://github.com/amsterdam https://github.com/amsterdam

jurbaas 6y ago

Nice to see this picked up! We're hiring technical people at the municipality, reach out to me via my profile if you're interested.

meerih 6y ago

So great to see this discussion. When developing these registers, the target has been to enable citizen participation and public oversight of city algorithms. And to help cities and other government organizations to systematically govern their algorithmic systems. With that target in mind, conversations like this, anchored on proper data on the systems themselves, really count. Thank you! I'd love to invite everyone to give feedback also via our surveys at both sites. Meeri / Saidot