5 ms·

UK passport photo checker shows bias against dark-skinned women

37 points by asplake 6y ago

NoOneCaresAtAll 6y ago
This is a tragedy ^ and I Cannot for the life of me Figure out why HackerNews Would downvote this comment
draugadrotten 6y ago
solution is clear: do not use white racist computers to evaluate passport photos. joking aside - better testing might help
- ben_w 6y ago
  Clear but not easy. I’d be unpleasantly surprised if this is caused by someone having a line saying “if black & rand()<0.15 { reject() }”… although given the recent thing with the 2^16-1 Excel limit for Covid reporting, it certainly isn’t impossible for a U.K. system to be that badly wrong.
  - iso947 6y ago
    Just because you aren’t explicitly racist doesn’t mean you have unconscious bias that you need to check and control. The same applies to computer algorithms. The real danger is when those algorithms are treated as without bias.
    - ben_w 6y ago
      Hence me writing “Clear but not easy.”
dutchmartin 6y ago
This looks oddly familiar to the medical field. There they often only test on men (1). Do here we see the classic example of not having tested the product on all possible clients. Also, if they had known about this flaw, they probably had been able to build a submit for manual review option. (1) https://www.theguardian.com/lifeandstyle/2015/apr/30/fda-clinical-trials-gender-gap-epa-nih-institute-of-medicine-cardiovascular-disease https://www.theguardian.com/lifeandstyle/2015/apr/30/fda-cli...
deleted 6y ago
[deleted]
bitcharmer 6y ago
Can someone with facial recognition experience chime in? I always thought that it's just harder to extract facial features from an objectively darker image. Or is it real bias in the algo or the training set of photos?
- mschuster91 6y ago
  > I always thought that it's just harder to extract facial features from an objectively darker image. Amateur photographer here. The key thing is to ask why the image of PoC is "objectively darker", and the answer to that is because exposure meters/auto programs in cameras are calibrated for correctly exposing white people.
  - verroq 6y ago
    Um no? For more colour correction, you can use the vectorscope to target the universal skin tone line which fixes contrast for all skin tones.
    - mschuster91 6y ago
      I'm talking about these "photo booths". A professional photographer of course can do a color correction in Photoshop, but there is no human involved in these.
mschuster91 6y ago
> Dark-skinned women are told their photos are poor quality 22% of the time, while the figure for light-skinned women is 14% That doesn't surprise me even a tiny bit, given that the problem will already present itself in the camera - exposure settings are calibrated for white-ish people, and automated photo booths don't have a human to correct it or offer multiple different exposures to the user. Scanning the photo has the same problem.
- fogihujy 6y ago
  There is definitely something to be said about the quality of pictures taken in photo booths, but the study specifically used pictures that were all of sufficiently high quality: > Photos of politicians were collected from parliaments around the world. The gender and skin tone were recorded for each photo using the Fitzpatrick skin-tone scale. Each image met the Home Office standards for a passport photo. That definitely suggests the software used for automatic image quality check simply isn't up to par.
- jmkd 6y ago
  While this sounds plausible - yet simply shifts the blame for the bias elsewhere in the technology supply chain - it does not adequately explain the experience of Elaine Owusu cited in the article. Her photo shows a clear-as-day mouth closed, which was repeatedly interpreted by the system as mouth open. No exposure settings would change that outcome.
slg 6y ago
As we turn more of our lives over to the results of these machine learning algorithms, we need to keep reminding developers that any system that is not designed specifically to be anti-racist can still end up racist due to poor training data. There was a similar issue just a couple weeks back with Twitter. [1] We can't just ignore these issues and assume that an algorithm is inherently unbiased. [1] - https://news.ycombinator.com/item?id=24530968 https://news.ycombinator.com/item?id=24530968
- mongol 6y ago
  I think calling it racist is making a wrong connotation. It is introducing a bias. Racism is something else.
  - iso947 6y ago
    I’m not racist, I’m just biased against people with dark skin?
    - mongol 6y ago
      Are you trained with poor training data? My comment was on parent's statement. I did not say racism does not exist, but it is not the only word for situations where there is a bias involving pepole's color.
      - slg 6y ago
        This is bias against people with darker skin that disadvantages their lives in a way that people with lighter skin do not experience. How is that not an example of racism?
        mongol 6y ago
        Equalling racism with anything that disadvantages pepole of a skin color, regardless of cause, devalues the word. Racism used to mean that there was a belief that certain races were superior to others, or a belief of that nature involved. It is simply not accurate to apply it to any circumstance regarding discrimination of skin color. You need to look at if there was an intent or a conscious prejudice.
verroq 6y ago
Or maybe it's harder to find contrast in a darker skinned face in bad lighting.
Bang2Bay 6y ago
either in-competent programmers that coded the system or bad training data that only had non-dark skinned pictures for training the system
jlawson 6y ago
They seem to simply take it on faith that it's impossible for darker-skinned people to just have a higher chance of taking poor photos. This could be mediated by culture of income or a hundred other things. They offer no way to evaluate the statistical validity of the conclusions drawn from "BBC Research". They looked at 1000 politicians, but what was the color breakdown of those? What was the actual sample size of dark-skinned people? Who coded the photos' skin colors and how? This is just utter garbage without any rigor at all. Their "BBC Research" indicates that male faces read much better than female, but they don't comment on this at all. Why did they choose the race angle and not the sex angle? Moreover there are real reasons why dark-skinned features are harder to read than light-skinned. It's the same reason things are hard to make out in the dark, or you can't tell the features of an object covered in vantablack. It's physics. Less light means less information. This whole article is the perfect distillation of low-effort motivated race baiting. To an astute reader it proves the opposite of its bias - if this is what is considered racial greivance worthy of breathless BBC articles, there must really be nothing significant going on to complain about. There is zero chance that anything like this would ever have been published if the results went the other way, no matter to what degree.
- another-dave 6y ago
  Definitely agree the level of detail in the article wouldn't get published in an academic journal, though they do mention: "Kirstie Whitaker, programme lead for tools, practices and systems at the Alan Turing Institute, reviewed our code, independently reproduced our results, and provided input on the validity of the reported outcomes." Would be interesting to see a proper write-up of the same study. That said, let's say that it is taken as given that it's harder for a ML set-up to determine good photos for dark skin. Should it be used at all then? Is it acceptable that it's harder for the ~15% of the population to get a passport than the rest of the population because of an algorithm in the loop? I don't think you can put it down to other factors like socio-economics as the Home Office reproduced this in their user testing before launch and acknowledged as a problem of the system (Vs the photos)
- deleted 6y ago
  [deleted]
- 6y ago
throw93232 6y ago
Is this really a bias? Or maybe they are less facial features and less contrast, that make detection harder? For example there is only one eye color and hair color.. And it was fixed with simple appeal. Rant start > I think this just derails discussion about how system is really broken. My friend applied for UK residency around Christmas. She matches all criteria, her husband studied and now works in UK, she is EU resident. Border service holds her passport for 11 months now!!! Only info we can get is automated paid call, for 15 euro they tell us it is in progress... Meanwhile she got pregnant and is about to deliver. All faster before border service can even answer single question... She is about to deliver, 2000 km from her husband, and can not even leave this country, because UK is holding her passport. And yes, she happens to be black as well... ...rant end
- tempodox 6y ago
  > Is this really a bias? Not making your software function properly for all the legitimate input it has to deal with is a bias. Not working to eliminate that bias because it affects a group you don't happen to be part of could even be called racist.
  - collyw 6y ago
    The vast majority of software has bugs especially around edge cases that aren't frequently encountered. Sometimes it isn't even worth fixing somethings as they are expected so infrequently, it's easier to fix things by hand or via some other hack.
    - tempodox 6y ago
      Are you asserting that people with dark skin need photo ID more “infrequently” than others? That it “isn't even worth fixing”? Then that's how the bias in your thinking translates directly into bad and racist software. You are arguing for the developer's convenience. Good software is about providing functionality to users, not developer's convenience.
      - 6y ago
bjornedstrom 6y ago
These kinds of issues are not unique to dark skin. Anecdote time: :-) As a Swedish/Finnish man (and a somewhat stereotypical nerd at that), I'm paler than probably 95% of the world population. Although facial recognition features (in the sense of trained AI models) often work fine on me given a good quality underlying photo, what often fails instead are preprocessing steps or color correction, especially in real-time systems. This can be the camera itself or software handling the image later. For example if I'm in a Google Hangout in a normally lit room and then I move into slight sunlight or a lamp, my face will overexpose into a blob that looks like a low quality photo of the sun. It renders white, like #ffffff proper white. It's not actually overexposed at the hardware/camera level, but the software "corrects" it that way. This typically never happens to other people.
- jmkd 6y ago
  While this may represent a failure of those systems to adequately cope with data outside of typical ranges, your anecdote is orthogonal to the notion that the system linked in this thread did not appear to place dark-skinned women firmly within those ranges, and thus represents a form of institutional bias, experienced as racism. Your inconvenient experience differs at a material level, because as a white-skinned man, many other systems are optimised for you.
- raxxorrax 6y ago
  This is an awesome feature. Of your face, I mean, not of the recognition software. At this points these complains are bullshit in my opinion. It shifts the debate about if we want to accept facial recognition to the side. It is like a free gift to those that sell these systems. So if facial recognition is actually bad for minorities, people bringing up these topics should not be their friends. My opinion, I know it is a strong one, but it is also more correct. Is a passport checker even needed for security would be a better discussion topic.
rswskg 6y ago
The same app couldn't detect my babies head. He's white. Even with two sets of image correction in the photo shop, we couldn't get it work. At what point can you say it's not racist, just not very good?

37 points by asplake 6y ago

This is a tragedy ^ and I Cannot for the life of me Figure out why HackerNews Would downvote this comment

solution is clear: do not use white racist computers to evaluate passport photos. joking aside - better testing might help

ben_w 6y ago
Clear but not easy. I’d be unpleasantly surprised if this is caused by someone having a line saying “if black & rand()<0.15 { reject() }”… although given the recent thing with the 2^16-1 Excel limit for Covid reporting, it certainly isn’t impossible for a U.K. system to be that badly wrong.
- iso947 6y ago
  Just because you aren’t explicitly racist doesn’t mean you have unconscious bias that you need to check and control. The same applies to computer algorithms. The real danger is when those algorithms are treated as without bias.
  - ben_w 6y ago
    Hence me writing “Clear but not easy.”

dutchmartin 6y ago

This looks oddly familiar to the medical field. There they often only test on men (1). Do here we see the classic example of not having tested the product on all possible clients. Also, if they had known about this flaw, they probably had been able to build a submit for manual review option. (1) https://www.theguardian.com/lifeandstyle/2015/apr/30/fda-clinical-trials-gender-gap-epa-nih-institute-of-medicine-cardiovascular-disease https://www.theguardian.com/lifeandstyle/2015/apr/30/fda-cli...

deleted 6y ago

[deleted]

bitcharmer 6y ago

Can someone with facial recognition experience chime in? I always thought that it's just harder to extract facial features from an objectively darker image. Or is it real bias in the algo or the training set of photos?

mschuster91 6y ago
> I always thought that it's just harder to extract facial features from an objectively darker image. Amateur photographer here. The key thing is to ask why the image of PoC is "objectively darker", and the answer to that is because exposure meters/auto programs in cameras are calibrated for correctly exposing white people.
- verroq 6y ago
  Um no? For more colour correction, you can use the vectorscope to target the universal skin tone line which fixes contrast for all skin tones.
  - mschuster91 6y ago
    I'm talking about these "photo booths". A professional photographer of course can do a color correction in Photoshop, but there is no human involved in these.

mschuster91 6y ago

> Dark-skinned women are told their photos are poor quality 22% of the time, while the figure for light-skinned women is 14% That doesn't surprise me even a tiny bit, given that the problem will already present itself in the camera - exposure settings are calibrated for white-ish people, and automated photo booths don't have a human to correct it or offer multiple different exposures to the user. Scanning the photo has the same problem.

fogihujy 6y ago
There is definitely something to be said about the quality of pictures taken in photo booths, but the study specifically used pictures that were all of sufficiently high quality: > Photos of politicians were collected from parliaments around the world. The gender and skin tone were recorded for each photo using the Fitzpatrick skin-tone scale. Each image met the Home Office standards for a passport photo. That definitely suggests the software used for automatic image quality check simply isn't up to par.
jmkd 6y ago
While this sounds plausible - yet simply shifts the blame for the bias elsewhere in the technology supply chain - it does not adequately explain the experience of Elaine Owusu cited in the article. Her photo shows a clear-as-day mouth closed, which was repeatedly interpreted by the system as mouth open. No exposure settings would change that outcome.

slg 6y ago

As we turn more of our lives over to the results of these machine learning algorithms, we need to keep reminding developers that any system that is not designed specifically to be anti-racist can still end up racist due to poor training data. There was a similar issue just a couple weeks back with Twitter. [1] We can't just ignore these issues and assume that an algorithm is inherently unbiased. [1] - https://news.ycombinator.com/item?id=24530968 https://news.ycombinator.com/item?id=24530968

mongol 6y ago
I think calling it racist is making a wrong connotation. It is introducing a bias. Racism is something else.
- iso947 6y ago
  I’m not racist, I’m just biased against people with dark skin?
  - mongol 6y ago
    Are you trained with poor training data? My comment was on parent's statement. I did not say racism does not exist, but it is not the only word for situations where there is a bias involving pepole's color.
    - slg 6y ago
      This is bias against people with darker skin that disadvantages their lives in a way that people with lighter skin do not experience. How is that not an example of racism?
      - mongol 6y ago
        Equalling racism with anything that disadvantages pepole of a skin color, regardless of cause, devalues the word. Racism used to mean that there was a belief that certain races were superior to others, or a belief of that nature involved. It is simply not accurate to apply it to any circumstance regarding discrimination of skin color. You need to look at if there was an intent or a conscious prejudice.

verroq 6y ago

Or maybe it's harder to find contrast in a darker skinned face in bad lighting.

Bang2Bay 6y ago

either in-competent programmers that coded the system or bad training data that only had non-dark skinned pictures for training the system

jlawson 6y ago

They seem to simply take it on faith that it's impossible for darker-skinned people to just have a higher chance of taking poor photos. This could be mediated by culture of income or a hundred other things. They offer no way to evaluate the statistical validity of the conclusions drawn from "BBC Research". They looked at 1000 politicians, but what was the color breakdown of those? What was the actual sample size of dark-skinned people? Who coded the photos' skin colors and how? This is just utter garbage without any rigor at all. Their "BBC Research" indicates that male faces read much better than female, but they don't comment on this at all. Why did they choose the race angle and not the sex angle? Moreover there are real reasons why dark-skinned features are harder to read than light-skinned. It's the same reason things are hard to make out in the dark, or you can't tell the features of an object covered in vantablack. It's physics. Less light means less information. This whole article is the perfect distillation of low-effort motivated race baiting. To an astute reader it proves the opposite of its bias - if this is what is considered racial greivance worthy of breathless BBC articles, there must really be nothing significant going on to complain about. There is zero chance that anything like this would ever have been published if the results went the other way, no matter to what degree.

another-dave 6y ago
Definitely agree the level of detail in the article wouldn't get published in an academic journal, though they do mention: "Kirstie Whitaker, programme lead for tools, practices and systems at the Alan Turing Institute, reviewed our code, independently reproduced our results, and provided input on the validity of the reported outcomes." Would be interesting to see a proper write-up of the same study. That said, let's say that it is taken as given that it's harder for a ML set-up to determine good photos for dark skin. Should it be used at all then? Is it acceptable that it's harder for the ~15% of the population to get a passport than the rest of the population because of an algorithm in the loop? I don't think you can put it down to other factors like socio-economics as the Home Office reproduced this in their user testing before launch and acknowledged as a problem of the system (Vs the photos)
deleted 6y ago
[deleted]
6y ago

throw93232 6y ago

Is this really a bias? Or maybe they are less facial features and less contrast, that make detection harder? For example there is only one eye color and hair color.. And it was fixed with simple appeal. Rant start > I think this just derails discussion about how system is really broken. My friend applied for UK residency around Christmas. She matches all criteria, her husband studied and now works in UK, she is EU resident. Border service holds her passport for 11 months now!!! Only info we can get is automated paid call, for 15 euro they tell us it is in progress... Meanwhile she got pregnant and is about to deliver. All faster before border service can even answer single question... She is about to deliver, 2000 km from her husband, and can not even leave this country, because UK is holding her passport. And yes, she happens to be black as well... ...rant end

tempodox 6y ago
> Is this really a bias? Not making your software function properly for all the legitimate input it has to deal with is a bias. Not working to eliminate that bias because it affects a group you don't happen to be part of could even be called racist.
- collyw 6y ago
  The vast majority of software has bugs especially around edge cases that aren't frequently encountered. Sometimes it isn't even worth fixing somethings as they are expected so infrequently, it's easier to fix things by hand or via some other hack.
  - tempodox 6y ago
    Are you asserting that people with dark skin need photo ID more “infrequently” than others? That it “isn't even worth fixing”? Then that's how the bias in your thinking translates directly into bad and racist software. You are arguing for the developer's convenience. Good software is about providing functionality to users, not developer's convenience.
    - 6y ago

bjornedstrom 6y ago

These kinds of issues are not unique to dark skin. Anecdote time: :-) As a Swedish/Finnish man (and a somewhat stereotypical nerd at that), I'm paler than probably 95% of the world population. Although facial recognition features (in the sense of trained AI models) often work fine on me given a good quality underlying photo, what often fails instead are preprocessing steps or color correction, especially in real-time systems. This can be the camera itself or software handling the image later. For example if I'm in a Google Hangout in a normally lit room and then I move into slight sunlight or a lamp, my face will overexpose into a blob that looks like a low quality photo of the sun. It renders white, like #ffffff proper white. It's not actually overexposed at the hardware/camera level, but the software "corrects" it that way. This typically never happens to other people.

jmkd 6y ago
While this may represent a failure of those systems to adequately cope with data outside of typical ranges, your anecdote is orthogonal to the notion that the system linked in this thread did not appear to place dark-skinned women firmly within those ranges, and thus represents a form of institutional bias, experienced as racism. Your inconvenient experience differs at a material level, because as a white-skinned man, many other systems are optimised for you.
raxxorrax 6y ago
This is an awesome feature. Of your face, I mean, not of the recognition software. At this points these complains are bullshit in my opinion. It shifts the debate about if we want to accept facial recognition to the side. It is like a free gift to those that sell these systems. So if facial recognition is actually bad for minorities, people bringing up these topics should not be their friends. My opinion, I know it is a strong one, but it is also more correct. Is a passport checker even needed for security would be a better discussion topic.

rswskg 6y ago

The same app couldn't detect my babies head. He's white. Even with two sets of image correction in the photo shop, we couldn't get it work. At what point can you say it's not racist, just not very good?