5 ms·

Terrible piece. It makes me wonder if this is a ‘submarine’ from a proprietary AI provider that wants to push the narrative that open source AI tools need stron

Terrible piece. It makes me wonder if this is a ‘submarine’ from a proprietary AI provider that wants to push the narrative that open source AI tools need stronger regulation vs proprietary AI tools. “The scary part is how easy open source is to use”, and “a mistake in open source is much more difficult to correct then an algorithm from a - accountable - company.” Shows how little the author understands of this field. It won’t be tensorflow that’s biased against minorities in resume selection, more likely the data was biased.

phkahler 5y ago
Yeah, they conflate the software part of AI with the trained systems. They specifically mention introducing bias into open source code.
jhgb 5y ago
> "The scary part is how easy open source is to use" That sounds like a great advertisement for FLOSS, really.
throwjd764glug 5y ago
Yes, proprietary systems are so accountable. It only took "the prosecution and conviction of hundreds of [innocent] sub-postmasters for alleged theft, false accounting and/or fraud, resulting in imprisonment, loss of reputation and livelihood, bankruptcy, divorce, and even suicide amongst those involved" before the UK Post Office's proprietary Horizon system was found to be the cause. So far no-one responsible for the computer system or the prosecution has faced any repercussions. https://en.wikipedia.org/wiki/British_Post_Office_scandal https://en.wikipedia.org/wiki/British_Post_Office_scandal
garrinm 5y ago
In fact I see the opposite situation: a closed source algorithm will be protected as a trade secret. So identifying a bias might be harder. And correcting it anyway will still require them to ship a new version. Meanwhile in open source there could be many users raising issues if a bias is suspected. Many people working to fix it regardless of other priorities. And then an update will be pushed.
- AlanYx 5y ago
  Under the proposed EU AI regulation, there's actually a requirement for source code to be provided to the regulators to review. Presumably it would be subject to an NDA to address trade secrecy concerns.
  - mattlondon 5y ago
    It is the dataset used for training that is often at fault, not the source code of the model. I don't know if the EU regulations require access to the data too, but that could be problematic in a practical sense ( e.g. do they deliver terabytes of raw data to the regulators?) and there may be privacy issues with companies sharing their data with regulators
    - datastoat 5y ago
      > It is the dataset used for training that is often at fault, not the source code of the model This remark is absolutely spot on. This is why I don't like the language around 'algorithmic bias' and 'algorithmic accountability' -- it puts too much attention on The Algorithm, which lets big tech companies deflect us from scrutiny of their training data. Your comment about the conflict with privacy is also spot on. Here's a paper (shameless plug!) about it: Show Us the Data: Privacy, Explainability, and Why the Law Can't Have Both [1]. What's interesting about the GDPR is that it's not purely about what regulators do: it's legislation that grants rights to people, rights that can be litigated in court. If I as a GDPR data subject demand an explanation of an ML decision about me, as I'm entitled to do, my lawyer could argue "the source code isn't an adequate explanation, we want to see the training data too". Then it'd be up to the court to make a ruling about my lawyer's request, not up to the regulator. The paper [1], co-authored by a data scientist (myself) and a lawyer, thinks through the privacy / explainability conflict as it might play out in court. [1] https://www.gwlr.org/show-us-the-data/ https://www.gwlr.org/show-us-the-data/
AlanYx 5y ago
The critical element that's missing from the article--but which likely motivated the article--is that the EU's new AI regulation proposal mandates compliance all along the supply chain (for regulated applications of AI). In practice, this will likely mean that using open source for these AI applications will become a lot more difficult in a legal sense, perhaps impossible, because there's no single entity that can speak to compliance of open source components (vetting each commit, etc.). Perhaps some orgs will emerge who are willing to monitor and oversee open source software and certify that it meets the requirements of the legislation, but it's unclear how that would work. I'm not endorsing this model, just explaining the concern. I don't think the tech community really fully appreciates how misguided a lot of the current approaches to AI regulation among policymakers are right now. (Not just this, but initiatives like the current CAHAI work towards a binding international law instrument on AI.) And yes, like the GDPR, the proposed EU AI regulation has an extraterritorial effect. Under the proposed AI regulation, if you're selling into the EU or to any EU customers but don't have an establishment in the EU, you have to actually appoint a representative to comply with these requirements.
- adolph 5y ago
  Securing supply chain is an important effort even outside of regulation. Here is a podcast about some recent open source efforts: Kubernetes Podcast 155: Software Supply Chain Security, with Priya Wadhwa https://kubernetespodcast.com/episode/155-software-supply-chain-security/ https://kubernetespodcast.com/episode/155-software-supply-ch...
- riedel 5y ago
  The AI act is really a strange effort to regulate technology horizontally without even being able to define it's scope. Many who have read it believe it ends up regulating all computer logic in critical infrastructure (while making this definition wide as well) . I have tried to comment on this through various channels (local government/associations), because it seems a really strange "horizontal" regulation that could have huge side effects. I hope more people will point out the problems of the current proposal. I am really a proponent of both the GDPR and AI ethics for most parts but this one is really going too far .

phkahler 5y ago

Yeah, they conflate the software part of AI with the trained systems. They specifically mention introducing bias into open source code.

jhgb 5y ago

> "The scary part is how easy open source is to use" That sounds like a great advertisement for FLOSS, really.

throwjd764glug 5y ago

Yes, proprietary systems are so accountable. It only took "the prosecution and conviction of hundreds of [innocent] sub-postmasters for alleged theft, false accounting and/or fraud, resulting in imprisonment, loss of reputation and livelihood, bankruptcy, divorce, and even suicide amongst those involved" before the UK Post Office's proprietary Horizon system was found to be the cause. So far no-one responsible for the computer system or the prosecution has faced any repercussions. https://en.wikipedia.org/wiki/British_Post_Office_scandal https://en.wikipedia.org/wiki/British_Post_Office_scandal

garrinm 5y ago

In fact I see the opposite situation: a closed source algorithm will be protected as a trade secret. So identifying a bias might be harder. And correcting it anyway will still require them to ship a new version. Meanwhile in open source there could be many users raising issues if a bias is suspected. Many people working to fix it regardless of other priorities. And then an update will be pushed.

AlanYx 5y ago
Under the proposed EU AI regulation, there's actually a requirement for source code to be provided to the regulators to review. Presumably it would be subject to an NDA to address trade secrecy concerns.
- mattlondon 5y ago
  It is the dataset used for training that is often at fault, not the source code of the model. I don't know if the EU regulations require access to the data too, but that could be problematic in a practical sense ( e.g. do they deliver terabytes of raw data to the regulators?) and there may be privacy issues with companies sharing their data with regulators
  - datastoat 5y ago
    > It is the dataset used for training that is often at fault, not the source code of the model This remark is absolutely spot on. This is why I don't like the language around 'algorithmic bias' and 'algorithmic accountability' -- it puts too much attention on The Algorithm, which lets big tech companies deflect us from scrutiny of their training data. Your comment about the conflict with privacy is also spot on. Here's a paper (shameless plug!) about it: Show Us the Data: Privacy, Explainability, and Why the Law Can't Have Both [1]. What's interesting about the GDPR is that it's not purely about what regulators do: it's legislation that grants rights to people, rights that can be litigated in court. If I as a GDPR data subject demand an explanation of an ML decision about me, as I'm entitled to do, my lawyer could argue "the source code isn't an adequate explanation, we want to see the training data too". Then it'd be up to the court to make a ruling about my lawyer's request, not up to the regulator. The paper [1], co-authored by a data scientist (myself) and a lawyer, thinks through the privacy / explainability conflict as it might play out in court. [1] https://www.gwlr.org/show-us-the-data/ https://www.gwlr.org/show-us-the-data/

AlanYx 5y ago

The critical element that's missing from the article--but which likely motivated the article--is that the EU's new AI regulation proposal mandates compliance all along the supply chain (for regulated applications of AI). In practice, this will likely mean that using open source for these AI applications will become a lot more difficult in a legal sense, perhaps impossible, because there's no single entity that can speak to compliance of open source components (vetting each commit, etc.). Perhaps some orgs will emerge who are willing to monitor and oversee open source software and certify that it meets the requirements of the legislation, but it's unclear how that would work. I'm not endorsing this model, just explaining the concern. I don't think the tech community really fully appreciates how misguided a lot of the current approaches to AI regulation among policymakers are right now. (Not just this, but initiatives like the current CAHAI work towards a binding international law instrument on AI.) And yes, like the GDPR, the proposed EU AI regulation has an extraterritorial effect. Under the proposed AI regulation, if you're selling into the EU or to any EU customers but don't have an establishment in the EU, you have to actually appoint a representative to comply with these requirements.

adolph 5y ago
Securing supply chain is an important effort even outside of regulation. Here is a podcast about some recent open source efforts: Kubernetes Podcast 155: Software Supply Chain Security, with Priya Wadhwa https://kubernetespodcast.com/episode/155-software-supply-chain-security/ https://kubernetespodcast.com/episode/155-software-supply-ch...
riedel 5y ago
The AI act is really a strange effort to regulate technology horizontally without even being able to define it's scope. Many who have read it believe it ends up regulating all computer logic in critical infrastructure (while making this definition wide as well) . I have tried to comment on this through various channels (local government/associations), because it seems a really strange "horizontal" regulation that could have huge side effects. I hope more people will point out the problems of the current proposal. I am really a proponent of both the GDPR and AI ethics for most parts but this one is really going too far .