7 ms·

Help EFF Track the Progress of AI and Machine Learning

238 points by sinak 9y ago

startupdiscuss 9y ago
Part of what we are learning, in AI/ML, is the number of problems where the ML is relevant and where it is not. (Or, to put it another way, where we can find statistical relevance between features and targets). So I think you need some kind of "meta-metric" that measures the growth of the taxonomy itself. And perhaps some kind of weighting for the impact of the solution. There is also an interaction effect (for instance, Natural Language Processing is powerful, and "common sense reasoning" is powerful, but put them together and you have a knockout), but I don't know how to go about measuring that.
- daveguy 9y ago
  I think you'd have to measure "common sense reasoning" first. With NLP you can have accuracy/precision scores. With common sense it's almost equivalent to "what you need to beat the turning test" and not easily measured. I think if you get that part you essentially have the NLP. In that those last few percent error of NLP are generally "common sense" issues.
- pde3 9y ago
  Yes, we've also thought about the "meta metric" idea. We may try to make one if we feel that the taxonomy is at all complete and trustworthy.
samstave 9y ago
Who is, or is seeking to be, the canonical source of truth for AI, ML etc policy/position/etc and why should/how do we trust them? In twenty years, what body will be directing the policies and laws regulations etc WRT to how humanity deals with essentially what is another "sentient" species? Edit.. just read this and apparently this is exactly what the EFF is attempting to do... But the question still remains: how do we trust these policies, how do we request/reject them? I don't want to deal with this the same way the legal system is currently set up, lawyers and the law is flawed in many respects and I don't think it's a good idea to map the old to the new and uncharted directly. (Apologies for the clunky language/terms.. please educate me on how to speak of this if you know)
- pde3 9y ago
  You're asking great questions :) EFF isn't necessarily a fan of the way that current institutions of governance or the law operate, but when those institutions attempt to interfere with the development of technology, we step in to try to mitigate the damage and make the case for sensible outcomes. In the case of general-purpose human level AI, which to be clear is an extremely speculative kind of technology that might not happen in our lifetimes, I don't think anybody knows how humanity would deal with it. If it does happen, I think the biggest responsibility of participants in that process would be to minimize the risk of instability and conflict while humans and the new species (possibly species, plural; possibly not a species at all), figured out how to relate to each other. How best to accomplish that is largely a very difficult and mostly unanswered research question, though you can find some pointers to some interesting early work in the safety section of the Notebook.
  - samstave 9y ago
    If we cant determine when/if we will get on parity with humans in AI capability... we should be focused on AI as a tool that will be used by those with the know-how-resources against those who do not have ability/access; Rich vs poor: * HFT will never be a common man's tool * Government surveillance * Corporate surveillance * Behaviorally informed/adjusted pricing etc... Basically ML and AI will be used, in large part, by those who can to exploit those who cannot (or at least those who wont) defend themselves. There will be no opting out. So, with that said, if we go from the bottom up - and there is no opting out, then what is the best that one can hope for? I'd say complete ownership and control of one's own "meta-cloud" -- Any data that is a resultant trail of any action I take as an individual should be owned by me, and I should be able to see it all, and delete or block it. Or, in the extreme case, shouldn't I be able to require that any information presented to me (such as an ad or a price) be required to inform me as to how that information was formulated: "You are seeing this price because the following factors were analysed..." The real question is, in the future, is there even such a concept as "off grid"
tectonic 9y ago
Nice use of a shared Jupyter Notebook for data gathering. https://www.eff.org/ai/metrics https://www.eff.org/ai/metrics
- shmageggy 9y ago
  Nice that it's open and shared, but kinda weird that they are storing what is essentially tabular data in code. Looks to me like it would be much more readable if the data were in csv files, and just the graph generating code were in, well, code.
  - pde3 9y ago
    The measurements for each metric are tabular, but the taxonomy of problems above them is currently tree (or forest) structured. CSV export or import for specific metrics would be very easy to add if you'd like it. We already have a rough JSON export of the data: https://raw.githubusercontent.com/AI-metrics/AI-metrics/master/export-api/v01/progress.json https://raw.githubusercontent.com/AI-metrics/AI-metrics/mast...
comboy 9y ago
The project is very interesting, but I'm not sure why they are doing it. How is that protecting user rights? This doesn't measure AI/ML progress that's available to state actors.
- mlinksva 9y ago
  > EFF’s interest in AI progress is primarily from a policy perspective. We want to know what types of AI we need to start engaging with on legal, political, and technical safety fronts. Beyond that, we’re also just excited to see how many things computers are learning to do over time. > Given that machine learning tools and AI techniques are increasingly part of our everyday lives, it is critical that journalists, policy makers, and technology users understand the state of the field. When improperly designed or deployed, machine learning methods can violate privacy, threaten safety, and perpetuate inequality and injustice. Stakeholders must be able to anticipate such risks and policy questions before they arise, rather than playing catch-up with the technology. To this end, it’s part of the responsibility of researchers, engineers, and developers in the field to help make information about their life-changing research widely available and understandable. We hope you’ll join us.
  - comboy 9y ago
    Yes, it's not that I didn't read the page. But even in case of corporations, it's not really possible to gauge what t's available to them, because they don't share everything. Well on the other hand whatever is available publicly, they have at least that. So maybe the whole research does make some sense in this context.
    - mlinksva 9y ago
      Perhaps forcing governments and corporations to share more will be explored if it seems that this publicly documented baseline is far from the state of the art. Impossible to gauge, perhaps, but even realizing that highlights the risk of public ignorance.
      - 9y ago
arikr 9y ago
For ease of reading/UI, I think they should use a different color for the bars that represent "human score" and those that represent "excellent performance" - so that someone doesn't skim and assume they are the same thing, given that currently both are represented as red dotted lines but mean two different things.
DiabloD3 9y ago
Dupe: https://news.ycombinator.com/item?id=14603250 https://news.ycombinator.com/item?id=14603250
DanielBMarkham 9y ago
I believe the time has come for an independent institute to track AI and Big Data technology applications and begin creating guidelines for both industry self-certification and regulation. The other paths available, ignoring it, fearmongering about it, and trying to fit it into other political movements, do not seem to me to be heading towards an acceptable outcome. It has to be a technology-heavy group, otherwise it won't create much value. It also has to be grounded in history, philosophy, and political science, otherwise it'll just be reactionary. And we have enough reactionary groups already.

238 points by sinak 9y ago

Part of what we are learning, in AI/ML, is the number of problems where the ML is relevant and where it is not. (Or, to put it another way, where we can find statistical relevance between features and targets). So I think you need some kind of "meta-metric" that measures the growth of the taxonomy itself. And perhaps some kind of weighting for the impact of the solution. There is also an interaction effect (for instance, Natural Language Processing is powerful, and "common sense reasoning" is powerful, but put them together and you have a knockout), but I don't know how to go about measuring that.

daveguy 9y ago
I think you'd have to measure "common sense reasoning" first. With NLP you can have accuracy/precision scores. With common sense it's almost equivalent to "what you need to beat the turning test" and not easily measured. I think if you get that part you essentially have the NLP. In that those last few percent error of NLP are generally "common sense" issues.
pde3 9y ago
Yes, we've also thought about the "meta metric" idea. We may try to make one if we feel that the taxonomy is at all complete and trustworthy.

samstave 9y ago

Who is, or is seeking to be, the canonical source of truth for AI, ML etc policy/position/etc and why should/how do we trust them? In twenty years, what body will be directing the policies and laws regulations etc WRT to how humanity deals with essentially what is another "sentient" species? Edit.. just read this and apparently this is exactly what the EFF is attempting to do... But the question still remains: how do we trust these policies, how do we request/reject them? I don't want to deal with this the same way the legal system is currently set up, lawyers and the law is flawed in many respects and I don't think it's a good idea to map the old to the new and uncharted directly. (Apologies for the clunky language/terms.. please educate me on how to speak of this if you know)

pde3 9y ago
You're asking great questions :) EFF isn't necessarily a fan of the way that current institutions of governance or the law operate, but when those institutions attempt to interfere with the development of technology, we step in to try to mitigate the damage and make the case for sensible outcomes. In the case of general-purpose human level AI, which to be clear is an extremely speculative kind of technology that might not happen in our lifetimes, I don't think anybody knows how humanity would deal with it. If it does happen, I think the biggest responsibility of participants in that process would be to minimize the risk of instability and conflict while humans and the new species (possibly species, plural; possibly not a species at all), figured out how to relate to each other. How best to accomplish that is largely a very difficult and mostly unanswered research question, though you can find some pointers to some interesting early work in the safety section of the Notebook.
- samstave 9y ago
  If we cant determine when/if we will get on parity with humans in AI capability... we should be focused on AI as a tool that will be used by those with the know-how-resources against those who do not have ability/access; Rich vs poor: * HFT will never be a common man's tool * Government surveillance * Corporate surveillance * Behaviorally informed/adjusted pricing etc... Basically ML and AI will be used, in large part, by those who can to exploit those who cannot (or at least those who wont) defend themselves. There will be no opting out. So, with that said, if we go from the bottom up - and there is no opting out, then what is the best that one can hope for? I'd say complete ownership and control of one's own "meta-cloud" -- Any data that is a resultant trail of any action I take as an individual should be owned by me, and I should be able to see it all, and delete or block it. Or, in the extreme case, shouldn't I be able to require that any information presented to me (such as an ad or a price) be required to inform me as to how that information was formulated: "You are seeing this price because the following factors were analysed..." The real question is, in the future, is there even such a concept as "off grid"

tectonic 9y ago

Nice use of a shared Jupyter Notebook for data gathering. https://www.eff.org/ai/metrics https://www.eff.org/ai/metrics

shmageggy 9y ago
Nice that it's open and shared, but kinda weird that they are storing what is essentially tabular data in code. Looks to me like it would be much more readable if the data were in csv files, and just the graph generating code were in, well, code.
- pde3 9y ago
  The measurements for each metric are tabular, but the taxonomy of problems above them is currently tree (or forest) structured. CSV export or import for specific metrics would be very easy to add if you'd like it. We already have a rough JSON export of the data: https://raw.githubusercontent.com/AI-metrics/AI-metrics/master/export-api/v01/progress.json https://raw.githubusercontent.com/AI-metrics/AI-metrics/mast...

comboy 9y ago

The project is very interesting, but I'm not sure why they are doing it. How is that protecting user rights? This doesn't measure AI/ML progress that's available to state actors.

mlinksva 9y ago
> EFF’s interest in AI progress is primarily from a policy perspective. We want to know what types of AI we need to start engaging with on legal, political, and technical safety fronts. Beyond that, we’re also just excited to see how many things computers are learning to do over time. > Given that machine learning tools and AI techniques are increasingly part of our everyday lives, it is critical that journalists, policy makers, and technology users understand the state of the field. When improperly designed or deployed, machine learning methods can violate privacy, threaten safety, and perpetuate inequality and injustice. Stakeholders must be able to anticipate such risks and policy questions before they arise, rather than playing catch-up with the technology. To this end, it’s part of the responsibility of researchers, engineers, and developers in the field to help make information about their life-changing research widely available and understandable. We hope you’ll join us.
- comboy 9y ago
  Yes, it's not that I didn't read the page. But even in case of corporations, it's not really possible to gauge what t's available to them, because they don't share everything. Well on the other hand whatever is available publicly, they have at least that. So maybe the whole research does make some sense in this context.
  - mlinksva 9y ago
    Perhaps forcing governments and corporations to share more will be explored if it seems that this publicly documented baseline is far from the state of the art. Impossible to gauge, perhaps, but even realizing that highlights the risk of public ignorance.
    - 9y ago

arikr 9y ago

For ease of reading/UI, I think they should use a different color for the bars that represent "human score" and those that represent "excellent performance" - so that someone doesn't skim and assume they are the same thing, given that currently both are represented as red dotted lines but mean two different things.

DiabloD3 9y ago

Dupe: https://news.ycombinator.com/item?id=14603250 https://news.ycombinator.com/item?id=14603250

DanielBMarkham 9y ago

I believe the time has come for an independent institute to track AI and Big Data technology applications and begin creating guidelines for both industry self-certification and regulation. The other paths available, ignoring it, fearmongering about it, and trying to fit it into other political movements, do not seem to me to be heading towards an acceptable outcome. It has to be a technology-heavy group, otherwise it won't create much value. It also has to be grounded in history, philosophy, and political science, otherwise it'll just be reactionary. And we have enough reactionary groups already.