4 ms·

Sidestepping comedy for a bit, there are a lot of inscrutable systems that we connect to 'things that matter' all the time. The financial systems themselves are

Sidestepping comedy for a bit, there are a lot of inscrutable systems that we connect to 'things that matter' all the time. The financial systems themselves are pretty damn inscrutable. Corporations are very often inscrutable.

Analemma_ 8y ago
But corporations and financial systems can often be analyzed in terms of the incentives of the humans involved, or at least the human incentives pushing back against changes to the crappy status quo. When they behave badly, they usually do so in predictable ways because there are humans in the loop somewhere. By contrast, there is nothing remotely resembling a human mind anywhere in machine learning, and the failure modes are often, by our standards, insane (like thinking a picture of random noise is a cat). That creates a whole new level of danger when connecting to "things that matter". And to the extent we already connect inscrutable systems to things that matter, we should be trying to make that problem better, not worse. "When you're in a hole, the first thing to do is stop digging"
Avshalom 8y ago
I feel like you're making his point for him.
eli_gottlieb 8y ago
To quote, "This has made a lot of people very angry, and been widely regarded as a bad move."
tptacek 8y ago
The word Mickens uses is "interpretable", which financial infrastructure is, and ML models are not.
- Anderkent 8y ago
  Financial IT is interpretable, maybe, but is the financial system itself? You have a lot of agents taking actions that you can't really interpret from outside - unless you say something vacuous like this person made this trade because they think it was good, at which point you may as well say this ai model made this decision because it thought it was good.
  - tptacek 8y ago
    What does "the financial system" mean? Obviously there's a level you can address with that term where, just like with ML, interpretability becomes an open question. But on most every level that is genuinely comparable to the role an ML model plays in a software product, finance is plenty interpretable.
badloginagain 8y ago
There is also a lot of regulation around those entities because they're known to be inscrutable and are expected to be if no regulation existed. Technologists tend to think that tech is value-neutral, and will therefore give good outcomes.

Analemma_ 8y ago

But corporations and financial systems can often be analyzed in terms of the incentives of the humans involved, or at least the human incentives pushing back against changes to the crappy status quo. When they behave badly, they usually do so in predictable ways because there are humans in the loop somewhere. By contrast, there is nothing remotely resembling a human mind anywhere in machine learning, and the failure modes are often, by our standards, insane (like thinking a picture of random noise is a cat). That creates a whole new level of danger when connecting to "things that matter". And to the extent we already connect inscrutable systems to things that matter, we should be trying to make that problem better, not worse. "When you're in a hole, the first thing to do is stop digging"

Avshalom 8y ago

I feel like you're making his point for him.

eli_gottlieb 8y ago

To quote, "This has made a lot of people very angry, and been widely regarded as a bad move."

tptacek 8y ago

The word Mickens uses is "interpretable", which financial infrastructure is, and ML models are not.

Anderkent 8y ago
Financial IT is interpretable, maybe, but is the financial system itself? You have a lot of agents taking actions that you can't really interpret from outside - unless you say something vacuous like this person made this trade because they think it was good, at which point you may as well say this ai model made this decision because it thought it was good.
- tptacek 8y ago
  What does "the financial system" mean? Obviously there's a level you can address with that term where, just like with ML, interpretability becomes an open question. But on most every level that is genuinely comparable to the role an ML model plays in a software product, finance is plenty interpretable.

badloginagain 8y ago

There is also a lot of regulation around those entities because they're known to be inscrutable and are expected to be if no regulation existed. Technologists tend to think that tech is value-neutral, and will therefore give good outcomes.