3 ms·
Huh. I can read and understand the abstract and the introduction, but I can't judge the work after a first pass. This is the kind of paper that cannot be easily
by cs702 5y ago
Huh. I can read and understand the abstract and the introduction, but I can't judge the work after a first pass. This is the kind of paper that cannot be easily skimmed, because it consists almost entirely of densely packed pages chock-full of highly abstract mathematical reasoning.
Not surprisingly, the authors are mathematicians. They claim to prove that
* there are well-conditioned problems for which suitable DNNs exist, but no training algorithm can find arbitrarily good approximations of those suitable DNNs;
* it's possible to find approximations of those suitable DNNs only if we sacrifice digits of accuracy -- i.e., the approximations cannot be arbitrarily good; and
* there is a class of DNNs they propose, which they call "fast iterative restated networks" or FIRENETs, that solve undetermined systems of linear equations over the complex numbers, with a good blend of stability (robustness to adversarial samples) and accuracy (within the claimed theoretical limits).
Finally, the authors provide open-source code (A+ for doing that, but... Matlab!!??): https://www.github.com/Comp-Foundations-and-Barriers-of-AI/firenet https://www.github.com/Comp-Foundations-and-Barriers-of-AI/f...
Does anyone else here understand the work better than me? I would love an informal explanation that appeals to intuition.
- lp251 5y agoThis is a follow-up to an earlier paper on instabilities in NN-based solutions to inverse problems: https://www.pnas.org/doi/10.1073/pnas.1907377117 https://www.pnas.org/doi/10.1073/pnas.1907377117 Your statements appear to be a good summary of the paper.
- cs702 5y ago> Your statements appear to be a good summary of the paper. Thanks. I found the authors' claims relatively easy to grok. What I'd like to understand, intuitively, is how they got there!
- lp251 5y agoAh, sorry, I can't help too much there. I skimmed through the proofs in the SI. Many of the tools they use are taken from the compressed sensing literature (robust null space property, sqrt LASSO paths, sample complexity for CS-MRI style problems...). I can point you to additional reading, if you'd like.
- cs702 5y agoThank you. I think I can find introductory materials on my own. This is now on my list of things about which I'd like to learn/relearn more. It's been a long time since I've had any superficial exposure to the topic; my recollection of it is vague and in all likelihood outdated. My memory is a bit "situational" -- I tend to remember best the things I've been thinking about and working on, but as time passes everything else sort of becomes blurrier, less precise in my mind.
- glitchc 5y agoWhat’s wrong with Matlab? It works very well for this, as it’s very stable for numerical modeling work.
- cs702 5y agoIt's closed-source; it lacks the kind of tooling/infrastructure you need for working with larger, state-of-the-art DNNs; and very little AI research is done with it, limiting the audience that can can use the code. Python -- or even Julia -- would have been a better choice.
- lp251 5y agoThe story has changed somewhat recently, but up until ~2020 most ML for Inverse Problems research was done in Matlab, especially for MRI and CT reconstruction. Thinking of FBPNet and ADMM-Net.
- lostmsu 5y agoI just recently stumbled upon a practical example of the first item in your list: try approximating sin(FREQ_SCALE * input) using a neural network and gradient descent for various FREQ_SCALE and ranges of input. You will find it to be completely powerless even though you can easily construct a single parameter "NN" with sin activation, that reproduces the function exactly when the parameter is initialized with FREQ_SCALE. The problem extends to any periodic functions. I am working on a blog post about it.