3 ms·

(Disclaimer: I skimmed the article, and have it on my to-be-read) When I first encountered the notion of adversarial examples, I thought it was a niche concern

(Disclaimer: I skimmed the article, and have it on my to-be-read) When I first encountered the notion of adversarial examples, I thought it was a niche concern. As this paper outlined, however, the growth of "machine-learning-as-a-service" companies (Amazon, Open AI, Microsoft, etc.) has actually rendered this a legitimate concern. From my skimming, I wanted to highlight their interesting point that "gradient-based post-processing may be limited" in mitigating a compromised model. These points really bring these concerns from an academic to business realm. Lastly, I'm delighted that they acknowledge their influences from the cryptographic community with respect to rigorously quantifying notions of "hardness" and "indistinguishable." Of note, they seem to base their undetectable backdoors on the assumption that the shortest vector problem is not in BQP. As I recently learned looking at the NIST post-quantum debacle, this has been a point of great contention. I've in all likelihood mischaracterized the paper, but I look forward to reading it!

ks1723 5y ago
As a side question: What is the NIST post-quantum debacle? Could you give some references?
- izzygonzalez 5y ago
  One of their post-quantum bets did not work out. https://news.ycombinator.com/item?id=30466063 https://news.ycombinator.com/item?id=30466063
  - kvathupo 5y ago
    Ah, that actually wasn't the debacle I had in mind; I'm not too familiar with the details of the Rainbow concerns unfortunately. With respect to the shortest vector problem (SVP) being a point of contention among NIST PQC participants, two of the round 3 finalists are based on lattice cryptography, with NTRU directly relying on the hardness of SVP. The two concerns are: 1. The risks of lattice-based cryptography are poorly understood [1], [2] 2. Research progress into attacks on lattice-based cryptography have been fruitful during the NIST PQC process [1], [3]. From what I've gathered as a layperson, much of these concerns have been voiced by Daniel J. Bernstein. Bernstein contributed to the NTRU Prime software [4], which was used in OpenSSH 9 (I'll circle back to this point). As a consequence of these two concerns, the main argument seems to be that NIST should at least provide warnings [6] on the risks of lattice cryptography, particularly with regard to the use of cyclotomics by one of the finalists [5]. A common thread amongst these criticisms seems to be a distrust of NIST guidelines (a point that is also echoed by this ML backdoor paper). This has evidently stirred some bad blood between NIST workers and Bernstein [7], [8]. I'm sure to there's more to the story (especially since Bernstein's NTRU prime was a NIST PQC candidate), but I suppose NIST isn't free from passive-aggressiveness? Within the context of this bad-blood, it's amusing that OpenSSH 9 uses Bernstein's NTRU Prime (doesn't use cyclotomics iirc), as opposed to one of NIST PQC's finalists. (DISCLAIMER: I'm a layperson, and I encourage people to read the sources themselves to make an informed opinion. People are welcome to correct. ) [1] - See the link to the "Risks of lattice KEMs" PDF at the top: https://ntruprime.cr.yp.to/warnings.html https://ntruprime.cr.yp.to/warnings.html [2] - https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/Fm4cDfsx65s/m/0dBVXOcSCAAJ https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/Fm4c... [3] - https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/4iaf9O-UjSo/m/c9xzBm8iBwAJ https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/4iaf... [4] - https://ntruprime.cr.yp.to/index.html https://ntruprime.cr.yp.to/index.html [5] - https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/7WhvubvoCVs/m/X2zhcNbaAQAJ https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/7Whv... [6] - https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/KFgw5_qCXiI/m/uVM4O6WpAwAJ https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/KFgw... [7] - https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/Fm4cDfsx65s/m/FRkvMfFpCAAJ https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/Fm4c... [8] - [PDF] - https://csrc.nist.gov/csrc/media/Projects/post-quantum-cryptography/documents/round-3/official-comments/NTRU-Prime-round3-official-comment.pdf https://csrc.nist.gov/csrc/media/Projects/post-quantum-crypt...
polygamous_bat 5y ago
> Lastly, I'm delighted that they acknowledge their influences from the cryptographic community with respect to rigorously quantifying notions of "hardness" and "indistinguishable." Fun fact, that is because they ARE primarily cryptography people! Goldwasser is known for Blum-Goldwasser or Goldwasser-Micali crypto systems, while Vaikuntanathan is known for Zero Knowledge computations, both materials from any standard cryptography textbook! (And they're great teachers, I was lucky enough to have them both as teachers in a class a few years back :) )

ks1723 5y ago

As a side question: What is the NIST post-quantum debacle? Could you give some references?

izzygonzalez 5y ago
One of their post-quantum bets did not work out. https://news.ycombinator.com/item?id=30466063 https://news.ycombinator.com/item?id=30466063
- kvathupo 5y ago
  Ah, that actually wasn't the debacle I had in mind; I'm not too familiar with the details of the Rainbow concerns unfortunately. With respect to the shortest vector problem (SVP) being a point of contention among NIST PQC participants, two of the round 3 finalists are based on lattice cryptography, with NTRU directly relying on the hardness of SVP. The two concerns are: 1. The risks of lattice-based cryptography are poorly understood [1], [2] 2. Research progress into attacks on lattice-based cryptography have been fruitful during the NIST PQC process [1], [3]. From what I've gathered as a layperson, much of these concerns have been voiced by Daniel J. Bernstein. Bernstein contributed to the NTRU Prime software [4], which was used in OpenSSH 9 (I'll circle back to this point). As a consequence of these two concerns, the main argument seems to be that NIST should at least provide warnings [6] on the risks of lattice cryptography, particularly with regard to the use of cyclotomics by one of the finalists [5]. A common thread amongst these criticisms seems to be a distrust of NIST guidelines (a point that is also echoed by this ML backdoor paper). This has evidently stirred some bad blood between NIST workers and Bernstein [7], [8]. I'm sure to there's more to the story (especially since Bernstein's NTRU prime was a NIST PQC candidate), but I suppose NIST isn't free from passive-aggressiveness? Within the context of this bad-blood, it's amusing that OpenSSH 9 uses Bernstein's NTRU Prime (doesn't use cyclotomics iirc), as opposed to one of NIST PQC's finalists. (DISCLAIMER: I'm a layperson, and I encourage people to read the sources themselves to make an informed opinion. People are welcome to correct. ) [1] - See the link to the "Risks of lattice KEMs" PDF at the top: https://ntruprime.cr.yp.to/warnings.html https://ntruprime.cr.yp.to/warnings.html [2] - https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/Fm4cDfsx65s/m/0dBVXOcSCAAJ https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/Fm4c... [3] - https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/4iaf9O-UjSo/m/c9xzBm8iBwAJ https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/4iaf... [4] - https://ntruprime.cr.yp.to/index.html https://ntruprime.cr.yp.to/index.html [5] - https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/7WhvubvoCVs/m/X2zhcNbaAQAJ https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/7Whv... [6] - https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/KFgw5_qCXiI/m/uVM4O6WpAwAJ https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/KFgw... [7] - https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/Fm4cDfsx65s/m/FRkvMfFpCAAJ https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/Fm4c... [8] - [PDF] - https://csrc.nist.gov/csrc/media/Projects/post-quantum-cryptography/documents/round-3/official-comments/NTRU-Prime-round3-official-comment.pdf https://csrc.nist.gov/csrc/media/Projects/post-quantum-crypt...

polygamous_bat 5y ago

> Lastly, I'm delighted that they acknowledge their influences from the cryptographic community with respect to rigorously quantifying notions of "hardness" and "indistinguishable." Fun fact, that is because they ARE primarily cryptography people! Goldwasser is known for Blum-Goldwasser or Goldwasser-Micali crypto systems, while Vaikuntanathan is known for Zero Knowledge computations, both materials from any standard cryptography textbook! (And they're great teachers, I was lucky enough to have them both as teachers in a class a few years back :) )