Yes, but since such examples are far from common (look at all the TCS people's jaws dropping at that constant and power) I fail to see how this implies N vs NP

Yes, but since such examples are far from common (look at all the TCS people's jaws dropping at that constant and power) I fail to see how this implies N vs NP is irrelevant except in platonic math fairytale land. Since you said you have a QI background, I'm guessing you're familiar with Landau's symmetry breaking framework for identifying phase transitions. Like P, it works perfectly in a lot of cases and substantiates our desired qualitative idea (phase transitions) with a quantitive property. However, as I'm sure you know, we've found plenty of cases in the realm of QI where it fails to identify phase transitions (e.g. topological order) that we would like to categorize as such. However, nobody would think of throwing it out entirely just because of these exceptions, since it is useful in the vast majority of cases. Such is the case with P and the notion of being "easy" to compute. Yes, some algorithms in P are actually qualitatively "hard" and some outside are actually qualitatively "easy". But in the vast majority cases, finding that a problem is in P means we've drastically reduced its difficulty, and proofs that a problem isn't in P establish bounds that make it unfeasible in the general case.