I think this is half-true. There's historically been a lot of cross-pollination between IAS and Princeton, which has resulted in some spectacular achievements

I think this is half-true. There's historically been a lot of cross-pollination between IAS and Princeton, which has resulted in some spectacular achievements such as the first electronic computer(the IAS machine; built a little bit after Feynman left I believe). IAS also has a lot of scholars who are working "remotely" at other institutions/colleges iirc. My general impression is that while there are a lot of slackers, there are also some really active and committed researchers, much like the rest of academia, so I'm not sure if we can reject the null hypothesis here so quickly.