The issue is not so much about many fonts, but about one big font. The first time a font is loaded it has to be analyzed and cached and for some fonts this lead

The issue is not so much about many fonts, but about one big font. The first time a font is loaded it has to be analyzed and cached and for some fonts this leads to excessive resource usage. You can avoid that in recent versions by using a HarfBuzz based font-shaper instead of the Lua version: Replace `\setmainfont{Some name}` with `\setmainfont[Renderer=HarfBuzz]{Some name}`. The first time a bug font is used you might still see a slight delay (~1 second?), but it's orders of magnitude faster and does not require so much resources.