I use use-package with nix but disable package.el being able to install anything. This way I get fast startup and the download problem you mention doesn't happ

I use use-package with nix but disable package.el being able to install anything. This way I get fast startup and the download problem you mention doesn't happen. If nix isn't your thing, I used to do the same with straight.el by checking in .emacs.d/straight and disabling install by setting repos to nil.