That's correct, but if you want to keep any files that services like that access, such as SSH keys/authorized_keys you need to give them access to your director

That's correct, but if you want to keep any files that services like that access, such as SSH keys/authorized_keys you need to give them access to your directories. Because of this you can't make your home directory only readable by your user, since then sshd cannot access `~/.ssh` to check your authorized_keys file. The same is true for running a web server in `~/public_html` and anything else. The agreed upon solution to this is to make the home directory readable, then split the files into a `~/public` and `~/private` directory, which is well accepted in the server space but completely non-adopted and not straightforward in the desktop space. It's technically possible to modify your $XDG_DATA_DIR and other similar configurations, but many programs and tools still don't care and will only look for and operate on configuration files in `~/.config` or `~/.local` or worse yet just `~/.foo`.