The page cache exists to speed up access and it's transparent to processes. If another user/process tries to access your SSH files directly, it'll go through t

The page cache exists to speed up access and it's transparent to processes. If another user/process tries to access your SSH files directly, it'll go through the traditional file permissions to determine if it has access or not. If the disk block is in the page cache AND access is allowed to that inode, then the kernel will retrieve the page from the cache and give it to the process. To read the whole page cache, you'd need code sitting in kernel space. If something manages to load itself in the kernel space (e.g. kernel module), you have bigger problems to worry about.