Most endpoint protection systems typically use a combination of looking at known hashes or bad things and active process monitoring to identify new bad things.

Most endpoint protection systems typically use a combination of looking at known hashes or bad things and active process monitoring to identify new bad things. They're not perfect, and have varying levels of success. It is better than nothing. I can't speak to Symantec, but I've worked for companies that provide endpoint protection and watched a live demo of novel "malware" getting caught escalating privileges.