some things some physicists do aren't empirically-based, yes that's correct. whether you call that "science" is a matter of terminology, but the important thing

some things some physicists do aren't empirically-based, yes that's correct. whether you call that "science" is a matter of terminology, but the important thing is whether it's high or low quality thinking, and the atmosphere and ethos has high or low quality discussion and criticism. i would say it is science, because the most important part of science is not the experiments, but the attitudes associated with science. do all the experiments you want, and without "scientific rigor" you will not get anywhere.