Public opinion and public trust absolutely do matter to the justices. They derive their mandate from the people. If the people really want to relieve a justice

Public opinion and public trust absolutely do matter to the justices. They derive their mandate from the people. If the people really want to relieve a justice of their duty, they will find a way - "the government" can't exist without any public support. The Supreme Court is only the Supreme Court if enough people say it is.