When Finder is active click on the View menu, press the option key and select "Customize Control Strip", then you can just mouse down to the touchbar and remove

When Finder is active click on the View menu, press the option key and select "Customize Control Strip", then you can just mouse down to the touchbar and remove the Siri button.