While I think OP's idea isn't great, Windows seems to have an acceptable solution here: you can continue to allow drag-selection, and when clicking while holdin

While I think OP's idea isn't great, Windows seems to have an acceptable solution here: you can continue to allow drag-selection, and when clicking while holding down Ctrl/Shift you add to a selection rather than opening. The first click for the drag will be on the background and not on an icon, so it can still start a selection. This is how it works on Windows when you enable one-click.