Mixing pip with another package manager has always seemed weird to me. You're just asking for things to conflict and break. I noticed with Homebrew that there

Mixing pip with another package manager has always seemed weird to me. You're just asking for things to conflict and break. I noticed with Homebrew that there was no way to untangle packages installed through pip and ones installed through Homebrew. After dealing with that mess once, I now make sure to use pip install --user. It can still cause things to break, but if that does happen it's at least easy to nuke the packages installed to my home directory.