Isn't email the only password that 'really' matters? Barring super-secure sites like banks, if you have the email password, you have every 'forgot my password'

Isn't email the only password that 'really' matters? Barring super-secure sites like banks, if you have the email password, you have every 'forgot my password' link available - which will typically either let you reset the password, locking the user out, or even send you the password itself! This fact made me stop caring too much about varying individual 'random site' passwords - I just make sure to use a unique for my email.