Why not use SugarCRM with its built in email. Use IMAP to fetch your email from Gmail, and send from within SugarCRM to your contacts. I have used SugarCRM this

Why not use SugarCRM with its built in email. Use IMAP to fetch your email from Gmail, and send from within SugarCRM to your contacts. I have used SugarCRM this way before and it worked ok. I am not a massive fan of the software, but it does seem to have the features you want. It has a limited activity stream too. You could always fork the code to make it do exactly what you want, since it is open source.