Missing notifications for Android I was the lead for Flock Android, a team messaging app which competes with the likes of Slack. We started receiving some weird

Missing notifications for Android I was the lead for Flock Android, a team messaging app which competes with the likes of Slack. We started receiving some weird complaints from users about missing notifications. Now since it was a business messaging app, notifications were critical. We tried out everything to reproduce this issue, including bombarding devices with notifications, trying out different networks but we just could not reproduce the issue. Worse we had a couple of incidents with devices in our team but logs showed absolutely nothing. Google GCM said that the notification was sent but there was no trace of the notification in the app's logs, it was as the the OS was swallowing notifications. And somehow popular apps like whatsapp, facebook were immune to these issues. The final nail in the coffin was the CEO himself missing notifications on some occassions. Google was unhelpful as usual with their non existent dev customer support. We were able to find a bunch of support articles on the internet where other apps had essentially listed down steps to turn off battery optimizations and that seemed to work ( Interestly whatsapp, facebook are usually added by default in these lists). But there was no way to do it automatically and we could also do it in response to a support ticket, which is painful for users. After a couple of days we were able to encounter a few missed notifications on a oneplus device which we had connected to a machine taking system logs. And voila we got a few lines of what was happening. 01-12 09:05:23.649 2719 2839 I ActivityManager: [BgDetect]chkExcessCpu level: 0 doKills: true auto_mode: false uptime: 183054 01-12 09:05:23.661 2719 2839 I ActivityManager: [BgDetect]detect excessive cpu on process to.talk(pid : 13406) level 0 usage 29 01-12 09:05:23.892 2719 2839 I ActivityManager: [BgDetect]force stop to.talk (uid 10151) level 0 01-12 09:05:23.893 2719 2839 I ActivityManager: Force stopping to.talk appid=10151 user=0: from pid 2719 01-12 09:05:23.893 2719 2839 I ActivityManager: Killing 13406:to.talk/u0a151 (adj 200): stop to.talk Apparently some process called BgDetect figured out that our app was taking too much CPU and decided to kill it. Android being open source, we figured that we should be able to get source for it. But not only could we not find BgDetect in Android sources, it was non existent on oneplus sources too. We then got in touch with the marketing team and reached out to some contacts at oneplus, who directed us to their dev team. They asked for our apk and voila , in a week we were whitelisted along with the likes of Whatsapp and Facebook. But the issue persisted on Xiaomi, Oppo and a bunch of chinese manufacturers and for those we still had to dish out steps to whitelist Flock in battery optimizations. We eventually figured out that moving from http to xmpp for android fcm notifications gave us delivery receipts for push notifications on device. On google phones and reputable manufacturers these delivery receipts also meant that app received the notification. On affected devices apparently the OS would give us a delivery receipt but not deliver it to the application. We built a double ack mechanism which also sent delivery receipts from the app. Based on the time difference between app delivery receipts and fcm delivery receipts we were able to figure out affected devices and send them directed bot messages to preemptively ask users to whitelist flock. You can read more about the double ack mechanism architechture here: https://hackernoon.com/notifications-in-android-are-horribly.. https://hackernoon.com/notifications-in-android-are-horribly.... Google still refuses to even acknowledge the issue while problems like these are widespread: https://bit.ly/2QSzTKK https://bit.ly/2QSzTKK