13 ms·

We've also run an app with 1 million+ users for a few years, and I can confirm that notifications are a wreck on both platforms, but IMO even more so on Android

by tdrp 5y ago

We've also run an app with 1 million+ users for a few years, and I can confirm that notifications are a wreck on both platforms, but IMO even more so on Android/FCM. In fact most of our calculations show that we have never gone past 85% delivery of high-pri Android notifications to valid FCM tokens. By high-pri here we mean direct chat messages in an active 1-on-1 chat conversation. This has caused many users to abandon the app since they assume the system to be unreliable. One of the main reasons seems to be that many Android phone vendors have simply whitelisted WhatsApp, Facebook and a couple of famous apps and then dumped every single other app into some kind of "kill if in background and completely forget about it" bucket. In some cases it can be fixed by asking the users to wander deep into the phone settings and toggle some switches. We brought the delivery rate from the low 80s to 85% by literally buying a bunch of the phone models we had heard had issues and trying to repro what was happening then popping up custom instructions for users like "looks like you are running this kind of phone which in general will not deliver notifications, please go to settings -> blah". But the whole thing has been a gigantic game of cat and mouse which really shouldn't be in the hands of individual app developers. In many cases the FCM system does not return any error code. I'd be curious to hear if anyone here has achieved 95%+ FCM delivery rates on Android apps.

cameronh90 5y ago
Until last week I would have been confused by this, but last week my Pixel 4 broke. While waiting for it to be repaired, I bought a cheap Chinese phone to use as a backup - not sure of the brand but it cost £200. I don't know what the fuck they did to the operating system, but notifications are completely unreliable. I think there's possibly some aggressive app termination going on, maybe for battery or memory usage. Either way, most of the time I would only get a notification once I open the app. Some apps seem to be more reliable than others - maybe due to whitelisting as you say. In hindsight, Google allowing these no-QA phones to use the Android branding was a big mistake.
- Too 5y ago
  Killing idle apps, including Services, is a feature in Android to conserve battery life and to give memory to Activities in more recent use. This might not be as noticeable on high-end-phones. A lot of shitty written apps are not aware of this and think they can run some while loop in the background to poll their own backend 10 times per second. Eventually that will get killed. The solution to this is the push apis, or for apps that truly need a background loop to present a persistent notification aka "foreground-service", so the user is aware something heavy is going on in the background. Hearing FCM, which is the suggested push solution, having delivery issues above is not very comfortable.
gdeglin 5y ago
There are some well-intended reasons for this, but unfortunately, they've resulted in many problems that don't have simple solutions. To help increase battery life, both Apple and Google provide a standard notification backend that keeps a single background connection open for every device and routes notifications to the device. Google and Apple require the use of their systems for push notifications (APNS and FCM). BUT... Google's services are blocked in mainland China. So, developers in that region have resorted to building their own push notification systems. No single standard has emerged, so apps may use several third-party FCM alternatives or entirely develop their own. Inevitably if you live in China and you have hundreds of apps on your device, with each one waning to keep its own background connection open, your battery won't last more than a few hours. As a result, OEMs in China modify the operating system on devices they sell to prevent apps from having background connections open. Users are expected to manually go into their settings and allow specific applications to keep connections open. Sometimes OEMs will have pre-set list of popular apps that are automatically included. If one of these devices is sold outside of China, it sees FCM in the same way as any other background connection and kills it. Now here's where it gets crazy: 1. Many devices sold on eBay and in other channels are devices that were intended to only be sold in China. The seller modifies the operating system to add Google's services to it and changes the default language to English. These devices tend to be particularly problematic because sellers intentionally mislabel them as international devices, even though they are still running a modified version of Android intended only for phones sold in China. 2. Many OEMs have chosen to keep some of their background connection killing logic even in devices sold outside of China. It's hard to say precisely why. Maybe they are competing for higher battery life, or perhaps they don't recognize the problems that it causes. This logic usually does some combination of either killing an app such that it can't wake up to receive a notification, or it interferes with the FCM connection to prevent any app from receiving notifications. 3. Building on #2 -- Many popular devices (e.g. modern Samsung phones) will ask users during the onboarding process if they want to "put apps to sleep after three days of not being used", and the option is "ON" by default. Users often don't realize that this setting will break apps that set background timers (e.g.: alarm apps) or that need to receive notifications. 4. The modifications made by these OEMs are often buggy. For example, user settings get reset after updates. Or the operating system seemingly randomly kills apps in such a way that prevents them from getting notifications. 5. Since OEMs often allow some popular apps to get around these restrictions, it creates an unfair situation between different companies. For example, Facebook might always work fine, but an up-and-coming social network will run into reliability issues. 6. In some cases, devices will appear to receive a push notification but will not display it to the end-user. This can be related to either power management preferences or OEM-specific "do not disturb" preferences. This is a great website with more information on this issue: https://dontkillmyapp.com/ https://dontkillmyapp.com/ (Disclosure: I'm one of the founders of OneSignal. We've been tracking and mitigating this problem by implementing "Delivery Confirmation", a feature that permits apps to send a webhook whenever the app displays notification so that developers can see the end-to-end delivery of each message and either retry their messages or use a different method of contacting the user)
mleonhard 5y ago
I'm developing an app with critical realtime notifications. Do you mind if I ask you a few questions? 1. Does your 85% delivery rate apply to FCM "notification messages" (displayed to user) or only to "data messages" (processed by app)? 2. Do some devices fail to deliver messages when your app is in the foreground? And does it happen often with US-based devices? 3. When the app is in the background, how long does it take for it to stop receiving notifications? I know the answer must vary by device. 4. How do you determine delivery rate of notification messages? I believe those are handled entirely by Android OS and app code runs only when they are tapped. I'm using Flutter and have not yet dug into the details of how Android processes and displays notifications. 5. Do you use FCM to deliver APNS messages? 6. How reliable are APNS messages to a foreground app for US users I'm considering launching iPhone-only. My app helps people make appointments (dates). It uses notifications to remind users of their upcoming and imminent appointments. The backend detects when a user has not tapped a critical notification and sends them an SMS. This should reach the user even if they didn't see the notifications or they uninstalled the app without cancelling their appointments. I also implemented real-time chat using FCM data messages with a slow poll fallback. If data messages are not reliable, then I will need to replace the slow poll with a realtime push service. That will be a significant cost in added complexity. I would like to put off that work. I'm a solo bootstrapper. Working with Android and FCM has been daunting. Would you be interested in mentoring me a few hours a month? I can pay you. My email is in my profile.

by tdrp 5y ago

cameronh90 5y ago

Until last week I would have been confused by this, but last week my Pixel 4 broke. While waiting for it to be repaired, I bought a cheap Chinese phone to use as a backup - not sure of the brand but it cost £200. I don't know what the fuck they did to the operating system, but notifications are completely unreliable. I think there's possibly some aggressive app termination going on, maybe for battery or memory usage. Either way, most of the time I would only get a notification once I open the app. Some apps seem to be more reliable than others - maybe due to whitelisting as you say. In hindsight, Google allowing these no-QA phones to use the Android branding was a big mistake.

Too 5y ago
Killing idle apps, including Services, is a feature in Android to conserve battery life and to give memory to Activities in more recent use. This might not be as noticeable on high-end-phones. A lot of shitty written apps are not aware of this and think they can run some while loop in the background to poll their own backend 10 times per second. Eventually that will get killed. The solution to this is the push apis, or for apps that truly need a background loop to present a persistent notification aka "foreground-service", so the user is aware something heavy is going on in the background. Hearing FCM, which is the suggested push solution, having delivery issues above is not very comfortable.

gdeglin 5y ago

There are some well-intended reasons for this, but unfortunately, they've resulted in many problems that don't have simple solutions. To help increase battery life, both Apple and Google provide a standard notification backend that keeps a single background connection open for every device and routes notifications to the device. Google and Apple require the use of their systems for push notifications (APNS and FCM). BUT... Google's services are blocked in mainland China. So, developers in that region have resorted to building their own push notification systems. No single standard has emerged, so apps may use several third-party FCM alternatives or entirely develop their own. Inevitably if you live in China and you have hundreds of apps on your device, with each one waning to keep its own background connection open, your battery won't last more than a few hours. As a result, OEMs in China modify the operating system on devices they sell to prevent apps from having background connections open. Users are expected to manually go into their settings and allow specific applications to keep connections open. Sometimes OEMs will have pre-set list of popular apps that are automatically included. If one of these devices is sold outside of China, it sees FCM in the same way as any other background connection and kills it. Now here's where it gets crazy: 1. Many devices sold on eBay and in other channels are devices that were intended to only be sold in China. The seller modifies the operating system to add Google's services to it and changes the default language to English. These devices tend to be particularly problematic because sellers intentionally mislabel them as international devices, even though they are still running a modified version of Android intended only for phones sold in China. 2. Many OEMs have chosen to keep some of their background connection killing logic even in devices sold outside of China. It's hard to say precisely why. Maybe they are competing for higher battery life, or perhaps they don't recognize the problems that it causes. This logic usually does some combination of either killing an app such that it can't wake up to receive a notification, or it interferes with the FCM connection to prevent any app from receiving notifications. 3. Building on #2 -- Many popular devices (e.g. modern Samsung phones) will ask users during the onboarding process if they want to "put apps to sleep after three days of not being used", and the option is "ON" by default. Users often don't realize that this setting will break apps that set background timers (e.g.: alarm apps) or that need to receive notifications. 4. The modifications made by these OEMs are often buggy. For example, user settings get reset after updates. Or the operating system seemingly randomly kills apps in such a way that prevents them from getting notifications. 5. Since OEMs often allow some popular apps to get around these restrictions, it creates an unfair situation between different companies. For example, Facebook might always work fine, but an up-and-coming social network will run into reliability issues. 6. In some cases, devices will appear to receive a push notification but will not display it to the end-user. This can be related to either power management preferences or OEM-specific "do not disturb" preferences. This is a great website with more information on this issue: https://dontkillmyapp.com/ https://dontkillmyapp.com/ (Disclosure: I'm one of the founders of OneSignal. We've been tracking and mitigating this problem by implementing "Delivery Confirmation", a feature that permits apps to send a webhook whenever the app displays notification so that developers can see the end-to-end delivery of each message and either retry their messages or use a different method of contacting the user)

mleonhard 5y ago

I'm developing an app with critical realtime notifications. Do you mind if I ask you a few questions? 1. Does your 85% delivery rate apply to FCM "notification messages" (displayed to user) or only to "data messages" (processed by app)? 2. Do some devices fail to deliver messages when your app is in the foreground? And does it happen often with US-based devices? 3. When the app is in the background, how long does it take for it to stop receiving notifications? I know the answer must vary by device. 4. How do you determine delivery rate of notification messages? I believe those are handled entirely by Android OS and app code runs only when they are tapped. I'm using Flutter and have not yet dug into the details of how Android processes and displays notifications. 5. Do you use FCM to deliver APNS messages? 6. How reliable are APNS messages to a foreground app for US users I'm considering launching iPhone-only. My app helps people make appointments (dates). It uses notifications to remind users of their upcoming and imminent appointments. The backend detects when a user has not tapped a critical notification and sends them an SMS. This should reach the user even if they didn't see the notifications or they uninstalled the app without cancelling their appointments. I also implemented real-time chat using FCM data messages with a slow poll fallback. If data messages are not reliable, then I will need to replace the slow poll with a realtime push service. That will be a significant cost in added complexity. I would like to put off that work. I'm a solo bootstrapper. Working with Android and FCM has been daunting. Would you be interested in mentoring me a few hours a month? I can pay you. My email is in my profile.