6 ms·

Google confirms an issue with WebView is crashing many Android apps (Fixed)

88 points by neversaydie 6y ago

aritmo 6y ago
There is a misconception that removing Android Webview is "fixing" the problem. The proper word is that removing the updated Android Webview is "a temporary workaround" until an updated Android Webview is released. You cannot remove Android Webview because it is an essential part of Android. By "removing" the package, you are removing the latest update and revert back to the old (and probably insecure) version of Android Webview that came with the initial Android version of the phone. In the end, Google eventually released a new package now, so you can perform the update of Android Webview to the latest version.
- jmartrican 6y ago
  Dont see an update option for this app in the app store. It says last release was March 12th. How exactly do I get this update?
  - 51Cards 6y ago
    I was able to just do a Chrome update to solve the issue on my phone.
  - numpad0 6y ago
    No action required if you’re not experiencing issues. If you’re affected and not able to update, trying to “uninstall updates” for “Android System WebView” will revert it to ROM-included unaffected version and solve it.
    - CodesInChaos 6y ago
      That sounds very risky, since untrusted code shown in any webview can then exploit long fixed chrome bugs.
      - DoomHotel 6y ago
        I uninstalled the updates and was then able to install the March 22 version. It didn't offer any update until I did that.
- kevincox 6y ago
  I assume that this is uninstalling updates, which means that you are back to the WebView that came with your phone. This obviously opens a whole new can of worms as now you may be years of security updates behind. I really wish that popular stores had an option to roll back one version until the next update came out.
  - ProAm 6y ago
    This is why auto-update is turned off for me. You just cant trust blindly updating things because there is no undo button for breaking changes.
    - kevincox 6y ago
      How do you evaluate updates? And how do you rollback? It seems like you are still basically relying on "find the APK somewhere" for rollbacks still.
      - ProAm 6y ago
        Really if I do not need new functionality and it's not a sensitive app (i.e. one that deals with funds, email, text, etc..) I just dont. Some of my APKs are REALLY REALLY old. If i get a new phone, I save them off and roll them over. If its for an app that has potentially significant security issues I'll wait a while and monitor the reviews and check the web before I update (I'll also backup the installed APK before updating) so I can rollback if I need to. But the number of apps on my phone for this is very slim.
- perryizgr8 6y ago
  It is definitely fixing the problem. As is the entire phone is unusable because you cannot open many of the most important apps.
BiteCode_dev 6y ago
The problem is, I didn't issue an update on my android phone this week. I don't even have a Google play account and use ApkPure. Yet this morning my applications started to break. How did the update occured on my phone ?
- SahAssar 6y ago
  IIRC system components can be OTA updated in the background. The only way to get around that is to use a de-googled android distro.
51Cards 6y ago
Experienced this this morning on my Pixel 1. Woke up to several apps in a crash loop every few seconds and most others crashing on startup. After a reboot didn't solve anything I thought perhaps my (older) phone had suffered a RAM failure since I hadn't updated anything in several days. After finding an article about the issue I updated Chrome which corrected the issue.
exabrial 6y ago
How does something like this make it to production? I couldn't reply to any work emails for the greater part of the evening yesterday. Adds more evidence to the case of "never use Google for your business".
- throwdbaaway 6y ago
  I guess the team has Q1 OKR to meet and/or error budget to spend? Either way, it's bad. I am not saying the bug would have been prevented if the team operates differently. Brown paper bag release happens. However, the way this incident blew up suggests that the bad release made it to too many users in too short of a time, exactly the behavior incentivized by meeting (or exceeding) an OKR while having some spare error budget.
  - gowld 6y ago
    What do you think "error budget" means? (Hint, it doesn't mean "amount of intentionally deployed errors we allow".) Do you believe WebView should never update because it's never OK to spend error budget?
    - room500 6y ago
      That is a strawman argument. Of course WebView should update. The question is whether the structure of incentives contributed to a risky release. If you "play the game", an error budget causes your releases to become riskier and riskier as the budget grows. This is optimal because a riskier release allows you to accelerate changes, allowing you to hit other OKR. The issue is that this will eventually cause very risky releases (where you would slow down otherwise). In a sense, an error budget does eventually become "amount of intentionally deployed errors" because you are incentivized to increase risk until an error does appear.
    - topher200 6y ago
      Interestingly, the Google SRE book[1] mentions scenarios where they practice exactly this: > The solution to this Chubby scenario is interesting: SRE makes sure that global Chubby meets, but does not significantly exceed, its service level objective. In any given quarter, if a true failure has not dropped availability below the target, a controlled outage will be synthesized by intentionally taking down the system. I doubt there are any customer-facing systems that practice this technique. Just super interesting that it exists. [1]: https://sre.google/sre-book/service-level-objectives/#xref_risk-management_global-chubby-planned-outage https://sre.google/sre-book/service-level-objectives/#xref_r...
amelius 6y ago
At least you can write/bundle your own render engine on Android, something which is not possible on iOS.
- chrismorgan 6y ago
  But if your app is now 200MB instead of 2MB, know that there are a significant number of users that simply won’t install your app (or will remove it) because they don’t have either space or bandwidth. Cheaper Android phones regularly have almost no spare space (like, well under 2GB available to user space when empty, even though the phone may have been advertised as having 8GB, and even that 2GB may well be partitioned stupidly so that you have less than 1GB available to install apps on).
  - throwaway53453 6y ago
    Isn't the whole point of WebView that Firefox et al can provide it as well as Chrome?
    - meibo 6y ago
      Yes, Firefox can act as a WebView implementation on any Android phone. Apps don't need to ship this.
      - chrismorgan 6y ago
        No it can’t. https://wiki.mozilla.org/Mobile/GeckoView https://wiki.mozilla.org/Mobile/GeckoView: “GeckoView serves a similar purpose to Android's built-in WebView, but it has its own APIs and is not a drop in replacement.” The purpose of WebView has certainly never been about using a different engine; rather, it’s to have only one copy of a browser for apps to use as a widget, rather than each app bundling its own browser which uses vast amounts of space and raises serious security concerns.
    - chrismorgan 6y ago
      The situation I’m responding to is the suggestion that you bundle your own browser rather than using one provided by Android System WebView or equivalent.
tantalor 6y ago
This started happening for me last night. Just updated webview to 89.0.4389.105 and everything works now.
redsky17 6y ago
Well, I'm glad to see this posted here... I was concerned that somehow my phone got infected with some kind of malware after a reboot didn't fix it. It's concerning to me that an update was pushed out that apparently wasn't tested very well.
Tade0 6y ago
My phone rebooted a few times during the last week. At first I thought it was just planned obsolescence killing a three year old device, but fortunately that doesn't seem to be the case.
- topher200 6y ago
  I don't believe that a system reboot is a symptom of this issue, just app crashes. Also, the bug started yesterday afternoon. I doubt this has anything to do with your system reboots.
jerome-jh 6y ago
Actually just noticed that on my wife's phone: 03-23 14:47:11.393 21102 21102 F DEBUG : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0 03-23 14:47:11.393 21102 21102 F DEBUG : Cause: null pointer dereference 03-23 14:47:11.393 21102 21102 F DEBUG : x0 0000000000000000 x1 00000077a53d71ef x2 7265646f63654400 x3 726f7463656c6553 [...] 03-23 14:47:11.395 21102 21102 F DEBUG : backtrace: 03-23 14:47:11.395 21102 21102 F DEBUG : #00 pc 00000000034521d0 /data/app/com.android.chrome-xLjRNvGCAeF2ZEpHu2gC4g==/base.apk (offset 0xa0f000) Pretty bad. Affecting various apps including GMail. Solved by updating Chrome. Luckily for them (and us) Google Play was not affected.
- oolonthegreat 6y ago
  > Cause: null pointer dereference that is actually pretty funny :')
js2 6y ago
Still seeing 100x the normal crash rate across all our Android apps that use Webview on devices that have either com.android.chrome or com.google.android.webview 89.0.4389.90 installed. Started yesterday around 2 PM US/Eastern.
DoomHotel 6y ago
I wonder if Google's QA is getting worse. I had a broken Android Google Maps update last week where street view had multiple severe bugs: it didn't even show the blue lines when enabling the street view layer, split screen didn't show the marker for the virtual observer's location, and tapping in the map in split screen just rotated the view instead of moving the viewer to the new location. Something this broken should never, ever make it out the door.
- b0afc375b5 6y ago
  I attended a webinar from a famous e-commerce platform. They mentioned that they no longer have QA and make use of CI/CD pipelines (edit: canary deployment, feature flags, etc.) to move fast and break things. I assumed that everyone else no longer has QA and just pushes emergency fixes in a JIT fashion. I also assumed that our company is traditional and backwards for still having QA.
  - blacktriangle 6y ago
    Dedicated QA is amazing and it blows my mind that people think that CI is a 1:1 replacement. CI is great, but testing is for the bugs you anticipate. Good QA people find all the bugs you never even thought of.
oolonthegreat 6y ago
does anyone know if they released any technical details? isn't webview open source?

88 points by neversaydie 6y ago

There is a misconception that removing Android Webview is "fixing" the problem. The proper word is that removing the updated Android Webview is "a temporary workaround" until an updated Android Webview is released. You cannot remove Android Webview because it is an essential part of Android. By "removing" the package, you are removing the latest update and revert back to the old (and probably insecure) version of Android Webview that came with the initial Android version of the phone. In the end, Google eventually released a new package now, so you can perform the update of Android Webview to the latest version.

jmartrican 6y ago
Dont see an update option for this app in the app store. It says last release was March 12th. How exactly do I get this update?
- 51Cards 6y ago
  I was able to just do a Chrome update to solve the issue on my phone.
- numpad0 6y ago
  No action required if you’re not experiencing issues. If you’re affected and not able to update, trying to “uninstall updates” for “Android System WebView” will revert it to ROM-included unaffected version and solve it.
  - CodesInChaos 6y ago
    That sounds very risky, since untrusted code shown in any webview can then exploit long fixed chrome bugs.
    - DoomHotel 6y ago
      I uninstalled the updates and was then able to install the March 22 version. It didn't offer any update until I did that.
kevincox 6y ago
I assume that this is uninstalling updates, which means that you are back to the WebView that came with your phone. This obviously opens a whole new can of worms as now you may be years of security updates behind. I really wish that popular stores had an option to roll back one version until the next update came out.
- ProAm 6y ago
  This is why auto-update is turned off for me. You just cant trust blindly updating things because there is no undo button for breaking changes.
  - kevincox 6y ago
    How do you evaluate updates? And how do you rollback? It seems like you are still basically relying on "find the APK somewhere" for rollbacks still.
    - ProAm 6y ago
      Really if I do not need new functionality and it's not a sensitive app (i.e. one that deals with funds, email, text, etc..) I just dont. Some of my APKs are REALLY REALLY old. If i get a new phone, I save them off and roll them over. If its for an app that has potentially significant security issues I'll wait a while and monitor the reviews and check the web before I update (I'll also backup the installed APK before updating) so I can rollback if I need to. But the number of apps on my phone for this is very slim.
perryizgr8 6y ago
It is definitely fixing the problem. As is the entire phone is unusable because you cannot open many of the most important apps.

BiteCode_dev 6y ago

The problem is, I didn't issue an update on my android phone this week. I don't even have a Google play account and use ApkPure. Yet this morning my applications started to break. How did the update occured on my phone ?

SahAssar 6y ago
IIRC system components can be OTA updated in the background. The only way to get around that is to use a de-googled android distro.

51Cards 6y ago

Experienced this this morning on my Pixel 1. Woke up to several apps in a crash loop every few seconds and most others crashing on startup. After a reboot didn't solve anything I thought perhaps my (older) phone had suffered a RAM failure since I hadn't updated anything in several days. After finding an article about the issue I updated Chrome which corrected the issue.

exabrial 6y ago

How does something like this make it to production? I couldn't reply to any work emails for the greater part of the evening yesterday. Adds more evidence to the case of "never use Google for your business".

throwdbaaway 6y ago
I guess the team has Q1 OKR to meet and/or error budget to spend? Either way, it's bad. I am not saying the bug would have been prevented if the team operates differently. Brown paper bag release happens. However, the way this incident blew up suggests that the bad release made it to too many users in too short of a time, exactly the behavior incentivized by meeting (or exceeding) an OKR while having some spare error budget.
- gowld 6y ago
  What do you think "error budget" means? (Hint, it doesn't mean "amount of intentionally deployed errors we allow".) Do you believe WebView should never update because it's never OK to spend error budget?
  - room500 6y ago
    That is a strawman argument. Of course WebView should update. The question is whether the structure of incentives contributed to a risky release. If you "play the game", an error budget causes your releases to become riskier and riskier as the budget grows. This is optimal because a riskier release allows you to accelerate changes, allowing you to hit other OKR. The issue is that this will eventually cause very risky releases (where you would slow down otherwise). In a sense, an error budget does eventually become "amount of intentionally deployed errors" because you are incentivized to increase risk until an error does appear.
  - topher200 6y ago
    Interestingly, the Google SRE book[1] mentions scenarios where they practice exactly this: > The solution to this Chubby scenario is interesting: SRE makes sure that global Chubby meets, but does not significantly exceed, its service level objective. In any given quarter, if a true failure has not dropped availability below the target, a controlled outage will be synthesized by intentionally taking down the system. I doubt there are any customer-facing systems that practice this technique. Just super interesting that it exists. [1]: https://sre.google/sre-book/service-level-objectives/#xref_risk-management_global-chubby-planned-outage https://sre.google/sre-book/service-level-objectives/#xref_r...

amelius 6y ago

At least you can write/bundle your own render engine on Android, something which is not possible on iOS.

chrismorgan 6y ago
But if your app is now 200MB instead of 2MB, know that there are a significant number of users that simply won’t install your app (or will remove it) because they don’t have either space or bandwidth. Cheaper Android phones regularly have almost no spare space (like, well under 2GB available to user space when empty, even though the phone may have been advertised as having 8GB, and even that 2GB may well be partitioned stupidly so that you have less than 1GB available to install apps on).
- throwaway53453 6y ago
  Isn't the whole point of WebView that Firefox et al can provide it as well as Chrome?
  - meibo 6y ago
    Yes, Firefox can act as a WebView implementation on any Android phone. Apps don't need to ship this.
    - chrismorgan 6y ago
      No it can’t. https://wiki.mozilla.org/Mobile/GeckoView https://wiki.mozilla.org/Mobile/GeckoView: “GeckoView serves a similar purpose to Android's built-in WebView, but it has its own APIs and is not a drop in replacement.” The purpose of WebView has certainly never been about using a different engine; rather, it’s to have only one copy of a browser for apps to use as a widget, rather than each app bundling its own browser which uses vast amounts of space and raises serious security concerns.
  - chrismorgan 6y ago
    The situation I’m responding to is the suggestion that you bundle your own browser rather than using one provided by Android System WebView or equivalent.

tantalor 6y ago

This started happening for me last night. Just updated webview to 89.0.4389.105 and everything works now.

redsky17 6y ago

Well, I'm glad to see this posted here... I was concerned that somehow my phone got infected with some kind of malware after a reboot didn't fix it. It's concerning to me that an update was pushed out that apparently wasn't tested very well.

Tade0 6y ago

My phone rebooted a few times during the last week. At first I thought it was just planned obsolescence killing a three year old device, but fortunately that doesn't seem to be the case.

topher200 6y ago
I don't believe that a system reboot is a symptom of this issue, just app crashes. Also, the bug started yesterday afternoon. I doubt this has anything to do with your system reboots.

jerome-jh 6y ago

Actually just noticed that on my wife's phone: 03-23 14:47:11.393 21102 21102 F DEBUG : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0 03-23 14:47:11.393 21102 21102 F DEBUG : Cause: null pointer dereference 03-23 14:47:11.393 21102 21102 F DEBUG : x0 0000000000000000 x1 00000077a53d71ef x2 7265646f63654400 x3 726f7463656c6553 [...] 03-23 14:47:11.395 21102 21102 F DEBUG : backtrace: 03-23 14:47:11.395 21102 21102 F DEBUG : #00 pc 00000000034521d0 /data/app/com.android.chrome-xLjRNvGCAeF2ZEpHu2gC4g==/base.apk (offset 0xa0f000) Pretty bad. Affecting various apps including GMail. Solved by updating Chrome. Luckily for them (and us) Google Play was not affected.

oolonthegreat 6y ago
> Cause: null pointer dereference that is actually pretty funny :')

js2 6y ago

Still seeing 100x the normal crash rate across all our Android apps that use Webview on devices that have either com.android.chrome or com.google.android.webview 89.0.4389.90 installed. Started yesterday around 2 PM US/Eastern.

DoomHotel 6y ago

I wonder if Google's QA is getting worse. I had a broken Android Google Maps update last week where street view had multiple severe bugs: it didn't even show the blue lines when enabling the street view layer, split screen didn't show the marker for the virtual observer's location, and tapping in the map in split screen just rotated the view instead of moving the viewer to the new location. Something this broken should never, ever make it out the door.

b0afc375b5 6y ago
I attended a webinar from a famous e-commerce platform. They mentioned that they no longer have QA and make use of CI/CD pipelines (edit: canary deployment, feature flags, etc.) to move fast and break things. I assumed that everyone else no longer has QA and just pushes emergency fixes in a JIT fashion. I also assumed that our company is traditional and backwards for still having QA.
- blacktriangle 6y ago
  Dedicated QA is amazing and it blows my mind that people think that CI is a 1:1 replacement. CI is great, but testing is for the bugs you anticipate. Good QA people find all the bugs you never even thought of.

oolonthegreat 6y ago

does anyone know if they released any technical details? isn't webview open source?