33 ms·

The title is "Why Google Pixel lags 10x more than Moto Z" but the content is just a microbenchmark of fsync() without anything resembling a measurement of UI la

by darren_ 10y ago

The title is "Why Google Pixel lags 10x more than Moto Z" but the content is just a microbenchmark of fsync() without anything resembling a measurement of UI lag or that the 10x slowdown in fsync() actually means a 10x (or any) increase in said lag. The author's previous blogpost (http://taras.glek.net/post/Laggy-phones-and-misleading-benchmarks/ http://taras.glek.net/post/Laggy-phones-and-misleading-bench...) contends that 4K writes are a good proxy for phone lag, but has no evidence or measurements either, just the author's contention. It could all well be true! And it's great the author's dug up some concrete areas the pixel team could potentially improve at (dumb question: couldn't the pixel be updated to use most of these FS tweaks? could an in-use pixel FS be converted to use f2fs?). But I don't think the author has done a good job demonstrating an actual relation to UI lag or anything to do with the phone's perceived performance. disclaimer: googler, but I do iOS things.

izacus 10y ago
So much clickbait :/ Under no condition the Pixel lags "10x as much as Moto" when actually using it. It's a single benchmark of an FS performance for applications that do I/O on main thread. Something that is actively discouraged by things like StrictMode checks.
- IshKebab 10y ago
  To be fair he did say the difference would be more obvious after a year of use.
  - amyjess 10y ago
    Yes, this has shades of the 2012 Nexus 7 flash degradation issues...
    - iainmerrick 10y ago
      I agree with the previous commenters that the clickbait headline really oversells this article, but this is still a useful investigation. The 2012 Nexus 7 is a good comparison. It had slow, cheapskate flash memory, which degraded over time. But why should that cause UI jitter? As several people have pointed out, well-written apps use separate threads for UI and I/O. There were some big UI jitter problems in the OS, but there was a big effort to fix those in Jellybean, which the Nexus 7 shipped with. The problem is that just using separate threads isn't enough to keep UI and I/O completely separate. The UI uses the GPU, and the GPU fights with the kernel over access to I/O resources. Maybe the UI thread or GPU driver has a cache miss and needs code paged in; maybe the OS is busy writing out another page. If the Motorola phone really does have 10x lower I/O latency, that seems very significant, but we don't know if it really affects the UI without measurements. Just setting 'nobarrier' on the file system sounds dangerous though. I'd be very wary of that unless they have really good arguments and measurements to show it's safe. I write Android apps, and I/O reliability is a big headache. I'd put the success rate for simple file system operations at only around 99.9% -- as soon as your users number in the thousands, there are always a few users whose phones fail in bizarre ways. Most of the problems show up on Samsung phones, but that might just be because they're the most popular brand. Apple hardware definitely ages much, much more gracefully than any Android hardware I've seen. That might be okay if everyone got a new phone every two years, but that's definitely not the case; tons of people are using old phones.
      - emn13 10y ago
        I'm skeptical that off-main-thread I/O is a panacea for high I/O latency. It's certainly a boon; to be sure - but at the end of the day you want that I/O to complete. Showing more frames while you wait for that I/O to complete is going to paper over (very) small delays, but even with a fluid GFX, an app can still feel frustratingly slow. In fact, smooth but overlong animations can feel more laggy simply because the time it takes for the phone to do what you actually wanted is longer. If 4k of I/O take 16ms or more with great regularity (and p99 seems rather frequent to me), you're going to regularly hit high multiples of that in any kind of mildly intensive workload (which writes a lot more than once, and a lot more than 4k). And at high multiples of that, no amount of off-thread I/O is going to be able to hide that latency. So sure: asynchronicity will make slowless a little less ugly. And that's about it. (On the other hand, fsync is a pretty-heavy handed tool, and he doesn't show the latency for non synced I/O - so maybe the problem is just that software is too conservative - I mean, how often do phones crash, and how bad is losing a small amount of data then?)
        iainmerrick 10y ago
        how often do phones crash, and how bad is losing a small amount of data then? Phones crash a lot. Losing data can be anything from a minor nuisance (lost a few seconds of work) to a massive headache (some important file was subtly corrupted).
        emn13 10y ago
        Corruption is an orthogonal issue - losing any amount of data, even a single bit, can lead to corruption. Avoiding corruption by lowering I/O latency is not a viable strategy - at least not given "realistic" latencies and realistic frequency of I/O. In theory - sure, if I/O were once per day, and latency 1ns, you might accept the low chance of corruption without mitigation. In practice, you almost certainly need some concept of atomic actions or transactions to deal will corruption. The more relevant question then isn't whether phones crash and cause corruption, it's whether the additional amount of committed transactions you lose by relaxing durability requirements matters. You're always at risk of losing those in flight - does it matter if you lose the photo you just took? It depends, but I suspect the linked articles fsyncs/second is certainly massively excessive. The "bad" case was after all still just 1/50th of a second, i.e. much less than the blink of an eye. My contention is that phones don't crash often enough to make sub-second durability worthwhile. Most phones I come across certainly crash less than once a day, and losing your past 1 second of actions once a day sounds acceptable to me. In short: the benchmark is not directly useful. Perhaps it correlates with some relevant aspect of I/O performance (but that's not clear), or perhaps software is buggy and actually tries to fsync more than 50 times a second (but then: fix that), but it may simply be a meaningless benchmark.
- deleted 10y ago
  [deleted]
IshKebab 10y ago
> couldn't the pixel be updated to use most of these FS tweaks That would require reformatting, so probably not.
- leni536 10y ago
  Well, fstransform can reformat filesystems in place, so theoretically it might be possible (it cleverly uses sparse files for this). Not with the filesystems mounted though, so the update should be applied at reboot and it potentially takes a long time. Also very risky, especially with a near full drive.
JepZ 10y ago
fsync benchmarks are stupid. (neither a Pixel fan or hater here)
- reynoldsbd 10y ago
  I am willing to buy your argument, but you need to tell me why
- mulletbum 10y ago
  Please tell us why? If you have expertise with it I would like to know the flaw, I have zero experience with it.
  - cowsandmilk 10y ago
    Basically, the benchmark is testing a phone that doesn't run the code for fsync vs. a phone that does. The phone that has a no-op for fsync will clearly be faster at not running any code than the one that actually does what the programmer instructed. It is like asking what is faster: def sin(x): return 0 def sin(x): actually calculate the sin(x) More generally, how often do you call fsync in your code? I very rarely call it and generally allow the operating system to write the file to disk when it is convenient. If calling fsync is critical to your application, then the Moto Z has broken your app by making fsync a no-op. If it isn't critical to your application, then why are you calling it?
    - tinco 10y ago
      If you use a database for storing your apps data like sqlite, which you should, then you're doing an fsync on virtually every insert/update operation. One of the main reasons for using a database is that it takes away the headache of deciding when and how to fsync. If you manage application state yourself, and the phone does power off or kill your app at an unfortunate time, you might up with a corrupted application state. If you're a big company like Facebook or Google with millions of users this will be happening multiple times per day, and produce a slew of complaints about your app. Depending on the exact mechanism and implementation, making fsync a no-op is not necessarily bad. As long as it can (with highest priority) fsync all of its buffers upon notification of loss of power and be done before all capacitance is lost then there should be no trouble at all. Of course, this requires some coordination between the hardware, the OS and the filesystem which might be non-trivial even for a fully integrated company like Samsung so it could be it simply doesn't work at all like that.
rkangel 10y ago
> but has no evidence or measurements either, just the author's contention He both produces some measurements he has made, and cites an academic paper that corroborates. It's not certain proof but it's a long way from 'no evidence'.
- btian 10y ago
  I just read the paper and it doesn't say or imply anything about lag. Can you point me to any evidence related to the title, i.e., Pixel lag is 10x worse than Moto Z?

by darren_ 10y ago

izacus 10y ago

So much clickbait :/ Under no condition the Pixel lags "10x as much as Moto" when actually using it. It's a single benchmark of an FS performance for applications that do I/O on main thread. Something that is actively discouraged by things like StrictMode checks.

IshKebab 10y ago
To be fair he did say the difference would be more obvious after a year of use.
- amyjess 10y ago
  Yes, this has shades of the 2012 Nexus 7 flash degradation issues...
  - iainmerrick 10y ago
    I agree with the previous commenters that the clickbait headline really oversells this article, but this is still a useful investigation. The 2012 Nexus 7 is a good comparison. It had slow, cheapskate flash memory, which degraded over time. But why should that cause UI jitter? As several people have pointed out, well-written apps use separate threads for UI and I/O. There were some big UI jitter problems in the OS, but there was a big effort to fix those in Jellybean, which the Nexus 7 shipped with. The problem is that just using separate threads isn't enough to keep UI and I/O completely separate. The UI uses the GPU, and the GPU fights with the kernel over access to I/O resources. Maybe the UI thread or GPU driver has a cache miss and needs code paged in; maybe the OS is busy writing out another page. If the Motorola phone really does have 10x lower I/O latency, that seems very significant, but we don't know if it really affects the UI without measurements. Just setting 'nobarrier' on the file system sounds dangerous though. I'd be very wary of that unless they have really good arguments and measurements to show it's safe. I write Android apps, and I/O reliability is a big headache. I'd put the success rate for simple file system operations at only around 99.9% -- as soon as your users number in the thousands, there are always a few users whose phones fail in bizarre ways. Most of the problems show up on Samsung phones, but that might just be because they're the most popular brand. Apple hardware definitely ages much, much more gracefully than any Android hardware I've seen. That might be okay if everyone got a new phone every two years, but that's definitely not the case; tons of people are using old phones.
    - emn13 10y ago
      I'm skeptical that off-main-thread I/O is a panacea for high I/O latency. It's certainly a boon; to be sure - but at the end of the day you want that I/O to complete. Showing more frames while you wait for that I/O to complete is going to paper over (very) small delays, but even with a fluid GFX, an app can still feel frustratingly slow. In fact, smooth but overlong animations can feel more laggy simply because the time it takes for the phone to do what you actually wanted is longer. If 4k of I/O take 16ms or more with great regularity (and p99 seems rather frequent to me), you're going to regularly hit high multiples of that in any kind of mildly intensive workload (which writes a lot more than once, and a lot more than 4k). And at high multiples of that, no amount of off-thread I/O is going to be able to hide that latency. So sure: asynchronicity will make slowless a little less ugly. And that's about it. (On the other hand, fsync is a pretty-heavy handed tool, and he doesn't show the latency for non synced I/O - so maybe the problem is just that software is too conservative - I mean, how often do phones crash, and how bad is losing a small amount of data then?)
      - iainmerrick 10y ago
        how often do phones crash, and how bad is losing a small amount of data then? Phones crash a lot. Losing data can be anything from a minor nuisance (lost a few seconds of work) to a massive headache (some important file was subtly corrupted).
        emn13 10y ago
        Corruption is an orthogonal issue - losing any amount of data, even a single bit, can lead to corruption. Avoiding corruption by lowering I/O latency is not a viable strategy - at least not given "realistic" latencies and realistic frequency of I/O. In theory - sure, if I/O were once per day, and latency 1ns, you might accept the low chance of corruption without mitigation. In practice, you almost certainly need some concept of atomic actions or transactions to deal will corruption. The more relevant question then isn't whether phones crash and cause corruption, it's whether the additional amount of committed transactions you lose by relaxing durability requirements matters. You're always at risk of losing those in flight - does it matter if you lose the photo you just took? It depends, but I suspect the linked articles fsyncs/second is certainly massively excessive. The "bad" case was after all still just 1/50th of a second, i.e. much less than the blink of an eye. My contention is that phones don't crash often enough to make sub-second durability worthwhile. Most phones I come across certainly crash less than once a day, and losing your past 1 second of actions once a day sounds acceptable to me. In short: the benchmark is not directly useful. Perhaps it correlates with some relevant aspect of I/O performance (but that's not clear), or perhaps software is buggy and actually tries to fsync more than 50 times a second (but then: fix that), but it may simply be a meaningless benchmark.
deleted 10y ago
[deleted]

IshKebab 10y ago

> couldn't the pixel be updated to use most of these FS tweaks That would require reformatting, so probably not.

leni536 10y ago
Well, fstransform can reformat filesystems in place, so theoretically it might be possible (it cleverly uses sparse files for this). Not with the filesystems mounted though, so the update should be applied at reboot and it potentially takes a long time. Also very risky, especially with a near full drive.

JepZ 10y ago

fsync benchmarks are stupid. (neither a Pixel fan or hater here)

reynoldsbd 10y ago
I am willing to buy your argument, but you need to tell me why
mulletbum 10y ago
Please tell us why? If you have expertise with it I would like to know the flaw, I have zero experience with it.
- cowsandmilk 10y ago
  Basically, the benchmark is testing a phone that doesn't run the code for fsync vs. a phone that does. The phone that has a no-op for fsync will clearly be faster at not running any code than the one that actually does what the programmer instructed. It is like asking what is faster: def sin(x): return 0 def sin(x): actually calculate the sin(x) More generally, how often do you call fsync in your code? I very rarely call it and generally allow the operating system to write the file to disk when it is convenient. If calling fsync is critical to your application, then the Moto Z has broken your app by making fsync a no-op. If it isn't critical to your application, then why are you calling it?
  - tinco 10y ago
    If you use a database for storing your apps data like sqlite, which you should, then you're doing an fsync on virtually every insert/update operation. One of the main reasons for using a database is that it takes away the headache of deciding when and how to fsync. If you manage application state yourself, and the phone does power off or kill your app at an unfortunate time, you might up with a corrupted application state. If you're a big company like Facebook or Google with millions of users this will be happening multiple times per day, and produce a slew of complaints about your app. Depending on the exact mechanism and implementation, making fsync a no-op is not necessarily bad. As long as it can (with highest priority) fsync all of its buffers upon notification of loss of power and be done before all capacitance is lost then there should be no trouble at all. Of course, this requires some coordination between the hardware, the OS and the filesystem which might be non-trivial even for a fully integrated company like Samsung so it could be it simply doesn't work at all like that.

rkangel 10y ago

> but has no evidence or measurements either, just the author's contention He both produces some measurements he has made, and cites an academic paper that corroborates. It's not certain proof but it's a long way from 'no evidence'.

btian 10y ago
I just read the paper and it doesn't say or imply anything about lag. Can you point me to any evidence related to the title, i.e., Pixel lag is 10x worse than Moto Z?