16 ms·

Why Google Pixel lags 10x more than Moto Z

484 points by tapper 10y ago

comex 10y ago
> The key option is nobarrier. This effectively makes fsync() a no-op and explains most of the difference in performance. In other words, because the Moto Z cheats. I suppose it's understandable... nobody should ever be blocking GUI animations on fsync, much less two of them in a row, but here we are.
- ChuckMcM 10y ago
  Not exactly a cheat. A calculated risk. A friend of mine were having the storage discussion on phones and realized that phones never really experience a 'power outage' in normal use. People use their phones, they get low in power, and then they plug them in. Or the battery gets so low the phone shuts itself off in an orderly manner. In the 'computer is a phone' perspective there is no sudden power loss unless the user yanks the battery out of the back. That is presumably a rare occurrence for your typical user. As a result, the big risk for phone file systms is that the phone crashes, but nearly every phone I've seen preserves memory contents as long as power is applied, so even a crashed phone, on reboot can pick apart the previous bits in memory and reconstruct what was going on before it crashed. Given that fairly unique to phone criteria I could see nobarrier as a legit option.
  - quotemstr 10y ago
    > reboot can pick apart the previous bits in memory and reconstruct what was going on before it crashed What you're proposing is possible in theory, but no general purpose operating system I've seen actually attempts to recover page cache state from uncleared RAM upon reboot.
    - jrockway 10y ago
      Linux has PRINTK_PERSIST [1] to recover the printk logs after (warm) reboot. [1] https://lkml.org/lkml/2012/3/13/45 https://lkml.org/lkml/2012/3/13/45
    - reitanqild 10y ago
      Something related though battery (or capacitor) backed write-cache on high end servers.
  - derefr 10y ago
    > unless the user yanks the battery out of the back. That is presumably a rare occurrence for your typical user. I've witnessed a number of phones with the "crumple zone"-alike feature of effectively ejecting the battery cover + battery out of the phone whenever you drop it.
    - ygra 10y ago
      I would hope dropping a phone is also a rare occurrence for typical phone users. Otherwise they're probably better off getting a Nokia 3210 instead of a smartphone with glass screen.
      - iopq 10y ago
        > I would hope dropping a phone is also a rare occurrence for typical phone users. I drop my phone all the time and I don't have a case. It's fine.
        awjr 10y ago
        I dropped my phone at the weekend without a case. I now have a shattered screen. :/ It's about the 5th time I've dropped my phone and this is the first time something like this has happened.
        josephg 10y ago
        Apparently the glass in phone screens develops micro fractures from being dropped. (Well, from hitting the ground). You can't see them with the naked eye, but they weaken the glass and they grow with each successive drop. It's probably not just bad luck that the glass happened to break after the Nth time you dropped it. It was bound to happen if you kept dropping the device.
        marvy 10y ago
        Me too, though the screen did get a small crack once. If there was a 1% chance of file system corruption each time I dropped my phone, it would have happened to me by now, I think. And usually when I drop my phone, the battery comes out. It used to happen only half the time, but I think the cover got loosened as a result of all the times I dropped it, so now the battery falls out pretty much every time.
  - digi_owl 10y ago
    I have noticed this with a cheap Window tablet i acquired recently. Left alone it will eventually hibernate, something my Android tablets have never attempted. Similarly, when it goes below 4% battery it will basically force a hibernate. That said, all mobile devices, laptops included, have a "force shutdown" option activated by holding the power button for x seconds.
  - bmm6o 10y ago
    This is true for the first few years, but aging batteries can cause problems. My phone is in a situation where under higher power draws (GPS + navigation) it will simply die when the battery falls to 25%. It's been doing this pretty reliably for a month or 2.
  - pvg 10y ago
    In the 'computer is a phone' perspective there is no sudden power loss unless the user yanks the battery out of the back. Or the battery is a couple of years old. Or the user lives someplace where it's cold outside.
    - tempestn 10y ago
      Neither of those should cause a sudden power loss. Even if your battery rapidly depletes, the phone will still shut itself down properly before it loses power.
      - pvg 10y ago
        It shouldn't yet sometimes it does.
- sbierwagen 10y ago
  fsync() has never been as reliable as the name implies, since flushing the disk cache kills performance system-wide. Fuzziness on what fsync does has... side effects: https://danluu.com/file-consistency/ https://danluu.com/file-consistency/ http://blog.httrack.com/blog/2013/11/15/everything-you-always-wanted-to-know-about-fsync/ http://blog.httrack.com/blog/2013/11/15/everything-you-alway... Filesystems optimized for flash, and for battery-backed systems in general, (laptops, phones) have some history: https://en.wikipedia.org/wiki/Flash_file_system https://en.wikipedia.org/wiki/Flash_file_system
  - quotemstr 10y ago
    The fundamental problem is that fsync is too powerful. Most of the time, what you really want is a write barrier --- you don't care about when writes happen, but you do care about their order. fsync is a write barrier, but it's also a synchronous flush. But don't listen to me. I like TxF too.
  - ioltas 10y ago
    Or in short, calling fsync() twice is a tweak under-known: the file itself needs to be flushed, as well as its parent directory.
- chillaxtian 10y ago
  > I suppose it's understandable... nobody should ever be blocking GUI animations on fsync, much less two of them in a row, but here we are. this is the thing that confused me.. why are apps doing disk i/o on the GUI thread?
  - elktea 10y ago
    Threads are hard, basically.
  - umanwizard 10y ago
    I think lots of stuff causes disk I/O that you might not even be aware of. For example, on iOS, +[CLLocationManager authorizationStatus] (checking if the app is allowed to use location or not) has to read a file [1]. Almost no app developers are aware of this. [1] I've been told this by guys who worked full-time on location stuff, but I haven't verified it myself and can't find it in docs, so take it with a grain of salt
    - azernik 10y ago
      fsync() in particular is usually only called on (well, after) disk writes though - I can see that being involved in all kinds of system API calls too, but hopefully the system makers would be savvy enough to use an actual write barrier instead of being fsync-happy.
  - quotemstr 10y ago
    Because they don't care enough not to. There are even developer tools to help applications not do work on the UI thread: https://developer.android.com/reference/android/os/StrictMode.html https://developer.android.com/reference/android/os/StrictMod...
  - digi_owl 10y ago
    Even if they don't, the kernel will likely drop everything to get that fsync out of the way. So a fsync happy app that is doing a file write in the background can cause the whole UI to lag because the kernel is busy dealing with said fsync call.
- busterarm 10y ago
  I wouldn't really say Moto Z cheats so much as that Google designed the entire software ecosystem of Android without ever talking to a hardware engineer. They simply do not care about optimizing the performance of the phone running their software. Otherwise they never would have chosen a garbage-collected language like Java for the platform in the first place (I understand they've ameliorated this concern recently). Or used ext4 like here.
  - c8g 10y ago
    or developer had to build separate app for x86, armv7, armv5, armv6, armv8, mips,...
    - deleted 10y ago
      [deleted]
    - busterarm 10y ago
      Really just arm* until the last couple of years. These choices were made forever ago.
  - sangnoir 10y ago
    > I wouldn't really say Moto Z cheats so much as that Google designed the entire software ecosystem of Android without ever talking to a hardware engineer Not sure if you're being serious or hyperbolic - but Google bought Android, Inc (an Andy Rubin startup). The team had a lot of Danger alumni - Danger being the makers of the HipTop - aka the original T-Mobile Sidekick (also running Java). I'm certain they knew a thing or two about hardware, they just made trade-offs you disagree with.
- rrego 10y ago
  So from reading the XFS FAQ linked in the article(http://xfs.org/index.php/XFS_FAQ#Write_barrier_support. http://xfs.org/index.php/XFS_FAQ#Write_barrier_support.), does this mean that Moto Z never flushes the cache to disk? Wouldn't the cache HAVE to be flushed as it fills up at some point to maintain coherence.
  - idrae 10y ago
    Yes, it has to be flushed from time to time. Normally this happens automatically when it is needed. fsync() just forces a direct flush.
- macspoofing 10y ago
  >In other words, because the Moto Z cheats. If you're not cheating, you're not doing it right.
Animats 10y ago
"They drove development of the filesystem specifically by Twitter/FB/etc workloads captured from the phone." Why would Twitter or Facebook apps need to write much to persistent storage? They don't do so in a browser.
- LoSboccacc 10y ago
  Phone memory is severely limited and they have to unload images as you scroll to get the scrolling smooth. Scrolling forward and backward is a common user habit, and netwok latency is high, so you gonna need a local cache. Lying on fsync doesn't seem a good option tho, even if developers abuse it.
  - Animats 10y ago
    But you don't need to fsync that. It's presumably in a temporary file, or you're just reading it.
    - refulgentis 10y ago
      You're the only one linking these two ideas together. The article expressly separates discussion of how Samsung gathered data to inform their filesystem design from discussion about Motorola's fsync implementation, and they're inherently separate concepts.
    - pjc50 10y ago
      I suspect what happens is the developer just dumps everything in a sqlite database, which will fsync for you on commit.
      - JBReefer 10y ago
        ^ I think this is it. App developers are taught to reflexively use the local db to store data, not files - but may not understand the difference performance characteristics of this db. Someone (like me before reading this post) working on an app for the first time might not realize they're thrashing out fsyncs with UPDATES.
ajdlinux 10y ago
Related: https://www.flamingspork.com/projects/libeatmydata/ https://www.flamingspork.com/projects/libeatmydata/
kibwen 10y ago
Offtopic: I love my old Moto G, but I've been reluctant to buy any new phones from Motorola since its purchase by Lenovo, due to Lenovo's history of shipping computers with rootkits and malware. Should I suspect that Motorola phones are compromised, or am I just being unreasonably paranoid?
- viraptor 10y ago
  In corporations like that don't expect a common theme. Different departments manage different devices and unless there's a company wide push for a culture change (which would leak to the public), I wouldn't expect changes any time soon. But on the other hand, if you are paranoid about security in Android, you should either go with a Google phone (quickest updates), or something like Copperhead.
  - amckinlay 10y ago
    I think the commenter is more worried that he'll be funding a company whose practices he doesn't support.
    - blunte 10y ago
      And with the consolidation of companies, it becomes increasingly difficult to find any company that does things right and is owned by a company that does things right (and has sibling companies that do things right).
- ccozan 10y ago
  Motorola is a rare example and even more interesting regarding the relation with Lenovo. Since they keep an almost pristine Android, and hardware is rock solid, I think the paranoia is not necessary. My Moto X force is really unbreakable. Another interesting bit was that, instead Lenovo to "eat" the Motorola, it seems that it will move all its mobile handsets to the Moto name and leave quite some large space to develop, while supporting it with lots of money. I was really really skeptical at the beginning and I also told myself not to buy anymore from Moto under Lenovo, but in the main time my opinion changed and seems they go, let's say, not in the wrong direction, especially the Moto Z and the mods.
relics443 10y ago
Kinda makes me regret getting a Pixel to replace my Moto Z. Except the reception was terrible on the Z. Oh well. How is it that no one has made a worthy successor to the OG Turbo. That was a monster of a phone across the board.
- seunosewa 10y ago
  Is your Pixel phone noticeably slower than the Moto Z?
  - YCode 10y ago
    The author mentions it should take like a year for it to become noticeable.
  - relics443 10y ago
    I forgot my sarcasm tags
- Nursie 10y ago
  Are you sure you mean the Moto Z? It was only out a couple of months ahead of the pixel...
  - relics443 10y ago
    I am sure. What's your point? That it's difficult to get new phones in such a short time? You're correct. Although, to be technical it was a Moto Z Force Droid Edition...
    - Nursie 10y ago
      Not difficult, just unusual. As someone who buys my phones I tend to hang on to them for about 18 months on average and have just got a Z...
codedokode 10y ago
I don't think this could be interpreted as a visual lag. fsync() is usually used by databases, not UI libraries. So the title is misleading. My chinese Android phone has a more annoying hardware lag - a delay between touching the screen and touch event processing is over 100 ms. Any drum app is unusable. And if you try to scroll something up and down fast it is easy to see how the content on the screen lags behind finger movements.
- daenney 10y ago
  > I don't think this could be interpreted as a visual lag. fsync() is usually used by databases, not UI libraries. As others have pointed out, a lot of apps do I/O related work in the UI thread, mostly b/c people don't care enough not to (and sometimes it's hard). Flipping a toggle somewhere can easily cause a write to a sqlite database to need to happen. And it's for far more than just databases. > So the title is misleading. I don't think the title is that misleading, it is very much and significantly faster at storage operations than the Pixel, and those matter quite a bit in your daily usage.
  - kllrnohj 10y ago
    > I don't think the title is that misleading, it is very much and significantly faster at storage operations than the Pixel, and those matter quite a bit in your daily usage. All that was tested was fsync(), not "storage operations", and the Moto Z was faster at fsync() because it turns off fsync. But it's not clear why this matters to anything. Lag is usually the result of file system reads being slow (paging in code/resources), not writes being slow, so how does fsync performance matter?
    - daenney 10y ago
      > All that was tested was fsync(), not "storage operations", and the Moto Z was faster at fsync() because it turns off fsync. That's not what was tested, that's only what is shown in the graph. The lack of fsync contributes to it. I wrote a little fio benchmark driver to fill all available device storage with random 4k writes, print perf stats along the way One of the other things clearly mentioned is that b/c of Google's use of an additional FUSE filesystem they take a 30% performance hit: This means that on the Pixel every user IO gets a round-trip back into user-space before hitting the NAND. Fuse burns more CPU and slows down IO by up to 30%. Sure, the nobarrier trick makes a lot of difference but it's not the only thing in that post, nor was that the actual benchmark.
Tepix 10y ago
I hadn't heard about f2fs before. Sounds useful even on a notebook with SSD.
- puzzlingcaptcha 10y ago
  FWIW I've been using it for the past year for a root filesystem of a Debian NAS running off a USB stick and I didn't have any problems.
  - pmontra 10y ago
    Thabks. I was about to ask if Incould replace ext4 with f2fs on the SSD of my Ubuntu laptop. Fsyncs are still at the mercy of the SSD firmware AFAIK. I found this https://ubuntuforums.org/showthread.php?t=2326934 https://ubuntuforums.org/showthread.php?t=2326934 I'm going to study it. Any other first hand experience here on HN?
- peterwwillis 10y ago
  It's been around since 2012, one of many Flash-specific filesystems. And sure, it can be helpful for flash disks, as other filesystems can too. But in some benchmarks f2fs performs worse than ext4, and in others shows it is not syncing to disk as often as it should be, putting data unnecessarily at risk. Oh, and the fsck program used to crash on any filesystem errors (hopefully they've fixed that by now (edit: they appear to have fixed it in March)) The performance boost in this case is mainly due to the nobarrier option, which is potentially dangerous unless you have something like a battery-backed RAID controller to act as a persistent disk cache. If you back up all your stuff, could be nice. (A similar hack I used to use to eliminate iops-related slowdown was to implement a tmpfs mount for small files that got written a lot, and just rsync them once a minute to disk) One of the benefits of f2fs is that you can select the allocation and cleaning algorithms that it uses based on the flash chip in use, so before using it on your system, you might want to tune it to your application. https://www.kernel.org/doc/Documentation/filesystems/f2fs.txt https://www.kernel.org/doc/Documentation/filesystems/f2fs.tx...
wodenokoto 10y ago
My old 4S runs annoyingly slow. According to the article,the NAND, as it ages will slow down the phone, so will replacing my storage make my iPhone fast again?
- sgt 10y ago
  You may also want to make sure you have a decent amount of free space. If your capacity is at 95% you will most likely experience an even slower 4S than what is typical.
- wingerlang 10y ago
  What iOS version are you running? It might be cheaper to just buy a new one.
  - wodenokoto 10y ago
    9.02 ... And yeah, I wouldn't be surprised either, but I was intrigued by the prospect.
    - wingerlang 10y ago
      Some people swear that jailbreaking + installing various "speed up" tweaks (be it animation reduction or daemon killers) will help. Personally I don't use those kinds of tweaks but it might be worth a try.
- angry_octet 10y ago
  Highly unlikely. The flash doesn't slow down, but its hidden pool of spare blocks diminishes as random blocks fail. Delete some junk apps and old photos and you will give the storage allocator more room to play with. Also, disable background apps unless you really need them. Also Settings -> General -> Accessibility -> Reduce Motion, Reduce Transparency. Delete cookies from Chrome/Safari. Also, Settings -> Messages -> Expires -> 5 Minutes (instead of never).
darren_ 10y ago
The title is "Why Google Pixel lags 10x more than Moto Z" but the content is just a microbenchmark of fsync() without anything resembling a measurement of UI lag or that the 10x slowdown in fsync() actually means a 10x (or any) increase in said lag. The author's previous blogpost (http://taras.glek.net/post/Laggy-phones-and-misleading-benchmarks/ http://taras.glek.net/post/Laggy-phones-and-misleading-bench...) contends that 4K writes are a good proxy for phone lag, but has no evidence or measurements either, just the author's contention. It could all well be true! And it's great the author's dug up some concrete areas the pixel team could potentially improve at (dumb question: couldn't the pixel be updated to use most of these FS tweaks? could an in-use pixel FS be converted to use f2fs?). But I don't think the author has done a good job demonstrating an actual relation to UI lag or anything to do with the phone's perceived performance. disclaimer: googler, but I do iOS things.
- izacus 10y ago
  So much clickbait :/ Under no condition the Pixel lags "10x as much as Moto" when actually using it. It's a single benchmark of an FS performance for applications that do I/O on main thread. Something that is actively discouraged by things like StrictMode checks.
  - IshKebab 10y ago
    To be fair he did say the difference would be more obvious after a year of use.
    - amyjess 10y ago
      Yes, this has shades of the 2012 Nexus 7 flash degradation issues...
      - iainmerrick 10y ago
        I agree with the previous commenters that the clickbait headline really oversells this article, but this is still a useful investigation. The 2012 Nexus 7 is a good comparison. It had slow, cheapskate flash memory, which degraded over time. But why should that cause UI jitter? As several people have pointed out, well-written apps use separate threads for UI and I/O. There were some big UI jitter problems in the OS, but there was a big effort to fix those in Jellybean, which the Nexus 7 shipped with. The problem is that just using separate threads isn't enough to keep UI and I/O completely separate. The UI uses the GPU, and the GPU fights with the kernel over access to I/O resources. Maybe the UI thread or GPU driver has a cache miss and needs code paged in; maybe the OS is busy writing out another page. If the Motorola phone really does have 10x lower I/O latency, that seems very significant, but we don't know if it really affects the UI without measurements. Just setting 'nobarrier' on the file system sounds dangerous though. I'd be very wary of that unless they have really good arguments and measurements to show it's safe. I write Android apps, and I/O reliability is a big headache. I'd put the success rate for simple file system operations at only around 99.9% -- as soon as your users number in the thousands, there are always a few users whose phones fail in bizarre ways. Most of the problems show up on Samsung phones, but that might just be because they're the most popular brand. Apple hardware definitely ages much, much more gracefully than any Android hardware I've seen. That might be okay if everyone got a new phone every two years, but that's definitely not the case; tons of people are using old phones.
mda 10y ago
Any real proof that this actually causes user visible issues at all?
- Mikeb85 10y ago
  No. Just that it might, depending on how the Pixel's NAND memory degrades over time. My money is on the battery crapping out first.
carrot 10y ago
What does it matter?
carrot 10y ago
What does it matter?
blunte 10y ago
I wonder if this is why my Sony Xperia Z5 is so sluggish sometimes, despite having a Snapdragon 810 8-core CPU...
- kimshibal 10y ago
  It's not about hardware. My new android has twice spec than my 4 year old android phone. Yet, I'm still using my old phone for some apps. New phone has better spec, but it still has laggy scroll.
- vetinari 10y ago
  Snapdragon 810 is a big.LITTLE architecture; you don't have 8 CPUs available at the same time. Only 4. Check your running applications, the answer to sluggishness might be there.
  - mirsadm 10y ago
    All 8 cores can run at the same time on the 810.
    - vetinari 10y ago
      You are right, 810 has HMP. I somehow missed that.
joosters 10y ago
Other comments are querying the relevance of fsync() as a 'lag' benchmark, but I want to query whether fsync() is even meaningful at multiple calls per second. I know fsync() ensures data gets written to disk, but why does anyone care that it can happen so often? When a device crashes, some data (prior to the sync) may be lost, but do we really need multiple checkpoints per second to ensure only sub-second data loss? I'd be content with a couple of minutes worth of loss even on my main PC, with its lack of battery backup. To enforce rapid syncs on a phone seems utterly pointless. Keep the syncs for meaningful checkpoints, like buying something in an app or marking a message as sent. Multiple fsync() calls per second are a total waste.
- jchrisa 10y ago
  One of the few times they make sense is in a database server application where you might have hundreds of thousands of connected clients.
- charleslmunger 10y ago
  Loss isn't the same thing as corruption, which is the real fear here. What you're looking for (I think) is SQLite in WAL mode, with PRAGMA SYNCHRONOUS=NORMAL. That's ACI but not D.
agumonkey 10y ago
I don't see the hard link between GUI smoothness and io. You can have perfectly smooth 12000fps rendering and crappy IO, everything will just be "loading...".
pas 10y ago
Is there an Android sample app that show how to do I/O properly (off the main thread, and so on)? I know there's some kind of system to prevent network I/O on the main thread... w... why... I dare to ask the obvious, w-wwhy isn't there a warning for simple disk I/O too?
- cuu508 10y ago
  Android's StrictMode can monitor and notify about both network IO and disk IO.
  - pas 10y ago
    I found that, and looks very powerful, but that's a bit more than a simple switch in Android Studio, yet less than a warning on the Play Console, because you have to add it to your code (so not a static analyzer).
archivator 10y ago
This articles conflates a lot of things but it also has the priorities somewhat wrong. 1) fsync cost. Yes, fsyncs are dangerously slow in any Android app. (SQLite for example is a common culprit. Shared Prefs are another). HOWEVER, it's possible that flushes cause reads to be queued behind them (either in the kernel or on the device itself) which is even worse because 2) Random read cost is super super important. Android mmap's literally everything and demand paging is particularly common AND horrendous as a workflow. To add insult to injury, Android does not madvise the byte code or the resources as MADV_RANDOM, so read-ahead (or read-around) kicks in and you end up paging in 16KB-32KB where you only wanted 4KB. Also, history has shown custom flash-based file system on Android to be a world of pain. yaffs, jffs have some pretty atrocious bugs/quirks. I'd much rather see the world unify on common file systems, optimized for flash-like storage, rather than OEMs shipping their own in-house broken file "systems" (I'm looking at you, Samsung).
- jhasse 10y ago
  Why can't f2fs be that common file system?
  - jdcarter 10y ago
    I just read the F2FS paper and it seems very well-designed to match the physical properties of flash, plus some interesting capabilities to keep hot/cold data separate. If there's something wrong with F2FS, let's fix it. This seems like a far better place to start from than any filesystem designed around the assumptions of a spinning disk.
  - TazeTSchnitzel 10y ago
    It's in the mainline Linux kernel now, it's hardly some proprietary obscure vendor thing.
    - archivator 10y ago
      That's fair, it's a better state than the previous attempts. Still, it's not as tested as, say btrfs and ext4. Can't wait to see its particular quirks.
terrywang 10y ago
Nexus 5X running rooted AOSP 7.1.1 /storage/emulated fuse /dev/block/dm-0 /data ext4 rw,seclabel,nosuid,nodev,noatime,noauto_da_alloc,errors=panic,data=ordered,inode_readahead_blks=8 (no nobarrier mount option) In theory Google should be able to easily change the /data ext4 mount option, why didn't Google?
bla2 10y ago
If you're calling fsync on your painting thread, you're going to have trouble hitting 60FPS no matter what. What a clickbaity post.
kasabali 10y ago
> ... noatime,nodiratime, ... noatime implies nodiratime. I shrug off when I see newbies copy pasting this to their /etc/fstab but this is in a mainstream Android device??
- kiallmacinnes 10y ago
  I personally hate options that both have their own behavior, and imply another options behaviour unnecessarily. I'll be explicit when I use these options, and call them all out. That way, when someone who's not intimately familiar with the code reads it, they can actually tell what it's doing at a glance.
  - kasabali 10y ago
    I won't comment on your general remark but I don't think it is relevant to this specific case. "noatime" has a simple meaning and a clear name that represents its function: it stops updating inode access times. If you don't want your filesystem updating access times you use this mount option and it's done. Functionality of "nodiratime" is the one that's more specific & esoteric and subset of "noatime", thus it has a more specific name. It's not that "noatime" implies "nodiratime" , but more like "nodiratime" implements a subset of functionality of "noatime". Nothing to hate really.
Grazester 10y ago
"try to avoid buying devices that will slow down to point of being unusable as NAND wears out (ala Nexus 7, Nexus 6)" Has anyone experienced this issue with their Nexus 6? My phone is more than 2 years old and I have no noticeable slowdown. The pixel might have the slower storage option but it has no effect on usability. From what I have read its UI performance is the best of any Android phone yet. "The Pixels are fast — noticeably faster than Samsung's Galaxy S7. On performance alone, these are easily the best Android phones you can buy." http://www.theverge.com/2016/10/18/13304090/google-pixel-phone-review-pixel-xl http://www.theverge.com/2016/10/18/13304090/google-pixel-pho...
- discordianfish 10y ago
  I've got a Nexus 5x and the performance was disappointing from the start. But maybe it just feels like that because I came from an iphone 5s.
  - Grazester 10y ago
    Oh no buddy the Nexus 5x had some issues when initially released. Google did work these issues out though
- lanestp 10y ago
  Actually, yes. At work we have a Nexus 6 testing device. It started off strong but a few months ago it started lagging badly on videos. It even struggles with gif length mp4s. The problem is notable because it is the only phone with this problem. Even our old Galaxy S4 performs better.
  - Grazester 10y ago
    What kind of videos? Do you have a link perhaps where I could test this?
    - lanestp 10y ago
      Giphys and rewarded video ads. We are a gaming company so you can see where our priorities lie!
moonlander 10y ago
This makes very little sense. The author shows a fsync() benchmark comparing an actualy fsync (Pixel on ext4 without nobarrier) with a nobarrier (no-op fsync) alternative. The only thing this benchmark shows is that a no-op is faster than the real thing.
- idbehold 10y ago
  The title of the article is "Why [the] Google Pixel lags 10x more than [the] Moto Z." The author showed that the Moto Z is using a no-op fsync while the Pixel is doing the real thing. I think the author appropriately explained and answered the title prompt.
  - moonlander 10y ago
    But the author fails to prove that fsync() is in any way the bottleneck for common I/O operations in everyday use of a smartphone.
    - idbehold 10y ago
      Was that the hypothesis the author was trying to prove? Because I don't believe it was.
      - monocasa 10y ago
        The title is literally "Why [the] Google Pixel lags 10x more than [the] Moto Z". If he doesn't show that fsync(3) is the gating factor, then the fact that the Moto Z nops out fsync doesn't mean that much.
- cbsmith 10y ago
  Saying "no-op is faster than fsync" is just not as click-baity a title. There is a good question though... do you really need fsync on Android? If you don't, why are you calling it?
digi_owl 10y ago
I suspect it is not the fsync directly that leads to lag. But that when a fsync comes through, Linux stops doing anything else for the duration.
- kasabali 10y ago
  Linux doesn't do no such thing. ext* filesystems do (especially ext3).
kozak 10y ago
So this might be the reason why some mobile devices (like my ASUS TF700T tablet) degrade in performance so badly over time. Interesting.
- givinguflac 10y ago
  It's certainly part of it, but there are other possible contributors. For example, the original Nexus 7 suffered from severe NAND degradation over time, as have some other devices.
  - Namidairo 10y ago
    Both of those devices are just the victims of Asus cutting corners with cheap flash and Tegra 3's not so great memory bandwidth.
colanderman 10y ago
The fundamental problem sounds like it's that SQLite insists on not running in its own thread. This, coupled with the fact that Linux has no way to issue a true (i.e., non-blocking) write barrier, means that there is no way to implement write barriers in user space. Whereas, if SQLite did issue I/O from a separate thread, one could easily implement an "async commit" function which guaranteed consistency and ordering but not necessarily durability (i.e. a write barrier). This would suffice I suspect for 95% of usage in applications: users will probably be OK if their phone loses the last few seconds of user input before an OS crash, so long as everything else is left intact. EDIT: In fact, Postgresql has an option to permit exactly this behavior: https://www.postgresql.org/docs/9.6/static/wal-async-commit.html https://www.postgresql.org/docs/9.6/static/wal-async-commit.... This is possible in Postgresql because, unlike SQLite, I/O does not run in the client thread.
- swift 10y ago
  It is possible to run SQLite entirely on a separate thread and avoid blocking the main thread on its I/O, though. It just requires that you're willing to ship the data you need to read and write between threads and handle the synchronization yourself. I've written applications that do this and it works very well, but it's certainly more work than if SQLite supported off-main-thread I/O natively.
  - colanderman 10y ago
    Of course. But that's more work that I bet 95% of app developers are willing/able to put in. (Heck I've been doing this stuff for years and still have to spend time analyzing my code to make sure I'm not inadvertently holding locks in places where I shouldn't be.)
- kllrnohj 10y ago
  > The fundamental problem sounds like it's that SQLite insists on not running in its own thread. Pretty much all apps DO have SQLite running on its own thread, or at least a background thread pool of some sort. Android gets very cranky at the developer if they don't do this - there's tons of warnings, both runtime and in the form of lint. Database access, or anything involving fsync(), is exceptionally rare on the UI thread or any user-latency-critical thread.
ElijahLynn 10y ago
I always wondered why my wife's Moto X is still buttery smooth after 3 years!! I love that phone. I have a Pixel but she still has her Moto X and it is really great at being smooth, plus the wave to wake and other gestures are really unmatched by the Pixel. Love this write up/research! Hopefully it will teach the Pixel team a few things, or maybe they already knew but will now have the ammo to take to Product and change things!!
bitmapbrother 10y ago
If f2fs is so superior to ext4 then why doesn't its creator, Samsung, use it on their phones?
- strcat 10y ago
  It's significantly better on some benchmarks and significantly worse on others. There's no clear winner and ext4 is much more widely tested and has way more development time focused on it. Looking at only a few aspects of performance can result in either of ext4 or f2fs being seen as a clear winner but it's not representative of the whole picture.
bitmapbrother 10y ago
From Tim Murray, performance engineer on Pixel: >that fsync blog post floating around is pretty much bogus. also nobody should use nobarrier, it's not safe at all https://twitter.com/t_murray/status/808373275860418560 https://twitter.com/t_murray/status/808373275860418560

484 points by tapper 10y ago

comex 10y ago

> The key option is nobarrier. This effectively makes fsync() a no-op and explains most of the difference in performance. In other words, because the Moto Z cheats. I suppose it's understandable... nobody should ever be blocking GUI animations on fsync, much less two of them in a row, but here we are.

ChuckMcM 10y ago
Not exactly a cheat. A calculated risk. A friend of mine were having the storage discussion on phones and realized that phones never really experience a 'power outage' in normal use. People use their phones, they get low in power, and then they plug them in. Or the battery gets so low the phone shuts itself off in an orderly manner. In the 'computer is a phone' perspective there is no sudden power loss unless the user yanks the battery out of the back. That is presumably a rare occurrence for your typical user. As a result, the big risk for phone file systms is that the phone crashes, but nearly every phone I've seen preserves memory contents as long as power is applied, so even a crashed phone, on reboot can pick apart the previous bits in memory and reconstruct what was going on before it crashed. Given that fairly unique to phone criteria I could see nobarrier as a legit option.
- quotemstr 10y ago
  > reboot can pick apart the previous bits in memory and reconstruct what was going on before it crashed What you're proposing is possible in theory, but no general purpose operating system I've seen actually attempts to recover page cache state from uncleared RAM upon reboot.
  - jrockway 10y ago
    Linux has PRINTK_PERSIST [1] to recover the printk logs after (warm) reboot. [1] https://lkml.org/lkml/2012/3/13/45 https://lkml.org/lkml/2012/3/13/45
  - reitanqild 10y ago
    Something related though battery (or capacitor) backed write-cache on high end servers.
- derefr 10y ago
  > unless the user yanks the battery out of the back. That is presumably a rare occurrence for your typical user. I've witnessed a number of phones with the "crumple zone"-alike feature of effectively ejecting the battery cover + battery out of the phone whenever you drop it.
  - ygra 10y ago
    I would hope dropping a phone is also a rare occurrence for typical phone users. Otherwise they're probably better off getting a Nokia 3210 instead of a smartphone with glass screen.
    - iopq 10y ago
      > I would hope dropping a phone is also a rare occurrence for typical phone users. I drop my phone all the time and I don't have a case. It's fine.
      - awjr 10y ago
        I dropped my phone at the weekend without a case. I now have a shattered screen. :/ It's about the 5th time I've dropped my phone and this is the first time something like this has happened.
        josephg 10y ago
        Apparently the glass in phone screens develops micro fractures from being dropped. (Well, from hitting the ground). You can't see them with the naked eye, but they weaken the glass and they grow with each successive drop. It's probably not just bad luck that the glass happened to break after the Nth time you dropped it. It was bound to happen if you kept dropping the device.
      - marvy 10y ago
        Me too, though the screen did get a small crack once. If there was a 1% chance of file system corruption each time I dropped my phone, it would have happened to me by now, I think. And usually when I drop my phone, the battery comes out. It used to happen only half the time, but I think the cover got loosened as a result of all the times I dropped it, so now the battery falls out pretty much every time.
- digi_owl 10y ago
  I have noticed this with a cheap Window tablet i acquired recently. Left alone it will eventually hibernate, something my Android tablets have never attempted. Similarly, when it goes below 4% battery it will basically force a hibernate. That said, all mobile devices, laptops included, have a "force shutdown" option activated by holding the power button for x seconds.
- bmm6o 10y ago
  This is true for the first few years, but aging batteries can cause problems. My phone is in a situation where under higher power draws (GPS + navigation) it will simply die when the battery falls to 25%. It's been doing this pretty reliably for a month or 2.
- pvg 10y ago
  In the 'computer is a phone' perspective there is no sudden power loss unless the user yanks the battery out of the back. Or the battery is a couple of years old. Or the user lives someplace where it's cold outside.
  - tempestn 10y ago
    Neither of those should cause a sudden power loss. Even if your battery rapidly depletes, the phone will still shut itself down properly before it loses power.
    - pvg 10y ago
      It shouldn't yet sometimes it does.
sbierwagen 10y ago
fsync() has never been as reliable as the name implies, since flushing the disk cache kills performance system-wide. Fuzziness on what fsync does has... side effects: https://danluu.com/file-consistency/ https://danluu.com/file-consistency/ http://blog.httrack.com/blog/2013/11/15/everything-you-always-wanted-to-know-about-fsync/ http://blog.httrack.com/blog/2013/11/15/everything-you-alway... Filesystems optimized for flash, and for battery-backed systems in general, (laptops, phones) have some history: https://en.wikipedia.org/wiki/Flash_file_system https://en.wikipedia.org/wiki/Flash_file_system
- quotemstr 10y ago
  The fundamental problem is that fsync is too powerful. Most of the time, what you really want is a write barrier --- you don't care about when writes happen, but you do care about their order. fsync is a write barrier, but it's also a synchronous flush. But don't listen to me. I like TxF too.
- ioltas 10y ago
  Or in short, calling fsync() twice is a tweak under-known: the file itself needs to be flushed, as well as its parent directory.
chillaxtian 10y ago
> I suppose it's understandable... nobody should ever be blocking GUI animations on fsync, much less two of them in a row, but here we are. this is the thing that confused me.. why are apps doing disk i/o on the GUI thread?
- elktea 10y ago
  Threads are hard, basically.
- umanwizard 10y ago
  I think lots of stuff causes disk I/O that you might not even be aware of. For example, on iOS, +[CLLocationManager authorizationStatus] (checking if the app is allowed to use location or not) has to read a file [1]. Almost no app developers are aware of this. [1] I've been told this by guys who worked full-time on location stuff, but I haven't verified it myself and can't find it in docs, so take it with a grain of salt
  - azernik 10y ago
    fsync() in particular is usually only called on (well, after) disk writes though - I can see that being involved in all kinds of system API calls too, but hopefully the system makers would be savvy enough to use an actual write barrier instead of being fsync-happy.
- quotemstr 10y ago
  Because they don't care enough not to. There are even developer tools to help applications not do work on the UI thread: https://developer.android.com/reference/android/os/StrictMode.html https://developer.android.com/reference/android/os/StrictMod...
- digi_owl 10y ago
  Even if they don't, the kernel will likely drop everything to get that fsync out of the way. So a fsync happy app that is doing a file write in the background can cause the whole UI to lag because the kernel is busy dealing with said fsync call.
busterarm 10y ago
I wouldn't really say Moto Z cheats so much as that Google designed the entire software ecosystem of Android without ever talking to a hardware engineer. They simply do not care about optimizing the performance of the phone running their software. Otherwise they never would have chosen a garbage-collected language like Java for the platform in the first place (I understand they've ameliorated this concern recently). Or used ext4 like here.
- c8g 10y ago
  or developer had to build separate app for x86, armv7, armv5, armv6, armv8, mips,...
  - deleted 10y ago
    [deleted]
  - busterarm 10y ago
    Really just arm* until the last couple of years. These choices were made forever ago.
- sangnoir 10y ago
  > I wouldn't really say Moto Z cheats so much as that Google designed the entire software ecosystem of Android without ever talking to a hardware engineer Not sure if you're being serious or hyperbolic - but Google bought Android, Inc (an Andy Rubin startup). The team had a lot of Danger alumni - Danger being the makers of the HipTop - aka the original T-Mobile Sidekick (also running Java). I'm certain they knew a thing or two about hardware, they just made trade-offs you disagree with.
rrego 10y ago
So from reading the XFS FAQ linked in the article(http://xfs.org/index.php/XFS_FAQ#Write_barrier_support. http://xfs.org/index.php/XFS_FAQ#Write_barrier_support.), does this mean that Moto Z never flushes the cache to disk? Wouldn't the cache HAVE to be flushed as it fills up at some point to maintain coherence.
- idrae 10y ago
  Yes, it has to be flushed from time to time. Normally this happens automatically when it is needed. fsync() just forces a direct flush.
macspoofing 10y ago
>In other words, because the Moto Z cheats. If you're not cheating, you're not doing it right.

Animats 10y ago

"They drove development of the filesystem specifically by Twitter/FB/etc workloads captured from the phone." Why would Twitter or Facebook apps need to write much to persistent storage? They don't do so in a browser.

LoSboccacc 10y ago
Phone memory is severely limited and they have to unload images as you scroll to get the scrolling smooth. Scrolling forward and backward is a common user habit, and netwok latency is high, so you gonna need a local cache. Lying on fsync doesn't seem a good option tho, even if developers abuse it.
- Animats 10y ago
  But you don't need to fsync that. It's presumably in a temporary file, or you're just reading it.
  - refulgentis 10y ago
    You're the only one linking these two ideas together. The article expressly separates discussion of how Samsung gathered data to inform their filesystem design from discussion about Motorola's fsync implementation, and they're inherently separate concepts.
  - pjc50 10y ago
    I suspect what happens is the developer just dumps everything in a sqlite database, which will fsync for you on commit.
    - JBReefer 10y ago
      ^ I think this is it. App developers are taught to reflexively use the local db to store data, not files - but may not understand the difference performance characteristics of this db. Someone (like me before reading this post) working on an app for the first time might not realize they're thrashing out fsyncs with UPDATES.

ajdlinux 10y ago

Related: https://www.flamingspork.com/projects/libeatmydata/ https://www.flamingspork.com/projects/libeatmydata/

kibwen 10y ago

Offtopic: I love my old Moto G, but I've been reluctant to buy any new phones from Motorola since its purchase by Lenovo, due to Lenovo's history of shipping computers with rootkits and malware. Should I suspect that Motorola phones are compromised, or am I just being unreasonably paranoid?

viraptor 10y ago
In corporations like that don't expect a common theme. Different departments manage different devices and unless there's a company wide push for a culture change (which would leak to the public), I wouldn't expect changes any time soon. But on the other hand, if you are paranoid about security in Android, you should either go with a Google phone (quickest updates), or something like Copperhead.
- amckinlay 10y ago
  I think the commenter is more worried that he'll be funding a company whose practices he doesn't support.
  - blunte 10y ago
    And with the consolidation of companies, it becomes increasingly difficult to find any company that does things right and is owned by a company that does things right (and has sibling companies that do things right).
ccozan 10y ago
Motorola is a rare example and even more interesting regarding the relation with Lenovo. Since they keep an almost pristine Android, and hardware is rock solid, I think the paranoia is not necessary. My Moto X force is really unbreakable. Another interesting bit was that, instead Lenovo to "eat" the Motorola, it seems that it will move all its mobile handsets to the Moto name and leave quite some large space to develop, while supporting it with lots of money. I was really really skeptical at the beginning and I also told myself not to buy anymore from Moto under Lenovo, but in the main time my opinion changed and seems they go, let's say, not in the wrong direction, especially the Moto Z and the mods.

relics443 10y ago

Kinda makes me regret getting a Pixel to replace my Moto Z. Except the reception was terrible on the Z. Oh well. How is it that no one has made a worthy successor to the OG Turbo. That was a monster of a phone across the board.

seunosewa 10y ago
Is your Pixel phone noticeably slower than the Moto Z?
- YCode 10y ago
  The author mentions it should take like a year for it to become noticeable.
- relics443 10y ago
  I forgot my sarcasm tags
Nursie 10y ago
Are you sure you mean the Moto Z? It was only out a couple of months ahead of the pixel...
- relics443 10y ago
  I am sure. What's your point? That it's difficult to get new phones in such a short time? You're correct. Although, to be technical it was a Moto Z Force Droid Edition...
  - Nursie 10y ago
    Not difficult, just unusual. As someone who buys my phones I tend to hang on to them for about 18 months on average and have just got a Z...

codedokode 10y ago

I don't think this could be interpreted as a visual lag. fsync() is usually used by databases, not UI libraries. So the title is misleading. My chinese Android phone has a more annoying hardware lag - a delay between touching the screen and touch event processing is over 100 ms. Any drum app is unusable. And if you try to scroll something up and down fast it is easy to see how the content on the screen lags behind finger movements.

daenney 10y ago
> I don't think this could be interpreted as a visual lag. fsync() is usually used by databases, not UI libraries. As others have pointed out, a lot of apps do I/O related work in the UI thread, mostly b/c people don't care enough not to (and sometimes it's hard). Flipping a toggle somewhere can easily cause a write to a sqlite database to need to happen. And it's for far more than just databases. > So the title is misleading. I don't think the title is that misleading, it is very much and significantly faster at storage operations than the Pixel, and those matter quite a bit in your daily usage.
- kllrnohj 10y ago
  > I don't think the title is that misleading, it is very much and significantly faster at storage operations than the Pixel, and those matter quite a bit in your daily usage. All that was tested was fsync(), not "storage operations", and the Moto Z was faster at fsync() because it turns off fsync. But it's not clear why this matters to anything. Lag is usually the result of file system reads being slow (paging in code/resources), not writes being slow, so how does fsync performance matter?
  - daenney 10y ago
    > All that was tested was fsync(), not "storage operations", and the Moto Z was faster at fsync() because it turns off fsync. That's not what was tested, that's only what is shown in the graph. The lack of fsync contributes to it. I wrote a little fio benchmark driver to fill all available device storage with random 4k writes, print perf stats along the way One of the other things clearly mentioned is that b/c of Google's use of an additional FUSE filesystem they take a 30% performance hit: This means that on the Pixel every user IO gets a round-trip back into user-space before hitting the NAND. Fuse burns more CPU and slows down IO by up to 30%. Sure, the nobarrier trick makes a lot of difference but it's not the only thing in that post, nor was that the actual benchmark.

Tepix 10y ago

I hadn't heard about f2fs before. Sounds useful even on a notebook with SSD.

puzzlingcaptcha 10y ago
FWIW I've been using it for the past year for a root filesystem of a Debian NAS running off a USB stick and I didn't have any problems.
- pmontra 10y ago
  Thabks. I was about to ask if Incould replace ext4 with f2fs on the SSD of my Ubuntu laptop. Fsyncs are still at the mercy of the SSD firmware AFAIK. I found this https://ubuntuforums.org/showthread.php?t=2326934 https://ubuntuforums.org/showthread.php?t=2326934 I'm going to study it. Any other first hand experience here on HN?
peterwwillis 10y ago
It's been around since 2012, one of many Flash-specific filesystems. And sure, it can be helpful for flash disks, as other filesystems can too. But in some benchmarks f2fs performs worse than ext4, and in others shows it is not syncing to disk as often as it should be, putting data unnecessarily at risk. Oh, and the fsck program used to crash on any filesystem errors (hopefully they've fixed that by now (edit: they appear to have fixed it in March)) The performance boost in this case is mainly due to the nobarrier option, which is potentially dangerous unless you have something like a battery-backed RAID controller to act as a persistent disk cache. If you back up all your stuff, could be nice. (A similar hack I used to use to eliminate iops-related slowdown was to implement a tmpfs mount for small files that got written a lot, and just rsync them once a minute to disk) One of the benefits of f2fs is that you can select the allocation and cleaning algorithms that it uses based on the flash chip in use, so before using it on your system, you might want to tune it to your application. https://www.kernel.org/doc/Documentation/filesystems/f2fs.txt https://www.kernel.org/doc/Documentation/filesystems/f2fs.tx...

wodenokoto 10y ago

My old 4S runs annoyingly slow. According to the article,the NAND, as it ages will slow down the phone, so will replacing my storage make my iPhone fast again?

sgt 10y ago
You may also want to make sure you have a decent amount of free space. If your capacity is at 95% you will most likely experience an even slower 4S than what is typical.
wingerlang 10y ago
What iOS version are you running? It might be cheaper to just buy a new one.
- wodenokoto 10y ago
  9.02 ... And yeah, I wouldn't be surprised either, but I was intrigued by the prospect.
  - wingerlang 10y ago
    Some people swear that jailbreaking + installing various "speed up" tweaks (be it animation reduction or daemon killers) will help. Personally I don't use those kinds of tweaks but it might be worth a try.
angry_octet 10y ago
Highly unlikely. The flash doesn't slow down, but its hidden pool of spare blocks diminishes as random blocks fail. Delete some junk apps and old photos and you will give the storage allocator more room to play with. Also, disable background apps unless you really need them. Also Settings -> General -> Accessibility -> Reduce Motion, Reduce Transparency. Delete cookies from Chrome/Safari. Also, Settings -> Messages -> Expires -> 5 Minutes (instead of never).

darren_ 10y ago

The title is "Why Google Pixel lags 10x more than Moto Z" but the content is just a microbenchmark of fsync() without anything resembling a measurement of UI lag or that the 10x slowdown in fsync() actually means a 10x (or any) increase in said lag. The author's previous blogpost (http://taras.glek.net/post/Laggy-phones-and-misleading-benchmarks/ http://taras.glek.net/post/Laggy-phones-and-misleading-bench...) contends that 4K writes are a good proxy for phone lag, but has no evidence or measurements either, just the author's contention. It could all well be true! And it's great the author's dug up some concrete areas the pixel team could potentially improve at (dumb question: couldn't the pixel be updated to use most of these FS tweaks? could an in-use pixel FS be converted to use f2fs?). But I don't think the author has done a good job demonstrating an actual relation to UI lag or anything to do with the phone's perceived performance. disclaimer: googler, but I do iOS things.

izacus 10y ago
So much clickbait :/ Under no condition the Pixel lags "10x as much as Moto" when actually using it. It's a single benchmark of an FS performance for applications that do I/O on main thread. Something that is actively discouraged by things like StrictMode checks.
- IshKebab 10y ago
  To be fair he did say the difference would be more obvious after a year of use.
  - amyjess 10y ago
    Yes, this has shades of the 2012 Nexus 7 flash degradation issues...
    - iainmerrick 10y ago
      I agree with the previous commenters that the clickbait headline really oversells this article, but this is still a useful investigation. The 2012 Nexus 7 is a good comparison. It had slow, cheapskate flash memory, which degraded over time. But why should that cause UI jitter? As several people have pointed out, well-written apps use separate threads for UI and I/O. There were some big UI jitter problems in the OS, but there was a big effort to fix those in Jellybean, which the Nexus 7 shipped with. The problem is that just using separate threads isn't enough to keep UI and I/O completely separate. The UI uses the GPU, and the GPU fights with the kernel over access to I/O resources. Maybe the UI thread or GPU driver has a cache miss and needs code paged in; maybe the OS is busy writing out another page. If the Motorola phone really does have 10x lower I/O latency, that seems very significant, but we don't know if it really affects the UI without measurements. Just setting 'nobarrier' on the file system sounds dangerous though. I'd be very wary of that unless they have really good arguments and measurements to show it's safe. I write Android apps, and I/O reliability is a big headache. I'd put the success rate for simple file system operations at only around 99.9% -- as soon as your users number in the thousands, there are always a few users whose phones fail in bizarre ways. Most of the problems show up on Samsung phones, but that might just be because they're the most popular brand. Apple hardware definitely ages much, much more gracefully than any Android hardware I've seen. That might be okay if everyone got a new phone every two years, but that's definitely not the case; tons of people are using old phones.

mda 10y ago

Any real proof that this actually causes user visible issues at all?

Mikeb85 10y ago
No. Just that it might, depending on how the Pixel's NAND memory degrades over time. My money is on the battery crapping out first.

carrot 10y ago

What does it matter?

carrot 10y ago

What does it matter?

blunte 10y ago

I wonder if this is why my Sony Xperia Z5 is so sluggish sometimes, despite having a Snapdragon 810 8-core CPU...

kimshibal 10y ago
It's not about hardware. My new android has twice spec than my 4 year old android phone. Yet, I'm still using my old phone for some apps. New phone has better spec, but it still has laggy scroll.
vetinari 10y ago
Snapdragon 810 is a big.LITTLE architecture; you don't have 8 CPUs available at the same time. Only 4. Check your running applications, the answer to sluggishness might be there.
- mirsadm 10y ago
  All 8 cores can run at the same time on the 810.
  - vetinari 10y ago
    You are right, 810 has HMP. I somehow missed that.

joosters 10y ago

Other comments are querying the relevance of fsync() as a 'lag' benchmark, but I want to query whether fsync() is even meaningful at multiple calls per second. I know fsync() ensures data gets written to disk, but why does anyone care that it can happen so often? When a device crashes, some data (prior to the sync) may be lost, but do we really need multiple checkpoints per second to ensure only sub-second data loss? I'd be content with a couple of minutes worth of loss even on my main PC, with its lack of battery backup. To enforce rapid syncs on a phone seems utterly pointless. Keep the syncs for meaningful checkpoints, like buying something in an app or marking a message as sent. Multiple fsync() calls per second are a total waste.

jchrisa 10y ago
One of the few times they make sense is in a database server application where you might have hundreds of thousands of connected clients.
charleslmunger 10y ago
Loss isn't the same thing as corruption, which is the real fear here. What you're looking for (I think) is SQLite in WAL mode, with PRAGMA SYNCHRONOUS=NORMAL. That's ACI but not D.

agumonkey 10y ago

I don't see the hard link between GUI smoothness and io. You can have perfectly smooth 12000fps rendering and crappy IO, everything will just be "loading...".

pas 10y ago

Is there an Android sample app that show how to do I/O properly (off the main thread, and so on)? I know there's some kind of system to prevent network I/O on the main thread... w... why... I dare to ask the obvious, w-wwhy isn't there a warning for simple disk I/O too?

cuu508 10y ago
Android's StrictMode can monitor and notify about both network IO and disk IO.
- pas 10y ago
  I found that, and looks very powerful, but that's a bit more than a simple switch in Android Studio, yet less than a warning on the Play Console, because you have to add it to your code (so not a static analyzer).

archivator 10y ago

This articles conflates a lot of things but it also has the priorities somewhat wrong. 1) fsync cost. Yes, fsyncs are dangerously slow in any Android app. (SQLite for example is a common culprit. Shared Prefs are another). HOWEVER, it's possible that flushes cause reads to be queued behind them (either in the kernel or on the device itself) which is even worse because 2) Random read cost is super super important. Android mmap's literally everything and demand paging is particularly common AND horrendous as a workflow. To add insult to injury, Android does not madvise the byte code or the resources as MADV_RANDOM, so read-ahead (or read-around) kicks in and you end up paging in 16KB-32KB where you only wanted 4KB. Also, history has shown custom flash-based file system on Android to be a world of pain. yaffs, jffs have some pretty atrocious bugs/quirks. I'd much rather see the world unify on common file systems, optimized for flash-like storage, rather than OEMs shipping their own in-house broken file "systems" (I'm looking at you, Samsung).

jhasse 10y ago
Why can't f2fs be that common file system?
- jdcarter 10y ago
  I just read the F2FS paper and it seems very well-designed to match the physical properties of flash, plus some interesting capabilities to keep hot/cold data separate. If there's something wrong with F2FS, let's fix it. This seems like a far better place to start from than any filesystem designed around the assumptions of a spinning disk.
- TazeTSchnitzel 10y ago
  It's in the mainline Linux kernel now, it's hardly some proprietary obscure vendor thing.
  - archivator 10y ago
    That's fair, it's a better state than the previous attempts. Still, it's not as tested as, say btrfs and ext4. Can't wait to see its particular quirks.

terrywang 10y ago

Nexus 5X running rooted AOSP 7.1.1 /storage/emulated fuse /dev/block/dm-0 /data ext4 rw,seclabel,nosuid,nodev,noatime,noauto_da_alloc,errors=panic,data=ordered,inode_readahead_blks=8 (no nobarrier mount option) In theory Google should be able to easily change the /data ext4 mount option, why didn't Google?

bla2 10y ago

If you're calling fsync on your painting thread, you're going to have trouble hitting 60FPS no matter what. What a clickbaity post.

kasabali 10y ago

> ... noatime,nodiratime, ... noatime implies nodiratime. I shrug off when I see newbies copy pasting this to their /etc/fstab but this is in a mainstream Android device??

kiallmacinnes 10y ago
I personally hate options that both have their own behavior, and imply another options behaviour unnecessarily. I'll be explicit when I use these options, and call them all out. That way, when someone who's not intimately familiar with the code reads it, they can actually tell what it's doing at a glance.
- kasabali 10y ago
  I won't comment on your general remark but I don't think it is relevant to this specific case. "noatime" has a simple meaning and a clear name that represents its function: it stops updating inode access times. If you don't want your filesystem updating access times you use this mount option and it's done. Functionality of "nodiratime" is the one that's more specific & esoteric and subset of "noatime", thus it has a more specific name. It's not that "noatime" implies "nodiratime" , but more like "nodiratime" implements a subset of functionality of "noatime". Nothing to hate really.

Grazester 10y ago

"try to avoid buying devices that will slow down to point of being unusable as NAND wears out (ala Nexus 7, Nexus 6)" Has anyone experienced this issue with their Nexus 6? My phone is more than 2 years old and I have no noticeable slowdown. The pixel might have the slower storage option but it has no effect on usability. From what I have read its UI performance is the best of any Android phone yet. "The Pixels are fast — noticeably faster than Samsung's Galaxy S7. On performance alone, these are easily the best Android phones you can buy." http://www.theverge.com/2016/10/18/13304090/google-pixel-phone-review-pixel-xl http://www.theverge.com/2016/10/18/13304090/google-pixel-pho...

discordianfish 10y ago
I've got a Nexus 5x and the performance was disappointing from the start. But maybe it just feels like that because I came from an iphone 5s.
- Grazester 10y ago
  Oh no buddy the Nexus 5x had some issues when initially released. Google did work these issues out though
lanestp 10y ago
Actually, yes. At work we have a Nexus 6 testing device. It started off strong but a few months ago it started lagging badly on videos. It even struggles with gif length mp4s. The problem is notable because it is the only phone with this problem. Even our old Galaxy S4 performs better.
- Grazester 10y ago
  What kind of videos? Do you have a link perhaps where I could test this?
  - lanestp 10y ago
    Giphys and rewarded video ads. We are a gaming company so you can see where our priorities lie!

moonlander 10y ago

This makes very little sense. The author shows a fsync() benchmark comparing an actualy fsync (Pixel on ext4 without nobarrier) with a nobarrier (no-op fsync) alternative. The only thing this benchmark shows is that a no-op is faster than the real thing.

idbehold 10y ago
The title of the article is "Why [the] Google Pixel lags 10x more than [the] Moto Z." The author showed that the Moto Z is using a no-op fsync while the Pixel is doing the real thing. I think the author appropriately explained and answered the title prompt.
- moonlander 10y ago
  But the author fails to prove that fsync() is in any way the bottleneck for common I/O operations in everyday use of a smartphone.
  - idbehold 10y ago
    Was that the hypothesis the author was trying to prove? Because I don't believe it was.
    - monocasa 10y ago
      The title is literally "Why [the] Google Pixel lags 10x more than [the] Moto Z". If he doesn't show that fsync(3) is the gating factor, then the fact that the Moto Z nops out fsync doesn't mean that much.
cbsmith 10y ago
Saying "no-op is faster than fsync" is just not as click-baity a title. There is a good question though... do you really need fsync on Android? If you don't, why are you calling it?

digi_owl 10y ago

I suspect it is not the fsync directly that leads to lag. But that when a fsync comes through, Linux stops doing anything else for the duration.

kasabali 10y ago
Linux doesn't do no such thing. ext* filesystems do (especially ext3).

kozak 10y ago

So this might be the reason why some mobile devices (like my ASUS TF700T tablet) degrade in performance so badly over time. Interesting.

givinguflac 10y ago
It's certainly part of it, but there are other possible contributors. For example, the original Nexus 7 suffered from severe NAND degradation over time, as have some other devices.
- Namidairo 10y ago
  Both of those devices are just the victims of Asus cutting corners with cheap flash and Tegra 3's not so great memory bandwidth.

colanderman 10y ago

The fundamental problem sounds like it's that SQLite insists on not running in its own thread. This, coupled with the fact that Linux has no way to issue a true (i.e., non-blocking) write barrier, means that there is no way to implement write barriers in user space. Whereas, if SQLite did issue I/O from a separate thread, one could easily implement an "async commit" function which guaranteed consistency and ordering but not necessarily durability (i.e. a write barrier). This would suffice I suspect for 95% of usage in applications: users will probably be OK if their phone loses the last few seconds of user input before an OS crash, so long as everything else is left intact. EDIT: In fact, Postgresql has an option to permit exactly this behavior: https://www.postgresql.org/docs/9.6/static/wal-async-commit.html https://www.postgresql.org/docs/9.6/static/wal-async-commit.... This is possible in Postgresql because, unlike SQLite, I/O does not run in the client thread.

swift 10y ago
It is possible to run SQLite entirely on a separate thread and avoid blocking the main thread on its I/O, though. It just requires that you're willing to ship the data you need to read and write between threads and handle the synchronization yourself. I've written applications that do this and it works very well, but it's certainly more work than if SQLite supported off-main-thread I/O natively.
- colanderman 10y ago
  Of course. But that's more work that I bet 95% of app developers are willing/able to put in. (Heck I've been doing this stuff for years and still have to spend time analyzing my code to make sure I'm not inadvertently holding locks in places where I shouldn't be.)
kllrnohj 10y ago
> The fundamental problem sounds like it's that SQLite insists on not running in its own thread. Pretty much all apps DO have SQLite running on its own thread, or at least a background thread pool of some sort. Android gets very cranky at the developer if they don't do this - there's tons of warnings, both runtime and in the form of lint. Database access, or anything involving fsync(), is exceptionally rare on the UI thread or any user-latency-critical thread.

ElijahLynn 10y ago

I always wondered why my wife's Moto X is still buttery smooth after 3 years!! I love that phone. I have a Pixel but she still has her Moto X and it is really great at being smooth, plus the wave to wake and other gestures are really unmatched by the Pixel. Love this write up/research! Hopefully it will teach the Pixel team a few things, or maybe they already knew but will now have the ammo to take to Product and change things!!

bitmapbrother 10y ago

If f2fs is so superior to ext4 then why doesn't its creator, Samsung, use it on their phones?

strcat 10y ago
It's significantly better on some benchmarks and significantly worse on others. There's no clear winner and ext4 is much more widely tested and has way more development time focused on it. Looking at only a few aspects of performance can result in either of ext4 or f2fs being seen as a clear winner but it's not representative of the whole picture.

bitmapbrother 10y ago

From Tim Murray, performance engineer on Pixel: >that fsync blog post floating around is pretty much bogus. also nobody should use nobarrier, it's not safe at all https://twitter.com/t_murray/status/808373275860418560 https://twitter.com/t_murray/status/808373275860418560