5 ms·

As careful as you may be, it’s practically impossible to write software that will remain perfectly fluid when the UI can be blocked by arbitrary processing. We

by panic 9y ago

As careful as you may be, it’s practically impossible to write software that will remain perfectly fluid when the UI can be blocked by arbitrary processing. Well, that's what they did! All UI events and layout on the original iPhone were handled on the main thread. I doubt asynchronous layout or event handling would have improved the experience on its single-core CPU. The key technical advantage the original iPhone had was Core Animation, which composited the laid-out views and applied animations to them in a separate process. It ensured that all views would appear at the correct position in their animations each frame with no jitter, and kept most of the per-frame work in one place. But the animations were all initiated on the same main application thread that handled events, performed layout, and so on.

yorwba 9y ago
> All UI events and layout on the original iPhone were handled on the main thread. But what else happens on the main thread? The way I understand the article, UI and input are delegated to a dedicated compositor thread to prevent the heavy processing on the main thread from interfering with responsiveness. I would assume that iOS also separates the timing-sensitive UI handling from anything that might take longer than a single frame to process. Either way, you end up with one thread doing the heavy lifting and another keeping the UI responsive.
- jsiepkes 9y ago
  I don't understand. Which GUI toolkit works differently then the way described in the article? Afaik all UI toolkits have a single GUI thread.
  - aaronbrethorst 9y ago
    Texture, née AsyncDisplayKit, for one: https://medium.com/@Pinterest_Engineering/re-architecting-pinterest-039-s-ios-app-e0a2d34a6ac2 https://medium.com/@Pinterest_Engineering/re-architecting-pi...
  - amirouche 9y ago
    I am not familiar with this kind of framework. Where can I learn more about it?
- zigzigzag 9y ago
  I found the article a little confusing to be honest. I wonder if the author has written a traditional widget toolkit that isn't Firefox oriented. In old widget toolkits, going back to the 90s here, there was a single UI thread per app that did all drawing and sending of commands to the graphics hardware. Keeping the UI responsive on such toolkits simply meant doing things as much as possible in the background. Touching the UI data structures from other threads was forbidden. This architecture was adopted due to painful experiences with attempts to build thread-safe toolkits in the 80s such as Motif and the original Win32 widget library. None of it worked very well. Motif apps tended to be deadlock prone and Win32 was just a total API nightmare because it tried to hide the thread affinity of the underlying widgets, but didn't do a good job of it. Some systems in the 90s like NeXT and BeOS started experimenting with moving the rendering into a separate process, the window server. Note that X Windows, despite having a window server, did not use "retained mode" rendering and still required the app to respond to do every repaint such as if an occluded window was moved to the top. Systems with this sort of retained mode rendering pushed "draw lists" into the window server so the OS could draw the window from memory without having to wait for the app to respond. This used more memory but meant that overall window UI stayed responsive and fluid even if apps were under heavy load. However, anything that could change the UI like needing to respond to user input, of course stayed in the app and on the UI thread. MacOS X introduced a variant of the design, which I know less about, but I believe it basically just stored fully rendered copies of the image. Very RAM intensive and one reason MacOS X was considered very slow and heavy in the early days, but it made it possible to do things like the genie effect and exposé later on where the window server could animate the contents of windows without the app needing to be responding. All that is OS level compositing. The app itself did not do any asynchronous compositing. So dragging windows around was fast, but animations inside the app didn't benefit. So the next level of asynchronicity is toolkits that push app level rendering into a separate thread too. iOS, JavaFX, modern versions of Qt and modern versions of Android work this way. In these toolkits, the app's GUI is still constructed and manipulated on the primary/UI thread, but when the main thread "renders" the UI, it doesn't directly draw it, it constructs a set of draw lists for the apps own use. Again, these draw lists look a bit like this: 1. Clear this area of the window to this colour. 2. Draw a gradient fill from here to there. 3. Draw texture id 1234 with that shader at these coordinates, at 50% opacity. 4. Invoke remembered draw list 111. 5. Remember this set of instructions as draw list 222. Once these lists are created they're handed off to a dedicated render thread which starts processing them and turning them into commands to the GPU via an API like OpenGL or Direct3D. Note that these APIs are, in turn, simply creating buffers of commands, which eventually get dispatched to the GPU hardware for actual rendering. Because the render thread doesn't run any callbacks into app code, and because it's cooperating with the GPU hardware to remember and cache things, it doesn't have that much actual work to do and can process simple animations very fast and reliably. However, responding to user input is still done on the main thread. If you block the main thread, your UI will continue to repaint and may exhibit simple behaviours like hover animations, but actually clicking buttons won't work. That's because the most common thing to do in response to user input is change the UI itself in some way, and that must still be done on the main thread. I hope that helps.
mjburgess 9y ago
Do you really mean single thread, or do you mean single core? It seems doubtful it was on a single thread with those response characteristics.
krona 9y ago
The Core Animation animation/render/compositor thread has realtime priority.
doomlaser 9y ago
Also no Java, so no arbitrary garbage collection sweeps causing hitches, and significantly less memory footprint overall.
- bitmapbrother 9y ago
  Android executes native ARM code. Additionally, average GC pause times are now 0.4ms in Android O. To clarify, that's not 0.4ms for every frame - that's every time the GC needs to do a STW. How many milliseconds does ARC lose during its reclamation of memory on iOS? As for memory usage, Android N did use a bit more memory than iOS apps, but that's also been reduced by Android O. https://www.youtube.com/watch?v=iFE2Utbv1Oo&index=110&list=PLOU2XLYxmsIKC8eODk_RNCWv3fBcLvMMy https://www.youtube.com/watch?v=iFE2Utbv1Oo&index=110&list=P... As for your claim that iOS apps have a significantly less memory footprint overall, well, you would also be wrong about that. https://youtu.be/lCFpgknkqRE?t=6m27s https://youtu.be/lCFpgknkqRE?t=6m27s
  - flukus 9y ago
    I've heard "java can be as fast as c" for 20 years now and "android is fast and smooth" for 10 years and my experience has always been the opposite. At the point I wouldn't believe them if it was true, there is no credibility left.
    - bitmapbrother 9y ago
      ART isn't a JVM. And Objective-C isn't as fast C. There's a reason all of the games and apps that require high performance are written C/C++ on both platforms.
      - beerbaron23 9y ago
        Errrr Android ART is a VM and no you can't write games in C/C++ on Android. While technically it can be done it's pretty much impossible to get it off the ground and running because of the VM in place. There is a website that lists the games people have created to try it successfully, they give links to couple of them which made it into the Android store and basically you can tell by all the feedback that the game won't load. You can try to run the games too and see if you're lucky, I wasn't.... Objective-C is very quite fast, several notches faster then Java apps

by panic 9y ago

yorwba 9y ago

> All UI events and layout on the original iPhone were handled on the main thread. But what else happens on the main thread? The way I understand the article, UI and input are delegated to a dedicated compositor thread to prevent the heavy processing on the main thread from interfering with responsiveness. I would assume that iOS also separates the timing-sensitive UI handling from anything that might take longer than a single frame to process. Either way, you end up with one thread doing the heavy lifting and another keeping the UI responsive.

jsiepkes 9y ago
I don't understand. Which GUI toolkit works differently then the way described in the article? Afaik all UI toolkits have a single GUI thread.
- aaronbrethorst 9y ago
  Texture, née AsyncDisplayKit, for one: https://medium.com/@Pinterest_Engineering/re-architecting-pinterest-039-s-ios-app-e0a2d34a6ac2 https://medium.com/@Pinterest_Engineering/re-architecting-pi...
- amirouche 9y ago
  I am not familiar with this kind of framework. Where can I learn more about it?
zigzigzag 9y ago
I found the article a little confusing to be honest. I wonder if the author has written a traditional widget toolkit that isn't Firefox oriented. In old widget toolkits, going back to the 90s here, there was a single UI thread per app that did all drawing and sending of commands to the graphics hardware. Keeping the UI responsive on such toolkits simply meant doing things as much as possible in the background. Touching the UI data structures from other threads was forbidden. This architecture was adopted due to painful experiences with attempts to build thread-safe toolkits in the 80s such as Motif and the original Win32 widget library. None of it worked very well. Motif apps tended to be deadlock prone and Win32 was just a total API nightmare because it tried to hide the thread affinity of the underlying widgets, but didn't do a good job of it. Some systems in the 90s like NeXT and BeOS started experimenting with moving the rendering into a separate process, the window server. Note that X Windows, despite having a window server, did not use "retained mode" rendering and still required the app to respond to do every repaint such as if an occluded window was moved to the top. Systems with this sort of retained mode rendering pushed "draw lists" into the window server so the OS could draw the window from memory without having to wait for the app to respond. This used more memory but meant that overall window UI stayed responsive and fluid even if apps were under heavy load. However, anything that could change the UI like needing to respond to user input, of course stayed in the app and on the UI thread. MacOS X introduced a variant of the design, which I know less about, but I believe it basically just stored fully rendered copies of the image. Very RAM intensive and one reason MacOS X was considered very slow and heavy in the early days, but it made it possible to do things like the genie effect and exposé later on where the window server could animate the contents of windows without the app needing to be responding. All that is OS level compositing. The app itself did not do any asynchronous compositing. So dragging windows around was fast, but animations inside the app didn't benefit. So the next level of asynchronicity is toolkits that push app level rendering into a separate thread too. iOS, JavaFX, modern versions of Qt and modern versions of Android work this way. In these toolkits, the app's GUI is still constructed and manipulated on the primary/UI thread, but when the main thread "renders" the UI, it doesn't directly draw it, it constructs a set of draw lists for the apps own use. Again, these draw lists look a bit like this: 1. Clear this area of the window to this colour. 2. Draw a gradient fill from here to there. 3. Draw texture id 1234 with that shader at these coordinates, at 50% opacity. 4. Invoke remembered draw list 111. 5. Remember this set of instructions as draw list 222. Once these lists are created they're handed off to a dedicated render thread which starts processing them and turning them into commands to the GPU via an API like OpenGL or Direct3D. Note that these APIs are, in turn, simply creating buffers of commands, which eventually get dispatched to the GPU hardware for actual rendering. Because the render thread doesn't run any callbacks into app code, and because it's cooperating with the GPU hardware to remember and cache things, it doesn't have that much actual work to do and can process simple animations very fast and reliably. However, responding to user input is still done on the main thread. If you block the main thread, your UI will continue to repaint and may exhibit simple behaviours like hover animations, but actually clicking buttons won't work. That's because the most common thing to do in response to user input is change the UI itself in some way, and that must still be done on the main thread. I hope that helps.

mjburgess 9y ago

Do you really mean single thread, or do you mean single core? It seems doubtful it was on a single thread with those response characteristics.

krona 9y ago

The Core Animation animation/render/compositor thread has realtime priority.

doomlaser 9y ago

Also no Java, so no arbitrary garbage collection sweeps causing hitches, and significantly less memory footprint overall.

bitmapbrother 9y ago
Android executes native ARM code. Additionally, average GC pause times are now 0.4ms in Android O. To clarify, that's not 0.4ms for every frame - that's every time the GC needs to do a STW. How many milliseconds does ARC lose during its reclamation of memory on iOS? As for memory usage, Android N did use a bit more memory than iOS apps, but that's also been reduced by Android O. https://www.youtube.com/watch?v=iFE2Utbv1Oo&index=110&list=PLOU2XLYxmsIKC8eODk_RNCWv3fBcLvMMy https://www.youtube.com/watch?v=iFE2Utbv1Oo&index=110&list=P... As for your claim that iOS apps have a significantly less memory footprint overall, well, you would also be wrong about that. https://youtu.be/lCFpgknkqRE?t=6m27s https://youtu.be/lCFpgknkqRE?t=6m27s
- flukus 9y ago
  I've heard "java can be as fast as c" for 20 years now and "android is fast and smooth" for 10 years and my experience has always been the opposite. At the point I wouldn't believe them if it was true, there is no credibility left.
  - bitmapbrother 9y ago
    ART isn't a JVM. And Objective-C isn't as fast C. There's a reason all of the games and apps that require high performance are written C/C++ on both platforms.
    - beerbaron23 9y ago
      Errrr Android ART is a VM and no you can't write games in C/C++ on Android. While technically it can be done it's pretty much impossible to get it off the ground and running because of the VM in place. There is a website that lists the games people have created to try it successfully, they give links to couple of them which made it into the Android store and basically you can tell by all the feedback that the game won't load. You can try to run the games too and see if you're lucky, I wasn't.... Objective-C is very quite fast, several notches faster then Java apps