7 ms·

Async Rust in Practice: Performance, Pitfalls, Profiling

248 points by uberdru 5y ago

dagmx 5y ago
I'd love to read this article but man this site is painful on mobile. Multiple popups from the top, header that overlays the text, the chat bubble at the bottom. It really reduces the amount of space for the actual content When comparing to Reader View in Safari, the native site has at least 40% less viewing area
- PeterCorless 5y ago
  Hi Dagmx! Peter Corless here from ScyllaDB. Sorry to hear about your mobile experience. I've notified our web team. If you want to screenshot your mobile view to share with me, please email me at peter @ scylladb [dot] com. You have my commitment we'll be working on improving the page. Aside from that though, hope you were able to glean something valuable from the article. Would love to hear your opinions.
  - DenseComet 5y ago
    I've been using Scylla for a project and have ended up reading quite a few of those blog posts. They are generally very well written and useful, but I've noticed the same sort of issues. There's the chat popup at the bottom, a banner for Scylla Summit, a cookie banner that doesn't seem to remember what I clicked, and a header with broken transparency. For me, Cloudflare is the benchmark. Cloudflare.com is very clearly a marketing site, likely run by their marketing team, but their technical content such as their blog and docs are on completely different subdomains with a clean design with no popups or banners. I think this is the best way to run tech focused blog, making both developers and marketing happy.
  - dmitriid 5y ago
    > If you want to screenshot your mobile view to share with me Just, you know, visit your own site on mobile. Or open Chrome dev tools and switch to mobile view. > You have my commitment we'll be working on improving the page. As with all in-your-face marketing shenanigans, I doubt you'll change anything in the long run.
    - PeterCorless 5y ago
      I did this, in fact. I also had other colleagues. A lot of UI/UX depends on the browser (Chrome, Firefox, Brave, Opera) and the OS. So I wanted to confirm precisely what he was seeing. And we're already working on fixes.
  - dmitriid 5y ago
    Three screenshots: https://imgur.com/a/XUpP9ha https://imgur.com/a/XUpP9ha Including the cookie banner that's illegal under GDPR (and is likely illegal under CCPA)
    - throwaway81523 5y ago
      Xkcd 624?
  - dagmx 5y ago
    Thanks for looking into it. I see others (dmitriid) have uploaded screenshots already, so I won't duplicate. But yes, otherwise the article itself was insightful and thank you for sharing your findings.
- dijit 5y ago
  Seems alright on my iPad. https://imgur.com/a/SJwg1E2 https://imgur.com/a/SJwg1E2
  - dagmx 5y ago
    That's a much larger screen than the largest iPhone. Hence why I specified mobile.
- r00fus 5y ago
  TBH, I've moved to setting Reader View as default for all Safari visits in iOS. You can whitelist sites or just undo reader view for that session if needed. Zero ads, and mostly I'm there for the text anyway.
- bschwindHN 5y ago
  On an iPhone SE 1, you literally get 5 lines of readable text, and 3 if the Safari navigation UI is showing. I also can't close the chat bubble because the X is off the screen to the right, and the scroll position jumps around near the top of the page as you scroll because some calculation or CSS is wrong.
gxt 5y ago
It's a great story and it's pleasant to see end-devs investigations contribute to the overall performance of the ecosystem since Tokio is so widely used. Cheers
carllerche 5y ago
Tokio author here. Generally speaking, I recommend strongly against using FuturesUnordered unless you know all the pitfalls. We are working on an alternative utility that should hopefully avoid the issues described here and others: https://github.com/tokio-rs/tokio/pull/4335 https://github.com/tokio-rs/tokio/pull/4335
- psarna 5y ago
  That's great news! Especially that the observed performance of the test program based on FuturesUnordered, even though it stopped being quadratic, it was still considerably slower than the task::unconstrained one, which suggests there's room for improvement. Probably due to the fact that you still pay with a constant number of polls (32) each time you go out of budget.
  - carllerche 5y ago
    IMO FuturesUnordered should stop executing futures when it sees a "yield". An explicit yield signals control should be returned to the runtime. FuturesUnordered does not respect this.
    - dathinab 5y ago
      IMHO the main problem is tokio introducing preempting behaviour which in subtle ways can mess with all kinds of normally fully valid rust futures. Sure it sometimes magically fixes problematic code, but it's in effect still a magic extension to how rust polling works which can have surprising side effects. In a certain way tokios coperative-preempting is not adequate to handle any future which multiplexes multiple futures but such futures are a stable of rust async since the get to go.
      - carllerche 5y ago
        I don't think it is an issue w/ the pre-emption code. I believe FuturesUnordered is just doing the wrong thing: not respecting yields.
psarna 5y ago
Hi, author of the post here. I'll also be talking about this issue in a little more detail at an online Rust Warsaw meetup tomorrow, feel free to join, especially if you're up for a live discussion later: https://www.meetup.com/Rust-Warsaw/events/282879405/ https://www.meetup.com/Rust-Warsaw/events/282879405/
- petr_tik 5y ago
  Can you please shed some light on rust's adoption at scylla given the cumulative team expertise in Cpp and unsafe programming like thread-per-core architectures. In my experience so far, highly proficient Cpp programmers tend to find rust constraints overly restrictive, because they kind of internalise the borrow checker and are comfortable enough to bend the rules sometimes. how receptive are fellow scyllians (is that the term?) to rust? how much scope is there for future projects to be in Rust over Cpp? Thanks for the honest and detailed write-up!
  - tialaramex 5y ago
    > comfortable enough to bend the rules sometimes. Mmm. The world we actually have suggests that this comfort is dangerous. Robert Browning was a famous poet. Certainly competent enough with English that you'd assume he knows what he's doing and can bend the rules. So, Dictionary makers were a little confused by the passage in Pippa Passes, "Then owls and bats / Cowls and twats / Monks and nuns in a cloister's moods / Adjourn to the oak-stump pantry". Why did Browning use the word "Twat" in this context? Turns out that he just had just assumed it was a word meaning a hat nuns wear... Oops. Of course this goof just means a high school literature teacher covering Browning has to decide whether to skip this part of Pippa Passes maybe "for time" and hope nobody asks about it, or explain to the class that even the best of us screw up sometimes. In Computer Software our mistakes are often not treated so kindly.
    - deleted 5y ago
      [deleted]
  - psarna 5y ago
eminence32 5y ago
Nice article, and nice analysis of the problem. I have a personal theory that once a codebase gets complicated enough (no matter the language, no matter sync or async), you'll run into subtle bugs or performance problems that require a very deep understanding of all the relevant libraries in order to resolve the problem. One worry that I have with async rust is that the executors are so complex that the "baseline complexity" starts rather high. If this theory is true, then one might expect async rust to run into such problems more often than comparable non-async code. I haven't personally written enough async code to have any data either way, except for a few unpleasant experiences while making errors when writing async code. To the extent that this theory is a problem that needs solving, I don't think there is a solution. But I do think that, over time, weird async footguns will become less and less frequent as projects like tokio continue their excellent engineering efforts to plug up any weak spots and make things more robust in general.
- marcus_cemes 5y ago
  > One worry that I have with async rust is that the executors are so complex that the "baseline complexity" starts rather high. I completely agree, however I like that you have the option to swap out the executor for your own if the situation requires it, which I think concerns a very small number of applications. In other languages with a baked-in runtime you would be left trying to come up with workarounds to make it play nicely.
- rr808 5y ago
  I think this is true for every language, not just rust. I'm really not convinced async is good for complicated systems. Things like green threads are much easier to design with.
- valenterry 5y ago
  > I have a personal theory that once a codebase gets complicated enough (no matter the language, no matter sync or async), you'll run into subtle bugs or performance problems You probably, unintentionally, defined "complicated enough" to be at the point where these bugs occur - so that's a tautology. I think the point of programming languages and techniques/libraries is to move this point of "complicated enough" as far out as possible. Without any libraries, it might be easier to understand, but it will happen so much earlier that it's not worth it. And the point of libraries is that not only do they allow you to push the problem out to a much later point in time - they also allow you to switch between projects and not lose all your knowledge. Even considering high upfront costs, tt's a big net worth imho.
throwaway81523 5y ago
I haven't even clicked the link yet, but that the Scylla devs are doing something with Rust already is interesting. Seastar is very cool though constrained by the limitations of C++. It will be great to find out what effect Rust has.
- AlexSW 5y ago
  Which limitations of C++, sorry?
scottlamb 5y ago
Nice article. I'm in the process of absorbing it. Question: > scylla-rust-driver issued at least 1 syscall per query, which might be the source of elevated latency – and with a super-fast network (of which loopback is a prime example) it also means that throughput suffers – a latency of 1ms means that we won’t be able to send more than 1000 requests per second. Is 1 ms measured? Buffering and reducing syscalls is good, but still: that seems horribly slow for a small or EWOULDBLOCK-returning read/write. Why would it be that bad?
- enedil 5y ago
  I believe this is just an example of what are the implications of 1 query per ms, not indication that it actually took 1 ms.
  - psarna 5y ago
    Indeed - 1ms was a nice round number, but it was also not that far from the real, sub-1ms number. But note that by "latency" here I mean not only the round trip, but the total measured time of executing a single request, including the time its task spends in Tokio (and its queues), and until its response is fully processed as well. Since the program sent requests with configurable concurrency set to ~1024 or more, the overall throughput was still satisfying. On the one hand it was (and still is) concerning that the observed latency per simple request was that high, on the other - it never really came up in our distributed tests, since there the network latency imposes a few milliseconds anyway. When we figure out the real source of this behavior, I'll be happy to describe the investigation process in a blog post (:

248 points by uberdru 5y ago

I'd love to read this article but man this site is painful on mobile. Multiple popups from the top, header that overlays the text, the chat bubble at the bottom. It really reduces the amount of space for the actual content When comparing to Reader View in Safari, the native site has at least 40% less viewing area

PeterCorless 5y ago
Hi Dagmx! Peter Corless here from ScyllaDB. Sorry to hear about your mobile experience. I've notified our web team. If you want to screenshot your mobile view to share with me, please email me at peter @ scylladb [dot] com. You have my commitment we'll be working on improving the page. Aside from that though, hope you were able to glean something valuable from the article. Would love to hear your opinions.
- DenseComet 5y ago
  I've been using Scylla for a project and have ended up reading quite a few of those blog posts. They are generally very well written and useful, but I've noticed the same sort of issues. There's the chat popup at the bottom, a banner for Scylla Summit, a cookie banner that doesn't seem to remember what I clicked, and a header with broken transparency. For me, Cloudflare is the benchmark. Cloudflare.com is very clearly a marketing site, likely run by their marketing team, but their technical content such as their blog and docs are on completely different subdomains with a clean design with no popups or banners. I think this is the best way to run tech focused blog, making both developers and marketing happy.
- dmitriid 5y ago
  > If you want to screenshot your mobile view to share with me Just, you know, visit your own site on mobile. Or open Chrome dev tools and switch to mobile view. > You have my commitment we'll be working on improving the page. As with all in-your-face marketing shenanigans, I doubt you'll change anything in the long run.
  - PeterCorless 5y ago
    I did this, in fact. I also had other colleagues. A lot of UI/UX depends on the browser (Chrome, Firefox, Brave, Opera) and the OS. So I wanted to confirm precisely what he was seeing. And we're already working on fixes.
- dmitriid 5y ago
  Three screenshots: https://imgur.com/a/XUpP9ha https://imgur.com/a/XUpP9ha Including the cookie banner that's illegal under GDPR (and is likely illegal under CCPA)
  - throwaway81523 5y ago
    Xkcd 624?
- dagmx 5y ago
  Thanks for looking into it. I see others (dmitriid) have uploaded screenshots already, so I won't duplicate. But yes, otherwise the article itself was insightful and thank you for sharing your findings.
dijit 5y ago
Seems alright on my iPad. https://imgur.com/a/SJwg1E2 https://imgur.com/a/SJwg1E2
- dagmx 5y ago
  That's a much larger screen than the largest iPhone. Hence why I specified mobile.
r00fus 5y ago
TBH, I've moved to setting Reader View as default for all Safari visits in iOS. You can whitelist sites or just undo reader view for that session if needed. Zero ads, and mostly I'm there for the text anyway.
bschwindHN 5y ago
On an iPhone SE 1, you literally get 5 lines of readable text, and 3 if the Safari navigation UI is showing. I also can't close the chat bubble because the X is off the screen to the right, and the scroll position jumps around near the top of the page as you scroll because some calculation or CSS is wrong.

gxt 5y ago

It's a great story and it's pleasant to see end-devs investigations contribute to the overall performance of the ecosystem since Tokio is so widely used. Cheers

carllerche 5y ago

Tokio author here. Generally speaking, I recommend strongly against using FuturesUnordered unless you know all the pitfalls. We are working on an alternative utility that should hopefully avoid the issues described here and others: https://github.com/tokio-rs/tokio/pull/4335 https://github.com/tokio-rs/tokio/pull/4335

psarna 5y ago
That's great news! Especially that the observed performance of the test program based on FuturesUnordered, even though it stopped being quadratic, it was still considerably slower than the task::unconstrained one, which suggests there's room for improvement. Probably due to the fact that you still pay with a constant number of polls (32) each time you go out of budget.
- carllerche 5y ago
  IMO FuturesUnordered should stop executing futures when it sees a "yield". An explicit yield signals control should be returned to the runtime. FuturesUnordered does not respect this.
  - dathinab 5y ago
    IMHO the main problem is tokio introducing preempting behaviour which in subtle ways can mess with all kinds of normally fully valid rust futures. Sure it sometimes magically fixes problematic code, but it's in effect still a magic extension to how rust polling works which can have surprising side effects. In a certain way tokios coperative-preempting is not adequate to handle any future which multiplexes multiple futures but such futures are a stable of rust async since the get to go.
    - carllerche 5y ago
      I don't think it is an issue w/ the pre-emption code. I believe FuturesUnordered is just doing the wrong thing: not respecting yields.

psarna 5y ago

Hi, author of the post here. I'll also be talking about this issue in a little more detail at an online Rust Warsaw meetup tomorrow, feel free to join, especially if you're up for a live discussion later: https://www.meetup.com/Rust-Warsaw/events/282879405/ https://www.meetup.com/Rust-Warsaw/events/282879405/

petr_tik 5y ago
Can you please shed some light on rust's adoption at scylla given the cumulative team expertise in Cpp and unsafe programming like thread-per-core architectures. In my experience so far, highly proficient Cpp programmers tend to find rust constraints overly restrictive, because they kind of internalise the borrow checker and are comfortable enough to bend the rules sometimes. how receptive are fellow scyllians (is that the term?) to rust? how much scope is there for future projects to be in Rust over Cpp? Thanks for the honest and detailed write-up!
- tialaramex 5y ago
  > comfortable enough to bend the rules sometimes. Mmm. The world we actually have suggests that this comfort is dangerous. Robert Browning was a famous poet. Certainly competent enough with English that you'd assume he knows what he's doing and can bend the rules. So, Dictionary makers were a little confused by the passage in Pippa Passes, "Then owls and bats / Cowls and twats / Monks and nuns in a cloister's moods / Adjourn to the oak-stump pantry". Why did Browning use the word "Twat" in this context? Turns out that he just had just assumed it was a word meaning a hat nuns wear... Oops. Of course this goof just means a high school literature teacher covering Browning has to decide whether to skip this part of Pippa Passes maybe "for time" and hope nobody asks about it, or explain to the class that even the best of us screw up sometimes. In Computer Software our mistakes are often not treated so kindly.
  - deleted 5y ago
    [deleted]
- psarna 5y ago

eminence32 5y ago

Nice article, and nice analysis of the problem. I have a personal theory that once a codebase gets complicated enough (no matter the language, no matter sync or async), you'll run into subtle bugs or performance problems that require a very deep understanding of all the relevant libraries in order to resolve the problem. One worry that I have with async rust is that the executors are so complex that the "baseline complexity" starts rather high. If this theory is true, then one might expect async rust to run into such problems more often than comparable non-async code. I haven't personally written enough async code to have any data either way, except for a few unpleasant experiences while making errors when writing async code. To the extent that this theory is a problem that needs solving, I don't think there is a solution. But I do think that, over time, weird async footguns will become less and less frequent as projects like tokio continue their excellent engineering efforts to plug up any weak spots and make things more robust in general.

marcus_cemes 5y ago
> One worry that I have with async rust is that the executors are so complex that the "baseline complexity" starts rather high. I completely agree, however I like that you have the option to swap out the executor for your own if the situation requires it, which I think concerns a very small number of applications. In other languages with a baked-in runtime you would be left trying to come up with workarounds to make it play nicely.
rr808 5y ago
I think this is true for every language, not just rust. I'm really not convinced async is good for complicated systems. Things like green threads are much easier to design with.
valenterry 5y ago
> I have a personal theory that once a codebase gets complicated enough (no matter the language, no matter sync or async), you'll run into subtle bugs or performance problems You probably, unintentionally, defined "complicated enough" to be at the point where these bugs occur - so that's a tautology. I think the point of programming languages and techniques/libraries is to move this point of "complicated enough" as far out as possible. Without any libraries, it might be easier to understand, but it will happen so much earlier that it's not worth it. And the point of libraries is that not only do they allow you to push the problem out to a much later point in time - they also allow you to switch between projects and not lose all your knowledge. Even considering high upfront costs, tt's a big net worth imho.

throwaway81523 5y ago

I haven't even clicked the link yet, but that the Scylla devs are doing something with Rust already is interesting. Seastar is very cool though constrained by the limitations of C++. It will be great to find out what effect Rust has.

AlexSW 5y ago
Which limitations of C++, sorry?

scottlamb 5y ago

Nice article. I'm in the process of absorbing it. Question: > scylla-rust-driver issued at least 1 syscall per query, which might be the source of elevated latency – and with a super-fast network (of which loopback is a prime example) it also means that throughput suffers – a latency of 1ms means that we won’t be able to send more than 1000 requests per second. Is 1 ms measured? Buffering and reducing syscalls is good, but still: that seems horribly slow for a small or EWOULDBLOCK-returning read/write. Why would it be that bad?

enedil 5y ago
I believe this is just an example of what are the implications of 1 query per ms, not indication that it actually took 1 ms.
- psarna 5y ago
  Indeed - 1ms was a nice round number, but it was also not that far from the real, sub-1ms number. But note that by "latency" here I mean not only the round trip, but the total measured time of executing a single request, including the time its task spends in Tokio (and its queues), and until its response is fully processed as well. Since the program sent requests with configurable concurrency set to ~1024 or more, the overall throughput was still satisfying. On the one hand it was (and still is) concerning that the observed latency per simple request was that high, on the other - it never really came up in our distributed tests, since there the network latency imposes a few milliseconds anyway. When we figure out the real source of this behavior, I'll be happy to describe the investigation process in a blog post (: