6 ms·
Principles for Fast Tokio Applications
- dist1ll 12d agoWhen you're at a point of tuning Tokio, consider taking a look at ef_vi/DPDK + SPDK
- rusbus 12d agoDo you have any resources worth referencing on this? I assume this isn't something that works with tokio more of a replace tokio?
- kev009 12d agoI don't think there is a ton of overlap. tokio is appropriate for general userspace apps, ranging anywhere from a CLI, GUI, API or web app. DPDK and SPDK are specialized fast paths for building network data paths and storage solutions that come with tradeoffs: DPDK uses poll mode drivers, outside of the operating system, which have various implications including busy waiting and taking over the interface. That is why DPDK is fast, no kernel/userspace context switching and copies, and the drivers are tuned for the polling model. But it's not a general purpose building block.
- dist1ll 12d agoFwiw with ef_vi you have full control over the event queue - you don't need to busy-spin it, you can choose whatever strategy you prefer. > tokio is appropriate for general userspace apps Yep, and for those I wouldn't recommend it. But tokio is also widely used in performance-critical infrastructure and web services. For those I'd say it can definitely be worth taking a second look at kernel bypass.
- kev009 11d agoef_vi is a solarflare proprietary feature which is now a support product, AMD moved on to Pensando. Once you move away from busy poll, you rapidly lost grounds to use DPDK. The PMD is a deliberate design to elide latency and lower interconnect taxes like PCIe traffic and cache/memory bandwidth by batching queue maintenance, that is the bargain made with a PMD. The field opens to OS native fast paths which have fewer downsides outside of that niche. Application developers are rarely concerned with this because it's far from where the bottleneck is for them.. a web service is rarely primarily a data mover, while a proxy is. Tokio has more in common with Golang than something like DPDK.
- Tsarp 12d agoOne great use of agentic coding is being able to add and very granular tracing instrumentation to help with these sort of optimizations.
- jeffbee 12d agoAlso a great way to make sure that your app spends most of its time in observability overhead. For example even the latency histogram that the OP mentions is wildly expensive.
- MomsAVoxell 12d agoIf you’re not using eBPF to trace your app you’re doing it wrong.
- jeffbee 12d agoThe low cost of eBPF tracing is another myth.
- MomsAVoxell 12d ago1) Its no myth, but you can definitely foot-bullet into doing it wrong, and 2) it's a far better path to take than in-app telemetry.
- deleted 12d ago[deleted]
- MobiusHorizons 12d agoDoesn’t that only work on Linux? And then only for things that make syscalls? Presumably people have to trace other slow paths sometime.
- nicoburns 12d agoOne legitimately great thing about LLMs is that it makes it feasible to add these kind of tracing instrumentations temporarily for profiling and then throw them away so they never reach source control let alone production.
- jeffbee 12d agoAll of the significant server applications I have encountered in the industry have suffered from the same problem, which surprised their authors but seemed obvious to me: the application was spending the majority of its CPU time doing meta-work like entering and leaving epoll, stealing work from itself, etc. There are principles for writing Tokio servers and these are good points in the OP but I think they are little-known and too easy to violate.
- cube00 12d agoI can't say I'm surprised when I see the 100+ function stack traces that Axum built on Tokio produces. Before you say Axum is "holding it wrong" the project lives under the tokio-rs GitHub org.
- rusbus 12d agoNote that most of those end up getting inlined in practice
- prydt 12d agoDo you have any references for these principles for writing Tokio servers? Or just a high level summary of what best practices look like?
- 5ersi 12d agoFor a true high performance you should use thread busy-spinning, CPU pinning and SPSC/MPSC ring buffers.
- Kenji 12d ago[dead]
- VorpalWay 12d agoIt all depends on what you are doing. I do embedded with strict realtime requirements. CPU pinning would not be an option. I have also done software that should use as little resources as possible (but still be quick) to coexist with other software on the same hardware. All of these are different, valid, meanings of high performance. You need context. An interactive IDE is yet another thing that needs to be high performance in yet another way.
- mahboi 12d agoAlso, using 100% CPU without a good reason can cause thermal throttling that makes it slower for the sections that actually need 100% CPU
- deleted 12d ago[deleted]
- HackerThemAll 12d agoSpinning is covered in the article.
- saghm 12d ago"Be careful with mutexes" is good advice, but I'm surprised it doesn't explicitly call out the various channels that tokio provides as alternatives (detailed here: https://docs.rs/tokio/latest/tokio/sync/index.html https://docs.rs/tokio/latest/tokio/sync/index.html). There are a variety of options that fit different use cases, and you don't even need to enable the runtime feature to use them (e.g. if you want to do a single check for completion rather than await). I'd estimate that at least half of the bottlenecks I've seen with mutexes when using tokio could have been avoided by not even using a mutex at all and instead passing the data that's truly needed across different tasks with some type of channel. The other trick I've used a few times that's a bit hacky but can get the job done is when reading a snapshot of the data under a mutex is enough without needing to prevent other changes; if that's the case, you can just clone the data and drop the mutex to allow other uses move forward at the cost of the data potentially being stale.
- rusbus 12d ago(I am OP) Both good call outs. Will update the article to include them
- saghm 12d agoAwesome! I was pretty confident you already were aware of both of those based on the level of knowledge needed for everything else in there, so I mostly was mentioning them here in case some people here might find them useful. Adding them in for others is even better though!
- CoolestBeans 12d agoTasks and channels is the way. You can get something that feels like programming a real preemptive concurrency model like BEAM languages or golang but with minimal overhead.
- eru 12d agoWhen you send a message in Erlang, nothing the recipient does with the message impacts anything on the sender side. That's good! In principle, they could have used something like copy-on-write for this, but in practice they really just make a copy of the bytes. Alas in Go, when you mutate what you received on a channel, you mutate the object the sender might still be holding. That's pretty annoying. It gets worse, because Golang has no way to declare something as `const` (like in C) nor that you are holding an immutable borrow (like in Rust). So you need to rely on conventions and perhaps a linter. Slighty less of a tangent: task and channels and software transactional memory (STM) are all great. I see mutexes as more of an implementation detail that you can use to implement these higher level abstractions (but they aren't the only way).
- iberator 12d agoWhat the hell is Tokio? Articles mentions it like once I was expecting some programing principles from Japan
- carllerche 12d agohttps://github.com/tokio-rs/tokio https://github.com/tokio-rs/tokio
- cogman10 12d agoTo further explain. Rust doesn't provide a runtime/framework for async/await, you have to bring your own. Tokio is (I believe) the most popular async/await framework for rust.
- kevinbaiv 12d ago[flagged]
- denizay 12d agoFast Tokioo, drift, drift, drift!
- SwtCyber 12d agoOne thing I appreciate here is treating scheduler fairness as something you spend, not something you get for free
- bilaly 11d agoWhen working on 20ms audio frames, Trusting MissedTickBehavior::Delay is not enough by itself. If you miss a tick, frames pile up. If you don't want that, you should drain all accumulated full frames on every single tick. Otherwise, a single missed tick can cause permanent latency.
- just60sec 11d ago[dead]