13 ms·

Tokio Console

788 points by hasheddan 5y ago

hcarvalhoalves 5y ago
Very interesting! I can't help but wonder why these technologies (the framework, and accompanying observability) aren't an OS-level feature; wondering if at this point we need an OS at all, and why not ship a minimal kernel + the binary of an application built w/ this to run directly on the VM.
- billfruit 5y ago
  Yes truly, is there something like this for linux threads and processes?
  - pkaye 5y ago
    The Linux Perf tool is pretty powerful. I've barely just scratched the surface myself. https://perf.wiki.kernel.org/index.php/Main_Page https://perf.wiki.kernel.org/index.php/Main_Page As as a side note, Go and pretty good profiling tools. You can see a trace of goroutine execution and why a goroutine got scheduled out and when GC kicks in. https://about.sourcegraph.com/go/an-introduction-to-go-tool-trace-rhys-hiltner/ https://about.sourcegraph.com/go/an-introduction-to-go-tool-...
  - mycoliza 5y ago
    A lot of the Tokio console UI was inspired by `htop`, which provides a pretty similar overview of processes and threads. It doesn't really have the same ability to inspect things like `pthread_mutex` and timerfds etc in the same way that the Tokio console can inspect the state of `tokio::sync::Mutex` and `tokio::time::Sleep`, though; although I wonder if something like that could be possible with eBPF...
- xtagon 5y ago
  It's been done, check out https://www.nerves-project.org/ https://www.nerves-project.org/ for an example of running a minimal kernel + Elixir (BEAM VM) on devices. As I understand it, it's powered by Buildroot and should be possible to do for other VMs/languages. Don't think I've seen anything like this for desktop OS's though!
- kristjansson 5y ago
  Unikernels are a thing e.g. https://mirage.io https://mirage.io. AFIACT uptake has been ... slow. You give up a lot to erase most or all of your OS, and only the most performance sensitive applications realize a benefit
- pjmlp 5y ago
  With languages with rich runtimes that is exactly the point. Actually in a way, POSIX is the missing C's runtime that wasn't made part of ISO C.
- ReactiveJelly 5y ago
  They're not OS-level because there isn't wide consensus on the right way to do things yet. Rust, C#, and JS all have similar concepts of async, but they're all slightly different. None of them would be trivial to adapt to other system langs like C and C++ - Rust requires compiler support to take apart async functions and put them back together as state machine, and the others lean on their GC. (And also compiler support, IIRC) I think there is a proposal to add coroutines in the new C++ standard, but I'm not sure how it would be done in the kernel. And sometimes I see people saying, "async is very bad, just use coroutines." Having only used Lua coroutines, I don't understand what the big difference is supposed to be. But mostly, these runtimes don't need OS-level support. Async is sort of a way to do concurrency without a kernel-level context switch for every task switch, right? If it's working so well in-process, why involve the OS at all? > wondering if at this point we need an OS at all Depends what you mean by OS. If you deploy in a container, of course your OS shares its kernel with the host. But for some user stories, (glares at Android) "OS" means all the software, including a Blink-based web browser, a plethora of GUI programs, and other things that I would rather call a "desktop environment" than an OS.
  - paavohtl 5y ago
    > Rust requires compiler support to take apart async functions and put them back together as state machine, and the others lean on their GC C# actually does the same thing, though it indeed still relies on GC.
- pcwalton 5y ago
  In Linux there's work being done on a language-agnostic lightweight threading model, User-Mode Concurrency Groups or UMCG: https://lwn.net/Articles/863386/ https://lwn.net/Articles/863386/ One could imagine a similar, runtime-independent console for UMCG. Note, however, that the programming model for such a runtime would be much more similar to 1:1 threading (i.e. blocking I/O with threads) than async/await.
- hayley-patton 5y ago
  Arguably, they are an OS-level feature, in the form of OS threads and your favourite task manager; but it is more expensive to run a very large number of OS threads, compared to async tasks or green threads. Per Dan Ingalls [1], an operating system is a collection of things that don't fit into a language. There shouldn't be one. [1] Design Principles Behind Smalltalk <https://www.cs.virginia.edu/~evans/cs655/readings/smalltalk.html https://www.cs.virginia.edu/~evans/cs655/readings/smalltalk....>
  - c-cube 5y ago
    What about people who want to use several programs on one computer, possibly even written in different languages? This idea is ridiculous unless you think everyone shares your dream of running everything in a smalltalk image.
    - hayley-patton 5y ago
      I don't think everyone shares _my_ dreams, which mostly don't involve Smalltalk images, but this seems quite feasible. And, from experience, making languages interoperate is a hassle, so "written in different languages" isn't all great. In the particular case of Smalltalk, the operational(?) semantics of which are specified by a virtual machine, one can just compile targeting the virtual machine. c.f compiling Smalltalk and Java to the Self virtual machine, or all the compilers targeting the JVM nowadays.
- im_down_w_otp 5y ago
  That's basically what we did with https://github.com/auxoncorp/ferros https://github.com/auxoncorp/ferros, Bundle Rust programs together as tasks to run atop the formally verified seL4 microkernel.
- erichocean 5y ago
  > wondering if at this point we need an OS at all People have written unikernels, but they're not that interesting anymore when you can trivially constrain Linux to a single core, keep the remaining cores for you app (no context switching), and keep all of the Linux-y goodness for admin and debugging (SSH, gdb, etc.). All of the performance, none of the admin and deployment headaches. Basically, unless you truly can't afford that extra core, unikernels are all downside at this point.
echelon 5y ago
The Rust ecosystem is on fire! This is such amazing tooling. There's so much best in class engineering going on with this language, Tokio/async runtimes, graphics, web libraries, etc.
- mycoliza 5y ago
  Thank you, that's really nice to hear!
xtagon 5y ago
Very cool. Reminds me of Erlang Observer.
dthul 5y ago
Can't wait to try this out! I always wanted the ability to see the tasks that are currently running or waiting. In the screenshots I can see that tasks have descriptive names. Does anybody know how to set the name for a task? tokio::spawn doesn't take a name parameter. Does it require `tracing`?
- lkt 5y ago
  Looks like they have added a builder for tokio::task[1] that will allow you to set a name. It's unstable at the moment so you would need to set the tokio_unstable cfg flag. [1]: https://github.com/tokio-rs/tokio/blob/master/tokio/src/task/builder.rs https://github.com/tokio-rs/tokio/blob/master/tokio/src/task...
  - carllerche 5y ago
    Exactly, this is the first release of Tokio Console. We will keep adding functionality over time, which will help inform the APIs in Tokio to better provide the necessary instrumentation. I expect the new APIs like the task builder will stabilize (no longer require the `tokio_unstable` flag) over the course of 2022.
artursapek 5y ago
This is awesome
kristianpaul 5y ago
Is this the beginning of a new way of deploying applications?
tpmx 5y ago
I think a submission title regarding something specifically about Rust development should contain the word Rust.
- dnautics 5y ago
  i think most people know tokio is rust (I'm not a rust dev)
  - tpmx 5y ago
    It's the german word for Tokyo. There is little reason for a non-rust dev to know this other meaning. I hate these ambigious HN vaguebait headlines and you should too.
    - awestroke 5y ago
      There is little reason for a non-german to know this other meaning
      - deleted 5y ago
        [deleted]
      - andrewshadura 5y ago
        Actually, that's how Tokyo is spelled in many languages besides German.
        lupin_sansei 5y ago
        It used to be spelled Tokio in English speaking countries before the 1950s https://english.stackexchange.com/questions/207014/why-was-tokyo-sometimes-called-tokio https://english.stackexchange.com/questions/207014/why-was-t...
    - dnautics 5y ago
      if an hn headline says something like "best practices for Phoenix" nobody expects it to refer to the city Phoenix, Arizona, USA. It's more likely Elixir Phoenix, the web framework. Or it could be one of a ton of other software projects called Phoenix. this is not a problem.
mycoliza 5y ago
Hi, I'm one of the main authors of `tokio-console`, so if folks have any questions, I'm happy to answer them!
- IMTDb 5y ago
  Is there any question we should ask ? About some challenge or interesting discovery / aspect that most of us wouldn't even realise ?
- staticassertion 5y ago
  What's the intended workflow? Would I run a console an all of my rust services, and then when debugging some prod issue connect to it? Or would I flip a switch? Or is it more for CLIs? Curious to hear, in general, how it's been used.
  - carllerche 5y ago
    This first release is geared primarily towards local debugging. That said, it is designed to be able to enable / disable instrumentation at runtime and it will be able to support connecting to a process in production, enable the instrumentation, and debug there. Right now, we wanted to get the first release out and start getting people using it and collect feedback to help prioritize future development.
    - staticassertion 5y ago
      Got it, thanks. A follow up - is this work funded in any way?
      - mycoliza 5y ago
        Tokio is a non-profit, community-supported project, although many of us work on it as part of our day jobs. If you want to support Tokio development, you can contribute to it on GitHub Sponsors (https://github.com/sponsors/tokio-rs https://github.com/sponsors/tokio-rs) and on OpenCollective (https://opencollective.com/tokio https://opencollective.com/tokio). I've also recently started accepting donations on my personal GitHub Sponsors page (https://github.com/sponsors/hawkw https://github.com/sponsors/hawkw) if you're interested in supporting my open-source work in particular.
marmada 5y ago
this is very exciting b/c it feels like one of the 1st language features centered around runtime debugging. Most languages don't seem to do a lot for run-time debugging. Being able to `gdb` and step-through on a local binary is a far-cry from detailed metrics / visualizations. We end up resorting to stuff like Honeycomb, but I'm waiting for the days of a programming language built for the ground-up for runtime debugging. Anyways, this feels like an important step in the right direction and I'm excited to try this out
- ReactiveJelly 5y ago
  I've been waiting years for some cooperation between languages and editors to achieve things Bret Victor wanted years ago, like "Just change this constant while the program is running", where you can have very tight iteration loops for things that don't need a full recompile. I mean, Visual Studio has some kind of hot reloading, but I'm not gonna VS, I want mainstream support for this fast iteration and deep runtime debugging.
  - stjohnswarts 5y ago
    gdb has allowed you to change runtime variable values for the couple of decades that I've been using it. e.g. "set var i=10" ? Or are you talking about something else?
    - nneonneo 5y ago
      It's very limited, and doesn't work for things like constants that might be compiled straight into the code. By contrast, a true hot-reload can swap out the actual code at runtime.
- w7 5y ago
  Go also has pretty good out of the box profiling (pprof[0]) and third-party runtime debugging (delv[1]) that can be used both remotely and local. These tools also have decent editor integration and can be use hand in hand: https://blog.jetbrains.com/go/2019/04/03/profiling-go-applications-and-tests/ https://blog.jetbrains.com/go/2019/04/03/profiling-go-applic... https://blog.jetbrains.com/go/2020/03/03/how-to-find-goroutines-during-debugging/ https://blog.jetbrains.com/go/2020/03/03/how-to-find-gorouti... [0] https://github.com/google/pprof https://github.com/google/pprof [1] https://github.com/go-delve/delve https://github.com/go-delve/delve
javajosh 5y ago
Praise, 2 suggestions and a question. Color coding OOM (order-of-magnitude) is smart, but the colors chosen seem quite hard to distinguish. Consider picking different default colors; also consider adding a redundant representation of scale, like an integer representing OOM scale. It is strange that a thoughtful design around OOM would also choose keep so many digits. If the goal is to summarize, then please throw those extra digits away. (e.g. 10.3992s is specifying the time down to the millisecond, but those extra ms are not meaningful at the second time scale.) What is this about the runtime polling tasks? Does that happen in Tokio? Why? I am only familiar with 2 async runtimes (node, vert.x) and I was under the impression that neither of these "poll their threads" in any meaningful way. Threads (or process or fiber or whatever you want to call it in this context) never initiate action on their own. They are ALWAYS waiting for something to happen to them, and that includes timeouts, which would be triggered by passing a thread a clock tick. The runtime's job is to centrally manage resources that must persist between thread behavior, so at most it is going to be polling external resources, and not it's own threads. Or maybe I don't understand what polling means here? I would interpret it as meaning "keep checking a well-known place for changes, and then do something if a change is found". But since async threads can't initiate any change on their own, this is nonsensical.
- carllerche 5y ago
  Also, thanks for the thought re: digit precision. I am tracking it here: https://github.com/tokio-rs/console/issues/224 https://github.com/tokio-rs/console/issues/224
- aliceryhl 5y ago
  Generally the way async Rust works is that you have a Future trait with a poll method, and if you call it the future will attempt to make progress if it can — e.g. if the task is a timer it will check the time and complete the task if so and otherwise return Pending. However, async Rust includes an additional concept: Wakers. When your runtime (Tokio) calls poll on a future, it gives the future a waker object, and when the future is ready to continue work, something needs to call wake on the waker. Once this happens, Tokio will poll the task again soon, and Tokio wont poll tasks that have not been woken. For example, for timers, Tokio includes a timer wheel (a sort of priority queue) with all the registered timers, and the timer wheel calls wake on the appropriate waker whenever a timer expires. Similarly with a message passing channel, the sender calls wake on the receiver's waker when a message is sent.
teknopurge 5y ago
this is great. thank you.
xrisk 5y ago
Has anybody done this for Python?
- alexchamberlain 5y ago
  trio has a similar utility I believe
7357 5y ago
1st time i laughed reading a readme for such a brandifious project.
- phab 5y ago
  What does brandifious mean? A google search doesn't return anything.
  - 7357 5y ago
    Thanks for asking, in short pure nonsense with the intention to attract derision. Google does know it? :-) Because I made it up. It is synonymous to grandiose without being pretentious and is originally meant to be used in some product marketing context. Are you Sir Ious?
nih0 5y ago
very cool, reminds me a bit of jmx which I miss alot back from my java days.
Thaxll 5y ago
I'm not sure about that, I don't use Rust but I don't see how it could be relevant because its real time, what you want is to dump things and analyze later, if you have to stay in front of the console to see what's happening it's not very useful. Is it currently possible to dump the trace and analyze it later with that console? ( like Go or Java does with JFR ). The idea is great it's the way to consume the information that I think misses the point.
- pastrami_panda 5y ago
  I only read the first paragraph but it sounds like it addresses this? > It gives you a live, easy-to-navigate view into the program's tasks and resources, summarizing both their current status and their historical behavior.
- pjscott 5y ago
  Often you can find a misbehaving process that somehow got into a weird state, one that you weren't tracing beforehand because you weren't expecting trouble and didn't want the overhead, and in those cases it can be tremendously useful to be able to attach some inspection tool to figure out what's going on so you can fix the underlying bug.
- carllerche 5y ago
  Dumping to a file for later analysis is on the roadmap.
- pcwalton 5y ago
  It's helpful if you're, for example, running unit or integration tests and see stuck threads.
seanp2k2 5y ago
Is there something like this for Golang? I've inherited a golang project at work and it's really not fun to debug compared to Python.
- ReactiveJelly 5y ago
  Is there even something like this for Python? I've used Tracy for C++, which does similar tracing on OS-level threads, but I don't know much about Python.
- mycoliza 5y ago
  Go has pprof (https://github.com/google/pprof https://github.com/google/pprof), which I've heard good things about --- and, the pprof data model was one of the influences I looked at when designing the Tokio console's wire format. But, I'm not sure if pprof has any similar UIs to the one we've implemented for the Tokio console; and I haven't actually used it all that much.
  - Groxx 5y ago
    pprof + runtime tracing spans is probably the best equivalent in Go-land, yeah. I have yet to see any library actually use those, but that may just be me being unlucky.
- pse_pan2233 5y ago
  Already mentioned: https://pkg.go.dev/net/http/pprof https://pkg.go.dev/net/http/pprof I've just seen that Goland (the IDE) has some nice integration: https://blog.jetbrains.com/go/2020/03/03/how-to-find-goroutines-during-debugging/ https://blog.jetbrains.com/go/2020/03/03/how-to-find-gorouti...
outside1234 5y ago
Thank you so much for building the things in the Tokio ecosystem including Console. So much great stuff!
iillexial 5y ago
Is there sth like this for Go?
- mjibson 5y ago
  Having programmed in both Rust and Go: I mostly didn't need what this does in Go because my problem was usually solved by panic'ing, which prints a stack trace of each go routine, allowing me to get enough of the way through figuring out my current problem (usually something was stuck waiting or sending). In tokio, there's no such print-stack behavior, so it's much harder to get a snapshot of what's going on. (I'm relatively new to tokio and Rust, so there's perhaps a handy "print all the tokio tasks and their stack traces" method, but I haven't come across it yet.) If folks use this console thing for perf reasons and not debug reasons, then yeah, maybe cool to have in Go.
- pse_pan2233 5y ago
  https://pkg.go.dev/net/http/pprof https://pkg.go.dev/net/http/pprof Goland (the IDE) has some nice integration: https://blog.jetbrains.com/go/2020/03/03/how-to-find-goroutines-during-debugging/ https://blog.jetbrains.com/go/2020/03/03/how-to-find-gorouti...
stjohnswarts 5y ago
Oh I've been waiting for this for sooooo long. Thank you tokio team. I think this is going to make my life a lot easier in the debugging sphere.
the__alchemist 5y ago
Question for y'all using Async/Await: Are you coding web servers/backends, and/or do you have a background in that? Testing a (transparent!) theory.
- qdot76367 5y ago
  I'm using it for hardware libraries that are mostly I/O bound. We're not C10K, we're barely C10, but it's easier than managing threads manually for me.
  - msdrigg 5y ago
    This made me laugh, but Im in the same boat haha
  - the__alchemist 5y ago
    Interesting! I use rust mainly for embedded. There's a framework called Embassy that uses Async. Mainly built by one guy who's pretty sharp. I prefer to use interrupts, DMA, timers etc directly instead of Async, but it seems that a large number of Rust programmers consider Async to be fundamental. I wasn't sure if that was just the web devs or what, but here are 2 counter-examples.
- mycoliza 5y ago
  At my day job (https://buoyant.io/ https://buoyant.io/), we're using it to write a reverse proxy/load balancer --- one of the use-cases where you really, absolutely do need asynchronous concurrency. :)
larodi 5y ago
All your HN is belong to rustaceans.
pse_pan2233 5y ago
The Tokio ecosystem is tremendously important for people considering Rust for professional networking stuff. It not only has great libs / features, but will be well maintained for years to come - this is my firm believe at least and of course what you need for professional projects (as opposed to personal).
realcr 5y ago
Thanks for making this! It looks super useful. Will I be able to use console or part of it with the rest of the rust async ecosystem? (for example, async-std or futures-rs), Or is is Tokio specific?
- carllerche 5y ago
  It does not have a hard dependency to Tokio. Any runtime can use tracing to emit the necessary instrumentation to work with Tokio Console.
Inufu 5y ago
This looks great, thank you for the hard work! Have you considered to also expose this information in an interactive web interface? Using a zoomable timeline view (https://www.tensorflow.org/tensorboard/tensorboard_profiling_keras https://www.tensorflow.org/tensorboard/tensorboard_profiling...), both for after the fact analysis (taking a fixed N second trace and then inspecting it) as well as interactive visualization (automatically scrolling timeline with option to pause and scrub).
- carllerche 5y ago
  We would love a web view but don’t have any ability to design or much experience with building web apps, so we stuck with a terminal UI.
- mycoliza 5y ago
  Yes, we've designed the overall architecture of the system to be modular so that the telemetry can be consumed by a number of different UIs --- we'd love to see someone write web interfaces and/or native GUIs for the console data. I have basically no web development experience whatsoever, though, so I went with the terminal app, because not having to learn JavaScript first made it a lot easier to get started :) We're also thinking about factoring out the Tokio Console command-line application's internal data model and client code into its own library (https://github.com/tokio-rs/console/issues/227 https://github.com/tokio-rs/console/issues/227) to make it easier to build other UIs on top of that.
Dowwie 5y ago
In action: https://www.youtube.com/watch?v=JGCewPUvF70 https://www.youtube.com/watch?v=JGCewPUvF70
deleted 5y ago
[deleted]
IshKebab 5y ago
This looks great though it's a shame it only has a TUI mode. The interface is pretty much begging for a proper GUI. I guess we're still a way away from having a good de fact Rust GUI toolkit.
- carllerche 5y ago
  Would love a real GUI but we have no design talent and stuck with something simpler.
LeSaucy 5y ago
Is there anything comparable to this with boost/asio?
zbentley 5y ago
That's just lovely. I'm a huge fan of programs exposing deep runtime state via normal (not debug mode only) instrumentation. Java has historically been amazing in this area, it's great to see other languages stepping up as well. Does anyone know of a similar visualizer for coroutines/threads for async Python?
henryaj 5y ago
Anyone know what the font is in those screenshots?
- mycoliza 5y ago
  It's https://typeof.net/Iosevka/ https://typeof.net/Iosevka/ (I took the screenshots).
meganium1 5y ago

788 points by hasheddan 5y ago

Very interesting! I can't help but wonder why these technologies (the framework, and accompanying observability) aren't an OS-level feature; wondering if at this point we need an OS at all, and why not ship a minimal kernel + the binary of an application built w/ this to run directly on the VM.

billfruit 5y ago
Yes truly, is there something like this for linux threads and processes?
- pkaye 5y ago
  The Linux Perf tool is pretty powerful. I've barely just scratched the surface myself. https://perf.wiki.kernel.org/index.php/Main_Page https://perf.wiki.kernel.org/index.php/Main_Page As as a side note, Go and pretty good profiling tools. You can see a trace of goroutine execution and why a goroutine got scheduled out and when GC kicks in. https://about.sourcegraph.com/go/an-introduction-to-go-tool-trace-rhys-hiltner/ https://about.sourcegraph.com/go/an-introduction-to-go-tool-...
- mycoliza 5y ago
  A lot of the Tokio console UI was inspired by `htop`, which provides a pretty similar overview of processes and threads. It doesn't really have the same ability to inspect things like `pthread_mutex` and timerfds etc in the same way that the Tokio console can inspect the state of `tokio::sync::Mutex` and `tokio::time::Sleep`, though; although I wonder if something like that could be possible with eBPF...
xtagon 5y ago
It's been done, check out https://www.nerves-project.org/ https://www.nerves-project.org/ for an example of running a minimal kernel + Elixir (BEAM VM) on devices. As I understand it, it's powered by Buildroot and should be possible to do for other VMs/languages. Don't think I've seen anything like this for desktop OS's though!
kristjansson 5y ago
Unikernels are a thing e.g. https://mirage.io https://mirage.io. AFIACT uptake has been ... slow. You give up a lot to erase most or all of your OS, and only the most performance sensitive applications realize a benefit
pjmlp 5y ago
With languages with rich runtimes that is exactly the point. Actually in a way, POSIX is the missing C's runtime that wasn't made part of ISO C.
ReactiveJelly 5y ago
They're not OS-level because there isn't wide consensus on the right way to do things yet. Rust, C#, and JS all have similar concepts of async, but they're all slightly different. None of them would be trivial to adapt to other system langs like C and C++ - Rust requires compiler support to take apart async functions and put them back together as state machine, and the others lean on their GC. (And also compiler support, IIRC) I think there is a proposal to add coroutines in the new C++ standard, but I'm not sure how it would be done in the kernel. And sometimes I see people saying, "async is very bad, just use coroutines." Having only used Lua coroutines, I don't understand what the big difference is supposed to be. But mostly, these runtimes don't need OS-level support. Async is sort of a way to do concurrency without a kernel-level context switch for every task switch, right? If it's working so well in-process, why involve the OS at all? > wondering if at this point we need an OS at all Depends what you mean by OS. If you deploy in a container, of course your OS shares its kernel with the host. But for some user stories, (glares at Android) "OS" means all the software, including a Blink-based web browser, a plethora of GUI programs, and other things that I would rather call a "desktop environment" than an OS.
- paavohtl 5y ago
  > Rust requires compiler support to take apart async functions and put them back together as state machine, and the others lean on their GC C# actually does the same thing, though it indeed still relies on GC.
pcwalton 5y ago
In Linux there's work being done on a language-agnostic lightweight threading model, User-Mode Concurrency Groups or UMCG: https://lwn.net/Articles/863386/ https://lwn.net/Articles/863386/ One could imagine a similar, runtime-independent console for UMCG. Note, however, that the programming model for such a runtime would be much more similar to 1:1 threading (i.e. blocking I/O with threads) than async/await.
hayley-patton 5y ago
Arguably, they are an OS-level feature, in the form of OS threads and your favourite task manager; but it is more expensive to run a very large number of OS threads, compared to async tasks or green threads. Per Dan Ingalls [1], an operating system is a collection of things that don't fit into a language. There shouldn't be one. [1] Design Principles Behind Smalltalk <https://www.cs.virginia.edu/~evans/cs655/readings/smalltalk.html https://www.cs.virginia.edu/~evans/cs655/readings/smalltalk....>
- c-cube 5y ago
  What about people who want to use several programs on one computer, possibly even written in different languages? This idea is ridiculous unless you think everyone shares your dream of running everything in a smalltalk image.
  - hayley-patton 5y ago
    I don't think everyone shares _my_ dreams, which mostly don't involve Smalltalk images, but this seems quite feasible. And, from experience, making languages interoperate is a hassle, so "written in different languages" isn't all great. In the particular case of Smalltalk, the operational(?) semantics of which are specified by a virtual machine, one can just compile targeting the virtual machine. c.f compiling Smalltalk and Java to the Self virtual machine, or all the compilers targeting the JVM nowadays.
im_down_w_otp 5y ago
That's basically what we did with https://github.com/auxoncorp/ferros https://github.com/auxoncorp/ferros, Bundle Rust programs together as tasks to run atop the formally verified seL4 microkernel.
erichocean 5y ago
> wondering if at this point we need an OS at all People have written unikernels, but they're not that interesting anymore when you can trivially constrain Linux to a single core, keep the remaining cores for you app (no context switching), and keep all of the Linux-y goodness for admin and debugging (SSH, gdb, etc.). All of the performance, none of the admin and deployment headaches. Basically, unless you truly can't afford that extra core, unikernels are all downside at this point.

echelon 5y ago

The Rust ecosystem is on fire! This is such amazing tooling. There's so much best in class engineering going on with this language, Tokio/async runtimes, graphics, web libraries, etc.

mycoliza 5y ago
Thank you, that's really nice to hear!

xtagon 5y ago

Very cool. Reminds me of Erlang Observer.

dthul 5y ago

Can't wait to try this out! I always wanted the ability to see the tasks that are currently running or waiting. In the screenshots I can see that tasks have descriptive names. Does anybody know how to set the name for a task? tokio::spawn doesn't take a name parameter. Does it require `tracing`?

lkt 5y ago
Looks like they have added a builder for tokio::task[1] that will allow you to set a name. It's unstable at the moment so you would need to set the tokio_unstable cfg flag. [1]: https://github.com/tokio-rs/tokio/blob/master/tokio/src/task/builder.rs https://github.com/tokio-rs/tokio/blob/master/tokio/src/task...
- carllerche 5y ago
  Exactly, this is the first release of Tokio Console. We will keep adding functionality over time, which will help inform the APIs in Tokio to better provide the necessary instrumentation. I expect the new APIs like the task builder will stabilize (no longer require the `tokio_unstable` flag) over the course of 2022.

artursapek 5y ago

This is awesome

kristianpaul 5y ago

Is this the beginning of a new way of deploying applications?

tpmx 5y ago

I think a submission title regarding something specifically about Rust development should contain the word Rust.

dnautics 5y ago
i think most people know tokio is rust (I'm not a rust dev)
- tpmx 5y ago
  It's the german word for Tokyo. There is little reason for a non-rust dev to know this other meaning. I hate these ambigious HN vaguebait headlines and you should too.
  - awestroke 5y ago
    There is little reason for a non-german to know this other meaning
    - deleted 5y ago
      [deleted]
    - andrewshadura 5y ago
      Actually, that's how Tokyo is spelled in many languages besides German.
      - lupin_sansei 5y ago
        It used to be spelled Tokio in English speaking countries before the 1950s https://english.stackexchange.com/questions/207014/why-was-tokyo-sometimes-called-tokio https://english.stackexchange.com/questions/207014/why-was-t...
  - dnautics 5y ago
    if an hn headline says something like "best practices for Phoenix" nobody expects it to refer to the city Phoenix, Arizona, USA. It's more likely Elixir Phoenix, the web framework. Or it could be one of a ton of other software projects called Phoenix. this is not a problem.

mycoliza 5y ago

Hi, I'm one of the main authors of `tokio-console`, so if folks have any questions, I'm happy to answer them!

IMTDb 5y ago
Is there any question we should ask ? About some challenge or interesting discovery / aspect that most of us wouldn't even realise ?
staticassertion 5y ago
What's the intended workflow? Would I run a console an all of my rust services, and then when debugging some prod issue connect to it? Or would I flip a switch? Or is it more for CLIs? Curious to hear, in general, how it's been used.
- carllerche 5y ago
  This first release is geared primarily towards local debugging. That said, it is designed to be able to enable / disable instrumentation at runtime and it will be able to support connecting to a process in production, enable the instrumentation, and debug there. Right now, we wanted to get the first release out and start getting people using it and collect feedback to help prioritize future development.
  - staticassertion 5y ago
    Got it, thanks. A follow up - is this work funded in any way?
    - mycoliza 5y ago
      Tokio is a non-profit, community-supported project, although many of us work on it as part of our day jobs. If you want to support Tokio development, you can contribute to it on GitHub Sponsors (https://github.com/sponsors/tokio-rs https://github.com/sponsors/tokio-rs) and on OpenCollective (https://opencollective.com/tokio https://opencollective.com/tokio). I've also recently started accepting donations on my personal GitHub Sponsors page (https://github.com/sponsors/hawkw https://github.com/sponsors/hawkw) if you're interested in supporting my open-source work in particular.

marmada 5y ago

this is very exciting b/c it feels like one of the 1st language features centered around runtime debugging. Most languages don't seem to do a lot for run-time debugging. Being able to `gdb` and step-through on a local binary is a far-cry from detailed metrics / visualizations. We end up resorting to stuff like Honeycomb, but I'm waiting for the days of a programming language built for the ground-up for runtime debugging. Anyways, this feels like an important step in the right direction and I'm excited to try this out

ReactiveJelly 5y ago
I've been waiting years for some cooperation between languages and editors to achieve things Bret Victor wanted years ago, like "Just change this constant while the program is running", where you can have very tight iteration loops for things that don't need a full recompile. I mean, Visual Studio has some kind of hot reloading, but I'm not gonna VS, I want mainstream support for this fast iteration and deep runtime debugging.
- stjohnswarts 5y ago
  gdb has allowed you to change runtime variable values for the couple of decades that I've been using it. e.g. "set var i=10" ? Or are you talking about something else?
  - nneonneo 5y ago
    It's very limited, and doesn't work for things like constants that might be compiled straight into the code. By contrast, a true hot-reload can swap out the actual code at runtime.
w7 5y ago
Go also has pretty good out of the box profiling (pprof[0]) and third-party runtime debugging (delv[1]) that can be used both remotely and local. These tools also have decent editor integration and can be use hand in hand: https://blog.jetbrains.com/go/2019/04/03/profiling-go-applications-and-tests/ https://blog.jetbrains.com/go/2019/04/03/profiling-go-applic... https://blog.jetbrains.com/go/2020/03/03/how-to-find-goroutines-during-debugging/ https://blog.jetbrains.com/go/2020/03/03/how-to-find-gorouti... [0] https://github.com/google/pprof https://github.com/google/pprof [1] https://github.com/go-delve/delve https://github.com/go-delve/delve

javajosh 5y ago

Praise, 2 suggestions and a question. Color coding OOM (order-of-magnitude) is smart, but the colors chosen seem quite hard to distinguish. Consider picking different default colors; also consider adding a redundant representation of scale, like an integer representing OOM scale. It is strange that a thoughtful design around OOM would also choose keep so many digits. If the goal is to summarize, then please throw those extra digits away. (e.g. 10.3992s is specifying the time down to the millisecond, but those extra ms are not meaningful at the second time scale.) What is this about the runtime polling tasks? Does that happen in Tokio? Why? I am only familiar with 2 async runtimes (node, vert.x) and I was under the impression that neither of these "poll their threads" in any meaningful way. Threads (or process or fiber or whatever you want to call it in this context) never initiate action on their own. They are ALWAYS waiting for something to happen to them, and that includes timeouts, which would be triggered by passing a thread a clock tick. The runtime's job is to centrally manage resources that must persist between thread behavior, so at most it is going to be polling external resources, and not it's own threads. Or maybe I don't understand what polling means here? I would interpret it as meaning "keep checking a well-known place for changes, and then do something if a change is found". But since async threads can't initiate any change on their own, this is nonsensical.

carllerche 5y ago
Also, thanks for the thought re: digit precision. I am tracking it here: https://github.com/tokio-rs/console/issues/224 https://github.com/tokio-rs/console/issues/224
aliceryhl 5y ago
Generally the way async Rust works is that you have a Future trait with a poll method, and if you call it the future will attempt to make progress if it can — e.g. if the task is a timer it will check the time and complete the task if so and otherwise return Pending. However, async Rust includes an additional concept: Wakers. When your runtime (Tokio) calls poll on a future, it gives the future a waker object, and when the future is ready to continue work, something needs to call wake on the waker. Once this happens, Tokio will poll the task again soon, and Tokio wont poll tasks that have not been woken. For example, for timers, Tokio includes a timer wheel (a sort of priority queue) with all the registered timers, and the timer wheel calls wake on the appropriate waker whenever a timer expires. Similarly with a message passing channel, the sender calls wake on the receiver's waker when a message is sent.

teknopurge 5y ago

this is great. thank you.

xrisk 5y ago

Has anybody done this for Python?

alexchamberlain 5y ago
trio has a similar utility I believe

7357 5y ago

1st time i laughed reading a readme for such a brandifious project.

phab 5y ago
What does brandifious mean? A google search doesn't return anything.
- 7357 5y ago
  Thanks for asking, in short pure nonsense with the intention to attract derision. Google does know it? :-) Because I made it up. It is synonymous to grandiose without being pretentious and is originally meant to be used in some product marketing context. Are you Sir Ious?

nih0 5y ago

very cool, reminds me a bit of jmx which I miss alot back from my java days.

Thaxll 5y ago

I'm not sure about that, I don't use Rust but I don't see how it could be relevant because its real time, what you want is to dump things and analyze later, if you have to stay in front of the console to see what's happening it's not very useful. Is it currently possible to dump the trace and analyze it later with that console? ( like Go or Java does with JFR ). The idea is great it's the way to consume the information that I think misses the point.

pastrami_panda 5y ago
I only read the first paragraph but it sounds like it addresses this? > It gives you a live, easy-to-navigate view into the program's tasks and resources, summarizing both their current status and their historical behavior.
pjscott 5y ago
Often you can find a misbehaving process that somehow got into a weird state, one that you weren't tracing beforehand because you weren't expecting trouble and didn't want the overhead, and in those cases it can be tremendously useful to be able to attach some inspection tool to figure out what's going on so you can fix the underlying bug.
carllerche 5y ago
Dumping to a file for later analysis is on the roadmap.
pcwalton 5y ago
It's helpful if you're, for example, running unit or integration tests and see stuck threads.

seanp2k2 5y ago

Is there something like this for Golang? I've inherited a golang project at work and it's really not fun to debug compared to Python.

ReactiveJelly 5y ago
Is there even something like this for Python? I've used Tracy for C++, which does similar tracing on OS-level threads, but I don't know much about Python.
mycoliza 5y ago
Go has pprof (https://github.com/google/pprof https://github.com/google/pprof), which I've heard good things about --- and, the pprof data model was one of the influences I looked at when designing the Tokio console's wire format. But, I'm not sure if pprof has any similar UIs to the one we've implemented for the Tokio console; and I haven't actually used it all that much.
- Groxx 5y ago
  pprof + runtime tracing spans is probably the best equivalent in Go-land, yeah. I have yet to see any library actually use those, but that may just be me being unlucky.
pse_pan2233 5y ago
Already mentioned: https://pkg.go.dev/net/http/pprof https://pkg.go.dev/net/http/pprof I've just seen that Goland (the IDE) has some nice integration: https://blog.jetbrains.com/go/2020/03/03/how-to-find-goroutines-during-debugging/ https://blog.jetbrains.com/go/2020/03/03/how-to-find-gorouti...

outside1234 5y ago

Thank you so much for building the things in the Tokio ecosystem including Console. So much great stuff!

iillexial 5y ago

Is there sth like this for Go?

mjibson 5y ago
Having programmed in both Rust and Go: I mostly didn't need what this does in Go because my problem was usually solved by panic'ing, which prints a stack trace of each go routine, allowing me to get enough of the way through figuring out my current problem (usually something was stuck waiting or sending). In tokio, there's no such print-stack behavior, so it's much harder to get a snapshot of what's going on. (I'm relatively new to tokio and Rust, so there's perhaps a handy "print all the tokio tasks and their stack traces" method, but I haven't come across it yet.) If folks use this console thing for perf reasons and not debug reasons, then yeah, maybe cool to have in Go.
pse_pan2233 5y ago
https://pkg.go.dev/net/http/pprof https://pkg.go.dev/net/http/pprof Goland (the IDE) has some nice integration: https://blog.jetbrains.com/go/2020/03/03/how-to-find-goroutines-during-debugging/ https://blog.jetbrains.com/go/2020/03/03/how-to-find-gorouti...

stjohnswarts 5y ago

Oh I've been waiting for this for sooooo long. Thank you tokio team. I think this is going to make my life a lot easier in the debugging sphere.

the__alchemist 5y ago

Question for y'all using Async/Await: Are you coding web servers/backends, and/or do you have a background in that? Testing a (transparent!) theory.

qdot76367 5y ago
I'm using it for hardware libraries that are mostly I/O bound. We're not C10K, we're barely C10, but it's easier than managing threads manually for me.
- msdrigg 5y ago
  This made me laugh, but Im in the same boat haha
- the__alchemist 5y ago
  Interesting! I use rust mainly for embedded. There's a framework called Embassy that uses Async. Mainly built by one guy who's pretty sharp. I prefer to use interrupts, DMA, timers etc directly instead of Async, but it seems that a large number of Rust programmers consider Async to be fundamental. I wasn't sure if that was just the web devs or what, but here are 2 counter-examples.
mycoliza 5y ago
At my day job (https://buoyant.io/ https://buoyant.io/), we're using it to write a reverse proxy/load balancer --- one of the use-cases where you really, absolutely do need asynchronous concurrency. :)

larodi 5y ago

All your HN is belong to rustaceans.

pse_pan2233 5y ago

The Tokio ecosystem is tremendously important for people considering Rust for professional networking stuff. It not only has great libs / features, but will be well maintained for years to come - this is my firm believe at least and of course what you need for professional projects (as opposed to personal).

realcr 5y ago

Thanks for making this! It looks super useful. Will I be able to use console or part of it with the rest of the rust async ecosystem? (for example, async-std or futures-rs), Or is is Tokio specific?

carllerche 5y ago
It does not have a hard dependency to Tokio. Any runtime can use tracing to emit the necessary instrumentation to work with Tokio Console.

Inufu 5y ago

This looks great, thank you for the hard work! Have you considered to also expose this information in an interactive web interface? Using a zoomable timeline view (https://www.tensorflow.org/tensorboard/tensorboard_profiling_keras https://www.tensorflow.org/tensorboard/tensorboard_profiling...), both for after the fact analysis (taking a fixed N second trace and then inspecting it) as well as interactive visualization (automatically scrolling timeline with option to pause and scrub).

carllerche 5y ago
We would love a web view but don’t have any ability to design or much experience with building web apps, so we stuck with a terminal UI.
mycoliza 5y ago
Yes, we've designed the overall architecture of the system to be modular so that the telemetry can be consumed by a number of different UIs --- we'd love to see someone write web interfaces and/or native GUIs for the console data. I have basically no web development experience whatsoever, though, so I went with the terminal app, because not having to learn JavaScript first made it a lot easier to get started :) We're also thinking about factoring out the Tokio Console command-line application's internal data model and client code into its own library (https://github.com/tokio-rs/console/issues/227 https://github.com/tokio-rs/console/issues/227) to make it easier to build other UIs on top of that.

Dowwie 5y ago

In action: https://www.youtube.com/watch?v=JGCewPUvF70 https://www.youtube.com/watch?v=JGCewPUvF70

deleted 5y ago

[deleted]

IshKebab 5y ago

This looks great though it's a shame it only has a TUI mode. The interface is pretty much begging for a proper GUI. I guess we're still a way away from having a good de fact Rust GUI toolkit.

carllerche 5y ago
Would love a real GUI but we have no design talent and stuck with something simpler.

LeSaucy 5y ago

Is there anything comparable to this with boost/asio?

zbentley 5y ago

That's just lovely. I'm a huge fan of programs exposing deep runtime state via normal (not debug mode only) instrumentation. Java has historically been amazing in this area, it's great to see other languages stepping up as well. Does anyone know of a similar visualizer for coroutines/threads for async Python?

henryaj 5y ago

Anyone know what the font is in those screenshots?

mycoliza 5y ago
It's https://typeof.net/Iosevka/ https://typeof.net/Iosevka/ (I took the screenshots).

meganium1 5y ago