8 ms·

Yazi: Fast terminal file manager based on async I/O

157 points by sbt567 3y ago

ElectronBadger 3y ago
After using it for brief while, I'd say that I like it. It will not replace may daily driver (Midnight Commander), but it's good for fast preview/rename/delete actions.
jauntywundrkind 3y ago
Io_uring capable? That'd be cool. "async Io" doesn't mean a ton to me, other than hopefully not blocking. It'd be cool to have a great well integrated terminal file manager that uses high performance io_uring. That said, I have a hard time figuring out when & where I'd want to use terminal file managers. I like the idea, but the flexibility of the shell is so unparalleled.
- pxeger1 3y ago
  How would the performance of io_uring would merit its complexity for a terminal file manager?
  - ithkuil 3y ago
    Wouldn't io_uring complexity be hidden behind the async io abstraction? Would the file manager even have to bother what underlying mechanism provides the async IO interface with the kernel? (epoll, kqueue, io_uring, etc) Are there know ways that io_uring would leak through such an abstraction?
    - pxeger1 3y ago
      I was assuming the file manager would have to include the code for the async abstraction. But of course it could be just part of the async runtime / standard library. Still, a file manager doesn't seem like it would really need the ability to be so massively concurrent.
      - remram 3y ago
        I'm also lost. Does it just mean that the UI stays responsive while a file copy happens in the background? It says "CPU tasks are spread across multiple threads", but what CPU tasks does a file manager perform?
        samus 3y ago
        Almost every UI framework out there is single-threaded and has facilities to help offloading such tasks into the background. Usually, to a worker thread pool. It's a bit fiddly, especially in a non-managed language that doesn't protect from dumb mistakes, but it can be easily done. io_uring is for use cases where the context switch overhead of system calls starts to matter. While async I/O can be justified for U/I programming, it's safe to say that a file manager won't ever benefit from io_uring.
        sxyazi 3y ago
        There is a Reddit post discussing this point: https://www.reddit.com/r/rust/comments/16fxr58/comment/k04u7z8/?context=3 https://www.reddit.com/r/rust/comments/16fxr58/comment/k04u7... > There are many scenarios that require CPU, such as image downscaling, image encoding, code highlighting, sorting large directories, etc. If you want to delve further into Yazi's inner workings, you can see https://github.com/sxyazi/yazi/issues/143 https://github.com/sxyazi/yazi/issues/143
        remram 3y ago
        What advantage does thread pool + io_uring have over just thread pool? Unless you have millions of CPU cores and therefore many downscaling/encoding/highlighting tasks starting/finishing every second, I still don't see what io_uring brings to the table. Running tasks in the background does NOT require io_uring.
        sxyazi 3y ago
        Hey, Yazi currently is NOT using io_uring, and the link provided above explains it thoroughly, please take a careful look.
- alpaca128 3y ago
  The reason I use a term file manager is because it makes some cumbersome things much smoother. With nnn I can navigate directories at least as quickly as with a mouse in a GUI, can create new directories/files by pressing just two keys, and batch rename selected files in a Vim textbuffer. Also it's useful as file browser in Vim. I don't use it for everything and I still copy/move files a lot without it. But it accelerates some things and is simple to use.
- sxyazi 3y ago
  Hi, based on the current architecture of Yazi, I don't think it would be difficult to add io_uring to it. This is something we can consider for our future work, as it's not the current performance bottleneck. For more details: https://github.com/sxyazi/yazi/issues/143 https://github.com/sxyazi/yazi/issues/143
  - samus 3y ago
    Nice write-up! It shows that you didn't just use async for the sake of it, but that you thought about the kind of user experience that you want to have, and how async helps you to get there.
say_it_as_it_is 3y ago
Is asyncio over engineering for this line of work? How much concurrency is really needed here and couldn't it have been addressed by threads instead?
- kookamamie 3y ago
  But you see, there's a problem right there - threads aren't cool anymore. Not that the user knew or cared, but it's important perceived concurrency is shoved through the lens of async, somehow.
- zelphirkalt 3y ago
  If you look at the video showing how it works, you will see, that it apparently starts displaying file content, while you navigate over files and directories. I think this is what happens asynchronously, in order to never block the ability to navigate. I am not sure I am a fan of this, because I don't actually need everything to start being rendered. Maybe rendering something when I press a specific button would be good enough and that wouldn't necessarily require async stuff.
- jauntywundrkind 3y ago
  Personally I don't get the blowback that's shown up a couple places. Why do we so readily convince ourselves against doing the right thing, picking good technology? "Over engineering" is a huge massive bat of a word that is applied to "correct" all kinds of spirited ambition. Fear Uncertainty & Doubt is a dangerous default mode to adapt. I do not feel that it should be such an ordeal, require such courage, to do the good & right things. Better to try & fail than risk whiling ones time on lukewarm efforts. What is the actual danger here? What are you afraid of? Why hold your biases against another's endeavor?
  - lukeschlather 3y ago
    Async creates race conditions and even if you show me a proof that says there will never be a deadlock or other unexpected behavior I generally will believe that either there is an error in your proof or the actual executing code is different from the algorithm you've proved correct in some significant way. Doing previews in a non-blocking way sounds really cool though, I'm not ragging on the concept, just saying I am mistrustful of async.
kosolam 3y ago
Awesome project. Will try it. I can see how it can be useful and fun to use it in remote ssh sessions. When DOS was a thing I just used Norton Commander. But with linux the state of existing file managers made me get used to working in plain command line. Yazi seems like a stride in a good direction.
haukurb 3y ago
How does it compare to ranger, besides not being written in Python?
- sxyazi 3y ago
  I think this has been listed at the head of the README, and includes a example video, take a look at it :)
pinetroey 3y ago
How does it compare to nnn?
xuhu 3y ago
In 1991 dos navigator had an integrated file manager and editor like no other console file manager has: you could alt-tab and copy/paste from editors into shell commands interactively. In 2023 I still need two extra terminals for vim and for bash. And text selection is mouse-only.
- zelphirkalt 3y ago
  Doesn't Emacs fit that bill? In Emacs you can run terminal emulators and various shells and from all of them you can copy and paste into other buffers or from other buffers into them.
  - lisplist 3y ago
    To add to this: I had lots of problems with Emacs terminal emulators until I switched to VTerm. Most things work like you’d expect in VTerm.
    - zelphirkalt 3y ago
      Yep, vterm is great. Only thing annoying me is, that I have to switch modes to copy anything. Wish it would work like `M-x shell` instead, where I can simply navigate upwards and when I move point back to the prompt, I can paste and continue with commands.
- forgotpwd16 3y ago
  Vim has split screen support and built-in virtual terminal supporting normal mode. For file manager can use fern written in pure vimscript. In 2023 you were just not aware of them.
- thesuperbigfrog 3y ago
  >> In 2023 I still need two extra terminals for vim and for bash. And text selection is mouse-only. Terminal multiplexers like tmux, GNU screen, and others allow for keyboard-based text selection and copy / paste. Many advanced editors like VS Code, Emacs, and other have built-in terminals and keyboard commands to select, copy, and paste text.
ayoisaiah 3y ago
I've tried using LF in the past, but it didn't stick. Will definitely give this a go, as I'm trying to move to an pure terminal workflow as closely as possible. https://github.com/gokcehan/lf https://github.com/gokcehan/lf
sxyazi 3y ago
Hi, I'm the author of Yazi, and I was pleasantly surprised to come across it while browsing Hacker News. I'm really excited!! If you're interested in the topics of concurrency, performance, and io_uring discussed in this thread, you can take a look at my new article where I provide a detailed explanation of "Why is Yazi Fast?" https://github.com/sxyazi/yazi/issues/143 https://github.com/sxyazi/yazi/issues/143
- ta988 3y ago
  That's impressive. I wish the tabs could be shown side by side.
- cptnapalm 3y ago
  This is... wow. I really like this.
- eviks 3y ago
  Do you plan to cache folder sizes?
- BaculumMeumEst 3y ago
  I used Yazi for about a day and loved it, but there's just enough ergonomics missing that I ended up switching back to nnn for now. My main issue I couldn't find a way to filter the list of files/directories for the current directory I'm in. Let's say I want to go to directory `cpp`, and it's 22 entries down the list of current files/folders. If I try to `fd` for `cpp`, I'm going to get back a ton of results from all over my hard drive. So it seems like I need to just scroll way down, which is a bummer. In nnn, you can just type `/` then `cpp` and it will filter results in CWD that match. A nice-to-have would be live fuzzy-find functionality for the fd/rg integration, but that is not so critical. The other thing I like about nnn is it retains `g` and `G` from vim to go to the top and bottom of the list of results respectively. I didn't see a similar operation in yazi. Otherwise, very awesome tool, the performance is excellent, I appreciate how you explicitly state the external shell programs you interface with, I love the built in previews, and the devicons look clean.
  - 3y ago
BaculumMeumEst 3y ago
It looks pretty cool! A nice feature of nnn is that you can set it up to `cd on quit`, meaning when you navigate to a different folder and then quit the program, your original shell navigates to that folder. Is that possible in Yazi?
- sxyazi 3y ago
  The README includes are you looking for: https://github.com/sxyazi/yazi#usage https://github.com/sxyazi/yazi#usage
  - BaculumMeumEst 3y ago
    Gah. I need to skim better. Thanks. Here is a translation for anyone else who uses fish. Just hit it with a :%s/^\n//g to fix the formatting. # Rename this file to match the name of the function # e.g. ~/.config/fish/functions/ya.fish # or, add the lines to the 'config.fish' file. function ya set tmp (mktemp -t "yazi-cwd.XXXXX") yazi --cwd-file="$tmp" if set cwd (cat -- "$tmp") && [ -n "$cwd" ] && [ "$cwd" != "$PWD" ] cd -- "$cwd" end rm -f -- "$tmp" end

157 points by sbt567 3y ago

After using it for brief while, I'd say that I like it. It will not replace may daily driver (Midnight Commander), but it's good for fast preview/rename/delete actions.

jauntywundrkind 3y ago

Io_uring capable? That'd be cool. "async Io" doesn't mean a ton to me, other than hopefully not blocking. It'd be cool to have a great well integrated terminal file manager that uses high performance io_uring. That said, I have a hard time figuring out when & where I'd want to use terminal file managers. I like the idea, but the flexibility of the shell is so unparalleled.

pxeger1 3y ago
How would the performance of io_uring would merit its complexity for a terminal file manager?
- ithkuil 3y ago
  Wouldn't io_uring complexity be hidden behind the async io abstraction? Would the file manager even have to bother what underlying mechanism provides the async IO interface with the kernel? (epoll, kqueue, io_uring, etc) Are there know ways that io_uring would leak through such an abstraction?
  - pxeger1 3y ago
    I was assuming the file manager would have to include the code for the async abstraction. But of course it could be just part of the async runtime / standard library. Still, a file manager doesn't seem like it would really need the ability to be so massively concurrent.
    - remram 3y ago
      I'm also lost. Does it just mean that the UI stays responsive while a file copy happens in the background? It says "CPU tasks are spread across multiple threads", but what CPU tasks does a file manager perform?
      - samus 3y ago
        Almost every UI framework out there is single-threaded and has facilities to help offloading such tasks into the background. Usually, to a worker thread pool. It's a bit fiddly, especially in a non-managed language that doesn't protect from dumb mistakes, but it can be easily done. io_uring is for use cases where the context switch overhead of system calls starts to matter. While async I/O can be justified for U/I programming, it's safe to say that a file manager won't ever benefit from io_uring.
      - sxyazi 3y ago
        There is a Reddit post discussing this point: https://www.reddit.com/r/rust/comments/16fxr58/comment/k04u7z8/?context=3 https://www.reddit.com/r/rust/comments/16fxr58/comment/k04u7... > There are many scenarios that require CPU, such as image downscaling, image encoding, code highlighting, sorting large directories, etc. If you want to delve further into Yazi's inner workings, you can see https://github.com/sxyazi/yazi/issues/143 https://github.com/sxyazi/yazi/issues/143
        remram 3y ago
        What advantage does thread pool + io_uring have over just thread pool? Unless you have millions of CPU cores and therefore many downscaling/encoding/highlighting tasks starting/finishing every second, I still don't see what io_uring brings to the table. Running tasks in the background does NOT require io_uring.
        sxyazi 3y ago
        Hey, Yazi currently is NOT using io_uring, and the link provided above explains it thoroughly, please take a careful look.
alpaca128 3y ago
The reason I use a term file manager is because it makes some cumbersome things much smoother. With nnn I can navigate directories at least as quickly as with a mouse in a GUI, can create new directories/files by pressing just two keys, and batch rename selected files in a Vim textbuffer. Also it's useful as file browser in Vim. I don't use it for everything and I still copy/move files a lot without it. But it accelerates some things and is simple to use.
sxyazi 3y ago
Hi, based on the current architecture of Yazi, I don't think it would be difficult to add io_uring to it. This is something we can consider for our future work, as it's not the current performance bottleneck. For more details: https://github.com/sxyazi/yazi/issues/143 https://github.com/sxyazi/yazi/issues/143
- samus 3y ago
  Nice write-up! It shows that you didn't just use async for the sake of it, but that you thought about the kind of user experience that you want to have, and how async helps you to get there.

say_it_as_it_is 3y ago

Is asyncio over engineering for this line of work? How much concurrency is really needed here and couldn't it have been addressed by threads instead?

kookamamie 3y ago
But you see, there's a problem right there - threads aren't cool anymore. Not that the user knew or cared, but it's important perceived concurrency is shoved through the lens of async, somehow.
zelphirkalt 3y ago
If you look at the video showing how it works, you will see, that it apparently starts displaying file content, while you navigate over files and directories. I think this is what happens asynchronously, in order to never block the ability to navigate. I am not sure I am a fan of this, because I don't actually need everything to start being rendered. Maybe rendering something when I press a specific button would be good enough and that wouldn't necessarily require async stuff.
jauntywundrkind 3y ago
Personally I don't get the blowback that's shown up a couple places. Why do we so readily convince ourselves against doing the right thing, picking good technology? "Over engineering" is a huge massive bat of a word that is applied to "correct" all kinds of spirited ambition. Fear Uncertainty & Doubt is a dangerous default mode to adapt. I do not feel that it should be such an ordeal, require such courage, to do the good & right things. Better to try & fail than risk whiling ones time on lukewarm efforts. What is the actual danger here? What are you afraid of? Why hold your biases against another's endeavor?
- lukeschlather 3y ago
  Async creates race conditions and even if you show me a proof that says there will never be a deadlock or other unexpected behavior I generally will believe that either there is an error in your proof or the actual executing code is different from the algorithm you've proved correct in some significant way. Doing previews in a non-blocking way sounds really cool though, I'm not ragging on the concept, just saying I am mistrustful of async.

kosolam 3y ago

Awesome project. Will try it. I can see how it can be useful and fun to use it in remote ssh sessions. When DOS was a thing I just used Norton Commander. But with linux the state of existing file managers made me get used to working in plain command line. Yazi seems like a stride in a good direction.

haukurb 3y ago

How does it compare to ranger, besides not being written in Python?

sxyazi 3y ago
I think this has been listed at the head of the README, and includes a example video, take a look at it :)

pinetroey 3y ago

How does it compare to nnn?

xuhu 3y ago

In 1991 dos navigator had an integrated file manager and editor like no other console file manager has: you could alt-tab and copy/paste from editors into shell commands interactively. In 2023 I still need two extra terminals for vim and for bash. And text selection is mouse-only.

zelphirkalt 3y ago
Doesn't Emacs fit that bill? In Emacs you can run terminal emulators and various shells and from all of them you can copy and paste into other buffers or from other buffers into them.
- lisplist 3y ago
  To add to this: I had lots of problems with Emacs terminal emulators until I switched to VTerm. Most things work like you’d expect in VTerm.
  - zelphirkalt 3y ago
    Yep, vterm is great. Only thing annoying me is, that I have to switch modes to copy anything. Wish it would work like `M-x shell` instead, where I can simply navigate upwards and when I move point back to the prompt, I can paste and continue with commands.
forgotpwd16 3y ago
Vim has split screen support and built-in virtual terminal supporting normal mode. For file manager can use fern written in pure vimscript. In 2023 you were just not aware of them.
thesuperbigfrog 3y ago
>> In 2023 I still need two extra terminals for vim and for bash. And text selection is mouse-only. Terminal multiplexers like tmux, GNU screen, and others allow for keyboard-based text selection and copy / paste. Many advanced editors like VS Code, Emacs, and other have built-in terminals and keyboard commands to select, copy, and paste text.

ayoisaiah 3y ago

I've tried using LF in the past, but it didn't stick. Will definitely give this a go, as I'm trying to move to an pure terminal workflow as closely as possible. https://github.com/gokcehan/lf https://github.com/gokcehan/lf

sxyazi 3y ago

Hi, I'm the author of Yazi, and I was pleasantly surprised to come across it while browsing Hacker News. I'm really excited!! If you're interested in the topics of concurrency, performance, and io_uring discussed in this thread, you can take a look at my new article where I provide a detailed explanation of "Why is Yazi Fast?" https://github.com/sxyazi/yazi/issues/143 https://github.com/sxyazi/yazi/issues/143

ta988 3y ago
That's impressive. I wish the tabs could be shown side by side.
cptnapalm 3y ago
This is... wow. I really like this.
eviks 3y ago
Do you plan to cache folder sizes?
BaculumMeumEst 3y ago
I used Yazi for about a day and loved it, but there's just enough ergonomics missing that I ended up switching back to nnn for now. My main issue I couldn't find a way to filter the list of files/directories for the current directory I'm in. Let's say I want to go to directory `cpp`, and it's 22 entries down the list of current files/folders. If I try to `fd` for `cpp`, I'm going to get back a ton of results from all over my hard drive. So it seems like I need to just scroll way down, which is a bummer. In nnn, you can just type `/` then `cpp` and it will filter results in CWD that match. A nice-to-have would be live fuzzy-find functionality for the fd/rg integration, but that is not so critical. The other thing I like about nnn is it retains `g` and `G` from vim to go to the top and bottom of the list of results respectively. I didn't see a similar operation in yazi. Otherwise, very awesome tool, the performance is excellent, I appreciate how you explicitly state the external shell programs you interface with, I love the built in previews, and the devicons look clean.
- 3y ago

BaculumMeumEst 3y ago

It looks pretty cool! A nice feature of nnn is that you can set it up to `cd on quit`, meaning when you navigate to a different folder and then quit the program, your original shell navigates to that folder. Is that possible in Yazi?

sxyazi 3y ago
The README includes are you looking for: https://github.com/sxyazi/yazi#usage https://github.com/sxyazi/yazi#usage
- BaculumMeumEst 3y ago
  Gah. I need to skim better. Thanks. Here is a translation for anyone else who uses fish. Just hit it with a :%s/^\n//g to fix the formatting. # Rename this file to match the name of the function # e.g. ~/.config/fish/functions/ya.fish # or, add the lines to the 'config.fish' file. function ya set tmp (mktemp -t "yazi-cwd.XXXXX") yazi --cwd-file="$tmp" if set cwd (cat -- "$tmp") && [ -n "$cwd" ] && [ "$cwd" != "$PWD" ] cd -- "$cwd" end rm -f -- "$tmp" end