3 ms·

> Huh. Okay. Maybe you could answer the actual question: why? That seems like a rather extreme position to take. it makes me feel physically sick and my hands

> Huh. Okay. Maybe you could answer the actual question: why? That seems like a rather extreme position to take. it makes me feel physically sick and my hands start shaking when a computer does not answer immediately > 2k lines does come back within 2 repaint frames on my 10 year old mac, which is about as fast as most video game responses to controller input. which repaint frames ? e.g. here on my 240hz monitor that would be 8.3 ms which i guess is tolerable for visual input. on a 60hz, that's 33ms which is noticeable when everything else around it is fast > PS and I'm still wondering what your proposed alternative is... ? it's in the original post - having an utility which does such common piping use cases without having to go through three processes - here doing a time sort | uniq | sort -nr of a 100mb file gives me 2/3 seconds for the first sort, 1s for the uniq, and again a couple seconds for the second sort ; the uniq can stream but the sorts can't, and a single command would likely be able to optimize this with a bit of algorithmic thinking

dahart 4y ago
> my hands start shaking when a computer does not answer immediately I’m sorry to hear that. It sounds like it could be an anxiety issue somewhat unrelated to what we are talking about? Does it help to stick to very small files so you never have to wait? Does it help when you get progress updates in the interface? I still don’t understand what you’re suggesting because the idea that you want to give arbitrarily sized inputs to a sort and have it come back instantly is unrealistic. You can only improve the overheads and bandwidth, you can never avoid waiting if you give larger inputs. No matter what solution you suggest, I can give you a file you’ll have to wait for. Speeding up the unix pipeline doesn’t appear to solve the problem you’re describing, that’s why I’m asking for more detail. > it’s in the original post What is? I’m not sure what you’re referring to. You only mean the imaginary idea to combine 3 processes into one? @yegle only appeared to ask for a shortcut for a common workflow, and not a more performant alternative. I’m asking you what your preferred practical implementation is. What can you do today to speed up this workflow? And how does that solve the problem of having to wait? > the uniq can stream but the sorts can’t Right, this is true regardless of what tool you use. You can eliminate the overheads of starting up 3 processes, but you can’t avoid the second sort, if you ask for a second sort. Might be worth observing that the overheads of 3 processes get relatively smaller the larger the inputs are. If you’re sorting a gigabyte or terabyte of text, using a single process will be barely better, if at all. Might also be worth looking at cold cache vs warm cache startup times for 3 processes. Does your 2/3 second change the second time? PS GNU sort has options that can improve its bandwidth. Maybe try these? https://www.reddit.com/r/commandline/comments/a7hq5n/psa_improving_gnu_sort_speed/ https://www.reddit.com/r/commandline/comments/a7hq5n/psa_imp... This is another reason to avoid writing your own tool, since sort may in reality have a lot of functionality that is not easy to duplicate, and may have ways to gain performance that is hard to beat.

dahart 4y ago

> my hands start shaking when a computer does not answer immediately I’m sorry to hear that. It sounds like it could be an anxiety issue somewhat unrelated to what we are talking about? Does it help to stick to very small files so you never have to wait? Does it help when you get progress updates in the interface? I still don’t understand what you’re suggesting because the idea that you want to give arbitrarily sized inputs to a sort and have it come back instantly is unrealistic. You can only improve the overheads and bandwidth, you can never avoid waiting if you give larger inputs. No matter what solution you suggest, I can give you a file you’ll have to wait for. Speeding up the unix pipeline doesn’t appear to solve the problem you’re describing, that’s why I’m asking for more detail. > it’s in the original post What is? I’m not sure what you’re referring to. You only mean the imaginary idea to combine 3 processes into one? @yegle only appeared to ask for a shortcut for a common workflow, and not a more performant alternative. I’m asking you what your preferred practical implementation is. What can you do today to speed up this workflow? And how does that solve the problem of having to wait? > the uniq can stream but the sorts can’t Right, this is true regardless of what tool you use. You can eliminate the overheads of starting up 3 processes, but you can’t avoid the second sort, if you ask for a second sort. Might be worth observing that the overheads of 3 processes get relatively smaller the larger the inputs are. If you’re sorting a gigabyte or terabyte of text, using a single process will be barely better, if at all. Might also be worth looking at cold cache vs warm cache startup times for 3 processes. Does your 2/3 second change the second time? PS GNU sort has options that can improve its bandwidth. Maybe try these? https://www.reddit.com/r/commandline/comments/a7hq5n/psa_improving_gnu_sort_speed/ https://www.reddit.com/r/commandline/comments/a7hq5n/psa_imp... This is another reason to avoid writing your own tool, since sort may in reality have a lot of functionality that is not easy to duplicate, and may have ways to gain performance that is hard to beat.