3 ms·

What kind of cost? Obviously, backups on the author's server are not something I'd ever do, but the "offline only mode" is there, apparently. TBH I was thinkin

What kind of cost? Obviously, backups on the author's server are not something I'd ever do, but the "offline only mode" is there, apparently. TBH I was thinking about doing this for a while now. History of my shell, which now is 3.8MB in size, is one of my competitive advantages as a developer (and a very nice thing to have as a power user). It accumulated steadily since ~2005, and with fzf searching the history is much faster than having to google, as long as I did what I want to do now even once in the distant past. I even wrote a utility to merge history files across multiple hosts[1], so I don't have to think "where did I last use that command", as I have everything on every host. The problem with this, however, is shell startup time. It started being noticeable a few years ago and is slowly getting irritating. The idea of converting the histfile into sqlite db crossed my mind more than once due to this. [1] https://github.com/piotrklibert/zsh-merge-hist https://github.com/piotrklibert/zsh-merge-hist

conradludgate 3y ago
One such cost could be database size. I currently have 45k history entries in my database and it sits at roughly 15MB in size due to database indices along with the additional data we store
- klibertp 3y ago
  But it's a cost shared by all open shells, right? Well, even if it was 15MB per shell session, it should still be worth it if the startup and searching is faster. For comparison: I use extended ZSH history format, which records a timestamp and duration of the call (and nothing else), and I have ~65k entries there, with history file size, as mentioned, 3.8MB. It could be an order of magnitude larger and I still wouldn't care, as long as it loads faster than it takes ZSH to parse its history file.
  - conradludgate 3y ago
    Yes, this is 15MB on disk size for your whole machine. We agree that the trade off is worth it for the utility and speed improvements
arthurcolle 3y ago
This is interesting, thanks I'm checking it out. I was thinking this could be a flash in the pan thing that would then be annoying to maintain, but obviously global history of everything you've done is definitely a super boon to productivity. How do you handle maintaining this as you transition through jobs, machines, etc? 3.8MB is obviouslly trivially small in size so you could store it on a potato, but like what is your workflow around maintaining these one off ad hoc "developer boost" type tools?
- klibertp 3y ago
  > How do you handle maintaining this as you transition through jobs, machines, etc? Currently, the tool reads ~/.mergerc, which is a JSON file with a list of SSH hosts to SCP history to and from. As long as the history file is in the same place (it tends to be on hosts that I setup, and otherwise I check in default locations) and the host has an entry in ~/.ssh/config, the tool will work. It's really just a wrapper for a few SCP invocations plus a history file (extended) format parser. Changing servers is just a change in the config file, but it's also helpful for changing jobs, because I can quickly add a bit of filtering before the merging happens. I had to erase some API keys and such a few times, adding `filter` call here: https://github.com/piotrklibert/zsh-merge-hist/blob/master/src/scala/Main.scala#L34 https://github.com/piotrklibert/zsh-merge-hist/blob/master/s... took care of it. > what is your workflow around maintaining these one off ad hoc "developer boost" type tools? Good question. I don't have such workflow, at all. When I commit to write something like this, I try to make sure that it has a scope limited enough so that it can be "completed" or "done". In this case, the tool builds on SSH/SCP and a file format that hasn't changed in the last 20 years (at least). So, once I had it working, there was nothing much to do with it after that. The only change I had to do recently was changing `+` to `*` in the parser, because somehow (not sure how, actually) an empty command made it into the file. But that's all I had to do in 5 years time. I'm not as extreme, but suckless.org philosophy appears to work well here. Here's another example: https://github.com/piotrklibert/nimlock https://github.com/piotrklibert/nimlock - it's a port, done because I wanted to do something in Nim, but it worked for me for years and I suspect it still works now (after going full remote I stopped needing it). There's nothing much that could break (well, Wayland would break it, but I don't use it), and so there's not much you need to do in terms of maintenance. As for language choices - these are basically random. I made the zsh-merge-hist in Scala simply because I was interested in Scala back then. I have little tools written in Nim, OCaml, Racket, Elisp, Raku - and even AWK (pretty nice language actually) and shell. That's another reason why making the tools any more complex than what's absolutely necessary would be a problem: the churn in the ecosystems tends to be too high for me to keep track of, especially since I'd need to track 10 of them. EDIT: I forgot, but obviously the most important "trick" is not giving a shit if these things work for anyone else but me :D > I'm checking it out If you have Java installed, `./gradlew installDist` should give you `./build/install/bin/zsh-merge-hist` executable to run. The ~/.mergerc (on the host the tool runs) should look like this: { "hosts": ["host1"], "tempDir": "/tmp/zsh-merge-hist", "sourcePath": "mgmnt/zsh_history" } where `sourcePath` is a path to history file relative to the home directory.

conradludgate 3y ago

One such cost could be database size. I currently have 45k history entries in my database and it sits at roughly 15MB in size due to database indices along with the additional data we store

klibertp 3y ago
But it's a cost shared by all open shells, right? Well, even if it was 15MB per shell session, it should still be worth it if the startup and searching is faster. For comparison: I use extended ZSH history format, which records a timestamp and duration of the call (and nothing else), and I have ~65k entries there, with history file size, as mentioned, 3.8MB. It could be an order of magnitude larger and I still wouldn't care, as long as it loads faster than it takes ZSH to parse its history file.
- conradludgate 3y ago
  Yes, this is 15MB on disk size for your whole machine. We agree that the trade off is worth it for the utility and speed improvements

arthurcolle 3y ago

This is interesting, thanks I'm checking it out. I was thinking this could be a flash in the pan thing that would then be annoying to maintain, but obviously global history of everything you've done is definitely a super boon to productivity. How do you handle maintaining this as you transition through jobs, machines, etc? 3.8MB is obviouslly trivially small in size so you could store it on a potato, but like what is your workflow around maintaining these one off ad hoc "developer boost" type tools?

klibertp 3y ago
> How do you handle maintaining this as you transition through jobs, machines, etc? Currently, the tool reads ~/.mergerc, which is a JSON file with a list of SSH hosts to SCP history to and from. As long as the history file is in the same place (it tends to be on hosts that I setup, and otherwise I check in default locations) and the host has an entry in ~/.ssh/config, the tool will work. It's really just a wrapper for a few SCP invocations plus a history file (extended) format parser. Changing servers is just a change in the config file, but it's also helpful for changing jobs, because I can quickly add a bit of filtering before the merging happens. I had to erase some API keys and such a few times, adding `filter` call here: https://github.com/piotrklibert/zsh-merge-hist/blob/master/src/scala/Main.scala#L34 https://github.com/piotrklibert/zsh-merge-hist/blob/master/s... took care of it. > what is your workflow around maintaining these one off ad hoc "developer boost" type tools? Good question. I don't have such workflow, at all. When I commit to write something like this, I try to make sure that it has a scope limited enough so that it can be "completed" or "done". In this case, the tool builds on SSH/SCP and a file format that hasn't changed in the last 20 years (at least). So, once I had it working, there was nothing much to do with it after that. The only change I had to do recently was changing `+` to `*` in the parser, because somehow (not sure how, actually) an empty command made it into the file. But that's all I had to do in 5 years time. I'm not as extreme, but suckless.org philosophy appears to work well here. Here's another example: https://github.com/piotrklibert/nimlock https://github.com/piotrklibert/nimlock - it's a port, done because I wanted to do something in Nim, but it worked for me for years and I suspect it still works now (after going full remote I stopped needing it). There's nothing much that could break (well, Wayland would break it, but I don't use it), and so there's not much you need to do in terms of maintenance. As for language choices - these are basically random. I made the zsh-merge-hist in Scala simply because I was interested in Scala back then. I have little tools written in Nim, OCaml, Racket, Elisp, Raku - and even AWK (pretty nice language actually) and shell. That's another reason why making the tools any more complex than what's absolutely necessary would be a problem: the churn in the ecosystems tends to be too high for me to keep track of, especially since I'd need to track 10 of them. EDIT: I forgot, but obviously the most important "trick" is not giving a shit if these things work for anyone else but me :D > I'm checking it out If you have Java installed, `./gradlew installDist` should give you `./build/install/bin/zsh-merge-hist` executable to run. The ~/.mergerc (on the host the tool runs) should look like this: { "hosts": ["host1"], "tempDir": "/tmp/zsh-merge-hist", "sourcePath": "mgmnt/zsh_history" } where `sourcePath` is a path to history file relative to the home directory.