8 ms·

Keeping a long shell history

176 points by ingve 3y ago

rzzzt 3y ago
Can fzf or the shell history file answer this question? zsh records timestamps and runtime if extended history is enabled. bash needs HISTTIMEFORMAT to be configured properly. I haven't used fzf so not sure if it can search by timestamp, I only see a line number for the matching command (presumably) in the gif.
- defrost 3y ago
  FWiW the authors setup: Here’s the ZSH configuration I used for many, many years: possibly cannot fully answer the question as there are settings to scrub duplicate commands ... meaning the target day will show commands unique to that day but not necessarily all commands from that day. It's a small quibble but potentially an issue for those that really need to Indiana Jones their command history for some specific clue.
  - rzzzt 3y ago
    I agree, especially if the focus is not on the command itself but the date. Atuin has a CLI switch for filtering command history by date, but I don't see any support in the ctrl-r UI eg. with an operator like "before:" or similar.
    - defrost 3y ago
      In my own case large clumps of similar commands hinge upon what was the CD <current directory> at the time ... and the command most likely in many cases to be scrubbed as duplicate is a cd <one of 50 common working base directories>. This leads to "I can see I tidied up a bunch of project files first monday of last month ... but which project was that??"
- misnome 3y ago
  A nice thing that zsh (and bash-preexec) has is that you can add arbitrary functions to be called on command execution. I work at a large facility with a shared filesystem and many computers, so often I need to answer questions like “What did I run the last time I was sitting at computer X”. I log: command, timestamp, PID (to separate terminal streams), hostname and cwd. I’ve set up ctrl-r to search this “.fullhistory” file instead.
  - salohcin 3y ago
    This sounds interesting! Can you elaborate further?
    - misnome 3y ago
      The function: if [[ -n ${ZSH_VERSION-} ]]; then # ZSH doesn't split command over multiple variables preexec_custom_history() { echo "$HOSTNAME:\"$PWD\" $$ $(date "+%Y-%m-%dT%H:%M:%S%z") $1" >> "$CUSTOM_HISTORY_FILE" } else preexec_custom_history() { echo "$HOSTNAME:\"$PWD\" $$ $(date "+%Y-%m-%dT%H:%M:%S%z") $*" >> "$CUSTOM_HISTORY_FILE" } fi # Add it to the array of functions to be invoked each time. preexec_functions+=(preexec_custom_history) It's harder to search for multiline but I haven't gotten to fixing that yet. I've thought about putting it into sqlite (pwd makes the file large) but since everything is on network filesystems I don't think that's a place sqlite has any guarantees; screwing up a single line of a bare log doesn't break anything. Edit: The fzf is kind of hacked together but is here: https://pastebin.com/sTxsvZAf https://pastebin.com/sTxsvZAf . It has a few bugs but nothing annoying enough to have made me fix it.
- 59nadir 3y ago
  Atuin (the tool mentioned later in the post) stores timestamps and CWDs for commands in a database, which is probably why the title is what it is. I've been using it for 6+ months and it's actually good. I also like having an sqlite database with my commands so I can search it even more freeform if I want.
mmsc 3y ago
I also like to use sudo chattr +a .sh_history to ensure that my shell history file isn’t deleted. Being append-only, it also ensures that some type of timing issue in my shell which means my HISTFILESIZE or the equivalent isn’t set to something larger than default, meaning my history isn’t deleted.
- codetrotter 3y ago
  If the shell is unaware that the file is append only, does it result in appending not just the commands run from this session but undesirably also ends up appending a full copy of a lot of the previous history?
  - blueflow 3y ago
    I guess it means that the open() syscall will require the O_APPEND flag. If the file is opened without that flag it will error out.
- mgdlbp 3y ago
  a weaker method is to set a non-default HISTFILE after HISTSIZE etc
permalac 3y ago
Good tool, but my history is so noisy I had to find another way. Lately I started to add a new habit in my shell actions. When a command has some cognitive requirements I create a script with some ${1:-default} values and I store them all in $PATH enabled local/bin All my scripts start with __ and then some capital letters indicating the topic. I can copy them to new machines, which is something I would not do with my history.
- bananskalhalk 3y ago
  Prefixing your scripts is an awesome tips. A colleague told me he prefixed all his commands with ',' and I've started to do the same. '_' might clash with 'hidden'(?) shell expansions etc. Sadly git doesn't allow for aliases with an comma-prefix, so all git aliases starts with a dot.
  - gumby 3y ago
    You can make symlinks or shell aliases that just reference git-add et al directly.
- thenbe 3y ago
  > my history is so noisy I had to find another way The fzf search syntax [1] can help, if you become familiar with it. It is also supported in atuin [2]. [1]: https://github.com/junegunn/fzf#search-syntax https://github.com/junegunn/fzf#search-syntax [2]: https://docs.atuin.sh/configuration/config/#fuzzy-search-syntax https://docs.atuin.sh/configuration/config/#fuzzy-search-syn...
- __MatrixMan__ 3y ago
  Nushell has a SQLite history mode which captures more metadata than is typical. I haven't looked into extending it, but it seems like a good setup for adding enough so that some of these cognitive requirements could be recovered from history rather than thought of up front. Like if I tend to go down the wrong path, fix it, and then go down the right path, maybe that's a pattern that can be identified and then when I search history it finds two or three commands which together set me on the right path in the first place. I mean, thinking ahead is always gonna be better, but I don't always.
natrys 3y ago
With a bit of prompt hacking I send information (shell session, pwd, command, timestamp) to a fifo, which is read by small Tcl script to ingest in a sqlite database. Aturin seems nice, probably does something similar. It's a hacky script but the idea is sound. Good thing about SQL is that I can fine tune the query before feeding to fzf. A combination of order by recency and frequency of commands with and without context (pwd) works very well for me. With proper indexing I am not close to feeling slowdown, but if that happens I can probably just export data to duckdb and use that.
tomatocracy 3y ago
The article briefly mentions atuin at the end. I've tried atuin but found it a little bit too heavyweight for me - instead I use zsh-histdb[0] (together with the fzf extension for it[1]) which allows you to easily answer this type of question - can highly recommend it. 0. https://github.com/larkery/zsh-histdb https://github.com/larkery/zsh-histdb 1. https://github.com/m42e/zsh-histdb-fzf https://github.com/m42e/zsh-histdb-fzf
- larusso 3y ago
  I tried atuin as well and switched back to my fzf setup. I setup my history to ignore duplicates. Makes it also easier to use the up arrow key. But that can have downsides depending on the user. For me it works great for years. The only reason I don‘t share the history along with other dotfiles is the problem that some commands read API keys and token from the options. And I don‘t yet have a system for that so that. Because other means like env variables can make the it harder to repeat said command month later.
Brian_K_White 3y ago
The other day there was another post about a bash history in a db, years worth and searchable and available on all machines. I do not get this. I have over 30 years certainly cobbled some complicated commands, but I never need an ancient one. If it's complicated and used more than one day, it's a script. Complicated or not and used a long time ago, it's no longer valid. These days "so old it's no longer valid" is barely a year. Back in the sco Unix days you could have 10 year old commands that are still valid references, but Linux never had that except for trivial things. There are exceptions but they are exceptions and so by definition not reasonable to focus on. I think that building essentially mini apps right on the command line and then carrying around years of shell history to reuse them is solving a problem the wrong way. A stupid and backwards way. It's losing sight of what the problem actually is and like growing one weird finger muscle to the size of a leg instead of realizing maybe you're not supposed to still be using a finger for that job.
- petters 3y ago
  It's useful for several other things. Automatic logging is one. Another: I compute the commands run most often (candidates for automation). > These days "so old it's no longer valid" is barely a year I don't think this is true (generally). At my work, we use old boring technology for most things (as we should). I don't see how it could be true enough to warrant not keeping a command line log.
- janalsncm 3y ago
  The author mentioned this objection and suggested that since the cost of keeping a long history is so low, there’s really no reason not to. Additionally, you may not know a command will be useful in the future at the time you execute it. I suppose everyone’s workflows are different though. I keep a long history but rarely use it.
- deleted 3y ago
  [deleted]
- jefftk 3y ago
  I do a lot of analyses as a series of shell commands. Sometimes the command is complex, other times it isn't but the specific arguments I'm using matter a lot. When I need to revisit an old analysis being able to see the commands I was running is incredibly useful for picking up where I left off or answering people's questions about exactly what I did. These are rarely commands so complex that they represent serious programming effort, but that's not why I want to save them. (My approach: https://www.jefftk.com/p/you-should-be-logging-shell-history https://www.jefftk.com/p/you-should-be-logging-shell-history)
saagarjha 3y ago
Bringing this advice up again because (unlike a lot of what I say) it’s actually something that I have personal experience with and I think it will greatly improve your life: https://news.ycombinator.com/item?id=33187749 https://news.ycombinator.com/item?id=33187749
- codethief 3y ago
  Related comment with a different solution to the same problem: https://news.ycombinator.com/item?id=39076976 https://news.ycombinator.com/item?id=39076976
timvisee 3y ago
Fun fact: if you don't want to put a command in history, prefix it with a space. This works in many shells.
- LaputanMachine 3y ago
  This feature can also be be enabled in macOS by adding the following line to ~/.zshrc: setopt HIST_IGNORE_SPACE
  - JadeNB 3y ago
    > This feature can also be be enabled in macOS by adding the following line to ~/.zshrc: > setopt HIST_IGNORE_SPACE Except that I think it's called `histignorespace` (I haven't caught up with when and how much zsh cares about lowercased names), I think this is just about zsh, not about macOS specifically.
- talhah 3y ago
  And if you're using fish shell, you can start a private session with fish --private or fish -P for short
- bruce343434 3y ago
  Doesn't work for bash
  - unhammer 3y ago
    You may need to enable it HISTCONTROL=ignorespace
opentokix 3y ago
[flagged]
RockRobotRock 3y ago
My setup is so sporadic it would be hard to remember to save shell history. Is there any way to upload this to a server automatically across multiple computers?
- petters 3y ago
  Private git repo?
- KolmogorovComp 3y ago
  I’ve not used it but Atuin has good reputation and support uploading encrypted history to their (or your) servers. https://atuin.sh/ https://atuin.sh/
  - RockRobotRock 3y ago
    Installed it on my machines, we'll see how it works out. Thank you!
- EasyMark 3y ago
  I use a private github for my dotfiles, run a cron job to back them up nightly
rwmj 3y ago
Reminds me of Stephen Wolfram (Mathematica author) who has recorded every keystroke he ever typed: https://writings.stephenwolfram.com/2012/03/the-personal-analytics-of-my-life/ https://writings.stephenwolfram.com/2012/03/the-personal-ana...
- mattigames 3y ago
  [dead]
- codethief 3y ago
  Well, having read (or rather: started to read/tried to read) many of Stephen's blog posts, I would argue that not every single one of his musings should be recorded. :-)
- gnyman 3y ago
  Thats interesting to learn. I think he is on to something. The one thing that computers are really good at is storing data (especially text) so I believe it should be a core function of any computer that it should never loose anything a user has input. Sadly the reality is far from the truth. The number of times I have entered something into a webform or an app, only to have it disappear for some weird reason...
mustak_im 3y ago
Did somebody say atuin! https://atuin.sh/ https://atuin.sh/
- deleted 3y ago
  [deleted]
- oarmstrong 3y ago
  Yes, the article did.
nomilk 3y ago
My .zsh_history doesn't go back far enough, but I can see my stack overflow history from exactly 1731 days ago (the search is surprisingly nuanced, and allows filtering by user and date range right down to the day - to see yours just replace my user id with yours in the following url): https://stackoverflow.com/search?q=user%3A5783745+created%3A2019-04-26..2019-04-27 https://stackoverflow.com/search?q=user%3A5783745+created%3A... Looks like on that day I was having trouble aligning a navbar (no surprise), and figured out how to render a view without a layout in rails 5.
deleted 3y ago
[deleted]
spelufo 3y ago
There's a way to configure readline (which bash and many other use) to use up arrow history, like Matlab or Julia do by default, where you hit the up arrow key to search the history for commands that start with what you've typed so far. It is a small thing, but I use it all the time. In ~/.inputrc: "\e[A": history-search-backward "\C-p": history-search-backward "\e[B": history-search-forward "\C-n": history-search-forward
- _0ffh 3y ago
  At some point in the past it took me nearly an hour trawling the internet to find this piece of code, though it's not quite identical. "\e[A": history-search-backward "\e[B": history-search-forward "\eOA": history-search-backward "\eOB": history-search-forward The only downside ist, don't over-rely on history muscle memory! Regularly every few months I use p+arrowUp and expect to get "ping" but get the exceptional "pip" instead.
- BeetleB 3y ago
  Incidentally, many distributions have this enabled, but with the PgUp keybinding instead of Up.
- beaugunderson 3y ago
  when I first found this years ago it changed my life. so much faster than ctrl-r to search the history... type a prefix and hit up a few times.
  - Ferret7446 3y ago
    How is it so much faster? It's only one additional keypress, vs having to reach for the arrow keys.
october8140 3y ago
This reminds me of people who have 9999 tabs open instead of using bookmarks.
- m463 3y ago
  I think this is more analogous to automatic bookmarks. open tabs is more like a cluttered desk. You know, I would love a browsing history (as a log file). I think it would be helpful and useful.
- EasyMark 3y ago
  and then freak out when their browser crashes and trashes the session. If you're that dependent on it you should probably add one of the many tab extensions that will save your list of open tabs to a backup every few minutes.
  - sureglymop 3y ago
    I do this. I use sidebery which creates snapshots of all open tabs. I usually have between 100 and ~600 tabs open. Once a month I go through the list and close those that are not relevant anymore (can easily select and close multiple tabs in sidebery). It works pretty well for me, it often reminds me of things I was working on but haven't yet finished. Since I have so many tabs, I also created a firefox extension to pin tabs on the right, so that they are close to the "new tab" button.
bhaak 3y ago
I'm using a snippet from this page: https://spin.atomicobject.com/2016/05/28/log-bash-history/ https://spin.atomicobject.com/2016/05/28/log-bash-history/ export PROMPT_COMMAND='if [ "$(id -u)" -ne 0 ]; then echo "$(date "+%Y-%m-%d %H:%M:%S") $(pwd) $(HISTTIMEFORMAT= history 1)" >> ~/.logs/bash-history-$(date "+%Y-%m-%d").log; fi' This give an easily greppable shell history. An entry looks like this: 2022-10-13 09:14:54 /home/bhaak 1000 vim ~/.bashrc
amoss 3y ago
I've been doing something similar for years. Originally I tried to keep everything in .bash_history, but over the years and different machines it inevitably gets wiped sooner or later. This helps (I use bash rather than zsh): # Sort out history$ export HISTIGNORE="&:ls:vi:bg:fg:history" export HISTCONTROL=ignoredups:ignorespace export HISTSIZE=100000 export HISTFILESIZE=100000 shopt -s histappend shopt -s checkwinsize But the long-term (multi-machine) solution is to keep another append-only copy, one per machine: function storehist() { # Bug in mac version of bash stops HISTCMD getting set in function called from PROMPT_COMMAND... CMDCNT=`history 1 | sed -e 's/\w* *$.*$/\1/'`; if [ "$CMDCNT" != "$LASTCMDIND" ] && [ -n "$LASTCMDIND" ] # First time, no prev, do not log random then DATE=`date '+%Y%b%d %H%M'`; if [ "$LASTCMDIND" != "$CMDCNT" ] then echo "$DATE $HOSTNAME:$BASHTTY $CMDCNT" >>~/.custom_history; fi LASTCMDIND="$CMDCNT"; fi LASTCMDIND="$CMDCNT"; } export PROMPT_COMMAND='storehist; .... A little bit over the top and crusty but it has worked like that for years so I've not updated it in a while. There is a separate sync script that puts the .custom_history from each machine / container into a folder on each machine with the orginating machine name in the file-name. This keeps ^-R working over a long period of time, but eventually that stops being a good way of find obscure commands from long ago and then something like this works: cat .histories/* | grep CMD | grep WERID_ARG | grep RANDOM_RELEVANT_STR | sort -u | less The overall cost of running some script on each prompt is negliable on a modern machine and as the OP notes the disk space does not amount to much. It becomes very useful over time. I have about 20MB of shell history that stretches back to '06. It is one of things that is easy to setup, and yet the value accrues steadily over time. Across the dozen or so machines involved I doubt that the archive would have survived long-term without the step to export it outside of the shell's real history file.
- riffraff 3y ago
  FWIW, I have export HISTCONTROL=ignoreboth which I think is shorthand for ignoredups:ignorespace
amphitheatre 3y ago
I find it peculiar to ask this question but then also deduplicate your history file because then you lose complete history and context. Sometimes I've had to issue series of commands to fix a particular problem (changing a kerberos password, fixing the expiry date of a gpg key, etc.) that I can backtrack through the history file to find what I did and how I resolved it, later having to do the same, and according to their settings the first instance would be removed. Personally, I can't remember the solution to everything I do so having that deduplicated log gives me the full history of what I did and how I resolved it, amongst other uses. Sure, removing duplicate 'ls' entries is fine for example, but for every command? Not so sure.
- jefftk 3y ago
  I also don't understand why they're doing it. On my long-running full history duplicates cost me a factor of 2 in space, and presumably the same in search time (but still perceptually instant).
- misternugget 3y ago
  You know what, you're right. And it's actually something that's been bugging me for a while, I just never got around to fixing it. I'm going to update my config now. Thanks! (Author of post)
Sheeny96 3y ago
Shoutout for https://atuin.sh/ https://atuin.sh/, shell history across multiple devices with tons of cool features
- tmountain 3y ago
  I use—and love—atuin. It beings the power of SQLite to your shell history with sync across multiple devices. It’s simply awesome and has helped me unravel historical install and configuration details that would otherwise be lost.
csydas 3y ago
learning about ctrl+r on zsh feels like magic for me -- I'm not a heavy terminal user but I am pretty comfortable with it, and like the author, I have a lot of common commands with specific flags I need to add depending on if I'm doing a push to a local repo or a public one, etc. I like features like this that build off of how _i_ use the shell, and it's up to me to figure out how I want to use it, and it's so simple but it makes using the shell all that much more efficient and it's easier for me to express my intent to the shell. just very cool, like learning a bit of the coreutils to make every day stuff in system administration easier.
Fileformat 3y ago
Everyone's memory is better than mine: I can't even remember the command I typed, so I need to hunt it down from the context: current directory, date or previous commands. I use a zsh precmd() along the lines of echo "$(date "+%Y-%m-%d.%H:%M:%S") $(pwd) $(fc -ln -1)" >> ~/bash/bash-history-$(date "+%Y-%m-%d").log; Should be doable with bash's PROMPT_COMMAND if you are still on bash
- tripflag 3y ago
  We have a lot in common :-P except I'm still on bash, and do rely on PROMPT_COMMAND like you mentioned -- https://github.com/9001/asm/blob/hovudstraum/etc/profile.d/env.sh#L2-L24 https://github.com/9001/asm/blob/hovudstraum/etc/profile.d/e... Lines 11-21 can be ignored, they detect if the folder you were in got moved/deleted from another shell, to avoid the confusing behavior you get in that case.
- JNRowe 3y ago
  zsh has a comprehensive hook system available for these types of tasks¹(zshaddhistory in this case). It gives you more options to control how/when the history is preserved, and allows you to be more selective in when it should be written. Possibly worth noting for others that you'd want to think this through if you're using any of the history eliding options(hist_ignore_space² for example), as one history file may contain secrets when you're really expecting that it wouldn't. There is also a better interface to work with hook functions through the add-zsh-hook mechanism³, which allows stacking multiple hook functions together. ¹ https://zsh.sourceforge.io/Doc/Release/Functions.html#index-history_002c-hook-when-line-is-saved https://zsh.sourceforge.io/Doc/Release/Functions.html#index-... ² https://zsh.sourceforge.io/Doc/Release/Options.html#History https://zsh.sourceforge.io/Doc/Release/Options.html#History ³ https://zsh.sourceforge.io/Doc/Release/User-Contributions.html#Manipulating-Hook-Functions https://zsh.sourceforge.io/Doc/Release/User-Contributions.ht...
- csdvrx 3y ago
  > Should be doable with bash's PROMPT_COMMAND if you are still on bash Already done, with also PS0 and a sqlite backend: https://github.com/csdvrx/bash-timestamping-sqlite https://github.com/csdvrx/bash-timestamping-sqlite
Kubuxu 3y ago
I use zsh-histdb which stores the history in SQLite with context. Then I have few different queries/ways to interact with it. For example I have a line completion (which also shows as grayed out text while I type) based on most frequently used commands with give prefix with additional weighting depending how close in the directory tree they were used. [0]: https://github.com/larkery/zsh-histdb https://github.com/larkery/zsh-histdb
- ruined 3y ago
  oh, incredible. this is exactly what i wished existed for years but assumed was too absurd/overkill to have ever been made, so i didn't even search for it
pid1wow 3y ago
A little known secret is that you can actually search not only backward (ctrl+r), but also forward (ctrl+s). The problem is ctrl+s is bound by default to (obscure terminal thing) that stops your terminal from outputting anything. This can be disabled with stty stop ^-, and then ctrl+s will work as intended.
jackhalford 3y ago
I recently reinstalled my personal laptop to distro hop. the only two files I migrated over were my pgp key and .bash_history, the bash history is around 8000 lines I don’t want to loose. sometimes I ‘sort | uniq’ the bash history to filter about duplicates
- sureglymop 3y ago
  I often use diff in this way to compare command outputs: diff <(somecmd | sort) <(othercmd | sort)
jmclnx 3y ago
I find there are maybe about 10 commands I use regularly that I care about being in history. The rest I never search history for. So a couple of thousand seems to do well by me.
donaldihunter 3y ago
Totally agree with this. I use https://github.com/larkery/zsh-histdb https://github.com/larkery/zsh-histdb slightly modified to work more smoothly for me. If I remember correctly, I tried Atuin but it messed up multi-line commands. Zsh-histdb handles them well.
abathur 3y ago
Just a general recommendation from someone who keeps a long history: if your shell loads a histfile, don't let it grow unbounded. It isn't massive, but there's real shell startup latency associated with loading lines from your histfile. A few hundred isn't really noteworth, but tens and eventually hundreds of thousands of lines can add up. (In my case, i store them in a database and I synthesize histfiles from the db. Keeping the permanent storage in a separate file also probably works.)
- m463 3y ago
  I think histappend might help Even if it were slower, I also think the overhead of something like that is more than made up for by the help it provides.
qwertygnu 3y ago
Perhaps an edge case, but what about all the commands you mistype or use the wrong flags for etc. If I want to recall a niche or complex command I used months ago I probably won't remember which of the 10 results that pop up was the right one.
- mvc 3y ago
  Usually the most recent one is the one with all the kinks ironed out.
- BeetleB 3y ago
  You can set your shell not to store commands that returned an error exit code.
38 3y ago
PowerShell here, I use this: $MaximumHistoryCount = 9999
dooglius 3y ago
In my experience, the default shell history settings don't work well with multi-shell environments such as tmux. What I have done is in ~/.bash_profile, I create a subdir of ~/.historydirs and put in each of my history files (HISTFILE, LESSHISTFILE, PY_HISTFILE, etc.) which will be unique for a given tmux pane. This means that I can not only search for an exact command, but I can also get useful contextual information related (e.g. What folder did I last cd to?)
jl6 3y ago
This, plus I log all terminal output: script -q "$LOGDIR/$LOGTIME.log" --timing="$LOGDIR/$LOGTIME.timing" The log files are noisy with ANSI codes, but they can be stripped, and I only refer to the logs extremely rarely (but when I do, it's extremely useful).
m463 3y ago
my .bashrc has: shopt -s histappend export HISTFILESIZE=1000000 export HISTSIZE=1000000 export HISTTIMEFORMAT="%m/%d/%Y %T " export HISTIGNORE="exec env*:history*" export HISTCONTROL="ignoreboth" It seems my history preceeds "recorded time", because timestamps start in 2016, which is probably when I discovered HISTTIMEFORMAT. 110k commands btw the "exec env*" is because emacs tramp uses that when remotely logging into a machine and fill the logs with junk. The fist thing I do on any machine is to set HISTSIZE and HISTFILESIZE to large numbers. It is wonderful to refer back to how you set up a machine from scratch for instance. How did I set up this. what did I set up. What packages did I install. I've always kind of wanted to create some sort of "annotated history" program. The idea would be, to ask you what you are doing (maybe in a separate window to the side) so you could properly document something fiddly.

176 points by ingve 3y ago

Can fzf or the shell history file answer this question? zsh records timestamps and runtime if extended history is enabled. bash needs HISTTIMEFORMAT to be configured properly. I haven't used fzf so not sure if it can search by timestamp, I only see a line number for the matching command (presumably) in the gif.

defrost 3y ago
FWiW the authors setup: Here’s the ZSH configuration I used for many, many years: possibly cannot fully answer the question as there are settings to scrub duplicate commands ... meaning the target day will show commands unique to that day but not necessarily all commands from that day. It's a small quibble but potentially an issue for those that really need to Indiana Jones their command history for some specific clue.
- rzzzt 3y ago
  I agree, especially if the focus is not on the command itself but the date. Atuin has a CLI switch for filtering command history by date, but I don't see any support in the ctrl-r UI eg. with an operator like "before:" or similar.
  - defrost 3y ago
    In my own case large clumps of similar commands hinge upon what was the CD <current directory> at the time ... and the command most likely in many cases to be scrubbed as duplicate is a cd <one of 50 common working base directories>. This leads to "I can see I tidied up a bunch of project files first monday of last month ... but which project was that??"
misnome 3y ago
A nice thing that zsh (and bash-preexec) has is that you can add arbitrary functions to be called on command execution. I work at a large facility with a shared filesystem and many computers, so often I need to answer questions like “What did I run the last time I was sitting at computer X”. I log: command, timestamp, PID (to separate terminal streams), hostname and cwd. I’ve set up ctrl-r to search this “.fullhistory” file instead.
- salohcin 3y ago
  This sounds interesting! Can you elaborate further?
  - misnome 3y ago
    The function: if [[ -n ${ZSH_VERSION-} ]]; then # ZSH doesn't split command over multiple variables preexec_custom_history() { echo "$HOSTNAME:\"$PWD\" $$ $(date "+%Y-%m-%dT%H:%M:%S%z") $1" >> "$CUSTOM_HISTORY_FILE" } else preexec_custom_history() { echo "$HOSTNAME:\"$PWD\" $$ $(date "+%Y-%m-%dT%H:%M:%S%z") $*" >> "$CUSTOM_HISTORY_FILE" } fi # Add it to the array of functions to be invoked each time. preexec_functions+=(preexec_custom_history) It's harder to search for multiline but I haven't gotten to fixing that yet. I've thought about putting it into sqlite (pwd makes the file large) but since everything is on network filesystems I don't think that's a place sqlite has any guarantees; screwing up a single line of a bare log doesn't break anything. Edit: The fzf is kind of hacked together but is here: https://pastebin.com/sTxsvZAf https://pastebin.com/sTxsvZAf . It has a few bugs but nothing annoying enough to have made me fix it.
59nadir 3y ago
Atuin (the tool mentioned later in the post) stores timestamps and CWDs for commands in a database, which is probably why the title is what it is. I've been using it for 6+ months and it's actually good. I also like having an sqlite database with my commands so I can search it even more freeform if I want.

mmsc 3y ago

I also like to use sudo chattr +a .sh_history to ensure that my shell history file isn’t deleted. Being append-only, it also ensures that some type of timing issue in my shell which means my HISTFILESIZE or the equivalent isn’t set to something larger than default, meaning my history isn’t deleted.

codetrotter 3y ago
If the shell is unaware that the file is append only, does it result in appending not just the commands run from this session but undesirably also ends up appending a full copy of a lot of the previous history?
- blueflow 3y ago
  I guess it means that the open() syscall will require the O_APPEND flag. If the file is opened without that flag it will error out.
mgdlbp 3y ago
a weaker method is to set a non-default HISTFILE after HISTSIZE etc

permalac 3y ago

Good tool, but my history is so noisy I had to find another way. Lately I started to add a new habit in my shell actions. When a command has some cognitive requirements I create a script with some ${1:-default} values and I store them all in $PATH enabled local/bin All my scripts start with __ and then some capital letters indicating the topic. I can copy them to new machines, which is something I would not do with my history.

bananskalhalk 3y ago
Prefixing your scripts is an awesome tips. A colleague told me he prefixed all his commands with ',' and I've started to do the same. '_' might clash with 'hidden'(?) shell expansions etc. Sadly git doesn't allow for aliases with an comma-prefix, so all git aliases starts with a dot.
- gumby 3y ago
  You can make symlinks or shell aliases that just reference git-add et al directly.
thenbe 3y ago
> my history is so noisy I had to find another way The fzf search syntax [1] can help, if you become familiar with it. It is also supported in atuin [2]. [1]: https://github.com/junegunn/fzf#search-syntax https://github.com/junegunn/fzf#search-syntax [2]: https://docs.atuin.sh/configuration/config/#fuzzy-search-syntax https://docs.atuin.sh/configuration/config/#fuzzy-search-syn...
__MatrixMan__ 3y ago
Nushell has a SQLite history mode which captures more metadata than is typical. I haven't looked into extending it, but it seems like a good setup for adding enough so that some of these cognitive requirements could be recovered from history rather than thought of up front. Like if I tend to go down the wrong path, fix it, and then go down the right path, maybe that's a pattern that can be identified and then when I search history it finds two or three commands which together set me on the right path in the first place. I mean, thinking ahead is always gonna be better, but I don't always.

natrys 3y ago

With a bit of prompt hacking I send information (shell session, pwd, command, timestamp) to a fifo, which is read by small Tcl script to ingest in a sqlite database. Aturin seems nice, probably does something similar. It's a hacky script but the idea is sound. Good thing about SQL is that I can fine tune the query before feeding to fzf. A combination of order by recency and frequency of commands with and without context (pwd) works very well for me. With proper indexing I am not close to feeling slowdown, but if that happens I can probably just export data to duckdb and use that.

tomatocracy 3y ago

The article briefly mentions atuin at the end. I've tried atuin but found it a little bit too heavyweight for me - instead I use zsh-histdb[0] (together with the fzf extension for it[1]) which allows you to easily answer this type of question - can highly recommend it. 0. https://github.com/larkery/zsh-histdb https://github.com/larkery/zsh-histdb 1. https://github.com/m42e/zsh-histdb-fzf https://github.com/m42e/zsh-histdb-fzf

larusso 3y ago
I tried atuin as well and switched back to my fzf setup. I setup my history to ignore duplicates. Makes it also easier to use the up arrow key. But that can have downsides depending on the user. For me it works great for years. The only reason I don‘t share the history along with other dotfiles is the problem that some commands read API keys and token from the options. And I don‘t yet have a system for that so that. Because other means like env variables can make the it harder to repeat said command month later.

Brian_K_White 3y ago

The other day there was another post about a bash history in a db, years worth and searchable and available on all machines. I do not get this. I have over 30 years certainly cobbled some complicated commands, but I never need an ancient one. If it's complicated and used more than one day, it's a script. Complicated or not and used a long time ago, it's no longer valid. These days "so old it's no longer valid" is barely a year. Back in the sco Unix days you could have 10 year old commands that are still valid references, but Linux never had that except for trivial things. There are exceptions but they are exceptions and so by definition not reasonable to focus on. I think that building essentially mini apps right on the command line and then carrying around years of shell history to reuse them is solving a problem the wrong way. A stupid and backwards way. It's losing sight of what the problem actually is and like growing one weird finger muscle to the size of a leg instead of realizing maybe you're not supposed to still be using a finger for that job.

petters 3y ago
It's useful for several other things. Automatic logging is one. Another: I compute the commands run most often (candidates for automation). > These days "so old it's no longer valid" is barely a year I don't think this is true (generally). At my work, we use old boring technology for most things (as we should). I don't see how it could be true enough to warrant not keeping a command line log.
janalsncm 3y ago
The author mentioned this objection and suggested that since the cost of keeping a long history is so low, there’s really no reason not to. Additionally, you may not know a command will be useful in the future at the time you execute it. I suppose everyone’s workflows are different though. I keep a long history but rarely use it.
deleted 3y ago
[deleted]
jefftk 3y ago
I do a lot of analyses as a series of shell commands. Sometimes the command is complex, other times it isn't but the specific arguments I'm using matter a lot. When I need to revisit an old analysis being able to see the commands I was running is incredibly useful for picking up where I left off or answering people's questions about exactly what I did. These are rarely commands so complex that they represent serious programming effort, but that's not why I want to save them. (My approach: https://www.jefftk.com/p/you-should-be-logging-shell-history https://www.jefftk.com/p/you-should-be-logging-shell-history)

saagarjha 3y ago

Bringing this advice up again because (unlike a lot of what I say) it’s actually something that I have personal experience with and I think it will greatly improve your life: https://news.ycombinator.com/item?id=33187749 https://news.ycombinator.com/item?id=33187749

codethief 3y ago
Related comment with a different solution to the same problem: https://news.ycombinator.com/item?id=39076976 https://news.ycombinator.com/item?id=39076976

timvisee 3y ago

Fun fact: if you don't want to put a command in history, prefix it with a space. This works in many shells.

LaputanMachine 3y ago
This feature can also be be enabled in macOS by adding the following line to ~/.zshrc: setopt HIST_IGNORE_SPACE
- JadeNB 3y ago
  > This feature can also be be enabled in macOS by adding the following line to ~/.zshrc: > setopt HIST_IGNORE_SPACE Except that I think it's called `histignorespace` (I haven't caught up with when and how much zsh cares about lowercased names), I think this is just about zsh, not about macOS specifically.
talhah 3y ago
And if you're using fish shell, you can start a private session with fish --private or fish -P for short
bruce343434 3y ago
Doesn't work for bash
- unhammer 3y ago
  You may need to enable it HISTCONTROL=ignorespace

opentokix 3y ago

[flagged]

RockRobotRock 3y ago

My setup is so sporadic it would be hard to remember to save shell history. Is there any way to upload this to a server automatically across multiple computers?

petters 3y ago
Private git repo?
KolmogorovComp 3y ago
I’ve not used it but Atuin has good reputation and support uploading encrypted history to their (or your) servers. https://atuin.sh/ https://atuin.sh/
- RockRobotRock 3y ago
  Installed it on my machines, we'll see how it works out. Thank you!
EasyMark 3y ago
I use a private github for my dotfiles, run a cron job to back them up nightly

rwmj 3y ago

Reminds me of Stephen Wolfram (Mathematica author) who has recorded every keystroke he ever typed: https://writings.stephenwolfram.com/2012/03/the-personal-analytics-of-my-life/ https://writings.stephenwolfram.com/2012/03/the-personal-ana...

mattigames 3y ago
[dead]
codethief 3y ago
Well, having read (or rather: started to read/tried to read) many of Stephen's blog posts, I would argue that not every single one of his musings should be recorded. :-)
gnyman 3y ago
Thats interesting to learn. I think he is on to something. The one thing that computers are really good at is storing data (especially text) so I believe it should be a core function of any computer that it should never loose anything a user has input. Sadly the reality is far from the truth. The number of times I have entered something into a webform or an app, only to have it disappear for some weird reason...

mustak_im 3y ago

Did somebody say atuin! https://atuin.sh/ https://atuin.sh/

deleted 3y ago
[deleted]
oarmstrong 3y ago
Yes, the article did.

nomilk 3y ago

My .zsh_history doesn't go back far enough, but I can see my stack overflow history from exactly 1731 days ago (the search is surprisingly nuanced, and allows filtering by user and date range right down to the day - to see yours just replace my user id with yours in the following url): https://stackoverflow.com/search?q=user%3A5783745+created%3A2019-04-26..2019-04-27 https://stackoverflow.com/search?q=user%3A5783745+created%3A... Looks like on that day I was having trouble aligning a navbar (no surprise), and figured out how to render a view without a layout in rails 5.

deleted 3y ago

[deleted]

spelufo 3y ago

There's a way to configure readline (which bash and many other use) to use up arrow history, like Matlab or Julia do by default, where you hit the up arrow key to search the history for commands that start with what you've typed so far. It is a small thing, but I use it all the time. In ~/.inputrc: "\e[A": history-search-backward "\C-p": history-search-backward "\e[B": history-search-forward "\C-n": history-search-forward

_0ffh 3y ago
At some point in the past it took me nearly an hour trawling the internet to find this piece of code, though it's not quite identical. "\e[A": history-search-backward "\e[B": history-search-forward "\eOA": history-search-backward "\eOB": history-search-forward The only downside ist, don't over-rely on history muscle memory! Regularly every few months I use p+arrowUp and expect to get "ping" but get the exceptional "pip" instead.
BeetleB 3y ago
Incidentally, many distributions have this enabled, but with the PgUp keybinding instead of Up.
beaugunderson 3y ago
when I first found this years ago it changed my life. so much faster than ctrl-r to search the history... type a prefix and hit up a few times.
- Ferret7446 3y ago
  How is it so much faster? It's only one additional keypress, vs having to reach for the arrow keys.

october8140 3y ago

This reminds me of people who have 9999 tabs open instead of using bookmarks.

m463 3y ago
I think this is more analogous to automatic bookmarks. open tabs is more like a cluttered desk. You know, I would love a browsing history (as a log file). I think it would be helpful and useful.
EasyMark 3y ago
and then freak out when their browser crashes and trashes the session. If you're that dependent on it you should probably add one of the many tab extensions that will save your list of open tabs to a backup every few minutes.
- sureglymop 3y ago
  I do this. I use sidebery which creates snapshots of all open tabs. I usually have between 100 and ~600 tabs open. Once a month I go through the list and close those that are not relevant anymore (can easily select and close multiple tabs in sidebery). It works pretty well for me, it often reminds me of things I was working on but haven't yet finished. Since I have so many tabs, I also created a firefox extension to pin tabs on the right, so that they are close to the "new tab" button.

bhaak 3y ago

I'm using a snippet from this page: https://spin.atomicobject.com/2016/05/28/log-bash-history/ https://spin.atomicobject.com/2016/05/28/log-bash-history/ export PROMPT_COMMAND='if [ "$(id -u)" -ne 0 ]; then echo "$(date "+%Y-%m-%d %H:%M:%S") $(pwd) $(HISTTIMEFORMAT= history 1)" >> ~/.logs/bash-history-$(date "+%Y-%m-%d").log; fi' This give an easily greppable shell history. An entry looks like this: 2022-10-13 09:14:54 /home/bhaak 1000 vim ~/.bashrc

amoss 3y ago

I've been doing something similar for years. Originally I tried to keep everything in .bash_history, but over the years and different machines it inevitably gets wiped sooner or later. This helps (I use bash rather than zsh): # Sort out history$ export HISTIGNORE="&:ls:vi:bg:fg:history" export HISTCONTROL=ignoredups:ignorespace export HISTSIZE=100000 export HISTFILESIZE=100000 shopt -s histappend shopt -s checkwinsize But the long-term (multi-machine) solution is to keep another append-only copy, one per machine: function storehist() { # Bug in mac version of bash stops HISTCMD getting set in function called from PROMPT_COMMAND... CMDCNT=`history 1 | sed -e 's/\w* *$.*$/\1/'`; if [ "$CMDCNT" != "$LASTCMDIND" ] && [ -n "$LASTCMDIND" ] # First time, no prev, do not log random then DATE=`date '+%Y%b%d %H%M'`; if [ "$LASTCMDIND" != "$CMDCNT" ] then echo "$DATE $HOSTNAME:$BASHTTY $CMDCNT" >>~/.custom_history; fi LASTCMDIND="$CMDCNT"; fi LASTCMDIND="$CMDCNT"; } export PROMPT_COMMAND='storehist; .... A little bit over the top and crusty but it has worked like that for years so I've not updated it in a while. There is a separate sync script that puts the .custom_history from each machine / container into a folder on each machine with the orginating machine name in the file-name. This keeps ^-R working over a long period of time, but eventually that stops being a good way of find obscure commands from long ago and then something like this works: cat .histories/* | grep CMD | grep WERID_ARG | grep RANDOM_RELEVANT_STR | sort -u | less The overall cost of running some script on each prompt is negliable on a modern machine and as the OP notes the disk space does not amount to much. It becomes very useful over time. I have about 20MB of shell history that stretches back to '06. It is one of things that is easy to setup, and yet the value accrues steadily over time. Across the dozen or so machines involved I doubt that the archive would have survived long-term without the step to export it outside of the shell's real history file.

riffraff 3y ago
FWIW, I have export HISTCONTROL=ignoreboth which I think is shorthand for ignoredups:ignorespace

amphitheatre 3y ago

I find it peculiar to ask this question but then also deduplicate your history file because then you lose complete history and context. Sometimes I've had to issue series of commands to fix a particular problem (changing a kerberos password, fixing the expiry date of a gpg key, etc.) that I can backtrack through the history file to find what I did and how I resolved it, later having to do the same, and according to their settings the first instance would be removed. Personally, I can't remember the solution to everything I do so having that deduplicated log gives me the full history of what I did and how I resolved it, amongst other uses. Sure, removing duplicate 'ls' entries is fine for example, but for every command? Not so sure.

jefftk 3y ago
I also don't understand why they're doing it. On my long-running full history duplicates cost me a factor of 2 in space, and presumably the same in search time (but still perceptually instant).
misternugget 3y ago
You know what, you're right. And it's actually something that's been bugging me for a while, I just never got around to fixing it. I'm going to update my config now. Thanks! (Author of post)

Sheeny96 3y ago

Shoutout for https://atuin.sh/ https://atuin.sh/, shell history across multiple devices with tons of cool features

tmountain 3y ago
I use—and love—atuin. It beings the power of SQLite to your shell history with sync across multiple devices. It’s simply awesome and has helped me unravel historical install and configuration details that would otherwise be lost.

csydas 3y ago

learning about ctrl+r on zsh feels like magic for me -- I'm not a heavy terminal user but I am pretty comfortable with it, and like the author, I have a lot of common commands with specific flags I need to add depending on if I'm doing a push to a local repo or a public one, etc. I like features like this that build off of how _i_ use the shell, and it's up to me to figure out how I want to use it, and it's so simple but it makes using the shell all that much more efficient and it's easier for me to express my intent to the shell. just very cool, like learning a bit of the coreutils to make every day stuff in system administration easier.

Fileformat 3y ago

Everyone's memory is better than mine: I can't even remember the command I typed, so I need to hunt it down from the context: current directory, date or previous commands. I use a zsh precmd() along the lines of echo "$(date "+%Y-%m-%d.%H:%M:%S") $(pwd) $(fc -ln -1)" >> ~/bash/bash-history-$(date "+%Y-%m-%d").log; Should be doable with bash's PROMPT_COMMAND if you are still on bash

tripflag 3y ago
We have a lot in common :-P except I'm still on bash, and do rely on PROMPT_COMMAND like you mentioned -- https://github.com/9001/asm/blob/hovudstraum/etc/profile.d/env.sh#L2-L24 https://github.com/9001/asm/blob/hovudstraum/etc/profile.d/e... Lines 11-21 can be ignored, they detect if the folder you were in got moved/deleted from another shell, to avoid the confusing behavior you get in that case.
JNRowe 3y ago
zsh has a comprehensive hook system available for these types of tasks¹(zshaddhistory in this case). It gives you more options to control how/when the history is preserved, and allows you to be more selective in when it should be written. Possibly worth noting for others that you'd want to think this through if you're using any of the history eliding options(hist_ignore_space² for example), as one history file may contain secrets when you're really expecting that it wouldn't. There is also a better interface to work with hook functions through the add-zsh-hook mechanism³, which allows stacking multiple hook functions together. ¹ https://zsh.sourceforge.io/Doc/Release/Functions.html#index-history_002c-hook-when-line-is-saved https://zsh.sourceforge.io/Doc/Release/Functions.html#index-... ² https://zsh.sourceforge.io/Doc/Release/Options.html#History https://zsh.sourceforge.io/Doc/Release/Options.html#History ³ https://zsh.sourceforge.io/Doc/Release/User-Contributions.html#Manipulating-Hook-Functions https://zsh.sourceforge.io/Doc/Release/User-Contributions.ht...
csdvrx 3y ago
> Should be doable with bash's PROMPT_COMMAND if you are still on bash Already done, with also PS0 and a sqlite backend: https://github.com/csdvrx/bash-timestamping-sqlite https://github.com/csdvrx/bash-timestamping-sqlite

Kubuxu 3y ago

I use zsh-histdb which stores the history in SQLite with context. Then I have few different queries/ways to interact with it. For example I have a line completion (which also shows as grayed out text while I type) based on most frequently used commands with give prefix with additional weighting depending how close in the directory tree they were used. [0]: https://github.com/larkery/zsh-histdb https://github.com/larkery/zsh-histdb

ruined 3y ago
oh, incredible. this is exactly what i wished existed for years but assumed was too absurd/overkill to have ever been made, so i didn't even search for it

pid1wow 3y ago

A little known secret is that you can actually search not only backward (ctrl+r), but also forward (ctrl+s). The problem is ctrl+s is bound by default to (obscure terminal thing) that stops your terminal from outputting anything. This can be disabled with stty stop ^-, and then ctrl+s will work as intended.

jackhalford 3y ago

I recently reinstalled my personal laptop to distro hop. the only two files I migrated over were my pgp key and .bash_history, the bash history is around 8000 lines I don’t want to loose. sometimes I ‘sort | uniq’ the bash history to filter about duplicates

sureglymop 3y ago
I often use diff in this way to compare command outputs: diff <(somecmd | sort) <(othercmd | sort)

jmclnx 3y ago

I find there are maybe about 10 commands I use regularly that I care about being in history. The rest I never search history for. So a couple of thousand seems to do well by me.

donaldihunter 3y ago

Totally agree with this. I use https://github.com/larkery/zsh-histdb https://github.com/larkery/zsh-histdb slightly modified to work more smoothly for me. If I remember correctly, I tried Atuin but it messed up multi-line commands. Zsh-histdb handles them well.

abathur 3y ago

Just a general recommendation from someone who keeps a long history: if your shell loads a histfile, don't let it grow unbounded. It isn't massive, but there's real shell startup latency associated with loading lines from your histfile. A few hundred isn't really noteworth, but tens and eventually hundreds of thousands of lines can add up. (In my case, i store them in a database and I synthesize histfiles from the db. Keeping the permanent storage in a separate file also probably works.)

m463 3y ago
I think histappend might help Even if it were slower, I also think the overhead of something like that is more than made up for by the help it provides.

qwertygnu 3y ago

Perhaps an edge case, but what about all the commands you mistype or use the wrong flags for etc. If I want to recall a niche or complex command I used months ago I probably won't remember which of the 10 results that pop up was the right one.

mvc 3y ago
Usually the most recent one is the one with all the kinks ironed out.
BeetleB 3y ago
You can set your shell not to store commands that returned an error exit code.

38 3y ago

PowerShell here, I use this: $MaximumHistoryCount = 9999

dooglius 3y ago

In my experience, the default shell history settings don't work well with multi-shell environments such as tmux. What I have done is in ~/.bash_profile, I create a subdir of ~/.historydirs and put in each of my history files (HISTFILE, LESSHISTFILE, PY_HISTFILE, etc.) which will be unique for a given tmux pane. This means that I can not only search for an exact command, but I can also get useful contextual information related (e.g. What folder did I last cd to?)

jl6 3y ago

This, plus I log all terminal output: script -q "$LOGDIR/$LOGTIME.log" --timing="$LOGDIR/$LOGTIME.timing" The log files are noisy with ANSI codes, but they can be stripped, and I only refer to the logs extremely rarely (but when I do, it's extremely useful).

m463 3y ago

my .bashrc has: shopt -s histappend export HISTFILESIZE=1000000 export HISTSIZE=1000000 export HISTTIMEFORMAT="%m/%d/%Y %T " export HISTIGNORE="exec env*:history*" export HISTCONTROL="ignoreboth" It seems my history preceeds "recorded time", because timestamps start in 2016, which is probably when I discovered HISTTIMEFORMAT. 110k commands btw the "exec env*" is because emacs tramp uses that when remotely logging into a machine and fill the logs with junk. The fist thing I do on any machine is to set HISTSIZE and HISTFILESIZE to large numbers. It is wonderful to refer back to how you set up a machine from scratch for instance. How did I set up this. what did I set up. What packages did I install. I've always kind of wanted to create some sort of "annotated history" program. The idea would be, to ask you what you are doing (maybe in a separate window to the side) so you could properly document something fiddly.