5 ms·
At 0.5GB that's 5k per entry -- what are you storing for each? For comparison, my (non-work) history since 2012 (plain text) is 181k entries, and takes 25MB.
by jefftk 3y ago
At 0.5GB that's 5k per entry -- what are you storing for each?
For comparison, my (non-work) history since 2012 (plain text) is 181k entries, and takes 25MB. I store the command along with when and where you ran it. (https://www.jefftk.com/p/logging-shell-history-in-zsh https://www.jefftk.com/p/logging-shell-history-in-zsh)
- tomxor 3y agoNot defending the idea of a db history but no db schema is going to beat plain text's 1 byte 0x0A per line delimiter.
- justinclift 3y ago*cough* compressed row data *cough*
- tomxor 3y ago*cough* log rotate and gzip *cough* :D
- justinclift 3y agoTouché. :)
- akkartik 3y agoAh, a fellow packrat! I have every command I ever typed into a shell since around 2005, and my history weighs in at 1 CD or 650MB (as of a couple of years ago) I'm probably being wasteful of space because I store each session in a separate file. I used to do a lot of data analysis at the shell back in the day, and found it useful to audit sequences of commands afterwards for mistakes, or to turn them into scripts.
- TedDoesntTalk 3y agoThis is so insane that I love it. Do you also save your belly button lint since 2005? Or nail clippings? :)
- akkartik 3y agoI'm only a digital packrat. Bits are so much cheaper to hoard, even deciding to throw something away is often more work.
- enriquto 3y agoThis is more like saving your old notebooks and drafts, only that they don't take any meaningful space. Or like having a revision control system. Do you rebase your git repos regularly to delete commits older than 6 months?
- trallnag 3y agoDo you regularly back up your history?
- akkartik 3y agoOh yes. It gets backed up along with everything else.
- bastawhiz 3y agoNot the op, but I'd guess it's the full text search index.
- jefftk 3y agoFor 100k entries you can grep them instantaneously; there's no need to maintain an index.
- bastawhiz 3y agoGrep only works if you want an exact string match. If you want to find words out of order or support features like stemming, fts is necessary.
- wpietri 3y agoMaybe I have some sort of disease, but while reading "find words out of order or support features like stemming" the regexs for that immediately flashed before my eyes, so I think "necessary" is a little strong there.
- jack_squat 3y agoFTS is not the same as regex.
- wpietri 3y agoI don't think I said it was. I was addressing the specific use cases mentioned. If there's another use case you think is important in searching command line history, feel free to describe it.
- justinclift 3y ago> feel free to describe it Didn't they already? eg stemming
- outcoldman 3y agoAs a lot of people mentioned. This is FTS index. So it is definitely way more blown up. Plus I do save a lot of additional information with it: pwd, session id, shell used, exit codes, whole command obviously. And to support icloud, also additional information for icloud entity id. And now when you point out, 5k per entry is a lot of data. But I am on with that. This information really important for me.