5 ms·

I have been using sqlite database since 2017 [1], has over 100,000 items in the database at this point. The database is almost 0.5GB, but I also use Full-Text-S

I have been using sqlite database since 2017 [1], has over 100,000 items in the database at this point. The database is almost 0.5GB, but I also use Full-Text-Search capabilities of sqlite. 2 years ago I have built a mac application [2], that syncs items via iCloud, and it only works on the macOS. I would highly recommend anyone who spend a lot of time in terminal to improve their shell history by using atuin or similar tools, cannot tell how many times it actually helped me to find some important information there, about how I did one thing or another. - - [1] https://www.outcoldman.com/en/archive/2017/07/19/dbhist/ https://www.outcoldman.com/en/archive/2017/07/19/dbhist/ - [2] https://loshadki.app/shellhistory/ https://loshadki.app/shellhistory/

jefftk 3y ago
At 0.5GB that's 5k per entry -- what are you storing for each? For comparison, my (non-work) history since 2012 (plain text) is 181k entries, and takes 25MB. I store the command along with when and where you ran it. (https://www.jefftk.com/p/logging-shell-history-in-zsh https://www.jefftk.com/p/logging-shell-history-in-zsh)
- tomxor 3y ago
  Not defending the idea of a db history but no db schema is going to beat plain text's 1 byte 0x0A per line delimiter.
  - justinclift 3y ago
    *cough* compressed row data *cough*
    - tomxor 3y ago
      *cough* log rotate and gzip *cough* :D
      - justinclift 3y ago
        Touché. :)
- akkartik 3y ago
  Ah, a fellow packrat! I have every command I ever typed into a shell since around 2005, and my history weighs in at 1 CD or 650MB (as of a couple of years ago) I'm probably being wasteful of space because I store each session in a separate file. I used to do a lot of data analysis at the shell back in the day, and found it useful to audit sequences of commands afterwards for mistakes, or to turn them into scripts.
  - TedDoesntTalk 3y ago
    This is so insane that I love it. Do you also save your belly button lint since 2005? Or nail clippings? :)
    - akkartik 3y ago
      I'm only a digital packrat. Bits are so much cheaper to hoard, even deciding to throw something away is often more work.
      - enriquto 3y ago
        This is more like saving your old notebooks and drafts, only that they don't take any meaningful space. Or like having a revision control system. Do you rebase your git repos regularly to delete commits older than 6 months?
  - trallnag 3y ago
    Do you regularly back up your history?
    - akkartik 3y ago
      Oh yes. It gets backed up along with everything else.
- bastawhiz 3y ago
  Not the op, but I'd guess it's the full text search index.
  - jefftk 3y ago
    For 100k entries you can grep them instantaneously; there's no need to maintain an index.
    - bastawhiz 3y ago
      Grep only works if you want an exact string match. If you want to find words out of order or support features like stemming, fts is necessary.
      - wpietri 3y ago
        Maybe I have some sort of disease, but while reading "find words out of order or support features like stemming" the regexs for that immediately flashed before my eyes, so I think "necessary" is a little strong there.
        jack_squat 3y ago
        FTS is not the same as regex.
        wpietri 3y ago
        I don't think I said it was. I was addressing the specific use cases mentioned. If there's another use case you think is important in searching command line history, feel free to describe it.
        justinclift 3y ago
        > feel free to describe it Didn't they already? eg stemming
- outcoldman 3y ago
  As a lot of people mentioned. This is FTS index. So it is definitely way more blown up. Plus I do save a lot of additional information with it: pwd, session id, shell used, exit codes, whole command obviously. And to support icloud, also additional information for icloud entity id. And now when you point out, 5k per entry is a lot of data. But I am on with that. This information really important for me.
SoftTalker 3y ago
Wow so it looks like I'm in the minority in having shell command history disabled? I have a small number of commands (20) that go into the live history of login shells for convenience, but nothing gets saved when I log out. If there's something I do repeatedly, I make an alias or a function for it.
- maleldil 3y ago
  I think you have an unusual take on this. Why do you disable history?
  - SoftTalker 3y ago
    People's history files are a great place to look for passwords and other secrets, mainly. I suppose that risk could be reduced by having the history file encrypted on disk, but I don't know of any shell that does that (can't honestly say I've really looked though).
    - philsnow 3y ago
      I get a ton of value out of having my shell history available (both for search but also to try to reconstruct steps of what I did yesterday when my memory is hazy) I guess you could set up an entropy scanner and flag history lines that have high entropy, but that might not be enough (low-entropy secrets) and might be bothersome (lots of false positives / things that are technically secrets but that you don't care if they're in your shell history).

jefftk 3y ago

At 0.5GB that's 5k per entry -- what are you storing for each? For comparison, my (non-work) history since 2012 (plain text) is 181k entries, and takes 25MB. I store the command along with when and where you ran it. (https://www.jefftk.com/p/logging-shell-history-in-zsh https://www.jefftk.com/p/logging-shell-history-in-zsh)

tomxor 3y ago
Not defending the idea of a db history but no db schema is going to beat plain text's 1 byte 0x0A per line delimiter.
- justinclift 3y ago
  *cough* compressed row data *cough*
  - tomxor 3y ago
    *cough* log rotate and gzip *cough* :D
    - justinclift 3y ago
      Touché. :)
akkartik 3y ago
Ah, a fellow packrat! I have every command I ever typed into a shell since around 2005, and my history weighs in at 1 CD or 650MB (as of a couple of years ago) I'm probably being wasteful of space because I store each session in a separate file. I used to do a lot of data analysis at the shell back in the day, and found it useful to audit sequences of commands afterwards for mistakes, or to turn them into scripts.
- TedDoesntTalk 3y ago
  This is so insane that I love it. Do you also save your belly button lint since 2005? Or nail clippings? :)
  - akkartik 3y ago
    I'm only a digital packrat. Bits are so much cheaper to hoard, even deciding to throw something away is often more work.
    - enriquto 3y ago
      This is more like saving your old notebooks and drafts, only that they don't take any meaningful space. Or like having a revision control system. Do you rebase your git repos regularly to delete commits older than 6 months?
- trallnag 3y ago
  Do you regularly back up your history?
  - akkartik 3y ago
    Oh yes. It gets backed up along with everything else.
bastawhiz 3y ago
Not the op, but I'd guess it's the full text search index.
- jefftk 3y ago
  For 100k entries you can grep them instantaneously; there's no need to maintain an index.
  - bastawhiz 3y ago
    Grep only works if you want an exact string match. If you want to find words out of order or support features like stemming, fts is necessary.
    - wpietri 3y ago
      Maybe I have some sort of disease, but while reading "find words out of order or support features like stemming" the regexs for that immediately flashed before my eyes, so I think "necessary" is a little strong there.
      - jack_squat 3y ago
        FTS is not the same as regex.
        wpietri 3y ago
        I don't think I said it was. I was addressing the specific use cases mentioned. If there's another use case you think is important in searching command line history, feel free to describe it.
        justinclift 3y ago
        > feel free to describe it Didn't they already? eg stemming
outcoldman 3y ago
As a lot of people mentioned. This is FTS index. So it is definitely way more blown up. Plus I do save a lot of additional information with it: pwd, session id, shell used, exit codes, whole command obviously. And to support icloud, also additional information for icloud entity id. And now when you point out, 5k per entry is a lot of data. But I am on with that. This information really important for me.

SoftTalker 3y ago

Wow so it looks like I'm in the minority in having shell command history disabled? I have a small number of commands (20) that go into the live history of login shells for convenience, but nothing gets saved when I log out. If there's something I do repeatedly, I make an alias or a function for it.

maleldil 3y ago
I think you have an unusual take on this. Why do you disable history?
- SoftTalker 3y ago
  People's history files are a great place to look for passwords and other secrets, mainly. I suppose that risk could be reduced by having the history file encrypted on disk, but I don't know of any shell that does that (can't honestly say I've really looked though).
  - philsnow 3y ago
    I get a ton of value out of having my shell history available (both for search but also to try to reconstruct steps of what I did yesterday when my memory is hazy) I guess you could set up an entropy scanner and flag history lines that have high entropy, but that might not be enough (low-entropy secrets) and might be bothersome (lots of false positives / things that are technically secrets but that you don't care if they're in your shell history).