5 ms·

I keep my personal notes in an app that I wrote which is backed by SQLite. It resembles a wiki. First, I tried tags. This seemed like a good idea and was a lot

I keep my personal notes in an app that I wrote which is backed by SQLite. It resembles a wiki. First, I tried tags. This seemed like a good idea and was a lot of fun at first, but eventually I got _really_ tired of having to curate tags for all of my notes and boy there were a lot of them! It's not something you can just do once either, because every time you update a note you have to remember to change the list of tags as well. I wound up with many articles whose tags no longer (or never did) match the content. To get myself out of the tag-curating business, I then tried the "folder" idea by separating areas of concerns into namespaces. So, all of the Python-related notes go under the "Python" namespace. All of the database notes into "Databases," and so on. All the did was shift the problem space. Which namespace does the note on SQLAlchemy go in? If an article does not fit into an existing namespace, should I create one for it now, or put it in the root and hope I remember to move it if a second related note comes along? To put it succinctly, my notes do not fit into a DAG. My third iteration dropped both tags and namespaces and implemented the FTS5 search built into SQLite. Two years later, I have no regrets. The only "curation" I have to worry about is giving each note an accurate title and breaking up large notes into smaller ones when it makes sense. Now when I need something I just search for it and it shows up in the results. Google for all of their other faults got one thing extremely right when it came to web content and email: curation is a fine hobby but for Really Getting Shit Done, you index the content well and then just search for it when you need it.

mro_name 5y ago
exactly the same sold me on spotlight search on macos. Have to sign "for all of their other faults got one thing extremely right" as well, sigh.
onion2k 5y ago
Search works brilliantly for your own notes because you know what you're looking for. You wrote the notes. It fails for problems that you can't describe well though, especially if you don't know the technical jargon to search for. Directories and tags can work for that problem because you can easily see relationships like a hierarchy of folders, or what other tags a tagged item has. Tags and folders also act as a prompt to classify and organise things, which occasionaly leads to discovering how things are connected. "How to find stuff" is a space with many solutions to similar looking but subtly different problems.
- xtiansimon 5y ago
  > “You wrote the notes.” HA! In my case I wrote my notes, but don’t remember what’s in ‘em. It’s endemic for learning. folders/tags is a false dichotomy.
telchar 5y ago
That's a nice summary of the problem and a solution I think I can agree with. One quibble is I think your notes could maybe fit into a DAG, but (as I think you meant) not a tree. In practice I don't think there's an easy way to do that - it would mean lots of sym or maybe hard linking and I don't know enough about filesystems to know if that's feasible. But the one area that curation carries a big advantage over search is in browsing. You can do maybe some things with topic modeling and recommendations to allow a kind of browsing, but with searching it's really hard to know whether you have thoroughly covered a part of the space via searching, while with hierarchical curation that is easy. Filling in that last part with a good solution would make searching a no brainer over curation I think, but IMO I don't think most search solutions try to handle that right now.
BoppreH 5y ago
> To put it succinctly, my notes do not fit into a DAG. I think you mean "tree" or "hierarchy"? A DAG would be a superset of tagging, and could place SQLAlchemy under both Python and Databases, as long as you consider the edges oriented (i.e. there's a parent-child relationship). Doesn't solve the tag maintenance issue, though.
- tylerhou 5y ago
  Technically, if their notes can't be represented by a DAG, they also can't be represented by a tree either.
  - jahewson 5y ago
    Right but the point is that the notes, as described, can actually be represented by a DAG. They just can’t be represented by a tree.
em-bee 5y ago
for me, folders are just a limited form of tagging. since i switched from mutt with its folders to sup which uses tags for email, i don't want to look back. first of all, my extensive list of mutt folders was trivially translated to equivalent tags, so i could continue as i was used to. but then, tagging allowed me to create additional tags as i needed them. i didn't tag everything, and sup also indexes all mails and also has search, so in fact i get the advantage of both tags and search (and i can save searches and treat them like virtual tags/folders as well). well, effectively, tags too, are just a specialized form of searching. the one thing i don't see is the problem with tags getting out of sync. yes, it happens. tags do become obsolete, but that is rarely a problem. it just means that i find get more results than i should otherwise. occasionally when i see way to many obsolete tags i go and clean up that particular category of tags. if it is just a few then i ignore them. and if i look at a tag and i don't find what i need, then i search for it, and add the tag when i find it so that next time i'll find it faster, because with more than 2 million emails in my archive search alone is not good enough.
- Quekid5 5y ago
  > for me, folders are just a limited form of tagging. I understand that this is a very subjective thing... but folders offer an exclusion which isn't immediately available via tagging. Tagging is implicitly an OR operation. Now, your search engine might offer ways to add "not X", but it's a tradeoff. It you have highly-compartmentalized bits of info you end up with the problem of "how much do have to explicitly exclude from my search?". It's complicated. Personally, I think we're missing a level of organization somehow.
  - em-bee 5y ago
    not sure i follow. can you give an example how exclusion works? all i see is that without using symlinks i can have only one folder/tag per item. i can choose to do that with tags too if i want, but i don't see how using folders helps with that other than forcing me to do it.
    - zdragnar 5y ago
viraptor 5y ago
> I wound up with many articles whose tags no longer (or never did) match the content. I'm curious about your tagging strategy. Could you show some example of this issue? I feel like my tags are very different since they're barely curated, but I can't imagine ending up with a "wrong" tag. I may miss some and have to add them later, but i can't remember ever removing or changing a tag. That's both in pinboard and in my scanned documents.
akvadrako 5y ago
I feel like you must be implicitly tagging your notes. Let's say you have notes about SQLite. If you don't put "SQLite" in the note, how will you find them?
- iqanq 5y ago
  select * from notes where upper(text) like '%SQLITE%' :P
  - akvadrako 5y ago
    My point is "sqlite" is effectively a tag.
    - guelo 5y ago
      Right. If you search for "database" instead of sqlite it will only show up if you happened to use that word in the notes. If you're searching notes from years ago you might not remember the exact keyword you need. You need a search engine with a powerful enough synonym/grammatical similarity capability.
mason55 5y ago
This is how I tell people to use Confluence. There’s no hope of ever having the general set of all articles organized in a way that makes sense so just try to make the article searchable. If it’s about an error, make sure the error message is in there. If it’s about configuration of a feature make sure the feature name and “configuration” is in the title. Also make sure to write a paragraph describing what the article is about and different times someone might need it. It’s like internal white hat SEO
- smcnally 5y ago
  Making docs discoverable is good, and this works well for retrieving mostly static documentation and tickets. It gets complicated when when resulting items are used for troubleshooting or new documentation and tickets. When results, e.g., include multiple tickets and documents referencing the same error or configuration, which do you use for troubleshooting? Which results are the authoritative / canonical / source of truth for documenting the latest configuration directives? Versioning used well for the input helps. Organization and ontology is additional context for using search results well.
lamontcg 5y ago
> So, all of the Python-related notes go under the "Python" namespace. All of the database notes into "Databases," and so on. All the did was shift the problem space. Which namespace does the note on SQLAlchemy go in? You don't need to have a perfect heirarchy, you just need to have a heirarchy that you'll remember. If you constrain yourself into trying to produce a Liskov free heirarchy you'll tie yourself in knots. I'd probably put SQLAlchemy under Python because it is python-specific. Databases should be kept for your notes on PostgreSQL administration (and you may have python scripts in there but they'll be to serve the purpose of administrating a concrete PostgreSQL install or something. And you don't need to find all the database stuff and collect it together, you just need to know where you put something. So largely put it in the folder that makes the most immediate sense to you and don't overthink it. Later if you hate it, refactor it. But remember that you're dealing with your own personal hash table for retrieval and not trying to implement group_by. The latter issue is where you'd need an index over your notes that was searchable, but I don't find myself needing it.
Buttons840 5y ago
That's a really valuable insight and well said, thanks for sharing. I wonder if tags could work like spam detection. For each tag mark a few things as "tag" and "not tag" and let an AI figure out everything else. If there's a tag you never look at, and so it's poorly trained, who cares? You never look at it. Commonly used tags will be better trained, because you will have marked more things as "yes tag" or "not tag".
tshaddox 5y ago
I like tags which are structured as a tree, so that you can tag something as “Python” and it will also acquire the parent tag “Programming”. When you add a new tag for the first time you still need to eventually curate it by giving it other tags as parents if applicable. I suppose this is roughly equivalent to a directory structure where each item can be in multiple directories at the same time.
charlieyu1 5y ago
How good is the search performance? I have used an a couple of apps that provide full text search, but it is very slow when the number of notes is large. I think the best solution is a mix of 3, folder structure with tags, full text search when needed

mro_name 5y ago

exactly the same sold me on spotlight search on macos. Have to sign "for all of their other faults got one thing extremely right" as well, sigh.

onion2k 5y ago

Search works brilliantly for your own notes because you know what you're looking for. You wrote the notes. It fails for problems that you can't describe well though, especially if you don't know the technical jargon to search for. Directories and tags can work for that problem because you can easily see relationships like a hierarchy of folders, or what other tags a tagged item has. Tags and folders also act as a prompt to classify and organise things, which occasionaly leads to discovering how things are connected. "How to find stuff" is a space with many solutions to similar looking but subtly different problems.

xtiansimon 5y ago
> “You wrote the notes.” HA! In my case I wrote my notes, but don’t remember what’s in ‘em. It’s endemic for learning. folders/tags is a false dichotomy.

telchar 5y ago

That's a nice summary of the problem and a solution I think I can agree with. One quibble is I think your notes could maybe fit into a DAG, but (as I think you meant) not a tree. In practice I don't think there's an easy way to do that - it would mean lots of sym or maybe hard linking and I don't know enough about filesystems to know if that's feasible. But the one area that curation carries a big advantage over search is in browsing. You can do maybe some things with topic modeling and recommendations to allow a kind of browsing, but with searching it's really hard to know whether you have thoroughly covered a part of the space via searching, while with hierarchical curation that is easy. Filling in that last part with a good solution would make searching a no brainer over curation I think, but IMO I don't think most search solutions try to handle that right now.

BoppreH 5y ago

> To put it succinctly, my notes do not fit into a DAG. I think you mean "tree" or "hierarchy"? A DAG would be a superset of tagging, and could place SQLAlchemy under both Python and Databases, as long as you consider the edges oriented (i.e. there's a parent-child relationship). Doesn't solve the tag maintenance issue, though.

tylerhou 5y ago
Technically, if their notes can't be represented by a DAG, they also can't be represented by a tree either.
- jahewson 5y ago
  Right but the point is that the notes, as described, can actually be represented by a DAG. They just can’t be represented by a tree.

em-bee 5y ago

for me, folders are just a limited form of tagging. since i switched from mutt with its folders to sup which uses tags for email, i don't want to look back. first of all, my extensive list of mutt folders was trivially translated to equivalent tags, so i could continue as i was used to. but then, tagging allowed me to create additional tags as i needed them. i didn't tag everything, and sup also indexes all mails and also has search, so in fact i get the advantage of both tags and search (and i can save searches and treat them like virtual tags/folders as well). well, effectively, tags too, are just a specialized form of searching. the one thing i don't see is the problem with tags getting out of sync. yes, it happens. tags do become obsolete, but that is rarely a problem. it just means that i find get more results than i should otherwise. occasionally when i see way to many obsolete tags i go and clean up that particular category of tags. if it is just a few then i ignore them. and if i look at a tag and i don't find what i need, then i search for it, and add the tag when i find it so that next time i'll find it faster, because with more than 2 million emails in my archive search alone is not good enough.

Quekid5 5y ago
> for me, folders are just a limited form of tagging. I understand that this is a very subjective thing... but folders offer an exclusion which isn't immediately available via tagging. Tagging is implicitly an OR operation. Now, your search engine might offer ways to add "not X", but it's a tradeoff. It you have highly-compartmentalized bits of info you end up with the problem of "how much do have to explicitly exclude from my search?". It's complicated. Personally, I think we're missing a level of organization somehow.
- em-bee 5y ago
  not sure i follow. can you give an example how exclusion works? all i see is that without using symlinks i can have only one folder/tag per item. i can choose to do that with tags too if i want, but i don't see how using folders helps with that other than forcing me to do it.
  - zdragnar 5y ago

viraptor 5y ago

> I wound up with many articles whose tags no longer (or never did) match the content. I'm curious about your tagging strategy. Could you show some example of this issue? I feel like my tags are very different since they're barely curated, but I can't imagine ending up with a "wrong" tag. I may miss some and have to add them later, but i can't remember ever removing or changing a tag. That's both in pinboard and in my scanned documents.

akvadrako 5y ago

I feel like you must be implicitly tagging your notes. Let's say you have notes about SQLite. If you don't put "SQLite" in the note, how will you find them?

iqanq 5y ago
select * from notes where upper(text) like '%SQLITE%' :P
- akvadrako 5y ago
  My point is "sqlite" is effectively a tag.
  - guelo 5y ago
    Right. If you search for "database" instead of sqlite it will only show up if you happened to use that word in the notes. If you're searching notes from years ago you might not remember the exact keyword you need. You need a search engine with a powerful enough synonym/grammatical similarity capability.

mason55 5y ago

This is how I tell people to use Confluence. There’s no hope of ever having the general set of all articles organized in a way that makes sense so just try to make the article searchable. If it’s about an error, make sure the error message is in there. If it’s about configuration of a feature make sure the feature name and “configuration” is in the title. Also make sure to write a paragraph describing what the article is about and different times someone might need it. It’s like internal white hat SEO

smcnally 5y ago
Making docs discoverable is good, and this works well for retrieving mostly static documentation and tickets. It gets complicated when when resulting items are used for troubleshooting or new documentation and tickets. When results, e.g., include multiple tickets and documents referencing the same error or configuration, which do you use for troubleshooting? Which results are the authoritative / canonical / source of truth for documenting the latest configuration directives? Versioning used well for the input helps. Organization and ontology is additional context for using search results well.

lamontcg 5y ago

> So, all of the Python-related notes go under the "Python" namespace. All of the database notes into "Databases," and so on. All the did was shift the problem space. Which namespace does the note on SQLAlchemy go in? You don't need to have a perfect heirarchy, you just need to have a heirarchy that you'll remember. If you constrain yourself into trying to produce a Liskov free heirarchy you'll tie yourself in knots. I'd probably put SQLAlchemy under Python because it is python-specific. Databases should be kept for your notes on PostgreSQL administration (and you may have python scripts in there but they'll be to serve the purpose of administrating a concrete PostgreSQL install or something. And you don't need to find all the database stuff and collect it together, you just need to know where you put something. So largely put it in the folder that makes the most immediate sense to you and don't overthink it. Later if you hate it, refactor it. But remember that you're dealing with your own personal hash table for retrieval and not trying to implement group_by. The latter issue is where you'd need an index over your notes that was searchable, but I don't find myself needing it.

Buttons840 5y ago

That's a really valuable insight and well said, thanks for sharing. I wonder if tags could work like spam detection. For each tag mark a few things as "tag" and "not tag" and let an AI figure out everything else. If there's a tag you never look at, and so it's poorly trained, who cares? You never look at it. Commonly used tags will be better trained, because you will have marked more things as "yes tag" or "not tag".

tshaddox 5y ago

I like tags which are structured as a tree, so that you can tag something as “Python” and it will also acquire the parent tag “Programming”. When you add a new tag for the first time you still need to eventually curate it by giving it other tags as parents if applicable. I suppose this is roughly equivalent to a directory structure where each item can be in multiple directories at the same time.

charlieyu1 5y ago

How good is the search performance? I have used an a couple of apps that provide full text search, but it is very slow when the number of notes is large. I think the best solution is a mix of 3, folder structure with tags, full text search when needed