6 ms·

I've been thinking along these lines, some other features I'd like: - ability to have certain sites run site-specific extra processing: i.e. youtube-dl youtube

I've been thinking along these lines, some other features I'd like: - ability to have certain sites run site-specific extra processing: i.e. youtube-dl youtube links - ability to have a list of sites to be archived periodically instead of once only. And the option to be notified when a site updates. even if it were run as a batch job - ability to ingest a PDF or ebook, identify all the URLs, snapshot all the URLS, present them as a list that links to the original, cached version, the page location - would also be nice if the data could be stored in a human readable structure in a normal filesystem, so your ability to use the data isn't dependent on your ability to run the tool. Overall I think it is an interesting project but the commercial potential is limited. EDIT: maybe the document processing and periodic check thing would make more sense as a higher level tool that depended on the bookmarking tool -- and the extra processing also might make more sense as a plugin type architecture.

StreakyCobra 9y ago
> would also be nice if the data could be stored in a human readable structure in a normal filesystem, so your ability to use the data isn't dependent on your ability to run the tool. This is really important yes! > Overall I think it is an interesting project but the commercial potential is limited. Indeed. This kind of tool is mostly limited to hackers, and it is not a big enough market to create a business model I suppose. This would have to be done for the love of open-source :-)
- apjana 9y ago
  There's an option to filter the print output. But yes, currently the data is stored in SQLite only. Yes, buku doesn't intend to be a commercial utility. I use bookmarks as a context pointer for everything I do. So I wrote buku. But it's written as a library so other projects can use it.
apjana 9y ago
Yes, buku can be used as a bookmarking engine in a bigger project that also scrapes data. It has REST APIs to do that.
dredmorbius 9y ago
On the filesystem, I'm thinking of some sort of (possibly virtual?) document filesystem structured by: * Title * Author * Subject * Date * Other? The problem with a fixed / structured filesystem is that it's largely inflexible. The next option would be to have some sort of hybrid -- a persistent data store plus, say, hardlinks or symlinks onto that store based on other elements, could work, but would be somewhat annoying to maintain (though not impossible, and a tools-based approach might well work). The advantage to a filesystem-based approach is that you'd be able to use any filesystem-based tools on it: find, grep, ls, cd, etc. This also points to the distinct limitations of filesystems and naming conventions for document-oriented systems, generally. They're OK for statically-defined computer systems. Sometimes (/proc, /sys, /dev, /dbus, ... are all exceptions for which virtualising the filesystem is a current fix), but when it comes to the human realm, the oversimplification and reliance on standards creates pain. A related problem is: how do you identify a given work, reasonably uniquely and reasonably persistently across variants? Taking a content hash works for git, but doesn't apply particularly well to a human-readable document where whitespace, casing, characterset, character substitutions (straight vs. curly quotes, "-" and "--" vs. en and em dash, etc.), translations, multiple output formats, etc., might all create unique (and unrelatable) hash fingerprints. In catologues, some tuple of author, title, and publication date generally suffices, and creates the general outline of a unique, but relatable, identity foundation. A book might have multiple editions or publishers, or multiple formats (ISDN relates these to the core work). Sub-parts (a chapter or section) might be included into other works. Etc. I'd really like to have the option, say, of creating and relating: * The source document. * Some standardised markup (Markdown, LaTeX, some BDSM HTML5 or similar format, etc.). * Generated outputs (PDF, PS, ePub, DJVU, Mobi, ASCII, etc.) * Metadata: Author, title, publisher, publication, date, URL, language, various identifiers (LoCCN, ISBN, DOI, ...), etc., subject(s), rating(s), review(s), citation(s). For research, this could be highly invaluable.
- StreakyCobra 9y ago
  > The problem with a fixed / structured filesystem is that it's largely inflexible. The next option would be to have some sort of hybrid -- a persistent data store plus, say, hardlinks or symlinks onto that store based on other elements, could work, but would be somewhat annoying to maintain (though not impossible, and a tools-based approach might well work). What your are describing here is called a semantic file system, and there are some implementations of this, like for instance https://www.tagsistant.net/ https://www.tagsistant.net/

StreakyCobra 9y ago

> would also be nice if the data could be stored in a human readable structure in a normal filesystem, so your ability to use the data isn't dependent on your ability to run the tool. This is really important yes! > Overall I think it is an interesting project but the commercial potential is limited. Indeed. This kind of tool is mostly limited to hackers, and it is not a big enough market to create a business model I suppose. This would have to be done for the love of open-source :-)

apjana 9y ago
There's an option to filter the print output. But yes, currently the data is stored in SQLite only. Yes, buku doesn't intend to be a commercial utility. I use bookmarks as a context pointer for everything I do. So I wrote buku. But it's written as a library so other projects can use it.

apjana 9y ago

Yes, buku can be used as a bookmarking engine in a bigger project that also scrapes data. It has REST APIs to do that.

dredmorbius 9y ago

On the filesystem, I'm thinking of some sort of (possibly virtual?) document filesystem structured by: * Title * Author * Subject * Date * Other? The problem with a fixed / structured filesystem is that it's largely inflexible. The next option would be to have some sort of hybrid -- a persistent data store plus, say, hardlinks or symlinks onto that store based on other elements, could work, but would be somewhat annoying to maintain (though not impossible, and a tools-based approach might well work). The advantage to a filesystem-based approach is that you'd be able to use any filesystem-based tools on it: find, grep, ls, cd, etc. This also points to the distinct limitations of filesystems and naming conventions for document-oriented systems, generally. They're OK for statically-defined computer systems. Sometimes (/proc, /sys, /dev, /dbus, ... are all exceptions for which virtualising the filesystem is a current fix), but when it comes to the human realm, the oversimplification and reliance on standards creates pain. A related problem is: how do you identify a given work, reasonably uniquely and reasonably persistently across variants? Taking a content hash works for git, but doesn't apply particularly well to a human-readable document where whitespace, casing, characterset, character substitutions (straight vs. curly quotes, "-" and "--" vs. en and em dash, etc.), translations, multiple output formats, etc., might all create unique (and unrelatable) hash fingerprints. In catologues, some tuple of author, title, and publication date generally suffices, and creates the general outline of a unique, but relatable, identity foundation. A book might have multiple editions or publishers, or multiple formats (ISDN relates these to the core work). Sub-parts (a chapter or section) might be included into other works. Etc. I'd really like to have the option, say, of creating and relating: * The source document. * Some standardised markup (Markdown, LaTeX, some BDSM HTML5 or similar format, etc.). * Generated outputs (PDF, PS, ePub, DJVU, Mobi, ASCII, etc.) * Metadata: Author, title, publisher, publication, date, URL, language, various identifiers (LoCCN, ISBN, DOI, ...), etc., subject(s), rating(s), review(s), citation(s). For research, this could be highly invaluable.

StreakyCobra 9y ago
> The problem with a fixed / structured filesystem is that it's largely inflexible. The next option would be to have some sort of hybrid -- a persistent data store plus, say, hardlinks or symlinks onto that store based on other elements, could work, but would be somewhat annoying to maintain (though not impossible, and a tools-based approach might well work). What your are describing here is called a semantic file system, and there are some implementations of this, like for instance https://www.tagsistant.net/ https://www.tagsistant.net/