7 ms·

Show HN: Archivy – Self-hosted knowledge base embedded into your filesystem

219 points by etherio 6y ago

jitl 6y ago
I love to see the web-page archiving. Curious about how you see your tool versus Obsidian. Have you checked out Obsidian? Similarly uses markdown and front matter on the local filesystem.
- masukomi 6y ago
  can't speak for the author but: * obsidian is a proprietary piece of software so you can't rely on it for long term archival. There's zero guarantee the company that makes it will exist next year and keep updating it to work with new operating systems. * obsidian doesn't download page contents * obsidian doesn't handle bookmarks in any notable way. While they both use markdown, and can store notes the similarity pretty much ends there.
  - osmarks 6y ago
    It does use flat files for storage, so there's not much lockin there.
    - groby_b 6y ago
      And the flat files are IIRC markdown, so it works just fine for long term archival.
    - masukomi 6y ago
      not true. If you buy into obsidian's features then you loose the visualized graph, a functional task list, all your [[internal links]] break etc... sure, you've got your core data, but now you've got to recreate obsidian if you want it to work the way you've become accustomed to, and presumably like since you would have used it long enough for this to be a concern. exporting your data is great, but it's only the beginning. For example: I can export my data from twitter too. It doesn't mean I still have a functional way to share short text thoughts, or the history of them, with their responses linked to other user's accounts or functional way to show all my tweets with a tag or or or ...
stfwn 6y ago
Cool! I really like the (upcoming) idea of watching your own online profiles on HN and such for upvotes and saving those pages. It’s essentially what I already use those upvotes for in my head, but not what they are in reality.
- etherio 6y ago
  Yeah I'm really excited about that idea of building a digital garden based on the stuff you enjoy online AND your notes.
  - pbronez 6y ago
    I wonder if there’s a reasonable way to extend this to structured quantified self data... Fitbit steps, calorie logs, chat logs, etc. That stuff won’t fit the “markdown with a header” format too well, unfortunately. Would probably need to add a SQLite or DuckDB [0] storage engine. [0] https://duckdb.org/ https://duckdb.org/
clashmeifyoucan 6y ago
Nice idea, it would be interesting if you could integrate the wayback machine so it archives the webpage you add. That way even if the original page gets deleted, a snapshot will be permanently archived.
- masukomi 6y ago
  a) it does better > If you add bookmarks, their webpages contents' will be saved to ensure that you will always have access to it b) It's not inline with the idea of archiving things you care about. If you don't control it you can't rely on it. You definitely can't rely on wayback machine to always exist. Someone's got to keep paying for those servers and it's not a huge profit center, and there have been questions regarding its survival before because of lack of money.
  - clashmeifyoucan 6y ago
    > a) it does better that's not very interesting because when I find an interesting read I rarely bookmark it. For me bookmarks are links that I visit often. > b) that is a fair point, however I think the wayback way is still not too shabby an idea. It ensures there's a copy snapshotted just in case, and yes it might not last forever but it's done a fantastic job so far so not trusting it now just because it'll not survive sounds a minor risk imho.
    - homarp 6y ago
      it's not "browser" bookmark. It's "add a bookmark" into the app see https://github.com/Uzay-G/archivy/blob/master/main/templates/bookmarks/new.html https://github.com/Uzay-G/archivy/blob/master/main/templates... it's just a form where you paste the URL of the site you're interested.
      - masukomi 6y ago
        so? I don't understand why this is a meaningful statement. Lots of us use Pinboard.in or similar "bookmark" services. They aren't "Browser bookmarks" they're just forms in some separate app too. We find them more useful than browser bookmarks. The pasting can easily be worked around with a simple bookmarklet. I'm not sure what point you're trying to make.
siraben 6y ago
Worth noting that this is written by a 15 year old. This looks interesting, nice integration with Elasticsearch as well. However, I see and have tried tools like this several times in the past (tiddlywiki, org-brain, etc.) and haven't been able to stay on it, always reverting to paper, or, more recently, reMarkable tablet notebooks. Is it just me or it requires quite a bit of motivation and dedication to stick to it?
- etherio 6y ago
  Yeah. The idea of paper notebooks that could use ocr to export the paper notebook to a digital notebook would be really cool because digital is more flexible but paper feels more natural.
  - apearson 6y ago
    Like a Livescribe pen?
  - DEADBEEFC0FFEE 6y ago
    There's a cheap product called whitelines. Which might be interesting to you.
    - Geezus_42 6y ago
      I don't understand the saleing point of whitelines. Their app just takes a picture and converts it to a PDF which is the same thing Adobe Scan, cam2scan, and probably a million other apps do except those apps don't require you buy special paper.
- ggrrhh_ta 6y ago
  I wish I had again the intellectual confidence I had when I was 15...
- ahnick 6y ago
  How do you like the reMarkable tablet? What does your workflow with it look like?
Naac 6y ago
A much more complete version of this is tiddlywiki: https://tiddlywiki.com/ https://tiddlywiki.com/
- etherio 6y ago
  I saw this, and it seemed interesting but I had a few problems with it: A) doesn't embrace the same idea of building your own digital garden that holds nt only your notes but also automatically syncs with different services like hn, pocket etc to save your digital presence locally. B) Flexibility of search. Archivy uses elastic search with a neat nlp pipeline to process data and allow it to be searched with accuracy. This is all configured in elastic-search.json and the user can configure it to his needs as he pleases. C) Ease of use and minimal interface. Archivy has a simple direct UI and its goal is NOT to become a note taking app nor does it pretend to. You can directly search at the top and then you have a tree view of your data organised in folders.
  - Naac 6y ago
    > A) doesn't embrace the same idea of building your own digital garden that holds nt only your notes but also automatically syncs with different services like hn, pocket etc to save your digital presence locally. How does something sync with HN? Running tiddlywiki as a service means it's always "in sync". It is easily extensible with browser addons if you want. It most certainly embraces the idea of building your own garden. People have done incredible things with tiddlywiki. >> B) Flexibility of search. Archivy uses elastic search with a neat nlp pipeline to process data and allow it to be searched with accuracy. This is all configured in elastic-search.json and the user can configure it to his needs as he pleases. Elastic search seems like such an overkill for a personal wiki/note taking app. Even for incredibly large wikis. With proper tagging ( and even without ) you will be surprised how fast tiddlywiki search is. And also, for a personal wiki, I would want the least amount of dependencies. I really don't want to have to set up and keep up to date elastic search. >> C) Ease of use and minimal interface. Archivy has a simple direct UI and its goal is NOT to become a note taking app nor does it pretend to. "Archivy is a self-hosted knowledge repository". That's the same goal as tiddlywiki. They both have a simple and minimal UI, although I agree Archivy looks more minimal. But I think that's because Archivy offers much less features.
djhworld 6y ago
Pretty cool. Instead of elastic search you could also use SQLite5 with its full text search support.
- nikisweeting 6y ago
  That + ripgrep is how we're planning to do full-text search in ArchiveBox.io, seems much more appealing than running a full ElasticSearch cluster.
  - cmckn 6y ago
    A grisly death is more appealing than running a full Elasticsearch cluster :)
  - makeworld 6y ago
    Can't wait to see it! Nice combo.
- pbronez 6y ago
  First, I love that this exists. I was playing around with exactly the same concept (text files w/structured front matter as a personal knowledge base) a few months ago, but didn’t get as far. I think the Elasticsearch dependency is overkill, specifically because it precludes deployment on a corporately-managed computer where you can run Python but not Elasticsearch. It would be great to have an option to use a pure python search tool like Woosh [0]. This would trade away some search power for significant portability gains. I might do a fork for this! [0] http://whoosh.readthedocs.io/en/latest/intro.html http://whoosh.readthedocs.io/en/latest/intro.html
  - etherio 6y ago
    I'm actually thinking of making this an option here: https://github.com/Uzay-G/archivy/issues/13 https://github.com/Uzay-G/archivy/issues/13. It'd be nice to have those two choices.
ainiriand 6y ago
I was absolutely coding exactly the same pet project. This one is so much better than mine! Kudos. Very good job.
deleted 6y ago
[deleted]
stakkur 6y ago
I accomplish this with Emacs + Org-roam. And it's a 'forever' file system, all text. https://www.orgroam.com/ https://www.orgroam.com/

219 points by etherio 6y ago

jitl 6y ago

I love to see the web-page archiving. Curious about how you see your tool versus Obsidian. Have you checked out Obsidian? Similarly uses markdown and front matter on the local filesystem.

masukomi 6y ago
can't speak for the author but: * obsidian is a proprietary piece of software so you can't rely on it for long term archival. There's zero guarantee the company that makes it will exist next year and keep updating it to work with new operating systems. * obsidian doesn't download page contents * obsidian doesn't handle bookmarks in any notable way. While they both use markdown, and can store notes the similarity pretty much ends there.
- osmarks 6y ago
  It does use flat files for storage, so there's not much lockin there.
  - groby_b 6y ago
    And the flat files are IIRC markdown, so it works just fine for long term archival.
  - masukomi 6y ago
    not true. If you buy into obsidian's features then you loose the visualized graph, a functional task list, all your [[internal links]] break etc... sure, you've got your core data, but now you've got to recreate obsidian if you want it to work the way you've become accustomed to, and presumably like since you would have used it long enough for this to be a concern. exporting your data is great, but it's only the beginning. For example: I can export my data from twitter too. It doesn't mean I still have a functional way to share short text thoughts, or the history of them, with their responses linked to other user's accounts or functional way to show all my tweets with a tag or or or ...

stfwn 6y ago

Cool! I really like the (upcoming) idea of watching your own online profiles on HN and such for upvotes and saving those pages. It’s essentially what I already use those upvotes for in my head, but not what they are in reality.

etherio 6y ago
Yeah I'm really excited about that idea of building a digital garden based on the stuff you enjoy online AND your notes.
- pbronez 6y ago
  I wonder if there’s a reasonable way to extend this to structured quantified self data... Fitbit steps, calorie logs, chat logs, etc. That stuff won’t fit the “markdown with a header” format too well, unfortunately. Would probably need to add a SQLite or DuckDB [0] storage engine. [0] https://duckdb.org/ https://duckdb.org/

clashmeifyoucan 6y ago

Nice idea, it would be interesting if you could integrate the wayback machine so it archives the webpage you add. That way even if the original page gets deleted, a snapshot will be permanently archived.

masukomi 6y ago
a) it does better > If you add bookmarks, their webpages contents' will be saved to ensure that you will always have access to it b) It's not inline with the idea of archiving things you care about. If you don't control it you can't rely on it. You definitely can't rely on wayback machine to always exist. Someone's got to keep paying for those servers and it's not a huge profit center, and there have been questions regarding its survival before because of lack of money.
- clashmeifyoucan 6y ago
  > a) it does better that's not very interesting because when I find an interesting read I rarely bookmark it. For me bookmarks are links that I visit often. > b) that is a fair point, however I think the wayback way is still not too shabby an idea. It ensures there's a copy snapshotted just in case, and yes it might not last forever but it's done a fantastic job so far so not trusting it now just because it'll not survive sounds a minor risk imho.
  - homarp 6y ago
    it's not "browser" bookmark. It's "add a bookmark" into the app see https://github.com/Uzay-G/archivy/blob/master/main/templates/bookmarks/new.html https://github.com/Uzay-G/archivy/blob/master/main/templates... it's just a form where you paste the URL of the site you're interested.
    - masukomi 6y ago
      so? I don't understand why this is a meaningful statement. Lots of us use Pinboard.in or similar "bookmark" services. They aren't "Browser bookmarks" they're just forms in some separate app too. We find them more useful than browser bookmarks. The pasting can easily be worked around with a simple bookmarklet. I'm not sure what point you're trying to make.

siraben 6y ago

Worth noting that this is written by a 15 year old. This looks interesting, nice integration with Elasticsearch as well. However, I see and have tried tools like this several times in the past (tiddlywiki, org-brain, etc.) and haven't been able to stay on it, always reverting to paper, or, more recently, reMarkable tablet notebooks. Is it just me or it requires quite a bit of motivation and dedication to stick to it?

etherio 6y ago
Yeah. The idea of paper notebooks that could use ocr to export the paper notebook to a digital notebook would be really cool because digital is more flexible but paper feels more natural.
- apearson 6y ago
  Like a Livescribe pen?
- DEADBEEFC0FFEE 6y ago
  There's a cheap product called whitelines. Which might be interesting to you.
  - Geezus_42 6y ago
    I don't understand the saleing point of whitelines. Their app just takes a picture and converts it to a PDF which is the same thing Adobe Scan, cam2scan, and probably a million other apps do except those apps don't require you buy special paper.
ggrrhh_ta 6y ago
I wish I had again the intellectual confidence I had when I was 15...
ahnick 6y ago
How do you like the reMarkable tablet? What does your workflow with it look like?

Naac 6y ago

A much more complete version of this is tiddlywiki: https://tiddlywiki.com/ https://tiddlywiki.com/

etherio 6y ago
I saw this, and it seemed interesting but I had a few problems with it: A) doesn't embrace the same idea of building your own digital garden that holds nt only your notes but also automatically syncs with different services like hn, pocket etc to save your digital presence locally. B) Flexibility of search. Archivy uses elastic search with a neat nlp pipeline to process data and allow it to be searched with accuracy. This is all configured in elastic-search.json and the user can configure it to his needs as he pleases. C) Ease of use and minimal interface. Archivy has a simple direct UI and its goal is NOT to become a note taking app nor does it pretend to. You can directly search at the top and then you have a tree view of your data organised in folders.
- Naac 6y ago
  > A) doesn't embrace the same idea of building your own digital garden that holds nt only your notes but also automatically syncs with different services like hn, pocket etc to save your digital presence locally. How does something sync with HN? Running tiddlywiki as a service means it's always "in sync". It is easily extensible with browser addons if you want. It most certainly embraces the idea of building your own garden. People have done incredible things with tiddlywiki. >> B) Flexibility of search. Archivy uses elastic search with a neat nlp pipeline to process data and allow it to be searched with accuracy. This is all configured in elastic-search.json and the user can configure it to his needs as he pleases. Elastic search seems like such an overkill for a personal wiki/note taking app. Even for incredibly large wikis. With proper tagging ( and even without ) you will be surprised how fast tiddlywiki search is. And also, for a personal wiki, I would want the least amount of dependencies. I really don't want to have to set up and keep up to date elastic search. >> C) Ease of use and minimal interface. Archivy has a simple direct UI and its goal is NOT to become a note taking app nor does it pretend to. "Archivy is a self-hosted knowledge repository". That's the same goal as tiddlywiki. They both have a simple and minimal UI, although I agree Archivy looks more minimal. But I think that's because Archivy offers much less features.

djhworld 6y ago

Pretty cool. Instead of elastic search you could also use SQLite5 with its full text search support.

nikisweeting 6y ago
That + ripgrep is how we're planning to do full-text search in ArchiveBox.io, seems much more appealing than running a full ElasticSearch cluster.
- cmckn 6y ago
  A grisly death is more appealing than running a full Elasticsearch cluster :)
- makeworld 6y ago
  Can't wait to see it! Nice combo.
pbronez 6y ago
First, I love that this exists. I was playing around with exactly the same concept (text files w/structured front matter as a personal knowledge base) a few months ago, but didn’t get as far. I think the Elasticsearch dependency is overkill, specifically because it precludes deployment on a corporately-managed computer where you can run Python but not Elasticsearch. It would be great to have an option to use a pure python search tool like Woosh [0]. This would trade away some search power for significant portability gains. I might do a fork for this! [0] http://whoosh.readthedocs.io/en/latest/intro.html http://whoosh.readthedocs.io/en/latest/intro.html
- etherio 6y ago
  I'm actually thinking of making this an option here: https://github.com/Uzay-G/archivy/issues/13 https://github.com/Uzay-G/archivy/issues/13. It'd be nice to have those two choices.

ainiriand 6y ago

I was absolutely coding exactly the same pet project. This one is so much better than mine! Kudos. Very good job.

deleted 6y ago

[deleted]

stakkur 6y ago

I accomplish this with Emacs + Org-roam. And it's a 'forever' file system, all text. https://www.orgroam.com/ https://www.orgroam.com/