5 ms·

This blog is a summary of a fun 1 month adventure I had with Knowledge Management Systems and building a POC that I thought had potential. It was inspired by so

This blog is a summary of a fun 1 month adventure I had with Knowledge Management Systems and building a POC that I thought had potential. It was inspired by so many of the products I see on Hacker News. I’ve open sourced all the code + written down some notes on my insights/ architecture. Just a warning, the code leaves much to be desired as this was a mini project over 2-3 weekends. At the moment, I have 3 key layers of the application. The first is Chrome Extension which allows data collection. It tracks all sorts of data such as attention, time on page, scroll/ click and hover behaviour. It then sends this data to the Electron app. The Electron app receives this data through an API and saves it down locally using PouchDB (data structure is compatible with Apache CouchBD and thus allows easy cloud storage). Within the Electron app, I use an Express/ NodeJS web server to expose endpoints for functionality with the front end/ receiving data from Chrome Extension. For the front end I use ReactJS. If you’re interested, please read the full article! There are lots of videos of the application in action.

xrd 6y ago
Do you then sync to a real CouchDB? Or, do you use PouchDB server on the backend, or otherwise? I'm interested in understanding how you eventually translate to Graph4j and how you do authentication. Where can I look for this? This is really, really cool that you open sourced this. Thank you.
- steve1820 6y ago
  No worries! I'm always leeching from the open source community so I'd thought to give back haha. The visualisation library I'm using is by Ant Design (Alibaba). You can see some examples below from the documentation. https://g6.antv.vision/en/examples/gallery https://g6.antv.vision/en/examples/gallery Re syncing with real CouchDB - I haven't implemented that yet but from what I read, its definitely possible with PouchDB. Re I have a bunch of Express APIs that expose PouchDB so I can do basic CRUD operations such as creating resources etc.
  - xrd 6y ago
    It is definitely possible to sync with CouchDB. I've just found that the uncertainty around how to do authz/authn (CouchDB recommends, now, that you do this in your own proxy layer) and the difficulties to make sure the entire process works, makes debugging tricky. I've tried with a variety of NodeJS+CouchDB and PouchDB-server and nothing gives me a solution which I don't have to babysit quite a bit.
ydj 6y ago
This is neat. I had a similar idea of tracking what we read online in the browser. In addition to just recording time spent and links followed, I would also archive every article read to automatically build up a personal library of the articles themselves. Additionally, light weight note taking on the articles themselves ala hypothe.is Unfortunately a new baby has drained all my time for such pursuits. Hope you develop your concept further!
- steve1820 6y ago
  Awesome that we came up with similar ideas! So right now, the Chrome Extension tracks all link clicks as well so the visualisation actually builds relationships between articles/ blogs/ anything else you read on the internet. I also had another idea that once we build our Knowledge Maps, we should be able to compare and share it will one another. For instance, I'd love to see what our software engineers around my age/ experience are reading and the insights they are drawing from articles. Obviously some sort of security/ privacy mechanism will need to be implemented as well.
steve1820 6y ago
Forgot to add: It also tracks what you click so it can build relationships between articles/ blogs/ anything else you read on the internet. For example, if I’m reading an article about big data/ ETL pipelines and then I click on a link from within the article to a resource on machine learning, this relationship should be tracked and digested.
TrailMarker 6y ago
Built something like that 5 years ago https://twitter.com/MindDriveCo/status/668518135880916992 https://twitter.com/MindDriveCo/status/668518135880916992 Have been working ever since to address many of the issues discussed in this thread and lot more. On the way to turning Engelbart's Conceptual Framework https://www.dougengelbart.org/content/view/138 https://www.dougengelbart.org/content/view/138 for Augmenting Human Intellect into a Practical (meta) Framework. "Go Meta Young Man' and join the effort to build a Kernel for Open Collective Intellect on the Decentralized Web. It is not a commercial venture, but wanting to contribute to Web 3 something comparable in impact to Ward Cunningham's Wiki, something that have the potential to trully change the workld as Ward Cunningham idea of the Wiki had. Primarily Enduring, Open, Co-evolvable, Bootstrapable, metadesignable, promoting edge-user autonomy, uneclosable collaboration and much more https://www.researchgate.net/publication/334126329_Weaving_a_Decentralized_Semantic_Web_of_Personal_Knowledge https://www.researchgate.net/publication/334126329_Weaving_a...
- steve1820 6y ago
  I'll have a read...very interesting!
ErikBjare 6y ago
This is a really cool use of browser history data. I'm building ActivityWatch [1] and have been thinking of building something similar on top of it. [1]: https://activitywatch.net https://activitywatch.net

xrd 6y ago

Do you then sync to a real CouchDB? Or, do you use PouchDB server on the backend, or otherwise? I'm interested in understanding how you eventually translate to Graph4j and how you do authentication. Where can I look for this? This is really, really cool that you open sourced this. Thank you.

steve1820 6y ago
No worries! I'm always leeching from the open source community so I'd thought to give back haha. The visualisation library I'm using is by Ant Design (Alibaba). You can see some examples below from the documentation. https://g6.antv.vision/en/examples/gallery https://g6.antv.vision/en/examples/gallery Re syncing with real CouchDB - I haven't implemented that yet but from what I read, its definitely possible with PouchDB. Re I have a bunch of Express APIs that expose PouchDB so I can do basic CRUD operations such as creating resources etc.
- xrd 6y ago
  It is definitely possible to sync with CouchDB. I've just found that the uncertainty around how to do authz/authn (CouchDB recommends, now, that you do this in your own proxy layer) and the difficulties to make sure the entire process works, makes debugging tricky. I've tried with a variety of NodeJS+CouchDB and PouchDB-server and nothing gives me a solution which I don't have to babysit quite a bit.

ydj 6y ago

This is neat. I had a similar idea of tracking what we read online in the browser. In addition to just recording time spent and links followed, I would also archive every article read to automatically build up a personal library of the articles themselves. Additionally, light weight note taking on the articles themselves ala hypothe.is Unfortunately a new baby has drained all my time for such pursuits. Hope you develop your concept further!

steve1820 6y ago
Awesome that we came up with similar ideas! So right now, the Chrome Extension tracks all link clicks as well so the visualisation actually builds relationships between articles/ blogs/ anything else you read on the internet. I also had another idea that once we build our Knowledge Maps, we should be able to compare and share it will one another. For instance, I'd love to see what our software engineers around my age/ experience are reading and the insights they are drawing from articles. Obviously some sort of security/ privacy mechanism will need to be implemented as well.

steve1820 6y ago

Forgot to add: It also tracks what you click so it can build relationships between articles/ blogs/ anything else you read on the internet. For example, if I’m reading an article about big data/ ETL pipelines and then I click on a link from within the article to a resource on machine learning, this relationship should be tracked and digested.

TrailMarker 6y ago

Built something like that 5 years ago https://twitter.com/MindDriveCo/status/668518135880916992 https://twitter.com/MindDriveCo/status/668518135880916992 Have been working ever since to address many of the issues discussed in this thread and lot more. On the way to turning Engelbart's Conceptual Framework https://www.dougengelbart.org/content/view/138 https://www.dougengelbart.org/content/view/138 for Augmenting Human Intellect into a Practical (meta) Framework. "Go Meta Young Man' and join the effort to build a Kernel for Open Collective Intellect on the Decentralized Web. It is not a commercial venture, but wanting to contribute to Web 3 something comparable in impact to Ward Cunningham's Wiki, something that have the potential to trully change the workld as Ward Cunningham idea of the Wiki had. Primarily Enduring, Open, Co-evolvable, Bootstrapable, metadesignable, promoting edge-user autonomy, uneclosable collaboration and much more https://www.researchgate.net/publication/334126329_Weaving_a_Decentralized_Semantic_Web_of_Personal_Knowledge https://www.researchgate.net/publication/334126329_Weaving_a...

steve1820 6y ago
I'll have a read...very interesting!

ErikBjare 6y ago

This is a really cool use of browser history data. I'm building ActivityWatch [1] and have been thinking of building something similar on top of it. [1]: https://activitywatch.net https://activitywatch.net