3 ms·
My version of this is https://dogsheep.github.io/ https://dogsheep.github.io/ - the idea is to pull your digital footprint from various different sources (Twitt
by simonw 5y ago
My version of this is https://dogsheep.github.io/ https://dogsheep.github.io/ - the idea is to pull your digital footprint from various different sources (Twitter, Foursquare, GitHub etc) into SQLite database files, then run Datasette on top to explore them.
On top of that I built a search engine called Dogsheep Beta which builds a full-text search index across all of the different sources and lets you search in one place: https://github.com/dogsheep/dogsheep-beta https://github.com/dogsheep/dogsheep-beta
You can see a live demonstration of that search engine on the Datasette website: https://datasette.io/-/beta?q=dogsheep https://datasette.io/-/beta?q=dogsheep
The key difference I see with Apollo is that Dogsheep separates fetching of data from search and indexing, and uses SQLite as the storage format. I'm using a YAML configuration to define how the search index should work: https://github.com/simonw/datasette.io/blob/main/templates/dogsheep-beta.yml https://github.com/simonw/datasette.io/blob/main/templates/d... - it defines SQL queries that can be used to build the index from other tables, plus HTML fragments for how those results should be displayed.
- tomcam 5y agoHoly crap you should submit as a Show HN
- simonw 5y agoIt's failed to make the homepage a few times in the past: https://hn.algolia.com/?q=dogsheep https://hn.algolia.com/?q=dogsheep - the one time it did make it was this one about Dogsheep Photos: https://news.ycombinator.com/item?id=23271053 https://news.ycombinator.com/item?id=23271053
- mosselman 5y agoSimon is not an unknown on HN.
- gizdan 5y agoWow! That's super cool. I will have to check this out at some point. Am I correct in understanding that the pocket tool actually imports the URLs contents? If not, how hard would it be to include the actual content of URLs? Specifically, I'll probably end up using something else (for me NextCloud bookmarks).
- simonw 5y agoSadly not - I'd love it to do that, but the Pocket API doesn't make that available. I've been contemplating building an add-on for Dogsheep that can do this for any given URL (from Pocket or other sources) by shelling out to an archive script such as https://github.com/postlight/mercury-parser https://github.com/postlight/mercury-parser - I collected some suggestions for libraries to use here: https://twitter.com/simonw/status/1401656327869394945 https://twitter.com/simonw/status/1401656327869394945 That way you could save a URL using Pocket or browser bookmarks or Pinboard or anything else that I can extract saved URLs from an a separate script could then archive the full contents for you.
- neolog 5y agoSingleFile and SingleFileZ are chrome extensions that export full web pages pretty effectively. https://chrome.google.com/webstore/detail/singlefile/mpiodijhokgodhhofbcjdecpffjipkle https://chrome.google.com/webstore/detail/singlefile/mpiodij... https://chrome.google.com/webstore/detail/singlefilez/offkdfbbigofcgdokjemgjpdockaafjg https://chrome.google.com/webstore/detail/singlefilez/offkdf...
- cxr 5y agoYou might be interested in checking out Perkeep or Zotero.
- ssivark 5y agoAny thoughts on perkeep? https://perkeep.org/ https://perkeep.org/ I see substantial overlap in the vision.
- simonw 5y agoPerkeep feels more like it's solving the long-term archiving problem. Dogsheep doesn't really touch on that much - it's based on the assumption that you'll always be able to pull the data from the original source. It's more about providing ways to explore and analyze (and combine) that data once you've extracted it.
- ssivark 5y agoTrue; but it seems like they might be natural complements, at least in the context of personal use. I guess you’re more focused on use cases like journalism (where a requirement like perkeep might practically become an extra hurdle), but I wonder whether the tools you’re building could be used more generally. Maybe one could create structured SQL tables for metadata of specific types (on perkeep content) to be able to (re)use any analysis tools in dogsheep :-? Either ways, cool project! :-)