5 ms·
It already seems to have some nice features, but for me the dream bookmark manager would be something really simple with two commands like: $ bookmark add http
by StreakyCobra 9y ago
It already seems to have some nice features, but for me the dream bookmark manager would be something really simple with two commands like:
$ bookmark add http://.. http://...
That will:
1. Download a static copy of the webpage in a single HTML file, with a PDF exported copy, that also take care of removing adds and unrelated content from the stored content.
2. Run something like http://smmry.com/ http://smmry.com/ to create a summary of the page in few sentences and store it.
3. Use NLP techniques to extract the principle keywords and use them as tags
And another command like:
$ bookmark search "..."
That will:
* Not use regexp or complicated search pattern, but instead;
* Search in titles, tags, AND page content smartly and interactively, and;
* Sort/filter results smartly by relevance, number of matches, frecency, or anything else useful
Storing everything in a git repository or simple file structure for easy synchronization, bonus point for browsers integrations.
- apjana 9y agoNeat idea, we'll surely consider it. Thanks!
- pkaye 9y agoThe autotagging feature would be so useful. I've been looking for something that does it for a long while.
- masukomi 9y agoPinboard.in doesn't "autotag" but it suggests tags and you can click them to have them added to the form you're saving. it's quick and easy.
- seoseokho 9y agoAt Wire, we've actually been working on the "autotagging" (& some more) for the past 2 years. We're focused on mobile and don't have a desktop version, yet. Our current version tags a page based on the keywords in metadata, title; we're still fine tuning it and will include the content in the future. The way Wire works is: instead of the conventional bookmarks, users just saves what they find and use the same search engine they use everyday to find it again. What the user saves is stored for offline viewing. In addition, we've also included an offline p2p feature that allows the user and an nearby offline friend to share what they've saved. Check it out on: https://goo.gl/xMgxfJ https://goo.gl/xMgxfJ
- StreakyCobra 9y agoGlad to hear! I like the HN community, it gather a lot of enthusiastic people!
- masukomi 9y agoexcept for the the exported pdf and ad removal and git you've basically described Pinboard.in I'm pretty sure it searches the content not just the title, tags, and comment you left. It saves a copy of the page so that the site disappearing doesn't mean you've lost the info. It's not downloadable but i don't know why i'd want to backup their backup anyway. It suggests tags / keywords (probably by harvesting the plethora of other people bookmarking things) . and it's got an API so you could make a command line client.
- jbmorgado 9y agoThe thing is, Pinboard can decide to just shut down and there it goes all your saved webpages. Besides, if you want to save the webpages and have full text search, you have to pay the premium package every year forever.
- aw3c2 9y agoa) https://pinboard.in/settings/backup https://pinboard.in/settings/backup b) Do you seriously not consider 25$ per year reasonable for such a useful service?* * I would pay that just to read maciej's fantastic presentations.
- StreakyCobra 9y agoAds removal and "git/simple file structure" are mandatory features for me (PDF export is optional though). But first and foremost, it should be an open-source tool, I forgot to mention it. I don't want to be stuck because (as already said) a company/website closes. Yes backups are possible, but when the tool is gone, it's gone, and backups are not really useful. People will have to spend days trying out to figure a new tool and how to import existing data. Open-source protect from such kind of issues. I don't say this tool is bad, it is probably really nice for some people. But at least for me it does not fulfill my wills/needs.
- benibela 9y agoHow would that deal with pictures?
- StreakyCobra 9y agoI'm not sur at which level you ask this. For storing the images, Base64 encoded in the HTML files. For making them useful, I don't think it's the most important feature, but if need be it is possible to think about using OCR for extracting text, or Deep Learning for describing them [1]. [1] https://research.googleblog.com/2014/11/a-picture-is-worth-thousand-coherent.html https://research.googleblog.com/2014/11/a-picture-is-worth-t...
- asaddhamani 9y agoThe average web page is around 2MB, 60% of which is images [1]. Base64 encoding makes the size even larger. Would that compromise on higher disk usage make sense for you? I like both the OCR and DL ideas. Shameless plug, I've been working on a bookmarking service (https://github.com/crestify/crestify https://github.com/crestify/crestify), and local archiving with images is something we're looking at add. Since it is something a lot of people seem to want, contributors are welcome :). [1] https://www.soasta.com/blog/page-bloat-average-web-page-2-mb/ https://www.soasta.com/blog/page-bloat-average-web-page-2-mb...
- StreakyCobra 9y agoThe system I propose would be for sure space-consuming. Removing the ads and other unneeded contents would probably help to shrink down the page size though. This is the price to pay to keep the bookmarked content available in the long-term, and I'm ready to pay this "price". Assuming 1 bookmark per day during 10 years, this will sum up around 8 Go. Storage is cheep enough nowadays so I can support these 8 Go over 10 years. This will probably not be mobile-friendly, but I'm almost never consulting bookmarks from my phone anyway. Your project seems interesting, thanks for sharing! This may be especially useful to see how you implemented the readability part.
- sergiotapia 9y agoI'll build this. It sounds like a useful and fun project to build. Will build it using Crystal to be able to ship a single binary with no dependencies. SQLite will probably be enough for this project so it'll ship with it's own DB.
- r3bl 9y agoAlso, Mozilla's Readability library[0] should help you out to extract only relevant content (this is what's behind Firefox's reading mode). So, the only semi-difficult part is the NLP. [0] https://github.com/mozilla/readability https://github.com/mozilla/readability
- StreakyCobra 9y agoLet us know the project URL if you start it, so we can follow your work! If I would be doing it, I would probably go with python because of the existing ecosystem: NLP, readability, beautifying, image processing or deep learning, … Probably everything is already existing, the goal would be to assemble the pieces like playing lego :-) in a much more complicated way of course. But I'm not doing it, so your choice is the correct :-) Have a nice time working on this project! PS: Even the image captioning is almost already implemented, amazing https://github.com/tensorflow/models/tree/master/im2txt https://github.com/tensorflow/models/tree/master/im2txt
- elevensies 9y agoI've been thinking along these lines, some other features I'd like: - ability to have certain sites run site-specific extra processing: i.e. youtube-dl youtube links - ability to have a list of sites to be archived periodically instead of once only. And the option to be notified when a site updates. even if it were run as a batch job - ability to ingest a PDF or ebook, identify all the URLs, snapshot all the URLS, present them as a list that links to the original, cached version, the page location - would also be nice if the data could be stored in a human readable structure in a normal filesystem, so your ability to use the data isn't dependent on your ability to run the tool. Overall I think it is an interesting project but the commercial potential is limited. EDIT: maybe the document processing and periodic check thing would make more sense as a higher level tool that depended on the bookmarking tool -- and the extra processing also might make more sense as a plugin type architecture.
- StreakyCobra 9y ago> would also be nice if the data could be stored in a human readable structure in a normal filesystem, so your ability to use the data isn't dependent on your ability to run the tool. This is really important yes! > Overall I think it is an interesting project but the commercial potential is limited. Indeed. This kind of tool is mostly limited to hackers, and it is not a big enough market to create a business model I suppose. This would have to be done for the love of open-source :-)
- apjana 9y agoThere's an option to filter the print output. But yes, currently the data is stored in SQLite only. Yes, buku doesn't intend to be a commercial utility. I use bookmarks as a context pointer for everything I do. So I wrote buku. But it's written as a library so other projects can use it.
- apjana 9y agoYes, buku can be used as a bookmarking engine in a bigger project that also scrapes data. It has REST APIs to do that.
- 9y ago
- dredmorbius 9y agoIf you want other output formats, there's little you can do to improve over pandoc. That will generate ePub, .mobi, DJVU, PDF, PS, and a multitude of other formats, on the fly. HTML is a valid input for most of those. The main problem isn't pandoc, but HTML -- the crap that passes for Web-compatible today is simply any asshat's bad idea. I see as highly useful something which looks at what's been downloaded and reduces it to a hugely simplified structure -- Markdown will almost always be sufficient. I've found, in writing my own strippers and manually re-writing HTML, that body content rarely amounts to more than paragraphs, italic/emphasis, and anchors/hrefs. Better-written content has internal structure via headers. Bold itself is almost never used within CMS systems for body text, it's almost always a tell for advertising or promotional content. The sad truth is that manual rewrite or re-tagging of pages, in Markdown is often the best option I've got for getting to something remotely reasonable. The good news is that that's actually a good tool for reading an article, even if you find that on reading, it's not worth keeping :)
- accordionclown 9y agoi agree with all that. lot of good wisdom there. as for html-to-markdown conversion, http://markdownrules.com http://markdownrules.com is good.