3 ms·
Do you just record n-grams indexed against the page url, are you then uploading that index? If you're not uploading it how is there no "local version" available
by pbhjpbhj 12y ago
Do you just record n-grams indexed against the page url, are you then uploading that index? If you're not uploading it how is there no "local version" available?
It's an interesting idea. Personally I have a script that wgets all the pages I bookmark and I very rarely use that content. What use cases are you anticipating?
- gwern 12y agoWhat does indexing mean here? I'm not sure what it means to 'index' but not 'store' webpages as the frontpage says.
- pbhjpbhj 12y agoI assumed they meant that they extracted meta-data about the page (page mentions "Winnie the Pooh" for example, like a book index) and kept that associated with a key (eg URL) but didn't actually store the content per se. I was hoping the answer to my question would include that sort of detail - what's stored and where.
- flippyhead 12y agoThat's exactly right. The indexed content is stored (as an inserted index) but not the plain text.
- boyter 12y agoIt would be useful for pulling back pages that have fallen off the internet and are not in the wayback machine. Thats a use case I have hit a few times. I even started backing up useful posts just in case they died. http://gigablast.com/rants.html http://gigablast.com/rants.html is an example. Its a really good insight into the creation of a search engine which really should be preserved. One that did dissapear but has since come back is this post http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra... which I went looking for a few years ago but had dissapeared from all search indexes.
- ctrijueque 12y agoPinboard offers an archival account (25$ a year) that works wonderfully for this. Take a look if you don't already know it. https://pinboard.in/faq/#archiving https://pinboard.in/faq/#archiving