3 ms·

Thanks. We first grab all browser history from your existing browsers and slurp it into a single sqlite database. We do this sequentially but since it's sqlite-

Thanks. We first grab all browser history from your existing browsers and slurp it into a single sqlite database. We do this sequentially but since it's sqlite-to-sqlite it's usually very qucik. For the full text portion we send out a request to the URL for the full HTML, distill it (similar to "reader" mode in some browsers) and add that distilled content to the full text index. The full-text generation happens with some concurrency but we intentionally didn't want to spam anyone's server so it's limited and takes a while to populate.

nikvdp 5y ago
Yup! Specifically we extract the full text for each website 8 URLs at a time right now, but you can still search over the titles and URLs while the full text is being collected
wuyishan 5y ago
how does the crawling works, if the URLs content is behind logins or similar?
- nikvdp 5y ago
  Hey wuyishan, right now it isn’t able to retrieve the full text of content that’s behind login walls, but we are looking at ways to allow it to handle those sites as well