5 ms·

Pagefind: Static Low-Bandwidth Search at Scale

76 points by chmaynard 4y ago

londons_explore 4y ago
So.... A complete index of my site is too large... So I could chunk the index up... For example, one index for words beginning with each letter of the alphabet. I could host the index for words beginning with 'a' at mysite.com/search/a But I could do yet smaller indexes! I could make one index for each word. Ie. use the URL mysite.com/search/abacus We could call these mini-indexes search result pages...
- kevincox 4y ago
  It's not quite that simple because the user can search for more than one word. Generating all searches for one or two words is probably feasible but not much more than that. After that you probably do want to just generate one index per word and merge them together at search time to dynamically generate the search result page.
- liambigelow 4y ago
  That is, in essence, what Pagefind is doing under the hood — but Pagefind also addresses the other reply in that it can combine these results when you search multiple words (among many other search benefits)
fanf2 4y ago
I recently deployed pagefind on my web site and it was pleasingly easy, and the result is reasonably nippy https://dotat.at/search.html https://dotat.at/search.html
xSxY3fj5gVCmvWE 4y ago
Nice, thanks! A while ago I was looking for a a search lib precisely like this for my blog. I really like the idea of a client-side search - it feels extremely responsive to use compared to server side. The bandwidth-saving chunking tech is a nice bonus that I didn't expect - the websites I've used seem to naively load the entire index.
fmajid 4y ago
Five years ago when I switched from WO to Hugo I wrote my own solution based on SQLite’s fts5 full-text search and using Hugo itself so the results match the theme of your site: https://github.com/fazalmajid/fts5index https://github.com/fazalmajid/fts5index
lloydatkinson 4y ago
I wonder how this compares to algolia? I recently added that to an open source project I released recently and the search results aren’t just not relevant they aren’t even close to correct. It works by first indexing your site by scraping it. Not sure why everyone seems to use it.
- liambigelow 4y ago
  I wrote out an explainer recently on HN answering exactly that: https://news.ycombinator.com/item?id=32172290 https://news.ycombinator.com/item?id=32172290
tbran 4y ago
This looks fantastic! I've used Lunr [0] (a client-side search engine), but as noted in the article, the index is huge and search is quite slow for a site with a lot of content. Splitting the index up is a great idea. Thank you! [0]: https://lunrjs.com/ https://lunrjs.com/
jacooper 4y ago
Im working on a Hugo website, and it already includes a search engine. Does Hugo include a search index ? And If so, is PageFind that much better? The theme I'm trying hugo-theme-stack.

76 points by chmaynard 4y ago

So.... A complete index of my site is too large... So I could chunk the index up... For example, one index for words beginning with each letter of the alphabet. I could host the index for words beginning with 'a' at mysite.com/search/a But I could do yet smaller indexes! I could make one index for each word. Ie. use the URL mysite.com/search/abacus We could call these mini-indexes search result pages...

kevincox 4y ago
It's not quite that simple because the user can search for more than one word. Generating all searches for one or two words is probably feasible but not much more than that. After that you probably do want to just generate one index per word and merge them together at search time to dynamically generate the search result page.
liambigelow 4y ago
That is, in essence, what Pagefind is doing under the hood — but Pagefind also addresses the other reply in that it can combine these results when you search multiple words (among many other search benefits)

fanf2 4y ago

I recently deployed pagefind on my web site and it was pleasingly easy, and the result is reasonably nippy https://dotat.at/search.html https://dotat.at/search.html

xSxY3fj5gVCmvWE 4y ago

Nice, thanks! A while ago I was looking for a a search lib precisely like this for my blog. I really like the idea of a client-side search - it feels extremely responsive to use compared to server side. The bandwidth-saving chunking tech is a nice bonus that I didn't expect - the websites I've used seem to naively load the entire index.

fmajid 4y ago

Five years ago when I switched from WO to Hugo I wrote my own solution based on SQLite’s fts5 full-text search and using Hugo itself so the results match the theme of your site: https://github.com/fazalmajid/fts5index https://github.com/fazalmajid/fts5index

lloydatkinson 4y ago

I wonder how this compares to algolia? I recently added that to an open source project I released recently and the search results aren’t just not relevant they aren’t even close to correct. It works by first indexing your site by scraping it. Not sure why everyone seems to use it.

liambigelow 4y ago
I wrote out an explainer recently on HN answering exactly that: https://news.ycombinator.com/item?id=32172290 https://news.ycombinator.com/item?id=32172290

tbran 4y ago

This looks fantastic! I've used Lunr [0] (a client-side search engine), but as noted in the article, the index is huge and search is quite slow for a site with a lot of content. Splitting the index up is a great idea. Thank you! [0]: https://lunrjs.com/ https://lunrjs.com/

jacooper 4y ago

Im working on a Hugo website, and it already includes a search engine. Does Hugo include a search index ? And If so, is PageFind that much better? The theme I'm trying hugo-theme-stack.