6 ms·

Show HN: Stork: A customizable, WASM-powered full-text search plugin for the web

137 points by jil 6y ago

amelius 6y ago
I hate it when stuff is marketed as magic, like in this case "impossibly fast web search". It's like marketeers aren't even trying to cover up their lies anymore.
- skulk 6y ago
  IIUC, it's "impossibly fast" if you assume the search -> see results flow requires a network round-trip (a very common assumption).
  - StavrosK 6y ago
    It's actually impossibly slow for me, as it didn't work at all on Firefox or Chrome. Chrome showed a progress bar but no results when searching for the recommended "liberty".
- adewinter 6y ago
  If you look elsewhere in the thread it seems like a solo individual worked on this for years and posted it here today. Your comment is needlessly negative and mean.
bobajeff 6y ago
I'll keep my eye on this. It sounds like this runs in the browser itself. Right now I'm using meilisearch for a project and would prefer to not use up a http port for the search engine. One thing I'll need is fuzzy search though.
- jil 6y ago
  Fuzzy search is definitely on my list! Keep an eye out in the near future...
- xemoka 6y ago
  Fuse.js (https://fusejs.io/ https://fusejs.io/ ) has been super simple for client side search for me, I’ve used it in a couple projects now.
somishere 6y ago
Like the idea but nothing happens when I type in the search box using ff85 (beta 4).
- remram 6y ago
  I get nothing either, on stable Firefox (84).
  - jil 6y ago
    Oh no! If it's any consolation, I think it might be the site itself, not the search plugin... the site has been acting up in unrelated ways. The site desperately needs a rewrite, but I didn't have time before this launch date. Refreshing might work, but no promises? Or you could try my personal site's search bar: https://jameslittle.me/search/ https://jameslittle.me/search/
    - remram 6y ago
      That other site works, your suspicion is probably correct.
      - brundolf 6y ago
        Same here- Firefox 84, search doesn't work on the OP but does work on the link above
        jil 6y ago
        Apologies for this! Someone's very kindly put up a PR[0] that I suspect will fix this, but it's past my bedtime so I won't get a chance to review/merge/relaunch it until tomorrow. [0]: https://github.com/jameslittle230/stork/pull/93 https://github.com/jameslittle230/stork/pull/93
    - ficklepickle 6y ago
      Same for me, Brave on Android. Your personal site works though.
jabo 6y ago
OP - congratulations on shipping! WASM-powered search caught my eye. Looks like the search index is downloaded and used locally in the browser, so this is as fast as search can get. One trade-off though is that you're limited to relatively small datasets. While this shouldn't be an issue for small-medium static sites, an index that needs to be downloaded over the wire will affect your page performance for larger sites / datasets. In those cases, you'd want to use a dedicated search server like Algolia or its open source alternative Typesense [1]. Both of them store the index server-side and only return the relevant records that are searched for. For eg: you'd probably not want to download 2M records over the wire to search through them locally [2]. You'd be better off storing the index server-side. [1] https://github.com/typesense/typesense https://github.com/typesense/typesense (shameless plug) [2] https://recipe-search.typesense.org/ https://recipe-search.typesense.org/
- t0astbread 6y ago
  Just a quick non-thought-through idea but would it be possible to build an index in a way that allows clients to download only parts of it based on what they search? I.e. the search client normalizes the query in some way and then requests only the relevant index pages. The index would probably be a lot larger on the server but if disk space is not an issue...? (Though at some point you have to ask yourself what the benefits of such an approach are compared to a search server.)
  - jil 6y ago
    I keep wanting to try that! Sharding the index would be really cool, and would definitely allow for a seemingly infinite index size. The barrier, as you were getting to, would be making sure index shard downloads could keep up with typing speed. Definitely something I'll try experimenting with in the future, so... watch this space, perhaps?
  - lgierth 6y ago
    Merkle Search Trees: Efficient State-Based CRDTs in Open Networks https://hal.inria.fr/hal-02303490/document https://hal.inria.fr/hal-02303490/document Peer-to-Peer Ordered Search Indexes https://0fps.net/2020/12/19/peer-to-peer-ordered-search-indexes/ https://0fps.net/2020/12/19/peer-to-peer-ordered-search-inde... (which adds useful context about the above) Sorry for just dropping these links, I should already be asleep :)
jil 6y ago
I've been working on Stork for just over a year! I recently wrote a blog post about what building it has been like -- if you're interested in the project, definitely check the blog post out! https://jameslittle.me/blog/2020/one-year-of-stork/ https://jameslittle.me/blog/2020/one-year-of-stork/
- deleted 6y ago
  [deleted]
Scaevolus 6y ago
How much does WASM matter for performance, or is it more about the convenience of only having to write Rust?
Scaevolus 6y ago
Does the query engine support any boolean operations, or is it just finding documents that contain every (stemmed) input word?
azangru 6y ago
How does it compare to lunr.js?
mceachen 6y ago
Interesting project! The fact that I have to fetch the stork.js from your CDN, and a random wasm blob gets pulled in as well, is unfortunately a deal killer for adoption, at least for me. A pointer to how to self-host might help address this.
- kodah 6y ago
  I'm visiting my parents on a low speed connection and it didn't work at first. I had to clear the cache and restart. I don't know if it's related to this, but I suspect so.
- jil 6y ago
  Self-hosted Stork is definitely possible, and definitely something I want to build! I've been focusing on a different type of user for the 1.0.0 (the user who wants to paste in a single script tag), and there are still some last experience improvements I want to make for those folks, but I completely recognize the value of self-hosting code like this and want to build something into Stork that will fill that niche, as well. For now (and I haven't tried this for self-hosting, so use at your own risk) -- the whole project (incl. the final JS file and the WASM blob) can be built from source. Tweaking the build a little bit would probably get you most of the way there.
barkingcat 6y ago
Can stork.js be served locally rather from the domain https://files.stork-search.net https://files.stork-search.net ?
tmpfs 6y ago
I have been following and using this for a while and it works great for small to medium sites. I built from source and self-host. Congratulations on gettiing it launched James!
ptrik 6y ago
Similar project: https://endler.dev/2019/tinysearch/ https://endler.dev/2019/tinysearch/
rcarmo 6y ago
Nice. I wonder how it works with around 8000 pages of content, will give it a whirl in a few weeks.

137 points by jil 6y ago

I hate it when stuff is marketed as magic, like in this case "impossibly fast web search". It's like marketeers aren't even trying to cover up their lies anymore.

skulk 6y ago
IIUC, it's "impossibly fast" if you assume the search -> see results flow requires a network round-trip (a very common assumption).
- StavrosK 6y ago
  It's actually impossibly slow for me, as it didn't work at all on Firefox or Chrome. Chrome showed a progress bar but no results when searching for the recommended "liberty".
adewinter 6y ago
If you look elsewhere in the thread it seems like a solo individual worked on this for years and posted it here today. Your comment is needlessly negative and mean.

bobajeff 6y ago

I'll keep my eye on this. It sounds like this runs in the browser itself. Right now I'm using meilisearch for a project and would prefer to not use up a http port for the search engine. One thing I'll need is fuzzy search though.

jil 6y ago
Fuzzy search is definitely on my list! Keep an eye out in the near future...
xemoka 6y ago
Fuse.js (https://fusejs.io/ https://fusejs.io/ ) has been super simple for client side search for me, I’ve used it in a couple projects now.

somishere 6y ago

Like the idea but nothing happens when I type in the search box using ff85 (beta 4).

remram 6y ago
I get nothing either, on stable Firefox (84).
- jil 6y ago
  Oh no! If it's any consolation, I think it might be the site itself, not the search plugin... the site has been acting up in unrelated ways. The site desperately needs a rewrite, but I didn't have time before this launch date. Refreshing might work, but no promises? Or you could try my personal site's search bar: https://jameslittle.me/search/ https://jameslittle.me/search/
  - remram 6y ago
    That other site works, your suspicion is probably correct.
    - brundolf 6y ago
      Same here- Firefox 84, search doesn't work on the OP but does work on the link above
      - jil 6y ago
        Apologies for this! Someone's very kindly put up a PR[0] that I suspect will fix this, but it's past my bedtime so I won't get a chance to review/merge/relaunch it until tomorrow. [0]: https://github.com/jameslittle230/stork/pull/93 https://github.com/jameslittle230/stork/pull/93
  - ficklepickle 6y ago
    Same for me, Brave on Android. Your personal site works though.

jabo 6y ago

OP - congratulations on shipping! WASM-powered search caught my eye. Looks like the search index is downloaded and used locally in the browser, so this is as fast as search can get. One trade-off though is that you're limited to relatively small datasets. While this shouldn't be an issue for small-medium static sites, an index that needs to be downloaded over the wire will affect your page performance for larger sites / datasets. In those cases, you'd want to use a dedicated search server like Algolia or its open source alternative Typesense [1]. Both of them store the index server-side and only return the relevant records that are searched for. For eg: you'd probably not want to download 2M records over the wire to search through them locally [2]. You'd be better off storing the index server-side. [1] https://github.com/typesense/typesense https://github.com/typesense/typesense (shameless plug) [2] https://recipe-search.typesense.org/ https://recipe-search.typesense.org/

t0astbread 6y ago
Just a quick non-thought-through idea but would it be possible to build an index in a way that allows clients to download only parts of it based on what they search? I.e. the search client normalizes the query in some way and then requests only the relevant index pages. The index would probably be a lot larger on the server but if disk space is not an issue...? (Though at some point you have to ask yourself what the benefits of such an approach are compared to a search server.)
- jil 6y ago
  I keep wanting to try that! Sharding the index would be really cool, and would definitely allow for a seemingly infinite index size. The barrier, as you were getting to, would be making sure index shard downloads could keep up with typing speed. Definitely something I'll try experimenting with in the future, so... watch this space, perhaps?
- lgierth 6y ago
  Merkle Search Trees: Efficient State-Based CRDTs in Open Networks https://hal.inria.fr/hal-02303490/document https://hal.inria.fr/hal-02303490/document Peer-to-Peer Ordered Search Indexes https://0fps.net/2020/12/19/peer-to-peer-ordered-search-indexes/ https://0fps.net/2020/12/19/peer-to-peer-ordered-search-inde... (which adds useful context about the above) Sorry for just dropping these links, I should already be asleep :)

jil 6y ago

I've been working on Stork for just over a year! I recently wrote a blog post about what building it has been like -- if you're interested in the project, definitely check the blog post out! https://jameslittle.me/blog/2020/one-year-of-stork/ https://jameslittle.me/blog/2020/one-year-of-stork/

deleted 6y ago
[deleted]

Scaevolus 6y ago

How much does WASM matter for performance, or is it more about the convenience of only having to write Rust?

Scaevolus 6y ago

Does the query engine support any boolean operations, or is it just finding documents that contain every (stemmed) input word?

azangru 6y ago

How does it compare to lunr.js?

mceachen 6y ago

Interesting project! The fact that I have to fetch the stork.js from your CDN, and a random wasm blob gets pulled in as well, is unfortunately a deal killer for adoption, at least for me. A pointer to how to self-host might help address this.

kodah 6y ago
I'm visiting my parents on a low speed connection and it didn't work at first. I had to clear the cache and restart. I don't know if it's related to this, but I suspect so.
jil 6y ago
Self-hosted Stork is definitely possible, and definitely something I want to build! I've been focusing on a different type of user for the 1.0.0 (the user who wants to paste in a single script tag), and there are still some last experience improvements I want to make for those folks, but I completely recognize the value of self-hosting code like this and want to build something into Stork that will fill that niche, as well. For now (and I haven't tried this for self-hosting, so use at your own risk) -- the whole project (incl. the final JS file and the WASM blob) can be built from source. Tweaking the build a little bit would probably get you most of the way there.

barkingcat 6y ago

Can stork.js be served locally rather from the domain https://files.stork-search.net https://files.stork-search.net ?

tmpfs 6y ago

I have been following and using this for a while and it works great for small to medium sites. I built from source and self-host. Congratulations on gettiing it launched James!

ptrik 6y ago

Similar project: https://endler.dev/2019/tinysearch/ https://endler.dev/2019/tinysearch/

rcarmo 6y ago

Nice. I wonder how it works with around 8000 pages of content, will give it a whirl in a few weeks.