3 ms·
Does this relate to http://elasticlunr.com/ http://elasticlunr.com/? mdBook uses the latter, and I was wondering if there's a similar library for Python with in
by stavros 5y ago
Does this relate to http://elasticlunr.com/ http://elasticlunr.com/? mdBook uses the latter, and I was wondering if there's a similar library for Python with index compatibility so I can provide my own search from them.
- heywhy 5y agoYes, it is a port of that library with some improvements.
- dnautics 5y agoYou might want to consider mapping an index to an ets table-based data structure instead of an immutable object managed by a GenServer, it will give you a way to share it between processes without having to awkwardly copy a potentially huge data structure all over the place.
- heywhy 5y agoI do have thoughts about performance too but I was following the "get it working then make improvements" route :). Thank you for the suggestions.
- dnautics 5y agoLove it. You're doing exactly the right thing.
- kuzee 5y agoThis makes sense, and I think you've taken the correct route. I look forward to trying this in one of my projects and comparing to my current postgres-only backed search strategy. For my use case losing the index between restarts isn't a deal breaker, so hopefully I'll have some useful feedback.
- heywhy 5y agoThat's great. I will be looking forward to this.
- linkdd 5y agoI'd say even using mnesia as an option for disc copies.
- dnautics 5y agomnesia had very difficult to debug consistency issues that can crop up. Have these been fixed?
- skrebbel 5y agoI don't understand how this works. Is data read from ETS somehow shared more efficiently than data shared via a regular message? (which iirc is always copied)
- eproxus 5y agoIt’s still copied, but a process can quickly become a bottleneck in parallel code (every request to a process is sequential). An ETS table can be concurrently read (and tweaked even further for that use case if desired).
- heywhy 5y agoLike eproxus mentioned, it's still been shared through normal process messaging but improvements will be made regarding this.
- dnautics 5y agoIt's still copied but if you are using an ets table you're likely only copying a small subset of the data per query instead of schlepping the whole index every time.
- simonw 5y agoIf you want to build search in Python you can get a very long way using the full text search engine built into SQLite, which is available in the Python standard library. I have some tools for using that here: https://sqlite-utils.datasette.io/en/stable/python-api.html#full-text-search https://sqlite-utils.datasette.io/en/stable/python-api.html#...
- stavros 5y agoThanks, but I didn't explain it well: I don't want to generally build search, I have a static site using mdBook and I want to use the mdBook indexes from Python to generate search results for that site.