4 ms·
Show HN: Podcastsaver.com – a search engine testbench dressed as a podcast site
Hey HN,
I submitted PodcastSaver (https://podcastsaver.com) before but the reason it's interesting now is that I've started converting it into a live search engine test-bench.
I've discussed it a bit here[0], and this idea has been kicking around in my head for a while so I got a chance to do some related writing about it with Supabase[1].
The basic idea is to use a modest piece of the podcast index[2] as a place to test out different new age search engines against each other.
So far there are two engines running:
- Postgres FTS + pg_trgm (tuned -- indices are there, I did some EXPLAINing earlier today to tighten things up, but still all built-in tech)
- Meilisearch (untuned -- just stand it up, give it resources and put in documents)
To that effect, I've added a "nerds" page you should peruse:
https://podcastsaver.com/nerds
On that page you can:
- choose your search engine
- choose whether to force disable the cache (obviously... you'd want that, for the results to mean anything, but for regular people surfing the cache is on!)
As far as actually getting the podcast search really good, there is a ton of curation left to do so it's a subpar consumer product still, but it's interesting from at least this angle! Going to add more search engines later but who knows when (this project was supposed to be short!).
I can't add every engine on the huge list of new-age search engines[3], but I can say that I will get to highlighting all of them in Awesome F/OSS[4]... Eventually.
[0]: https://news.ycombinator.com/item?id=33316029
[1]: https://supabase.com/blog/postgres-full-text-search-vs-the-rest
[2]: https://podcastindex.org/
[3]: https://news.ycombinator.com/item?id=33317232
[4]: https://awsmfoss.com
- AkshatJ27 4y agoClickable link: https://podcastsaver.com https://podcastsaver.com
- ffhhj 4y agoVery interesting project. Do you plan including other search engines such as Elasticsearch, Solr, Pinecone, Vespa?
- hardwaresofton 4y agoNone of those in particular but see this comment for the ones I do want to add! https://news.ycombinator.com/item?id=33316029 https://news.ycombinator.com/item?id=33316029
- hardwaresofton 4y agoLooks like I didn't give Meilisearch enough lattitude -- It's still ingesting and I didn't give it enough disk: [2022-10-24T20:04:38Z ERROR meilisearch_lib::tasks::update_loop] an error occurred while processing an update batch: Internal error: Quota exceeded (os error 122) Looks like I'll also have to re-feed it but I reduced the batch size so hopefully it's a little happier with that.
- qdequelen 4y agoHello, I'm the Meilisearch CEO, your issues could be because you sent your data without configuring your index, and it's what I read from your comment. Just change your setting to not index URLs. Check out the doc: https://docs.meilisearch.com/reference/api/settings.html#update-searchable-attributes https://docs.meilisearch.com/reference/api/settings.html#upd... With the payload: `["title", "descriptionHTML"]` It will change everything!
- hardwaresofton 4y agoHey thanks for letting me know! I did add a few fields that weren’t directly searched on, because I wanted to be a bit more fair across other search engines (postgres is holding the whole document and the indexes). I’m going to change the configuration and see how that goes. One thing I’d love help with (that would make an awesome recipe section for your docs site) are the best practices around bulk insertion! I couldn’t tell if there was an actual benefit to using addDocuments() vs addDocumentsInBatches().
- qdequelen 4y agoIf you remove the URLs from indexation, it'll generally save a ton of place and will be much, much faster to index. We are thinking about not indexing URLs by default; you can help us by explaining your use case here -> https://github.com/meilisearch/product/discussions/553 https://github.com/meilisearch/product/discussions/553 Just a detail, if you're making a `du -sh` on your computer, the size on the disk will stay unchanged because we are doing soft deletion ;). Don't worry. It will be physically deleted after a while if you need it in the future. If you kept the default configuration of Meilisearch, the maximum size of the HTTP payload is 100Mb (for security). You change it here -> https://docs.meilisearch.com/learn/configuration/instance_options.html#payload-limit-size https://docs.meilisearch.com/learn/configuration/instance_op... addDocumentsInBatches() is just an helper to send your big json array into multiple parts, not absolutely sure you'll need it. (Code -> https://github.com/meilisearch/meilisearch-js/blob/807a6d827b6a14c34481cb9fefcad205d170402d/src/indexes.ts#L411 https://github.com/meilisearch/meilisearch-js/blob/807a6d827...)
- cusx 4y agoThe text are unreadable on the nerds page in Dark Mode. https://podcastsaver.com/nerds https://podcastsaver.com/nerds
- hardwaresofton 4y agoA cardinal sin! Apologies I finished that page JUST before posting, will fix.
- hardwaresofton 4y agoUPDATE: this is fixed now!