6 ms·
Not sure, but won't argue that vehemently. I build my own crawler in a few days using pgvector, go-colly, and some embedding. It's a bit over-engineered because
by RamblingCTO 3y ago
Not sure, but won't argue that vehemently. I build my own crawler in a few days using pgvector, go-colly, and some embedding. It's a bit over-engineered because of go-python interop, but I'm happy with that. And with a subscription model like kagi you should be able to pay for the storage. I also think that a search engine doesn't need to crawl nor store the whole web. You don't need to crawl pinterest, facebook, instagram, tiktok, and known SEO farms or spam blogs.
I'm too busy to productize this just yet, but might do down the line.
- 1123581321 3y agoThat's a neat project, but it wouldn't satisfy a sizable percentage of searches as a Bing API replacement. It needs much more crawling, processing and a quality maintenance and improvement operation. I think you should keep going with it. Perhaps you could find a way to contribute to engines like Kagi; they already incorporate some smaller and/or weirder indexes.
- carlosjobim 3y agoA ton of businesses and events are only on facebook or only on instagram. Probably the same with tiktok. So they are essential if you want to provide a complete search engine.
- RamblingCTO 3y agoWell, the business case would be a knowledge base for LLMs. I think that there's a lot of tinkering to be done to fine-tune such things anyway.
- carlosjobim 3y agoI'm not sure I understood you for reply. For example if you're looking for a specific sushi restaurant in your town, their Instagram page might be the best result.
- RamblingCTO 3y agoI got you! My search engine would be for things like chatGPT in the first place. They look up knowledge. But maybe you understood that. Anyway, your point is valid. As I said, it sure needs some fine-tuning ;)