3 ms·
Ranking Algorithm: The search uses SQLite FTS5 with BM25 ranking (Okapi BM25 probabilistic retrieval model), not PageRank. Current ranking factors: - Fiel
by joeg_usa 11mo ago
Ranking Algorithm:
The search uses SQLite FTS5 with BM25 ranking (Okapi BM25 probabilistic retrieval model), not PageRank. Current ranking factors:
- Field boosts: Title gets 20x weight, Tags 5x, Content 1x
- Quality boosts: +2 for meta descriptions, +3 for well-structured content (200-10k chars)
- No term proximity currently - FTS5 does boolean matching but not phrase distance scoring (though it is easy enough to do if necessary).
- No link graph/PageRank - we don't analyze inbound links between pages
Term proximity and link-based authority scoring (like our WitRank domain scoring system) are potential future enhancements. (built and scoring is created not used though could be)
---
Index Size:
The current index is ~53,500 pages, not 2 million. Only 56 pages actually contain "cheese" in the crawled content, so getting 45 results is accurate. The demo is smaller because:
1. The index is live and actively growing (currently ~100 docs/min crawl rate)
2. We're crawling real web content organically, not seeding from a dump
3. Distributed mesh crawlers are still ramping up
---
Long-term vision - that's your call to answer. The architecture currently supports both use cases:
- General web search: public crawling, distributed mesh nodes, browser-based PWA crawlers
- Private/enterprise search: SQLite-based, self-hostable, single-writer architecture