Y
HN Search
Hacker News Search
new
|
comments
|
top
|
jobs
marginalia_nu
searching PlanetScale…
1.
▲
2.
▲
3.
▲
4.
▲
5.
▲
6.
▲
1811 ms
·
6301.
▲
by
marginalia_nu
5y ago
Yeah it's hosted by cloudflare. I'm currently IP-blocking them, as because they keep prompting my crawler with a captcha, presumably because it's made millions of requests from their CDN. Some rigmarole getting recognized as
6302.
▲
by
marginalia_nu
5y ago
I just started with my website and did a crawl. Subsequently I've been seeding it with the best results form my previous crawls. It's a directed search so it doesn't seem to need a particularly solid seed to get decent result
6303.
▲
by
marginalia_nu
5y ago
Well,... yet. Load average is at 1.2, not that bad. But the services are getting a solid workout.
6304.
▲
by
marginalia_nu
5y ago
I wrote it myself from scratch. I have some metadata in mariadb, but the index is bespoke. A design sketch of the index is that it uses one file with sorted URL IDs, one with IDs of N-grams (i.e. words and word-pairs) referring to ranges in
6305.
▲
by
marginalia_nu
5y ago
Some people like a flashy UI, the modern look is important for them. It's ok to have aesthetic preferences, let's not pretend we don't all have them. In the end, it's a niche search engine I've made, the intended au
6306.
▲
by
marginalia_nu
5y ago
I do indeed index the web myself. Not the entire web, just a subset of it. The crawler quickly loses interest in javascript:y websites and only indexes at depth those websites that are simple. It also focuses on websites in English, Swedi
6307.
▲
by
marginalia_nu
5y ago
Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)