5 ms·

Huge props for the Crystal lang team. Crystal powers 90% of the Kagi search backend (reminder being Python). Highlights are great performance and concurency ha

Huge props for the Crystal lang team. Crystal powers 90% of the Kagi search backend (reminder being Python). Highlights are great performance and concurency handling. Biggest downsides at this moment are compilation speed (does not take advantage of multi CPU cores) and debugging tools. Overall our experience has been fantastic (we adopted it while still in beta) and the pace the language is developing is great.

widdershins 4y ago
Ah, that's interesting information. I've been a (paid) user of Kagi these last few weeks and I'm really enjoying its snappiness. Of course, no ads or trackers must help - when I look at the Network tab in developer tools the comparison with a google search is stark. ~4 requests vs. google's ~50.
- freediver 4y ago
  We obssess over performance, thanks for noticing that :)
  - Nextgrid 4y ago
    I find Kagi consistently faster to respond than DDG.
californical 4y ago
Wow! I’ve been paying for Kagi for as long as it’s been possible to, love the project & the search results are truly so much better than I’ve gotten from Big G for most queries. Never would’ve expected it to be using Crystal though! That’s really neat. Are there open source bits? I’d love to see some enterprise-level Crystal examples
- freediver 4y ago
  We have plans for open sourcing our compontets, including a blog post from our lead dev about how we use Crystal, but never seem to find the time - always some bigger fish to fry.
sph 4y ago
What do you mean by great concurrency? Last I checked multithreading was still hidden behind an experimental compiler flag, which turns me off a bit...
- kitd 4y ago
  Not OP, but the concurrency model is similar to Go with its use of channels. At the moment the runtime is single-core, but the chosen model should permit multi-core (when they enable it by default) with little to no changes to user source. Tbh, I prefer it that way round than try to retrofit multi-core onto a model that wasn't designed for it. But yes, single-core feels a bit behind the curve in 2022.
- straight-shoota 4y ago
  concurrency != multi-threading. Crystal's concurrency is based on fibers (green threads) which works very well even on a single thread (it scales to multiple threads, as well) And for many use cases with high parallelization and little shared state (including web applications such as a search engine), multi-threading isn't necessary. Synchronization overhead is a serious performance killer. You can get much more performance out of running N processes with a single thread instead of a single process running N threads.
  - zw963 4y ago
    Hope a lot of people can see this, detailed explanation come from Crystal core developers. because a lot of beginners complain that Crystal's fiber doesn't work on multiple cores/multiple threads and then give up using it, it's probably not as important as people think. Even many beginners mistakenly believe that the Crystal language does not support the creation of operating system native Thread， this is a huge misunderstanding for Crystal.
- samuell 4y ago
  I did a small comparison of implementing pipeline concurrency in Go and Crystal: https://rillabs.com/posts/crystal-concurrency-easier-syntax-than-golang https://rillabs.com/posts/crystal-concurrency-easier-syntax-... Crystal pretty much provides the same (great) concurrency functionality with light-weight threads and channels etc, but with a cleaner syntax and in my experience less gotchas around closing channels in the wrong place and order etc. The main downsides compared to Go I've seen is: - (As mentioned) Compile times - Far less compelling cross-compilation story
jm4 4y ago
Just want to say I am very happy with Kagi. I've been a paying subscriber since there was an announcement here a few months ago. The results are typically very good, the speed is unreal, and features like lenses and personalized results are so awesome. It's so nice not having search results filled with Stack Overflow spam sites, and if anything slips through I can block it.
simonebrunozzi 4y ago
Funny, I submitted this on HN because I've read that Kagi is powered by Crystal, and then went and took a look at it. p.s. hi Vlad :)

widdershins 4y ago

Ah, that's interesting information. I've been a (paid) user of Kagi these last few weeks and I'm really enjoying its snappiness. Of course, no ads or trackers must help - when I look at the Network tab in developer tools the comparison with a google search is stark. ~4 requests vs. google's ~50.

freediver 4y ago
We obssess over performance, thanks for noticing that :)
- Nextgrid 4y ago
  I find Kagi consistently faster to respond than DDG.

californical 4y ago

Wow! I’ve been paying for Kagi for as long as it’s been possible to, love the project & the search results are truly so much better than I’ve gotten from Big G for most queries. Never would’ve expected it to be using Crystal though! That’s really neat. Are there open source bits? I’d love to see some enterprise-level Crystal examples

freediver 4y ago
We have plans for open sourcing our compontets, including a blog post from our lead dev about how we use Crystal, but never seem to find the time - always some bigger fish to fry.

sph 4y ago

What do you mean by great concurrency? Last I checked multithreading was still hidden behind an experimental compiler flag, which turns me off a bit...

kitd 4y ago
Not OP, but the concurrency model is similar to Go with its use of channels. At the moment the runtime is single-core, but the chosen model should permit multi-core (when they enable it by default) with little to no changes to user source. Tbh, I prefer it that way round than try to retrofit multi-core onto a model that wasn't designed for it. But yes, single-core feels a bit behind the curve in 2022.
straight-shoota 4y ago
concurrency != multi-threading. Crystal's concurrency is based on fibers (green threads) which works very well even on a single thread (it scales to multiple threads, as well) And for many use cases with high parallelization and little shared state (including web applications such as a search engine), multi-threading isn't necessary. Synchronization overhead is a serious performance killer. You can get much more performance out of running N processes with a single thread instead of a single process running N threads.
- zw963 4y ago
  Hope a lot of people can see this, detailed explanation come from Crystal core developers. because a lot of beginners complain that Crystal's fiber doesn't work on multiple cores/multiple threads and then give up using it, it's probably not as important as people think. Even many beginners mistakenly believe that the Crystal language does not support the creation of operating system native Thread， this is a huge misunderstanding for Crystal.
samuell 4y ago
I did a small comparison of implementing pipeline concurrency in Go and Crystal: https://rillabs.com/posts/crystal-concurrency-easier-syntax-than-golang https://rillabs.com/posts/crystal-concurrency-easier-syntax-... Crystal pretty much provides the same (great) concurrency functionality with light-weight threads and channels etc, but with a cleaner syntax and in my experience less gotchas around closing channels in the wrong place and order etc. The main downsides compared to Go I've seen is: - (As mentioned) Compile times - Far less compelling cross-compilation story

jm4 4y ago

Just want to say I am very happy with Kagi. I've been a paying subscriber since there was an announcement here a few months ago. The results are typically very good, the speed is unreal, and features like lenses and personalized results are so awesome. It's so nice not having search results filled with Stack Overflow spam sites, and if anything slips through I can block it.

simonebrunozzi 4y ago

Funny, I submitted this on HN because I've read that Kagi is powered by Crystal, and then went and took a look at it. p.s. hi Vlad :)