5 ms·

Real world example for Crystal: our overnight sitemap generation (built using Ruby, as part of our Rails app) took 6 hours to generate 20 million links. At some

by prh8 3y ago

Real world example for Crystal: our overnight sitemap generation (built using Ruby, as part of our Rails app) took 6 hours to generate 20 million links. At some point due to infra changes, it stopped running to completion. Since then, we ported it over to Crystal, with the main code virtually unchanged (library to help generate sitemaps has matching API). Just had to set up basic database mapping for a few tables. That version is now in production, and it now processes 30 million links in about 30 minutes.

throwaway3838g 3y ago
Just curious, what makes Crystal so much faster for this use case?
- prh8 3y ago
  The biggest thing is the massive speed improvement compared to working with bloated ActiveRecord models, associations, plus some memory bloat. The Crystal implementation is massively fast dealing with database data.
- moe 3y ago
  Crystal performance is close to Go-lang. It's just very fast across the board. Consequently, just like Go-lang, it beats Ruby by an order of magnitude on pretty much any benchmark. Newcomers often assume Crystal must be sluggish like Ruby because "how could a language with such a convenient syntax be so fast", but it really is not. https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/
  - Alifatisk 3y ago
    > Crystal performance is close to Go-lang. It's just very fast across the board. I know Go is known for it concurrency, does Crystal come close to that aswell?
    - npn 3y ago
      in fact Crystal concurrency is the same with Go, both using coroutine technique. though Go should be more performance because it is built for that feature.
      - Alifatisk 3y ago
        Interesting, I might give Crystal a try!
CoolCold 3y ago
Hell, Ruby is that slow?!
- prh8 3y ago
  It's not specifically Ruby, but also using ActiveRecord, loading bloated models, associations, and some business logic. There is also a large amount of memory bloat with the sitemap library when working with large datasets. The Crystal sitemap library has a way to avoid that same bloat. With Crystal, all of those things are massively faster.
- pkd 3y ago
  Since a library is mentioned, it is possible that the Ruby library was particularly inefficient compared to Crystal. Crystal is definitely faster than Ruby, but I expect differences to diminish in IO-bound operations (which sitemap generation sounds like it would be).
  - npn 3y ago
    Crystal supports directly interacting with the IO streams, so even if it is IO-bound it still can be a lot faster.
    - pkd 3y ago
      What does Crystal's IO stream support provide that Ruby's does not? Both expose very similar IO stream interfaces from my experience.

by prh8 3y ago

throwaway3838g 3y ago

Just curious, what makes Crystal so much faster for this use case?

prh8 3y ago
The biggest thing is the massive speed improvement compared to working with bloated ActiveRecord models, associations, plus some memory bloat. The Crystal implementation is massively fast dealing with database data.
moe 3y ago
Crystal performance is close to Go-lang. It's just very fast across the board. Consequently, just like Go-lang, it beats Ruby by an order of magnitude on pretty much any benchmark. Newcomers often assume Crystal must be sluggish like Ruby because "how could a language with such a convenient syntax be so fast", but it really is not. https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/
- Alifatisk 3y ago
  > Crystal performance is close to Go-lang. It's just very fast across the board. I know Go is known for it concurrency, does Crystal come close to that aswell?
  - npn 3y ago
    in fact Crystal concurrency is the same with Go, both using coroutine technique. though Go should be more performance because it is built for that feature.
    - Alifatisk 3y ago
      Interesting, I might give Crystal a try!

CoolCold 3y ago

Hell, Ruby is that slow?!

prh8 3y ago
It's not specifically Ruby, but also using ActiveRecord, loading bloated models, associations, and some business logic. There is also a large amount of memory bloat with the sitemap library when working with large datasets. The Crystal sitemap library has a way to avoid that same bloat. With Crystal, all of those things are massively faster.
pkd 3y ago
Since a library is mentioned, it is possible that the Ruby library was particularly inefficient compared to Crystal. Crystal is definitely faster than Ruby, but I expect differences to diminish in IO-bound operations (which sitemap generation sounds like it would be).
- npn 3y ago
  Crystal supports directly interacting with the IO streams, so even if it is IO-bound it still can be a lot faster.
  - pkd 3y ago
    What does Crystal's IO stream support provide that Ruby's does not? Both expose very similar IO stream interfaces from my experience.