3 ms·

Some of the listed optimizations were: > We carefully vet what we eager-load depending on the type of request and we optimize towards reducing instances of N+1

by pqdbr 6y ago

Some of the listed optimizations were: > We carefully vet what we eager-load depending on the type of request and we optimize towards reducing instances of N+1 queries. > Reducing Memory Allocations > Implementing Efficient Caching Layers All of those steps seem pretty standard ways of optimizing a Rails application. I wished the article made it clearer why they decided to pursue such a complex route (the whole custom Lua/nginx routing and two applications instead of a monolith). Shopify surely has tons of Rails experts and I assume they pondered a lot before going for this unusual rewrite, so of course they have their reasons, but I really didn't understand (from the article) what they accomplished here that they couldn't have done in the Rails monolith. You don't need to ditch Rails if you just don't want to use ActiveRecord.

deleted 6y ago
[deleted]
pqdbr 6y ago
Someone replied but deleted right when I was posting this answer, so I'm replying to myself: What I didn't understand was why the listed performance optimizations couldn't be implemented in the monolith itself and ensued the development of a new application, which is still Ruby. In a production env, the request reaches the Rails controller pretty fast. I know for a fact that the view layer (.html.erb) can be a little slow if you compare it to, say, just a `render json:`, but if you're still going to be sending fully-rendered HTML pages over the wire, the listed optimizations (caching, query optimization and memory allocation) could all be implemented in Rails itself to a huge extent, and that's what I'd love to know more about.
- nthj 6y ago
  They talk about reducing memory allocations. My guess is the rest of the app is very large and they’re benefiting from not sharing memory and GC with that. Of course, everything you said is true for a small-to-medium sized Rails application. They likely could have explored a separate Rails app to meet this goal, but then they have to maintain the dependency tree and security risks twice. And if the Rails core refactors away any optimizations they make, they have to maintain and integrate with those. There’s definitely some wiggle room and a judgement call here but their custom implementation has merit.
- cbothner 6y ago
  Don't forget that a Shopify store is 100% customizable by merchants using Liquid (Turing complete, not that you should try). There is no .html.erb layer. Think of Storefront Renderer as a Liquid interpreter using optimized presenters for the business models.
  - benatkin 6y ago
    Liquid is designed so template authors don't have to be trusted. That's great and I wish it were more common. Here's an example of a disclaimer that should be attached to most templating languages: https://mozilla.github.io/nunjucks/templating.html https://mozilla.github.io/nunjucks/templating.html
    - harrisonjackson 6y ago
      Seconded! I've had to do a lot of weird stuff in django to get around this for our user templates.
pushrax 6y ago
(contributor here) The project does still use code from Rails. Some parts of ActiveSupport in particular are really not worth rewriting, it works fine and has a lot of investment already. The MVC part of Rails is not used for this project, because the storefront of Shopify works in a very different way than a CRUD app, and doesn’t benefit nearly as much. Custom code is a lot smaller and easier to understand and optimize. Outside of storefront, Shopify still benefits a lot from Rails MVC. I’ll also add that storefront serves a majority of requests made to Shopify but it’s a surprisingly tiny fraction of the actual code.
- joshmn 6y ago
  > because the storefront of Shopify works in a very different way than a CRUD app Any interesting/successful patterns you can share/resources you can share on said patterns?
  - pushrax 6y ago
    “Not a CRUD app” isn’t a design decision, it’s just that storefront is almost entirely read-only, and the views are merchant-provided Liquid code. Most of a shop's data can be accessed on any page; data dependencies are in large part defined by the view, not the controller.
    - joshmn 6y ago
      Oh I Know. As a Rails developer for 10 years, I'm always interested what patterns are developed when straying away from what the original guts of Rails.
  - tekstar 6y ago
    Shopify's storefront is based around a liquid renderer instance. If you look up how objects are added to the liquid context that is pretty similar to the overall pattern (or at least was back when I worked there, hi pushrax :)
    - pushrax 6y ago

by pqdbr 6y ago

deleted 6y ago

[deleted]

pqdbr 6y ago

Someone replied but deleted right when I was posting this answer, so I'm replying to myself: What I didn't understand was why the listed performance optimizations couldn't be implemented in the monolith itself and ensued the development of a new application, which is still Ruby. In a production env, the request reaches the Rails controller pretty fast. I know for a fact that the view layer (.html.erb) can be a little slow if you compare it to, say, just a `render json:`, but if you're still going to be sending fully-rendered HTML pages over the wire, the listed optimizations (caching, query optimization and memory allocation) could all be implemented in Rails itself to a huge extent, and that's what I'd love to know more about.

nthj 6y ago
They talk about reducing memory allocations. My guess is the rest of the app is very large and they’re benefiting from not sharing memory and GC with that. Of course, everything you said is true for a small-to-medium sized Rails application. They likely could have explored a separate Rails app to meet this goal, but then they have to maintain the dependency tree and security risks twice. And if the Rails core refactors away any optimizations they make, they have to maintain and integrate with those. There’s definitely some wiggle room and a judgement call here but their custom implementation has merit.
cbothner 6y ago
Don't forget that a Shopify store is 100% customizable by merchants using Liquid (Turing complete, not that you should try). There is no .html.erb layer. Think of Storefront Renderer as a Liquid interpreter using optimized presenters for the business models.
- benatkin 6y ago
  Liquid is designed so template authors don't have to be trusted. That's great and I wish it were more common. Here's an example of a disclaimer that should be attached to most templating languages: https://mozilla.github.io/nunjucks/templating.html https://mozilla.github.io/nunjucks/templating.html
  - harrisonjackson 6y ago
    Seconded! I've had to do a lot of weird stuff in django to get around this for our user templates.

pushrax 6y ago

(contributor here) The project does still use code from Rails. Some parts of ActiveSupport in particular are really not worth rewriting, it works fine and has a lot of investment already. The MVC part of Rails is not used for this project, because the storefront of Shopify works in a very different way than a CRUD app, and doesn’t benefit nearly as much. Custom code is a lot smaller and easier to understand and optimize. Outside of storefront, Shopify still benefits a lot from Rails MVC. I’ll also add that storefront serves a majority of requests made to Shopify but it’s a surprisingly tiny fraction of the actual code.

joshmn 6y ago
> because the storefront of Shopify works in a very different way than a CRUD app Any interesting/successful patterns you can share/resources you can share on said patterns?
- pushrax 6y ago
  “Not a CRUD app” isn’t a design decision, it’s just that storefront is almost entirely read-only, and the views are merchant-provided Liquid code. Most of a shop's data can be accessed on any page; data dependencies are in large part defined by the view, not the controller.
  - joshmn 6y ago
    Oh I Know. As a Rails developer for 10 years, I'm always interested what patterns are developed when straying away from what the original guts of Rails.
- tekstar 6y ago
  Shopify's storefront is based around a liquid renderer instance. If you look up how objects are added to the liquid context that is pretty similar to the overall pattern (or at least was back when I worked there, hi pushrax :)
  - pushrax 6y ago