3 ms·

A number of years ago, I worked on a team (~20 engineers in total) that successfully carved off two relatively independent portions of a large Rails app using e

by straws 6y ago

A number of years ago, I worked on a team (~20 engineers in total) that successfully carved off two relatively independent portions of a large Rails app using engines. I'm happy to see that Shopify is also using that strategy. I'm curious to know more what sorts of challenges they have around managing dependencies across engines — I think what we were doing was fairly vanilla Rails, and we didn't have the opportunity to run into those sorts of issues.

exterm 6y ago
The answer to that question could probably fill another blog post :D Long story short, Rails and dependency inversion equals lots of friction. The whole framework is built on the assumption that it's OK to access everything from everywhere, and over the years we've built lots of tooling on top of those assumptions. E.g. we heavily use https://github.com/Shopify/identity_cache https://github.com/Shopify/identity_cache with active record associations that cross component boundaries. We also have a GraphQL implementation that is pretty closely coupled to the active record layer and _really_ wants to reach into all the components directly. All of those problems can be overcome, but this is definitely an area where we have to working against "established" Rails culture, and our own assumptions from the past.
- sandGorgon 6y ago
  What's the difference between "componentization+engines" and microservices? From a deployment perspective are your engines deployed and scaled independently?
  - exterm 6y ago
    components are - same database - same runtime - same deployment - same repository That said, I don't think this is an either/or. It's a spectrum. you can have components within the same runtime and repository that have separate databases, or components that are using the same database but live in separate repos, etc. From one monolithic app towards fully separated microservices is a spectrum, and I think developers should be enabled to move freely around that spectrum.
    - sandGorgon 6y ago
      I think components are the better option. Because it allows for separation of concerns without introducing deployment ...or worse : political complexity. I call them Micro-SDKs.
- straws 6y ago
  I hope to hear more in the future! Do you envision any extension points to the way engines are implemented that could better enforce boundaries? In our engines, there was nothing that referenced another engine's resources, leaving the main application to handle route mapping and ActiveRecord associations between app models and engine's models. I feel like the use-case for engines has long been around supporting framework like functionality (Devise, Spree, etc), but I wonder if there are changes to be made that better support modularization for large apps.
  - exterm 6y ago
    > extension points to the way engines are implemented that could better enforce boundaries Can you expand on that? I'm not sure I follow.
kogus 6y ago
This is tangental, but I want applaud the mindset that produces the phrase "the opportunity to run into those sorts of issues".
- straws 6y ago
  I meant it more as a testament to how far you can get with a Rails app before needing to consider using the power tools :^)
JohnBooty 6y ago
I worked on a large Rails monolith a few years back with a similarly-sized team and we took the "components+engines" approach too.... and it was a bit of a nightmare, honestly. It sort of felt like the worst of both worlds, relative to monoliths or microservices. I strongly suspect, but cannot prove, that we would have been better off simply transitioning to "macroservices" -- breaking the monolith up into several (as opposed to dozens) of reasonably sized pieces. • We were encouraged to componitize everything. When I left, we were up to a few dozen components, and the number was climbing rapidly. I'm not sure if the approach itself was the problem, or if the flux during the transition period was the real pain point. • We had no real enforcement of interfaces between components. It was so easy to break things in other peoples' components. • Theoretically that breakage would be caught by tests. But to catch that breakage, you needed to run the complete test suite (30-60 minutes) rather than simply testing your own component • Essentially, it felt like we were suffering all the disadvantages of microservices, with the exception of coordinating deployments; from a devops perspective it was still just a single monolithic deployment • We still had many of the problems associated with monoliths, such as slow deployments, long test suite times, and extremely high per-instance RAM usage • Various small tooling and debugging issues related to using Rails but going too far "off the Rails" I'm looking forward to digging into the linked article and learning how Shopify solved those issues. They seem to have quite a bit of engineering firepower at their disposal. Our management did not allow us to dedicate a lot of resources to internal engineering concerns like this. (We essentially had one guy figuring it all out himself, and due to internal politics he was forbidden from considering a microservices or "macro services / multiple monoliths" approach. He was talented and did the best he could, considering)
- exterm 6y ago
  > When I left, we were up to a few dozen components, and the number was climbing rapidly. I should have included this in the blog post: The number of components _needs_ to be kept small. Shopify's main monolith is 2.8 million lines of code in 37 components, and I'd actually like to get that number _down_. I like to compare this to the main navigation that we present to our merchants. It's useful if it has 8 entries. It's not useful if it has 400. In a way, components are the main navigation to our code base. A developer should be able to look at what's in our "components" folder and get a general impression of what the system's capabilities are.

by straws 6y ago

exterm 6y ago

The answer to that question could probably fill another blog post :D Long story short, Rails and dependency inversion equals lots of friction. The whole framework is built on the assumption that it's OK to access everything from everywhere, and over the years we've built lots of tooling on top of those assumptions. E.g. we heavily use https://github.com/Shopify/identity_cache https://github.com/Shopify/identity_cache with active record associations that cross component boundaries. We also have a GraphQL implementation that is pretty closely coupled to the active record layer and _really_ wants to reach into all the components directly. All of those problems can be overcome, but this is definitely an area where we have to working against "established" Rails culture, and our own assumptions from the past.

sandGorgon 6y ago
What's the difference between "componentization+engines" and microservices? From a deployment perspective are your engines deployed and scaled independently?
- exterm 6y ago
  components are - same database - same runtime - same deployment - same repository That said, I don't think this is an either/or. It's a spectrum. you can have components within the same runtime and repository that have separate databases, or components that are using the same database but live in separate repos, etc. From one monolithic app towards fully separated microservices is a spectrum, and I think developers should be enabled to move freely around that spectrum.
  - sandGorgon 6y ago
    I think components are the better option. Because it allows for separation of concerns without introducing deployment ...or worse : political complexity. I call them Micro-SDKs.
straws 6y ago
I hope to hear more in the future! Do you envision any extension points to the way engines are implemented that could better enforce boundaries? In our engines, there was nothing that referenced another engine's resources, leaving the main application to handle route mapping and ActiveRecord associations between app models and engine's models. I feel like the use-case for engines has long been around supporting framework like functionality (Devise, Spree, etc), but I wonder if there are changes to be made that better support modularization for large apps.
- exterm 6y ago
  > extension points to the way engines are implemented that could better enforce boundaries Can you expand on that? I'm not sure I follow.

kogus 6y ago

This is tangental, but I want applaud the mindset that produces the phrase "the opportunity to run into those sorts of issues".

straws 6y ago
I meant it more as a testament to how far you can get with a Rails app before needing to consider using the power tools :^)

JohnBooty 6y ago

I worked on a large Rails monolith a few years back with a similarly-sized team and we took the "components+engines" approach too.... and it was a bit of a nightmare, honestly. It sort of felt like the worst of both worlds, relative to monoliths or microservices. I strongly suspect, but cannot prove, that we would have been better off simply transitioning to "macroservices" -- breaking the monolith up into several (as opposed to dozens) of reasonably sized pieces. • We were encouraged to componitize everything. When I left, we were up to a few dozen components, and the number was climbing rapidly. I'm not sure if the approach itself was the problem, or if the flux during the transition period was the real pain point. • We had no real enforcement of interfaces between components. It was so easy to break things in other peoples' components. • Theoretically that breakage would be caught by tests. But to catch that breakage, you needed to run the complete test suite (30-60 minutes) rather than simply testing your own component • Essentially, it felt like we were suffering all the disadvantages of microservices, with the exception of coordinating deployments; from a devops perspective it was still just a single monolithic deployment • We still had many of the problems associated with monoliths, such as slow deployments, long test suite times, and extremely high per-instance RAM usage • Various small tooling and debugging issues related to using Rails but going too far "off the Rails" I'm looking forward to digging into the linked article and learning how Shopify solved those issues. They seem to have quite a bit of engineering firepower at their disposal. Our management did not allow us to dedicate a lot of resources to internal engineering concerns like this. (We essentially had one guy figuring it all out himself, and due to internal politics he was forbidden from considering a microservices or "macro services / multiple monoliths" approach. He was talented and did the best he could, considering)

exterm 6y ago
> When I left, we were up to a few dozen components, and the number was climbing rapidly. I should have included this in the blog post: The number of components _needs_ to be kept small. Shopify's main monolith is 2.8 million lines of code in 37 components, and I'd actually like to get that number _down_. I like to compare this to the main navigation that we present to our merchants. It's useful if it has 8 entries. It's not useful if it has 400. In a way, components are the main navigation to our code base. A developer should be able to look at what's in our "components" folder and get a general impression of what the system's capabilities are.