7 ms·

This article is timely for me! I think about this a lot while working on Notion — and I’m drafting an RFC for a change that future engineers will either praise

by jitl 4y ago

This article is timely for me! I think about this a lot while working on Notion — and I’m drafting an RFC for a change that future engineers will either praise or curse. Right now the eng team is ~150, so we’re exiting the period when an experienced senior engineer (ie me) can easily make a profound tech stack change. Our database (Postgres) and backend stack (Express/Typescript) are already quite solidified. There is still time to change our front-end stack, although React is probably a given, we can change our API contract layer, our state management pattern, and possibly our client-side replicated ORM. It’s tricky to weigh continued speed today versus long-term speed. I think a lot about the tradeoff of “invest in infra improvements below the interface line” and “change the interface now, while there’s still time” when it comes to our ORM/state manager. We have a bunch of interesting challenges across all layers (see https://www.notion.so/blog/data-model-behind-notion https://www.notion.so/blog/data-model-behind-notion) so it’s not obvious what changes today will really change the constraints on our solutions tomorrow. My current philosophy is that we need to add new abstraction layers now if there’s a chance we need them, before detangling later becomes an insurmountable lift.

majormajor 4y ago
Do you need (more?) new abstraction layers or do you need extremely clear un-abstracted separations of concern? In the cases of trying to build abstraction layers in advance "in case we need them" that I've worked on, they rarely are sufficiently right-shaped for what the future feature need ends up being. Having more direct access to the primitives usually works better in terms of avoiding "the business logic is split across three layers of service calls" type of situations.
- jitl 4y ago
  We share the same record types (primitives?) across the stack - but because of that sharing, those types are hard to evolve, especially when it comes to their semantics. Too much code interprets then directly, which ossifies those primitives. How to enforce separation of concern when such data structure is shared liberally? > the business logic is split across three layers of service calls we are thankfully very far from this world.
  - 8n4vidtmkvmk 4y ago
    convert the records at the service boundaries to something better suited for that service. then everything doesn't depend on each other
    - jitl 4y ago
      Your advice is "add abstractions", which is basically the opposite of the grandparent commenter's advice. Right now I am on board the "add abstractions" train.
      - 8n4vidtmkvmk 4y ago
        I suppose, but converting it to another structure isn't a huge abstraction. Maybe it can complicate things if the conversion isn't obvious
aledalgrande 4y ago
Modularization will give you the flexibility to untangle the spaghetti and swap pieces if needed. Nothing has to be forever. Worst is fixing data though, not code.
- jitl 4y ago
  I’m curious what you imagine when you say “modularization.” It might be different from what I imagine.
  - aledalgrande 4y ago
    In my mind there's two types of modules: - technical, e.g. db abstraction layer - business, e.g. checkout What I'm suggesting is having this structure to help you with future swaps/refactors, because they will stay very localized. Of course where to cut a module out of a codebase is a skill in itself. Cut too much and you have redirection everywhere, cut too little and you have multiple responsibilities, bigger PRs and harder refactors.
bluesnowmonkey 4y ago
A vote for adding the abstractions now you think you’ll need later. I found once we crossed around the 1000 engineer mark, the main abstractions were so baked in that refactoring projects started taking years. Longer than the typical employee tenure, which effectively meant they couldn’t be done. 1. People generally don’t even start projects they won’t be around to finish, and 2. A refactoring project that turns over all its staff forgets what it was trying to do and grinds to a halt one way or another. (For instance because the junior staff, only ones willing to get involved with a long embattled project, don’t have the political capital to keep it funded in terms of headcount.)

by jitl 4y ago

majormajor 4y ago

Do you need (more?) new abstraction layers or do you need extremely clear un-abstracted separations of concern? In the cases of trying to build abstraction layers in advance "in case we need them" that I've worked on, they rarely are sufficiently right-shaped for what the future feature need ends up being. Having more direct access to the primitives usually works better in terms of avoiding "the business logic is split across three layers of service calls" type of situations.

jitl 4y ago
We share the same record types (primitives?) across the stack - but because of that sharing, those types are hard to evolve, especially when it comes to their semantics. Too much code interprets then directly, which ossifies those primitives. How to enforce separation of concern when such data structure is shared liberally? > the business logic is split across three layers of service calls we are thankfully very far from this world.
- 8n4vidtmkvmk 4y ago
  convert the records at the service boundaries to something better suited for that service. then everything doesn't depend on each other
  - jitl 4y ago
    Your advice is "add abstractions", which is basically the opposite of the grandparent commenter's advice. Right now I am on board the "add abstractions" train.
    - 8n4vidtmkvmk 4y ago
      I suppose, but converting it to another structure isn't a huge abstraction. Maybe it can complicate things if the conversion isn't obvious

aledalgrande 4y ago

Modularization will give you the flexibility to untangle the spaghetti and swap pieces if needed. Nothing has to be forever. Worst is fixing data though, not code.

jitl 4y ago
I’m curious what you imagine when you say “modularization.” It might be different from what I imagine.
- aledalgrande 4y ago
  In my mind there's two types of modules: - technical, e.g. db abstraction layer - business, e.g. checkout What I'm suggesting is having this structure to help you with future swaps/refactors, because they will stay very localized. Of course where to cut a module out of a codebase is a skill in itself. Cut too much and you have redirection everywhere, cut too little and you have multiple responsibilities, bigger PRs and harder refactors.

bluesnowmonkey 4y ago

A vote for adding the abstractions now you think you’ll need later. I found once we crossed around the 1000 engineer mark, the main abstractions were so baked in that refactoring projects started taking years. Longer than the typical employee tenure, which effectively meant they couldn’t be done. 1. People generally don’t even start projects they won’t be around to finish, and 2. A refactoring project that turns over all its staff forgets what it was trying to do and grinds to a halt one way or another. (For instance because the junior staff, only ones willing to get involved with a long embattled project, don’t have the political capital to keep it funded in terms of headcount.)