5 ms·

Reverse engineering of Linear's sync engine

175 points by flashblaze 1y ago

tonetheman 1y ago
Whatever a linear sync engine actually is... sigh. Needs more information.
- ralfhn 1y ago
  Linear.app is a product management tool like Jira. The article describes how they sync their data to their backend
- deleted 1y ago
  [deleted]
nologic98 1y ago
Is this applicable for a consumer mobile app to use for a local-first architecture (either conceptually or literally)?
- isaachinman 1y ago
  You could achieve something almost identical with Replicache + (Mobx or Orama). Only mentioning Mobx because it's what Linear uses. That level of the implementation is interchangeable.
- artman 1y ago
  Most certainly, if the data that the mobile app consumes is bounded and the same data is accessed frequently. Uber for example could have benefited from a sync architecture immensely (I tried to implement one back in the day, but was too late to the party as hypergrowth blocked any attempts at switching architectures). Sync architectures are not only great from a user experience point of view, but also for developer productivity and velocity. Sync takes care of a slew of problems that makes feature development slow. I gave a talk on this at last year's Local First conf https://www.youtube.com/watch?v=VLgmjzERT08&t=4s https://www.youtube.com/watch?v=VLgmjzERT08&t=4s.
  - bhl 1y ago
    Ecosystem for local-first and mobile is pretty immature, at least for Swift. In comparison to the web where there's so many libraries e.g. Zero, LiveStore, LiveBlocks, I've yet to find a good GRDB (sqlite abstraction) integration / client. Offline-first is definitely very strong, but now how do I get data into a remote database with conflict resolution support?
    - satvikpendem 1y ago
      I simply eschewed a relational database and instead used a CRDT like Yrs, Loro, Automerge, etc as my main source of truth. The benefit is that they work well on mobile as well as every other platform, given they're all written in Rust.
    - artman 1y ago
      For inspiration, you might want to look at what we open sourced at Uber, https://github.com/uber-archive/jetstream-ios https://github.com/uber-archive/jetstream-ios and https://github.com/uber-archive/jetstream/wiki/Protocol https://github.com/uber-archive/jetstream/wiki/Protocol. While pretty immature and quite outdated nowadays, it did power one prototype in production and has a lot of the same concepts that we later used in Linear's sync engine.
mappu 1y ago
If you're using AI to write all those em-dashes, please add a disclaimer. For humans i would say a shorter summary is Linear.app syncs a client IndexedDB with the server using naive last-write-wins, no conflict detection, no OT, no CRDT. There's a global sync ID that the server is in control of. Most of the article describes minutae of the json schema.
- evaneykelen 1y ago
  On macOS, typing two consecutive hyphens automatically gets converted to an em-dash in many applications: no AI involved necessarily.
  - notpushkin 1y ago
    I’ve built a custom layout for that (and a bunch of other symbols I frequently use). ⌥ hyphen for en-dash, ⌥ ⇧ hyphen for em-dash (and ⌥ M is for minus): https://typo.ale.sh/ https://typo.ale.sh/ (The idea isn’t new, of course: the default macOS layout’s 3rd layer is absolutely bonkers. I think Ilya Birman was the first: https://ilyabirman.net/typography-layout/ https://ilyabirman.net/typography-layout/)
    - jdxcode 1y ago
      those are the default macos keybindings for en-dash and em-dash characters
      - notpushkin 1y ago
        Good point, totally forgot about that :/
- pottertheotter 1y ago
  Never thought someone would be anti em-dashes.
  - dheatov 1y ago
bitpush 1y ago
What's the closest opensource library that implements this sync (or similar) scheme? ElectricSQL? ZeroSync? Firebase? Something else?
- jaccola 1y ago
  Firebase (Firestore is the DB) is the best I've ever used but not open source. MongoDB has Realm which achieves similar, is open source and is OK.
  - ochiba 1y ago
    Realm's sync functionality (Atlas Device Sync) has been deprecated by MongoDB: https://www.mongodb.com/docs/atlas/app-services/sync/device-sync-deprecation/ https://www.mongodb.com/docs/atlas/app-services/sync/device-...
- rapnie 1y ago
  There is a proliferation of sync tools and little standardization. Here some in local-first space [0]. Martin Kleppmann in his talk last year spoke [1] the need for a generic sync protocol, which was very interesting. [0] https://localfirstweb.dev/ https://localfirstweb.dev/ [1] https://www.youtube.com/watch?v=NMq0vncHJvU&t=1016s https://www.youtube.com/watch?v=NMq0vncHJvU&t=1016s
- drunkan 1y ago
  Someone maintains a list here https://www.localfirst.fm/landscape https://www.localfirst.fm/landscape LiveStore shows recreating linear as one of their examples though I haven’t tried it. It was on the front page recently https://news.ycombinator.com/item?id=44105412 https://news.ycombinator.com/item?id=44105412
- deleted 1y ago
  [deleted]
devmakasana 1y ago
Linear’s sync engine maintains a local, in-memory object graph (backed by MobX) and persists all changes to IndexedDB, allowing immediate, offline-first updates. We build same experience at www.teamcamp.app
artman 1y ago
I think the impressive part here isn't Linear's sync engine, but the fact that Evan Hu went through painstakingly reverse-engineer the engine by inspecting traffic and obfuscated code and was able to write documentation that is correct and more complete than what Linear publishes internally.
- wzhudev 1y ago
  Thanks for the kind words! This study was a lot of fun to me.
jtwaleson 1y ago
I have a first attempt at a sync engine for my app, but it's very primitive. Just a websocket that sends updates based on database triggers. If you miss one, you have to do a full reload. I know I'll need something better in a year or so. Any advice on what route to take with creating a sync engine for a product like mine? Self-hosted, single binary web app (Rust) + Postgres db. Frontend is based on VueJS. I've looked at the readme of Yjs and was considering that. I'm a solo dev for now. I'm tempted to feed Cursor this description of the reverse engineered solution of Linear, but I doubt it'll be successful.
- jgeurts 1y ago
  Take a look at Electric SQL: https://electric-sql.com/ https://electric-sql.com/
- jasonjmcghee 1y ago
  Yjs isn't a sync engine, it's a data structure for managing distributed concurrent updates and ensuring they are conflict free. Whether you use it feels orthogonal to the problem you're describing. --- For a minimal scope solution, have you considered making a table in your database where you log each update? Then you can keep an id of your most recent update locally and on websocket reconnection ask for the updates after your current change. Similar to how in-app notifications work. --- For local-first, you can use things like: https://tinybase.org/ https://tinybase.org/ https://electric-sql.com/ https://electric-sql.com/ https://livestore.dev/ https://livestore.dev/ But they are pretty foundational. You use them as your storage layer in the front end. So worth considering the scope of the change.
  - jtwaleson 1y ago
    Thanks, that helps! Like I said I had only very briefly looked at Yjs. The thought of an "updates" table has crossed my mind yes, but after some time you want a "materialized view" instead of replaying the history from the start, and that's where it gets complicated ;) I'll take a look at those alternatives. I'd rather have something stable than having to re-invent the wheel. Thanks again!
- 1y ago
hipgoat 1y ago
[flagged]

175 points by flashblaze 1y ago

tonetheman 1y ago

Whatever a linear sync engine actually is... sigh. Needs more information.

ralfhn 1y ago
Linear.app is a product management tool like Jira. The article describes how they sync their data to their backend
deleted 1y ago
[deleted]

nologic98 1y ago

Is this applicable for a consumer mobile app to use for a local-first architecture (either conceptually or literally)?

isaachinman 1y ago
You could achieve something almost identical with Replicache + (Mobx or Orama). Only mentioning Mobx because it's what Linear uses. That level of the implementation is interchangeable.
artman 1y ago
Most certainly, if the data that the mobile app consumes is bounded and the same data is accessed frequently. Uber for example could have benefited from a sync architecture immensely (I tried to implement one back in the day, but was too late to the party as hypergrowth blocked any attempts at switching architectures). Sync architectures are not only great from a user experience point of view, but also for developer productivity and velocity. Sync takes care of a slew of problems that makes feature development slow. I gave a talk on this at last year's Local First conf https://www.youtube.com/watch?v=VLgmjzERT08&t=4s https://www.youtube.com/watch?v=VLgmjzERT08&t=4s.
- bhl 1y ago
  Ecosystem for local-first and mobile is pretty immature, at least for Swift. In comparison to the web where there's so many libraries e.g. Zero, LiveStore, LiveBlocks, I've yet to find a good GRDB (sqlite abstraction) integration / client. Offline-first is definitely very strong, but now how do I get data into a remote database with conflict resolution support?
  - satvikpendem 1y ago
    I simply eschewed a relational database and instead used a CRDT like Yrs, Loro, Automerge, etc as my main source of truth. The benefit is that they work well on mobile as well as every other platform, given they're all written in Rust.
  - artman 1y ago
    For inspiration, you might want to look at what we open sourced at Uber, https://github.com/uber-archive/jetstream-ios https://github.com/uber-archive/jetstream-ios and https://github.com/uber-archive/jetstream/wiki/Protocol https://github.com/uber-archive/jetstream/wiki/Protocol. While pretty immature and quite outdated nowadays, it did power one prototype in production and has a lot of the same concepts that we later used in Linear's sync engine.

mappu 1y ago

If you're using AI to write all those em-dashes, please add a disclaimer. For humans i would say a shorter summary is Linear.app syncs a client IndexedDB with the server using naive last-write-wins, no conflict detection, no OT, no CRDT. There's a global sync ID that the server is in control of. Most of the article describes minutae of the json schema.

evaneykelen 1y ago
On macOS, typing two consecutive hyphens automatically gets converted to an em-dash in many applications: no AI involved necessarily.
- notpushkin 1y ago
  I’ve built a custom layout for that (and a bunch of other symbols I frequently use). ⌥ hyphen for en-dash, ⌥ ⇧ hyphen for em-dash (and ⌥ M is for minus): https://typo.ale.sh/ https://typo.ale.sh/ (The idea isn’t new, of course: the default macOS layout’s 3rd layer is absolutely bonkers. I think Ilya Birman was the first: https://ilyabirman.net/typography-layout/ https://ilyabirman.net/typography-layout/)
  - jdxcode 1y ago
    those are the default macos keybindings for en-dash and em-dash characters
    - notpushkin 1y ago
      Good point, totally forgot about that :/
pottertheotter 1y ago
Never thought someone would be anti em-dashes.
- dheatov 1y ago

bitpush 1y ago

What's the closest opensource library that implements this sync (or similar) scheme? ElectricSQL? ZeroSync? Firebase? Something else?

jaccola 1y ago
Firebase (Firestore is the DB) is the best I've ever used but not open source. MongoDB has Realm which achieves similar, is open source and is OK.
- ochiba 1y ago
  Realm's sync functionality (Atlas Device Sync) has been deprecated by MongoDB: https://www.mongodb.com/docs/atlas/app-services/sync/device-sync-deprecation/ https://www.mongodb.com/docs/atlas/app-services/sync/device-...
rapnie 1y ago
There is a proliferation of sync tools and little standardization. Here some in local-first space [0]. Martin Kleppmann in his talk last year spoke [1] the need for a generic sync protocol, which was very interesting. [0] https://localfirstweb.dev/ https://localfirstweb.dev/ [1] https://www.youtube.com/watch?v=NMq0vncHJvU&t=1016s https://www.youtube.com/watch?v=NMq0vncHJvU&t=1016s
drunkan 1y ago
Someone maintains a list here https://www.localfirst.fm/landscape https://www.localfirst.fm/landscape LiveStore shows recreating linear as one of their examples though I haven’t tried it. It was on the front page recently https://news.ycombinator.com/item?id=44105412 https://news.ycombinator.com/item?id=44105412
deleted 1y ago
[deleted]

devmakasana 1y ago

Linear’s sync engine maintains a local, in-memory object graph (backed by MobX) and persists all changes to IndexedDB, allowing immediate, offline-first updates. We build same experience at www.teamcamp.app

artman 1y ago

I think the impressive part here isn't Linear's sync engine, but the fact that Evan Hu went through painstakingly reverse-engineer the engine by inspecting traffic and obfuscated code and was able to write documentation that is correct and more complete than what Linear publishes internally.

wzhudev 1y ago
Thanks for the kind words! This study was a lot of fun to me.

jtwaleson 1y ago

I have a first attempt at a sync engine for my app, but it's very primitive. Just a websocket that sends updates based on database triggers. If you miss one, you have to do a full reload. I know I'll need something better in a year or so. Any advice on what route to take with creating a sync engine for a product like mine? Self-hosted, single binary web app (Rust) + Postgres db. Frontend is based on VueJS. I've looked at the readme of Yjs and was considering that. I'm a solo dev for now. I'm tempted to feed Cursor this description of the reverse engineered solution of Linear, but I doubt it'll be successful.

jgeurts 1y ago
Take a look at Electric SQL: https://electric-sql.com/ https://electric-sql.com/
jasonjmcghee 1y ago
Yjs isn't a sync engine, it's a data structure for managing distributed concurrent updates and ensuring they are conflict free. Whether you use it feels orthogonal to the problem you're describing. --- For a minimal scope solution, have you considered making a table in your database where you log each update? Then you can keep an id of your most recent update locally and on websocket reconnection ask for the updates after your current change. Similar to how in-app notifications work. --- For local-first, you can use things like: https://tinybase.org/ https://tinybase.org/ https://electric-sql.com/ https://electric-sql.com/ https://livestore.dev/ https://livestore.dev/ But they are pretty foundational. You use them as your storage layer in the front end. So worth considering the scope of the change.
- jtwaleson 1y ago
  Thanks, that helps! Like I said I had only very briefly looked at Yjs. The thought of an "updates" table has crossed my mind yes, but after some time you want a "materialized view" instead of replaying the history from the start, and that's where it gets complicated ;) I'll take a look at those alternatives. I'd rather have something stable than having to re-invent the wheel. Thanks again!
1y ago

hipgoat 1y ago

[flagged]