14 ms·
The data model behind Notion's flexibility
- jitl 5y agoAuthor here. I'm happy to answer any questions you might have.
- Madeindjs 5y agoReally interessing. This seem to be a really good use case for a NoSQL database. Am I wrong ?
- thawab 5y agoPostgreSQL has Ltree http://patshaughnessy.net/2017/12/13/saving-a-tree-in-postgres-using-ltree http://patshaughnessy.net/2017/12/13/saving-a-tree-in-postgr...
- jitl 5y agoLtree is interesting, but if I understand correctly, to move a parent block, I'd also need to update the path column in all the child blocks -- at our scale such write amplification is a non-starter.
- RyanGoosling 5y agoYes, you're wrong. You're wrong because you need to JOIN a massive tree of blocks, to form the graph the author is referring to. You can break out the "block" model into several tables and represent it in a relational database that way. NoSQL = NO JOIN? Hope that helps.
- jitl 5y agoWe don't use JOIN for the content tree; I don't think I've seen one in any of our queries.
- RyanGoosling 5y agoWhat do your queries look like? Are you using an ORM?
- jitl 5y agoWe don't use an ORM. Notion's codebase on the back-end is much more functional than object-oriented, in the sense that we have many more code that looks like `transformTheData(theData, theChangeToMake): ResultingData` than we have classes or methods. We do lean very heavily on the TypeScript type system and try to make invalid states unrepresentable.
- bionhoward 5y agohave you tried "data last" FP like `transformTheData(theChangeToMake, theData): ResultingData` instead? I learned this from Ramda.JS, makes it way easier to leverage currying, ex `change = transformTheData(theChangeToMake); change(theData)`
- setr 5y agoI don’t actually see a graph represented anywhere in the article; the author references wanting a graph at the start, but the only thing I’m seeing described are trees of nested blocks. Even the properties list seems to be a grab-bag of KV pairs that gets permanently attached to a block once initialized, to support roundtripping Which is pretty much the ideal scenario for a document store. The article describes Notion as being very strictly hierarchal
- RyanGoosling 5y agoA block has many properties. A property has a name, and a value. The underlying persisted data doesn't necessarily have to be a bag of KV pairs. A block is related to its parent and descendant blocks. These relations are suitably represented in a relational database, not a document store. EDIT: In graph theory, a tree is an undirected, connected and acyclic graph.
- setr 5y agoA document store is basically optimized for specifically hierarchal data situations — a tree. The data structure you’re describing, and what the article describes, is precisely that: a tree. When comparing a document store versus a RDBMS, in terms of suitability and appropriateness, the distinction is primarily along the lines of a tree, versus an arbitrary graph (by which I mean that an RDBMS is more powerful, and more general, but not inherently as optimal in either performance, “scalability”, or UX in the places where a document store makes sense. More specifically, the way the article describes it, you’re not interested in “give me every block of type X” — you’re only interested in “given block Y, what type is it?”. That is, the question is one-way, and fits cleanly in a hierarchal format of a document store. The only question posed that operates in the reverse direction is permissions, though even that’s a little odd, since it seems to me it should only go “downwards” as well — a block’s permission scope is the sum of all of its parents, and you can store it there upon iteration. > The underlying persisted data doesn't necessarily have to be a bag of KV pairs. It doesn’t have to be... but it can be, and appears to be. > A block is related to its parent and descendant blocks. Right; the singular parent, and the multiple children. A tree. > In graph theory, a tree is an undirected, connected and acyclic graph. When discussing trees and graphs, I think it’s obvious a distinction is being made between a graph forming a tree, and graph forming a not-tree (more complex than a tree). When I say that a square is easier to encode than a rectangle, I do not mean that a square is not a rectangle, but that a rectangle is not a square — that a square’s more specific properties give us opportunity to simplify/optimize (I only need to store one length to represent it). A database can encode a tree just fine, but that doesn’t mean it’s the best tool to do so. There are other properties to a document store I don’t care for, and I don’t like them in general (like the implicit schema, and total lack of data consistency validation by the data store, and the fact that you often don’t truly have a tree), but representing a tree is what’s been described, and it’s exactly what they’re specialized for. If you want to argue against it, you need to specify why you think this isn’t a tree, because I feel it’s quite obvious it is.
- ellimilial 5y ago@setr explained it really well. A side note, NoSQL also includes graph databases, dedicated to this type of node/relationship traversal.
- DarraghBurke 5y agoI'm going to be writing a blog post for my engineering team soon and I'm really impressed by the storytelling and presentation here. How did you plan out your blog? Were there any resources or examples you found useful during the writing process? Thank you!
- jitl 5y agoI'm glad you enjoyed it! It took a long time to put together. I started out by writing down how the entire system works with enough explanation for a technically-minded non-engineer, and enough detail to satisfy a newly hired infra engineer. That rough draft was about 10,000 words. From there we started looking for a narrative. We extracted out the sections you see in the final post, and removed a lot of the superfluous technical detail so we didn't end up with technology buzzword soup; for example we cut discussion of Postgres, Memcached, etc etc, how we host the web servers; the kind of details that don't actually matter to the narrative. The illustrations were in the post from the beginning as Mermaid diagrams (https://mermaid-js.github.io/mermaid-live-editor/ https://mermaid-js.github.io/mermaid-live-editor/). As we got close to publication we polished them up in Figma. This is really the first engineering blog post we've put out, there was a fair amount of figuring-out-how-to-do-it going on. Now that we've had the experience, we're starting to write up our playbook internally.
- deadbyte 5y agoWould love to read the Director's Cut for more behind-the-scenes on storage, caching, infrastructure!
- euroderf 5y agoDid you... dogfood it ?
- mritchie712 5y agoI'd read a follow up with the Postgres part. I did a cmd+f for Postgres guessing that's what you used and was disappointed not to find it.
- burlesona 5y agoVery interested in your data store. How do you store, query, and search across documents? Also, are you adding presentational tables any time soon? :)
- jitl 5y agoOur source-of-truth data store is Postgres, with a Memcached cache on top. Most of our queries are "pointer chasing" - we follow a reference from one record in memory to fetch another record from the data store. To optimize recursive pointer-chasing queries, we cache the set of visited pointers in Memcached. We use Elasticsearch for search features like QuickFind. > Also, are you adding presentational tables any time soon? :) Sorry, can't talk about future plans like that :)
- johnknowles 5y agoI'm curious if Notion has any plans to make the "type" property user-extensible. Given the current data-structure, which decouples the block data from the way its rendered through the type property, a user has to define only one template for rendering arrangements of UI components (boxes, bullets, etc), titles and children. Extension could operate even at the level of derivation, where users could extend current base types with custom styling (color, font, size, border, etc) and child layout. As a plus, derivation would allow for blocks to be shared, with a fallback default rendering if users don't share custom types. Given the multi-dimensional nature of the uses of Notion (for work, personal projects, life management, etc), having types that were specific to their domain (grocery list, monthly budget table, contact card) would be a useful tool to semantically separate blocks by their presentation.
- RyanGoosling 5y agoWhy not extend this to create forms/surveys as well in your application? You can render a checkboxInput block type as a <input type="checkbox" /> in a form. Your backend would match up the form POST parameter names with the associated form input names and that would be your "Submission".
- bhl 5y agoWhat’s the reasoning why Notion doesn’t have partial text selection across blocks? I originally thought it was an issue with each block being it’s separate contenteditable, but if you wrap all of the contenteditable blocks in another contenteditable div, partial text selection works (at least on Chrome).
- gg2222 5y agoI'm wondering ... so when a user requests a page which will have a full hierarchy of blocks, how are the db queries done. Do you first query the root to see it's content blocks, then make additional queries to load the root's children block, then make additional queries to get those blocks children blocks (ie. recursively) until there are no more children? Does that result in too many database queries? Or do you have other ways to optimize it?
- andyjohnson0 5y agoIt appears that Notion uses Postgres, which supports the recursive/hierarchical queries that are part of standard SQL [1]. While I don't know for sure that Notion uses this, it seems likely. [1] https://en.wikipedia.org/wiki/Hierarchical_and_recursive_queries_in_SQL https://en.wikipedia.org/wiki/Hierarchical_and_recursive_que...
- fellowniusmonk 5y agoI've not used notion before and this is a great write up... Having built my own private-ish collaborative editor/note/todo/sharing/pm tool back in 2012 (that uses indent level permissions) I don't see how you could have a non "block" implementation? I also don't see how they are having so much performance difficulty moving up the inheritance tree at the client level? Does anyone else who has worked on these kinds of problems take a different approach? This being HN and a space I develop in I am actively curious if any other architecture are in use generally? Are there approaches outside of blocks for tools with this level of granularity? Really cool of Notion to share these kinds of "inside ball" deep dives.
- burlesona 5y agoI don’t think you can do this without a block-based approach. That part isn’t super novel, for example I wrote textbook editing software circa 2014 which worked largely the same way. What notion has done that is impressive IMO is made the editing experience enough like the MS-Word “inline” editing that people seem to understand it and use it without minding that it’s a block editor. The project I worked on back in the day wasn’t quite as slick, and there were a lot of users who got frustrated when things like copying text across multiple paragraphs didn’t always do what they had in mind (imagine a text selection from the middle of one paragraph (block) to the middle of another - that’s complicated to deal with). Nice to see Notion take this model mainstream with such a nice UX.
- erjiang 5y agoI built a no-code web-app creator that also has a concept of blocks and I can relate to this. Each paragraph is a different block, and I still haven't quite figured out the best way of handling cross-block interactions like your example of cross-block text selection. I think I would need to redo the browser-based editor, because even if I solved e.g. cross-block selection across blocks of the same type, it's unclear how it should work across, say, a text block and a video-embed block.
- Artistry121 5y ago
- jonpurdy 5y agoThis is a great explainer on how and why Notion is so powerful, and certainly helps explain why it doesn't feel like the fastest tool (because it's inherently more complicated and more powerful than most). I've tried using it for personal notes but it's speed and lack of offline prevented me from migrating to it. But I think it's the best choice for any organization that needs a collaborative docuement/wiki/Confluence-esque tool. Not to mention that I think it has the best default fonts, spacing, and general aesthetic of any tool I can think of.
- wafflez 5y agoMay I ask what you _do_ use for your personal notes instead? I'm always on the lookout for a new tool or workflow for my notes.
- imilk 5y agoGoogle Keep has been pretty sufficient for my needs. Nice search feature, able to capture things quickly when on a call or out and about. Not the most fully featured, but good for remembering things a few days or a week from now
- nxc18 5y agoI’m not the person you asked, but I evaluated and passed on Notion (and Coda, and Airtable) for similar reasons. I need a system that syncs, works offline, helps capture structured-ish data, is quick+flexible, and has fabulous linking. I discovered and am currently using Obsidian.Md. I use Ulysses to access and edit on iPhone+iPad. I love that it is markdown-based, linking is super easy, and everything is fast. Cons: doesn’t work super well with native macOS features (e.g. services), native app for iOS/Android is still in private beta, and I don’t like the separation between edit and view modes. Also it could probably use a stronger UI for database-like features (e.g. tags). I wish there were better options for inking on mobile, but that’s not really their thing. Previously was a OneNote devotee, but as I get into more complicated community engagement work, I needed something that could cross-link and find patterns.
- 5y ago
- polote 5y agoNotion is clearly a feat of engineering. The first time that you modify a link name, and it is automatically modified in all others connected users whether it is in the menu, inside a document or wherever it is. You are just impressed. You can also drag and drop links from a document to a menu. I'm just worried that so much engineering complexity is going to make things slow to move. We will see in a few years if that was a good idea or not
- dheera 5y agoIt's one of those things that they don't really need to move. The product is perfect as-is. Maybe a more full-fledged API/plugin system would be nice, but that's it. The only people that really want them to move and change things are investors, not customers. Same with Dropbox. A folder that syncs is all I need from them as a customer. Everything else they try to do is just getting in my way.
- jitl 5y ago> The product is perfect as-is. > The only people that really want them to move and change things are investors, not customers. I'm glad the Notion works for you, but I think we have a long way to go to reach perfection. To pick an example from the post, even indentation needs work. There's also always more work to do to make the product faster and easier to use.
- dmingod666 5y agoImages in wide-page view will get stretched and pasted to 100% width no matter what their actual size is. You need to manually resize them to look normal. I raised a bug for this, they said it was intended behavior. Like yea, ofcourse.
- peteretep 5y agoThe product also desperately needs simple tables
- 5y ago
- fredley 5y agoI think the power of Notion became apparent to me when I created a Page, accidentally changed it to some other type, then ctrl+z-ed it back to a Page, with all its contents intact. Seriously impressive. I wish search wasn't so crap though.
- ternaryoperator 5y ago> accidentally changed it to some other type This is the problem with Notion, in my experience at least. The docs are so limited and the maneuverability so unusual that it's very easy to do something like completely change the type of document accidentally.
- louisvgchi 5y agoI liked the part where they talked about the blocks with the blocks and the block blocks and how they all block together. I thought the reference to field pioneers was a bit much, I wouldn’t have the blocks to do that with a straight face. I wish these app makers (see also Figma, AirTable) would refrain from making their self-aggrandizing “we’re bringing computing to the masses”; what a load of blocks. What you’re doing is creating a product (another silo in waiting) serving a market that presently is interested. This block model is also not unique (see WordPress and Drupal): Notion may be refining some of that in a more seamless experience, but this is a different kind of statement. Perhaps “we added collaboration to blocks” would be a possible claim for uniqueness. Aside from this, I found the technical decisions interesting and worth a read.
- colinplamondon 5y agoI think it really is that unique - Notion blending read/write is pretty revolutionary for an app like this. Wiki's are obsessed with toggles between read/write mode. Those apps you mention - Drupal, Wordpress, etc - have write modes that exist in backends with alien experiences to the read modes. By unifying read/write into a single continuous UI, with built-in relational databases usable by non-technical users, Notion really does let people create damned powerful custom apps. If you squint, you can see how Notion looks a lot like the next version of the Internet. By hyper-focusing on company intranets they miss out on the power of connected workspaces ("domains"), but the thinking is there. I'll be seriously surprised if Notion isn't looked back on as a breakthrough paradigm shift.
- smoldesu 5y ago> I'll be seriously surprised if Notion isn't looked back on as a breakthrough paradigm shift. To that end, I'd be surprised if they're even remembered much at all. I cancelled my subscription after being seriously disappointed in the feature-set.
- colinplamondon 5y agoIf you want to set the line, I’ll take the over on “Notion succeeds”
- posharma 5y agoAm I the only one who thinks the data model is very complex? How does it compare to a document based data model? What are the pros and cons of each?
- jitl 5y agoYou're not the only one. The major issue with the document model is storage. How big can a document grow? How do you persist it, and make small changes inside it? How do you handle "hot" documents that are very popular? Moving data between documents or having part of a document reference part of another document are complex. Complexity comes from building derived data that looks at slices of a document. With a block-oriented model, your records are much more manageably sized. It's easier to reference or move data between documents, but in turn, you need to do these kinds of recursive shenanigans because your individual records are smaller in scope. Complexity comes from derived data that composes blocks together into a document.
- posharma 5y agoThanks!
- rkangel 5y agoNotion clicked for me when I realised that everything is a "page with some attributes". Blocks are interesting at the engineering level, but as a user I found I was normally thinking at the level of pages. Each page is some content (optional) with a set of arbitrary attributes of different types, e.g. date, string, number etc. When you create a table and populate it, you are creating a set of pages all of whom have the same attributes and the table is just a view of those pages. You can then create a calendar that's a different way of presenting those same pages. You can also deal with those pages directly and put content in, and reference them from elsewhere. It's a really simple data model, and it's pretty flexible. We built all our processes on Notion initially and then moved them off when they got to an appropriate scale. I'm glad the API exist now, because then that approach can be incremental rather than a wholesale move. All the ways of displaying things - tables, calendars etc. are just displaying collections of pages.
- colinplamondon 5y agoThis! The design decision to feel "default-text" is awesome - text is a great foundation for a "custom workspace" product. But, it's only when the bulk of content is in Tables that things get extremely powerful. Once Notion has in-line text references, ala Coda, the sky is really the limit. I think Notion nailed the UX and speed / performance. Even if it's slow compared to Apple Notes, it's lightning fast compared to Coda, where "Documents" are extremely heavy, and relatively isolated. Notion has a clean route towards eating Coda's most powerful functionality ([an incredible charting and formula](https://coda.io/formulas https://coda.io/formulas)). That's the kind of thing that a company as well-financed as Notion can dedicate a team to, and make incredible progress on. Similarly, They can eat Coda's in-line text references, so your text can live-link to information in tables. A matter of resource allocation. Coda, on the other hand, started from an "easier application builder" approach, like Retool for Excel experts. That's brilliant and powerful, but the UX creates a bigger dividing line between consumers and creators. Notion feels enough like a normal word processor that consumers start doing crazy things without realizing it.
- jitl 5y agoThe amount of care I see our users investing in their Notion spaces on Reddit is inspiring. As is seeing Notion show up on my TikTok feed!?
- gumby 5y agoNot sure how "unique" this model is; for example Claris Works was built out of an even more powerful block model (they called them frames) back in the late 1980s: > We came up with a frame-based approach. Most of the functionality particular to the various application types was packaged up into "frames": word processing frames, graphics frames, etc. These frames were then used as building blocks to make documents of the appropriate types, in a unified programming framework. E.g., a word processing document was essentially a bunch of text frames, one per page, linked together. (Doing this neatly was a big challenge - many subsequent efforts at building a component-based architecture (e.g. OpenDoc) have failed to take into account the top-level user interface requirements.) The result was that not only was most of the code shared across the document types, but the application was also truly integrated - the frames could be embedded in each other. E.g., you could plop a spreadsheet frame right into your word processing document. Text objects in a graphics document had a full-featured word processing engine behind them. The database form editor used the built-in graphics environment. Etc. > One related cool thing we had was a "shared graphical context" mechanism: sometimes, stuff would wind up being displayed in multiple frames at once. E.g., maybe you're dragging an object across a page break in a document with multiple pages (like this). We developed a general architecture for displaying actions live in multiple contexts. Of course, a lot of this kind of stuff is old hat today, but it was new and exciting in 1989. Some creative programming was required to do these things efficiently on the hardware of the time. > There were some cool features that didn't make it into the shipping product. For example, originally spreadsheet formulas were much more powerful: you could relate, e.g., graphical object positions and document properties to spreadsheet cells. So you could have the result of a calculation move objects around graphically, or vice-versa. (Further work in this direction led to a novel constraint-based programming paradigm called MOOSE, which I may resurrect some day...) (From http://groups.csail.mit.edu/mac/users/bob/clarisworks.php#onourown http://groups.csail.mit.edu/mac/users/bob/clarisworks.php#on... ) I've been kicking around the idea of writing a CRDT-based editor using this model.
- jitl 5y agoI think it makes us unique among the popular editors of today, but we're very aware that the block model isn't original. Execution matters much more than innovation.
- mattwad 5y agoBeen working with multiple document API's recently for my start-up. It's been a real challenge to understand how different document stores compare -- Confluence uses undocumented XML markup that "mostly" mirrors HTML, Google Docs gives you sometimes semantically-incorrect HTML with lots of inline styles, Slack has its own funny Markdown, etc. Then there's Notion blocks. I had to write my own HTML-to-Notion block parser, but I think I prefer working with it over everything else because structured data is just easier. One interesting gotcha: if you want to completely overwrite a page you currently have to remove every content block one at a time, and then add each new line one at a time. This is using their undocumented API, but the new API doesn't even offer deleting content yet. I'm eager to switch over but it may be a while...
- jitl 5y agoHopefully we get content editing soon! Most of the integrations I want to build need to edit text.
- jerrygoyal 5y agoAs author of a Notion browser extension, "everything is block" concept made it significantly easy to understand the notion doc structure and tweak the UI to add multiple customisations on top of it. For instance showing a fixed static ToC, etc. https://gourav.io/notion-boost https://gourav.io/notion-boost
- abraxaz 5y agoIs there a reason why you did not use rdf for representation and some rdf aware encoding like jsonld for serialization? Would be significantly easier for others to work with, could easily query it with SPARQL.
- jitl 5y agoAn early version of Notion (before my time) used Neo4j, but it turned out to be very slow for the kinds of data access Notion does.
- abraxaz 5y agoYou don't have to use neo4j or any graph database to use RDF. It is just your current model seems very graph based and actually not that difficult to map to RDF, it would likely be possible to do with a jsonld context, and if you provided such a context then it would make your data a lot easier for others to consume.
- georgewfraser 5y agoSalesforce and JIRA both did something similar: their underlying database schema is very generic, basically keys and values, allowing arbitrary logical schemas to be defined at runtime. Yet in both cases, they ended up not really taking advantage of this flexibility. The logical schema of both systems is a very ordinary relational schema that could have been implemented directly on the database, with much better performance. I wonder if the Notion developers made a serious attempt to build on top of a more conventional structured schema, and found it really was unworkable?
- jitl 5y ago> I wonder if the Notion developers made a serious attempt to build on top of a more conventional structured schema, and found it really was unworkable? This is an interesting question. I actually think Notion's data model is much more conventional than the data stores behind document editors like Google Docs or Figma. Blocks are "just" (these quotes are doing a lot of work) rows in Postgres. We use JSON for properties for flexibility and for user-defined property schemas. We could use an entity-attribute-value (https://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80%93value_model https://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80...) table as you describe for that, but such a table would complicate our caching techniques. It would also be enormous, but, it might be time for another think about that since we finished sharding.
- georgewfraser 5y agoI was thinking more along the lines of, specific tables for different classes of entity, with a schema that’s as fixed as possible, so you get the most compact and indexable representation.
- paulsutter 5y agoNotion is great for about a month, then the glacial performance and forced-hierarchical structure really start to grate. But by that time you’ve already bought it
- throayobviousl 5y agoNotion is cool but I went back to trello because its so damn slow. The ios app would refresh all the time and fail to sync. The web app was generally buggy.
- ano88888 5y agoI tried notion several times and gave up. It is not as good as excel or google sheet. I want the data table view to be as powerful and efficient as google sheet or excel plus other notion views. Right now, it is very difficult to even style the the cells with different colors. I think it will be much better for existing google sheet to add other notion functions. Don't know why google stops innovating.
- leifg 5y agoI totally agree but I also have the opposite problem. You can’t have simple tables in Notion (you know, like a comparison table without structured data). You always have to create what they call a “database”
- wdb 5y agoI used a similar approach when working on an editor/notes taker in Flash. Worked quite nicely but even back (early 2000s) then it wasn't really a new idea.
- victor106 5y agoGreat write up. I have a requirement to allow users to create adhoc tables within a web application. I was wohoe beat to represent this on database. Any thoughts?
- gofreddygo 5y agoTables (ad-hoc ones) are a beast. Code that aspires to do tame them, looks simple at first then evolve into crazy monsters that die at the altar of excel. Here's a rough progression of (not entirely) imaginary user requirements for an imaginary, general purpose table rendering software. "I just have a bunch of values, can we make this into a table?" "That looks good. You know what would make it great ? Header rows. Just make them stay there while the table scrolls." "Hey! can you give me some icons on the header row for each column that I can click and sort the data ?" "Please give me some pagination!" "Can you fix the sorting logic? A row with id 10 shows up before the row with 2?" "Can you support merging cells ? Please !" "Can you support re-ordering of columns ?" "I want to adjust the width of my columns but the first column should be fixed width" ... A year later... "This is great but you know its kinda slow. My users prefer to do this in excel and paste it here. Hey, here's a great idea, can you just parse data pasted from excel and just make a table from it ?"
- renke1 5y agoIt's always nice to learn how tools implement this. Especially interesting to me, because I am currently working on a similar tool, although more focused on the plain text aspect (think Markdown + widgets). I am facing slightly different challenges because widgets have to live in a plain text context. For instance, having a widget (say interactive diagram) as part of a Markdown blockquote. On the other hand I don't have to deal with nested blocks (yet), because regular Markdown elements are just text (with fancy styling though). Another challenge is something like $a+b=c$ where it is shown as rendered LaTeX when the user's caret is not touching that part of the document. Also, contenteditable is hard.
- julienreszka 5y agoThe webpage crashed as soon as I opened it. No idea what happened exactly.
- m_mueller 5y agoyou start a blog post with Engelbart, Nelson & Kay and you have my attention. At last someone who understands where the whole field of digital HID was started.
- hayri 5y agoA graph is so natural for pages, web pages are graphs too. So they choose a well fitting model for the problem, congrats