8 ms·

Show HN: Serverless collaborative notion-level note editor using CRDT in GO

98 points by kangmh 2y ago

terpimost 2y ago
It would be nice to see a good search implementation and cross device sync :)
- kangmh 2y ago
  File search is possible with the "/" key, but what you're referring to is searching within notes, right? For cross-device capability, currently, it's possible using shared folders. If there's a significant number of stars, considering adding these features might be worthwhile!
memset 2y ago
Awesome that you’ve used wails! Is it basically svelte for the front end? Also looks like you’ve written your own block editor - how did you decide on that approach vs using an existing one?
- kangmh 2y ago
  1. React for the frontend. 2. CRDT was essential for device-to-device encryption, and I wanted a fully customizable block editor. So, creating it myself was the only option.
chipdart 2y ago
Edit: good point. In the absence of a delete option, I've removed content to remove noise.
- gnabgib 2y ago
  Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes. Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
geekodour 2y ago
i came across somewhere on twitter that if two people are editing the document on notion, only the first one gets saved or something. I didn't get into the details of it but in that case this is already ahead in some ways.
- tlarkworthy 2y ago
  False, I for sure have realtime collaberated with people on the same doc. I thought maybe it might be on a individual block level, but then from the Notion docs https://www.notion.so/help/collaborate-within-a-workspace https://www.notion.so/help/collaborate-within-a-workspace, "You can edit the same block as someone else at the same time. Unlike other document software, content is not locked if someone else is making changes to it. The most recent change will be reflected on the page."
  - deleted 2y ago
    [deleted]
  - kangmh 2y ago
    "The most recent change will be reflected on the page" implies overwriting, so different opinions may arise depending on the perspective. Anyway, Notion is addressing the shortcomings of overwriting with features like history.
    - tlarkworthy 2y ago
      yeah, that sentence does imply a last-write-wins in the case of simultaneous keystrokes. But it's at the block level and not the document level.
rubymamis 2y ago
I'm too lazy to build it myself, pre-built binaries will be nice as I would like to give it a spin (I also created a block editor from scratch, in Qt C++ and QML[1] so it's always nice to see new block editors on the block (pun intended)). [1] https://www.get-plume.com https://www.get-plume.com
- mritchie712 2y ago
  Plume looks awesome! > Right now, all these plain text strings are stored in a SQLite database locally on your computer. But we have plans to remove the reliance on a custom database and to store all notes as simple .txt files inside a folder. I'm curious why would you want to move away from SQLite?
  - rubymamis 2y ago
    Thanks! I'm not 100% sure about this, still hearing feedback from people on both ends. Some reasons for the switch would be: 1. File versioning 2. Easier self-hosted syncing 3. Better data portability (although we already support exporting to .txt) I would love to hear counterpoints. Currently, SQLite works pretty well, and our custom database is pretty simple (plus, our database code is also open source[1]). [1] https://github.com/nuttyartist/notes/blob/master/src/dbmanager.cpp https://github.com/nuttyartist/notes/blob/master/src/dbmanag....
    - omnimus 2y ago
      Indexing/search/relations/performance would be some counterpoints that SQlite already solved. I have experience with flat file CMS systems (similar usecase) and at some point of complexity you end up rewriting your own database. Once every file has UUID an you have text file indexes of these than you realize you might have gone too far. But not all projects need that and the argument for version control is big.
      - rubymamis 2y ago
        Thanks for the input! I'll definitely take this into consideration and research more before deciding. Currently staying with SQlite and pretty happy with it.
rcarmo 2y ago
Nice, but I'm a bit wary of data format storage. Can this work just using Markdown-like files on every platform? There's a hint that iCloud storage is used separately.
- kangmh 2y ago
  Yeah. Because it's based on just files, you can work across multiple platforms using shared folders like iCloud storage. However, for merging during simultaneous edits, each note is composed of multiple JSON files and folders.
  - rcarmo 2y ago
    Hmm. So what if I need Markdown files to be always available for local editing by other apps?
    - kangmh 2y ago
      Not support it currently. However, a single markdown file can be treated as one block. Therefore, all the clues are ready in nb-crdt and nb-editor. One tricky part is the frequent replacement and conflicts of xxx.md files for file-based synchronization. There may be significant differences between file systems. Anyway, I think it could achieve eventual consistency by automatically generating folders like xxx.mds in the same path to manage it.
      - rcarmo 2y ago
        Yeah, but you lost me then. Either Markdown is the source of truth (and you do the CRDT when one is edited and reconstitute it after) or I won't store my notes in your app :)
        Kabukks 2y ago
        I've heard that CRDT needs to be the source of truth for certain features to work (handling users going online/offline?). I was never quite sure why it would not work to just create a new CRDT document from another source of truth (markdown, database, ...) when starting an editing session. Maybe someone else can explain.
canadiantim 2y ago
So this is for desktop rather than web?
- cryptonector 2y ago
  TFA says: > nb-editor enables nb-crdt(nb-crdt-go) synchronization and collaboration just through a shared folder, eliminating the need for a server. This does not mean that there's no server though, since the folder could get shared via SMB, NFS, etc.
eviks 2y ago
Is there any effort on creating some common open CRDT-based file format that multiple editors/tools can work with?
- voiceblue 2y ago
  Yes. Yjs is one such project: https://github.com/yjs/yjs https://github.com/yjs/yjs

98 points by kangmh 2y ago

terpimost 2y ago

It would be nice to see a good search implementation and cross device sync :)

kangmh 2y ago
File search is possible with the "/" key, but what you're referring to is searching within notes, right? For cross-device capability, currently, it's possible using shared folders. If there's a significant number of stars, considering adding these features might be worthwhile!

memset 2y ago

Awesome that you’ve used wails! Is it basically svelte for the front end? Also looks like you’ve written your own block editor - how did you decide on that approach vs using an existing one?

kangmh 2y ago
1. React for the frontend. 2. CRDT was essential for device-to-device encryption, and I wanted a fully customizable block editor. So, creating it myself was the only option.

chipdart 2y ago

Edit: good point. In the absence of a delete option, I've removed content to remove noise.

gnabgib 2y ago
Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes. Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

geekodour 2y ago

i came across somewhere on twitter that if two people are editing the document on notion, only the first one gets saved or something. I didn't get into the details of it but in that case this is already ahead in some ways.

tlarkworthy 2y ago
False, I for sure have realtime collaberated with people on the same doc. I thought maybe it might be on a individual block level, but then from the Notion docs https://www.notion.so/help/collaborate-within-a-workspace https://www.notion.so/help/collaborate-within-a-workspace, "You can edit the same block as someone else at the same time. Unlike other document software, content is not locked if someone else is making changes to it. The most recent change will be reflected on the page."
- deleted 2y ago
  [deleted]
- kangmh 2y ago
  "The most recent change will be reflected on the page" implies overwriting, so different opinions may arise depending on the perspective. Anyway, Notion is addressing the shortcomings of overwriting with features like history.
  - tlarkworthy 2y ago
    yeah, that sentence does imply a last-write-wins in the case of simultaneous keystrokes. But it's at the block level and not the document level.

rubymamis 2y ago

I'm too lazy to build it myself, pre-built binaries will be nice as I would like to give it a spin (I also created a block editor from scratch, in Qt C++ and QML[1] so it's always nice to see new block editors on the block (pun intended)). [1] https://www.get-plume.com https://www.get-plume.com

mritchie712 2y ago
Plume looks awesome! > Right now, all these plain text strings are stored in a SQLite database locally on your computer. But we have plans to remove the reliance on a custom database and to store all notes as simple .txt files inside a folder. I'm curious why would you want to move away from SQLite?
- rubymamis 2y ago
  Thanks! I'm not 100% sure about this, still hearing feedback from people on both ends. Some reasons for the switch would be: 1. File versioning 2. Easier self-hosted syncing 3. Better data portability (although we already support exporting to .txt) I would love to hear counterpoints. Currently, SQLite works pretty well, and our custom database is pretty simple (plus, our database code is also open source[1]). [1] https://github.com/nuttyartist/notes/blob/master/src/dbmanager.cpp https://github.com/nuttyartist/notes/blob/master/src/dbmanag....
  - omnimus 2y ago
    Indexing/search/relations/performance would be some counterpoints that SQlite already solved. I have experience with flat file CMS systems (similar usecase) and at some point of complexity you end up rewriting your own database. Once every file has UUID an you have text file indexes of these than you realize you might have gone too far. But not all projects need that and the argument for version control is big.
    - rubymamis 2y ago
      Thanks for the input! I'll definitely take this into consideration and research more before deciding. Currently staying with SQlite and pretty happy with it.

rcarmo 2y ago

Nice, but I'm a bit wary of data format storage. Can this work just using Markdown-like files on every platform? There's a hint that iCloud storage is used separately.

kangmh 2y ago
Yeah. Because it's based on just files, you can work across multiple platforms using shared folders like iCloud storage. However, for merging during simultaneous edits, each note is composed of multiple JSON files and folders.
- rcarmo 2y ago
  Hmm. So what if I need Markdown files to be always available for local editing by other apps?
  - kangmh 2y ago
    Not support it currently. However, a single markdown file can be treated as one block. Therefore, all the clues are ready in nb-crdt and nb-editor. One tricky part is the frequent replacement and conflicts of xxx.md files for file-based synchronization. There may be significant differences between file systems. Anyway, I think it could achieve eventual consistency by automatically generating folders like xxx.mds in the same path to manage it.
    - rcarmo 2y ago
      Yeah, but you lost me then. Either Markdown is the source of truth (and you do the CRDT when one is edited and reconstitute it after) or I won't store my notes in your app :)
      - Kabukks 2y ago
        I've heard that CRDT needs to be the source of truth for certain features to work (handling users going online/offline?). I was never quite sure why it would not work to just create a new CRDT document from another source of truth (markdown, database, ...) when starting an editing session. Maybe someone else can explain.

canadiantim 2y ago

So this is for desktop rather than web?

cryptonector 2y ago
TFA says: > nb-editor enables nb-crdt(nb-crdt-go) synchronization and collaboration just through a shared folder, eliminating the need for a server. This does not mean that there's no server though, since the folder could get shared via SMB, NFS, etc.

eviks 2y ago

Is there any effort on creating some common open CRDT-based file format that multiple editors/tools can work with?

voiceblue 2y ago
Yes. Yjs is one such project: https://github.com/yjs/yjs https://github.com/yjs/yjs