4 ms·
Are tokens different from applying an ID to a node? In this syntax something like --- #token Some Node That is Referenced Often other node A (#token) oth
by tonerow 4y ago
Are tokens different from applying an ID to a node? In this syntax something like
---
#token Some Node That is Referenced Often
other node A
(#token)
other node B
(#token)
other node C
other node D
(#token)
- catapart 4y agoOh! Sorry! "Token" is an intermediary object that acts as a sorting category. So if your markup was: ``` #parentA Start 'Welcome Node' #childA /1 "this node has a line break, so its content is in quotes" ``` That could be "tokened" as [ { tokenType: "ID" value: "parentA" decorators: [{ "character": "#", indexes: [0]}] }, { tokenType: "CONTENT" value: "Start" decorators: null }, { tokenType: "TITLE" value: "Welcome Node" decorators: [{character: "'", indexes: [0, 13]}] }, ... ] So it could be that you can read multiple different syntaxes into the same tokenizations. Like ``` #parentKey parent #childKey child #childKey -> #parentKey /north ``` could be equivalent to ``` #parentKey parent #childKey child /north ```
- tonerow 4y agoI see. That's an interesting idea! I can definitely see how tokenization makes filtering parts of the graph easier.