17 ms·

Ask HN: How to learn about text editor architectures and implementations?

• buescher 5y ago
  It's from a completely different time and point of view than "full-stack development", vi, or emacs, but Petter Hesselberg's Programming Industrial Strength Windows: Shrink-Wrap Your App! presents a complete text editor for Windows as an example of a complete "real-world" Windows application.
• ketanmaheshwari 5y ago
  I would suggest start with getting familiar with the C programming language. I suggest use Beej's guide for C. Next, pick up the "The Linux Programming Interface" book by Michael Kerrisk. As you read the book you will get better at understanding source code of at least Vim and Emacs if not others.
• cx0der 5y ago
  Neatpad https://www.catch22.net/tuts/neatpad https://www.catch22.net/tuts/neatpad This is a series that I used to follow as I too wanted to implement my own text editor to learn the underlying architecture.
• jhallenworld 5y ago
  Read technical documentation for existing editors, for example: https://sourceforge.net/p/joe-editor/mercurial/ci/default/tree/docs/hacking.md https://sourceforge.net/p/joe-editor/mercurial/ci/default/tr...
• teddyh 5y ago
  You want this: The Craft of Text Editing —or— Emacs for the Modern World –by– Craig A. Finseth https://www.finseth.com/craft/ https://www.finseth.com/craft/
  • buescher 5y ago
    And the earlier EMACS: The Extensible, Customizable Display Editor by rms https://www.gnu.org/software/emacs/emacs-paper.html https://www.gnu.org/software/emacs/emacs-paper.html which is more of a 10,000 foot view.
    • s3arch 5y ago
      This is awesome.
  • compressedgas 5y ago
    I like Finseth's 1980 thesis better than the 1991 book Theory and practice of text editors, or, A cookbook for an Emacs https://dspace.mit.edu/handle/1721.1/15905 https://dspace.mit.edu/handle/1721.1/15905
• danking00 5y ago
  I think this might be the kind of content you seek: https://news.ycombinator.com/item?id=11244103 https://news.ycombinator.com/item?id=11244103 It’s about data structures for editable text. Surprisingly complex!
• RapperWhoMadeIt 5y ago
  As far as I know, some of these terminal text editors use the ncurses library to handle their "frontend", the way in which they properly display text in your terminal. As mentioned in another comment it is utmost necessary that you feel comfortable reading code in C (with a decent knowledge of syscalls) in order for you to understand the programs' source code. But when you have already achieved that level, you can start learning about ncurses with this basic tutorial [0]. [0] https://tldp.org/HOWTO/NCURSES-Programming-HOWTO/ https://tldp.org/HOWTO/NCURSES-Programming-HOWTO/
• cellularmitosis 5y ago
  Gary Bernhardt of destroyallsoftware did a screencast episode about that: https://www.destroyallsoftware.com/screencasts/catalog/text-editor-from-scratch https://www.destroyallsoftware.com/screencasts/catalog/text-... Note that it’s behind a paywall (his catalog is likely worth it for a curious hacker such as yourself). Watch his “compiler from scratch” for free to get a sense of what his casts are like: https://www.destroyallsoftware.com/screencasts/catalog/a-compiler-from-scratch https://www.destroyallsoftware.com/screencasts/catalog/a-com...
• atfzl 5y ago
  xi-editor https://xi-editor.io/docs.html https://xi-editor.io/docs.html This is written in rust and has docs about rope data structure and editor architecture.
• tester34 5y ago
  Honestly nothing gives as much experience/knowledge as trying to write it yourself and then reading about how other people do it e.g VS Code blog https://code.visualstudio.com/blogs/2018/03/23/text-buffer-reimplementation https://code.visualstudio.com/blogs/2018/03/23/text-buffer-r...
• Someone 5y ago
  https://news.ycombinator.com/item?id=20603567 https://news.ycombinator.com/item?id=20603567, discussing https://viewsourcecode.org/snaptoken/kilo/index.html https://viewsourcecode.org/snaptoken/kilo/index.html probably will give you some hints. Architecture-wise, you can start with an ordered list of lines, with each line stored as a string. Features that complicate things are: - supporting large documents and staying speedy (“replace all” is a good test case) - supporting line wrapping or proportional fonts (makes it harder to translate between screen locations and (line, character) offsets) - supporting Unicode (makes it harder to translate between screen locations and (line, byte position) offsets) - syntax-colouring - plug-ins - regular expression based search (fairly simple for single-line search _if_ you store each line as a string; harder for custom data structures, as you can’t just use a regexp library) - supporting larger-than-memory files (especially on systems without virtual memory, but I think that’s somewhat of a lost art) - safely saving documents even if the disk doesn’t have space for two files (a lost art. Might not even have been fully solved, ever) Edit: you also want to look at https://news.ycombinator.com/item?id=11244103 https://news.ycombinator.com/item?id=11244103, discussing https://ecc-comp.blogspot.com/2015/05/a-brief-glance-at-how-5-text-editors.html https://ecc-comp.blogspot.com/2015/05/a-brief-glance-at-how-...
  • mkhnews 5y ago
    Yea, a double-linked list of lines or what some call a gap-buffer. And how to display it all is another big part.
• grisha 5y ago
  An excellent tutorial for terminal based text editor from scratch: https://viewsourcecode.org/snaptoken/kilo/ https://viewsourcecode.org/snaptoken/kilo/
  • timw4mail 5y ago
    There's also a Rust-native version: https://www.philippflenker.com/hecto/ https://www.philippflenker.com/hecto/
• amichail 5y ago
  For WYSIWYG scientific/math editors, see the open source TeXmacs.
• jdefelice 5y ago
  Not sure if this is entirely what you are after but Andreas Kling of SerenityOS builds an IDE from scratch[1] for his own operating system. While I don't personally know C++ I found the videos interesting and fun to watch. [1] DevTools hacking playlist - https://www.youtube.com/playlist?list=PLMOpZvQB55bfeIHSA71J81OZi3F19lh3d https://www.youtube.com/playlist?list=PLMOpZvQB55bfeIHSA71J8...
• jmiskovic 5y ago
  Lately I've been studying lite editor which is mostly Lua code. It's quite easy to read the code and follow along. Before that I wrote my own editor which kind of works, but it mostly made me appreciate the lite's design. https://github.com/rxi/lite https://rxi.github.io/lite_an_implementation_overview.html https://rxi.github.io/a_simple_undo_system.html
• bmitc 5y ago
  I’m interested in this as well and know nothing but know of a few things. I know of The Craft of Text Editing. https://www.finseth.com/craft/ https://www.finseth.com/craft/ There’s also the Racket editors, which includes a text editor control, in the Racket Graphical Interface Toolkit. It’s what is used to implement the DrRacket IDE. https://docs.racket-lang.org/gui/editor-overview.html https://docs.racket-lang.org/gui/editor-overview.html
• deleted 5y ago
  [deleted]
• jesperlang 5y ago
  The rope data structure is an interesting concept worth checking out! https://en.wikipedia.org/wiki/Rope_(data_structure) https://en.wikipedia.org/wiki/Rope_(data_structure)
  • evanmoran 5y ago
    Glad to see this here. This is was my original thought too as one of the original data structures for text editing. It's used to quickly insert/delete within a very large string. Has anyone used other mechanisms for large document types? It seems like an array of paragraphs, each containing a simple string could probably handle most editing tasks and might be easier to layout and manage. Curious what other data structures people have used!
    • raphlinus 5y ago
      In my opinion, the rope is easily superior to these other types, but it also depends on whether your language supports abstractions well. The problem with an "array of paragraphs" is that it helps when your problem involves the paragraph boundary, but gets in the way when it doesn't. For example, pressing backspace at the beginning of paragraph 2 causes a merge of paragraphs 1 and 2, which is not trivial. With a rope, it's the same as deleting a backspace within a simple string, once you have a good rope library under you. The other big reason to prefer a rope is that the worst case complexity is excellent. Basically all incremental operations are O(log n). With an "array of paragraphs" you get various pathological performance cases such as a huge number of small paragraphs or one very big one. A good rope implementation is not trivial, but when done right it hides its internal complexity from the layer above. And it's a solved problem. There are at least two or three solid rope crates for Rust (just to pick the language I'm most familiar with), and will likely be more, as people find it fun to implement.
      • jhallenworld 5y ago
        I would say the key is to abstract the lower level data structure, so that it could be (relatively) easily swapped-out. So for example, don't start with a doubly-linked list of lines or for sure your upper level code is going to have pointers to line headers. Better to have some kind of smart pointer which is an abstract index into the edit buffer and provide functions like "find next line", "find previous line". BTW, with any of these tree-based structures (like rope) you can store other meta-data in the the headers to make a fast index. For example, I would put a newline count in the rope headers, to make find a particular line a fast operation.
• scandox 5y ago
  You could start by looking at something super simple like Kilo: https://github.com/antirez/kilo https://github.com/antirez/kilo Even I could understand this one pretty well and that's no small matter.
  • fcatalan 5y ago
    I loved following this one a few summers ago and even went beyond the end of the tutorial adding functionality and trying more advanced data structures. I think I would have ended with a completely usable hyper-personal editor but unicode support didn't look like fun.
  • dakra 5y ago
    There is also a nice tutorial that guides you through building the kilo editor: https://viewsourcecode.org/snaptoken/kilo/ https://viewsourcecode.org/snaptoken/kilo/
    • incanus77 5y ago
      This. I was going to come here to post this specifically.
  • sam_lowry_ 5y ago
    The screencast by antirez is a joy to watch.
    • deleted 5y ago
      [deleted]
• nurbel 5y ago
  Have you seen The Architecture of Open Source Applications book [1] ? There is a chapter on Eclipse and its plugin system in it [2]. Lots of other interesting content not really related to text editing however. [1]: http://aosabook.org/en/index.html http://aosabook.org/en/index.html [2]: http://aosabook.org/en/eclipse.html http://aosabook.org/en/eclipse.html
• danielbarla 5y ago
  Object oriented programming and design patterns in particular get a bad rap these days, however, the original Design Patterns book [1] has a case study chapter about designing a WYSIWYG document editor. Also, one of the authors, Erich Gamma, joined Microsoft in 2011, and works on the Monaco suite of components that VS Code is built on top of. So, while I am sure there's a fair bit of difference in the years since they wrote that book, as well as the needs of implementation in JS, I'd say it's a fairly good deep dive into some of the topics from one of the actual architects behind it. Fair warning though, it's a fairly hard book to read. For a lighter, more fun intro to design patterns in particular, I always recommend Game Programming Patterns [2] [1] https://www.amazon.com/Design-Patterns-Elements-Reusable-Object-Oriented/dp/0201633612 https://www.amazon.com/Design-Patterns-Elements-Reusable-Obj... [2] https://gameprogrammingpatterns.com/contents.html https://gameprogrammingpatterns.com/contents.html
  • forinti 5y ago
    I was once tasked with writing a simple text editor. I knew this book inside out so I decided to try putting the text into a tree the way it describes. This made selecting text quite hard. So I gave up and just put the whole thing into an array. It made everything a lot easier.
• armchairhacker 5y ago
  See RSyntaxTextArea (open-source code editor) and the underlying JTextArea / swing code. There are a few weird design decisions especially in swing, but overall it's very easy to read and understand.
• whartung 5y ago
  If you're interested in visual editors and if you're looking for something perhaps more accessible (and I can't honestly say how much it really is, I have not looked at it) then consider taking a look at 'less', the pager. Less does almost everything an "editor" does, at least visually, except change text. It pages through text, forward and backward, line by line, it handles lines that are too long, tab expansion, it searches, even has an extended command set. (Can less do syntax coloring?) It also handles files too big for memory. These are all editor problems. Mind its solutions may not be optimized for an editor, but it's certainly smaller. Today, modern machines "suffer" from "too much" performance which actually frees you from not having to worry so much about the actual backing store, especially early on. Do you really intend to be editing a 2GB file? Honestly, how big is an average text file? And how many billion cycles per second does a modern CPU handle? Sucking the entire file in to RAM, and just pushing stuff around with block moves will take you very far on a modern machine. Not that you should not look at the other data structures (there are many), but you don't have to start there, depending on where your interest lies. Also consider hunting down the book "Software Tools". There's two editions, the original and "Software Tools in Pascal". It's by Kernighan and Plauger. They go through in detail and write a version of the 'ed' line editor. And if you really want to work on an editor, the CP/M world would love a new one. There, it's all about efficiency.
  • nicoburns 5y ago
    > Do you really intend to be editing a 2GB file? I don't know about most people, but I deal with such files (usually JSON) on a weekly basis. It's surely one of the things that makes implementing text editors tricky.
    • hutzlibu 5y ago
      Most people won't, they will just get annoyed, if they accidently open a video and the editor hangs up for some time, trying to process it. And I regulary work with JSON files the size of some MB, which already puts some editors into a real struggle. Out of curiosity, why do you have so big files? (A whole DB as a json file?) And what editor do you use for them?
• antidnan 5y ago
  Prosemirror is a good place to start. https://prosemirror.net/docs/guide/ https://prosemirror.net/docs/guide/ https://marijnhaverbeke.nl/blog/prosemirror.html https://marijnhaverbeke.nl/blog/prosemirror.html They have a well architected model, including plugins to extend functionality, see https://tiptap.dev/ https://tiptap.dev/ which is built ontop of Prosemirror bonus: The author talks about collaborative editing here: https://marijnhaverbeke.nl/blog/collaborative-editing.html https://marijnhaverbeke.nl/blog/collaborative-editing.html
  • codethief 5y ago
    I came here to mention Marijn Haverbeke's blog! His articles (about ProseMirror but also CodeMirror) are full of insights!
  • evilhackerdude 5y ago
    Seconded! Marijn has thought about this stuff a _lot_.
• epberry 5y ago
  Here's 2 articles on building a SQL editor. - https://arctype.com/blog/sql-query-editor-switch/ https://arctype.com/blog/sql-query-editor-switch/ - https://arctype.com/blog/indexeddb-localstorage/ https://arctype.com/blog/indexeddb-localstorage/ Disclaimer: I wrote the 2nd one. My general takeaway is that it gets interesting quickly as the amount of text edited increases.
• marttt 5y ago
  Rob Pike has published several great papers about sam, the Plan 9 text editor he wrote. 1. General overview of the editor; maybe scroll to the "Implementation" section here: http://doc.cat-v.org/plan_9/4th_edition/papers/sam/ http://doc.cat-v.org/plan_9/4th_edition/papers/sam/ Some of the references at the end of that paper may also be relevant or interesting. 2. Tutorial for the command language: http://doc.cat-v.org/bell_labs/sam_lang_tutorial/sam_tut.pdf http://doc.cat-v.org/bell_labs/sam_lang_tutorial/sam_tut.pdf 3. And explanations on structural regular expressions that sam uses: http://doc.cat-v.org/bell_labs/structural_regexps/se.pdf http://doc.cat-v.org/bell_labs/structural_regexps/se.pdf 4. Pike's paper representing Acme, the (sort-of) follow-up editor of sam: http://doc.cat-v.org/plan_9/4th_edition/papers/acme/ http://doc.cat-v.org/plan_9/4th_edition/papers/acme/
• mathieubordere 5y ago
  A small `vi` implementation that might be understandable -> https://git.busybox.net/busybox/tree/editors/vi.c https://git.busybox.net/busybox/tree/editors/vi.c
• mxstbr 5y ago
  If you're curious about looking at something novel and ambitious, Zed has a really interesting writeup of their… unusual architecture involving a GPU-powered UI framework and CRDTs written in Rust: https://zed.dev/ https://zed.dev/
• bakul 5y ago
  I have a different suggestion. Start with a single line. define an api for move, insert and delete. figure out how to display this line, associate the cursor with the current position and how to map keystrokes to these simple functions. Add commands to load/save this one line file. Also some convenience functions (move to star/end of a line etc). Add simple search, search and replace. Next extend this to as many lines as you can display on your screen. Add more commands. Add commands to operate on a sequence of lines. Next allow arbitrary number of lines, and arbitrary length lines (now you can see only a rectangular slice of the file). Add commands to move that window, move lines around etc. Resist the urge to micro optimize or do so early. Just use the simplest data structures that help you write the clearest code. Later you can profile the code and fix slow parts. I suggest this as it will force you to solve problems yourself as opposed reading about other people’s solutions. Basically learn by doing. Learn by struggling to come up with solutions and data structures, thereby developing some insight. The stepwise development should help focus on a small subset of problems at a time. Don’t be afraid of changing data structures as you gain knowledge. In fact write code to make it easy! Use a language that won’t trip you up in low level issues such as memory management. The api will make testing easier. Armed with this knowledge you’ll better appreciate and understand other people’s solutions as well!
  • s3arch 5y ago
    I really appreciate your thoughts. The process of incremental building things. Figure out what needs to be built first and then implement it. Finally compare your work with others. To be frank I used to feel that there must be some sort of deep intellectual concepts lying around, and whatever I try or implement is just dumb. But now after what you described the process and technique seems not so complicated.
    • hellectronic 5y ago
      Yeah its good advice. All Architectures are build like this. Inkrementally.
  • b3morales 5y ago
    Similar suggestion: if you're trying to learn the architecture of a big established project like Vim or Emacs, don't start at the top and try to follow everything. Start with a small feature you would like to tweak, implement, or understand. How does Emacs render the modeline, for example. Just finding the relevant code in the repo will give you some info. Read through it, follow its calls and the data structures it touches. Getting used to the code style might take a bit. You probably won't understand or retain everything about how this piece works, but in hacking on it you will naturally brush up against other areas and build up your familiarity with the project practices and larger structure. And you can use that as a stepping stone to check out another area of functionality.
• bebop 5y ago
  Ewig is an interesting implementation using immutable data structures. https://github.com/arximboldi/ewig https://github.com/arximboldi/ewig Very proof of concept, tries to be a little emacs like. Might be worth checking out.
  • otikik 5y ago
    +1 to this, also give a look at its author's presentation about it: https://www.youtube.com/watch?v=sPhpelUfu8Q https://www.youtube.com/watch?v=sPhpelUfu8Q
• maxk42 5y ago
  vim is kinda hard to follow and emacs is probably too complex. Start with some simpler open-source editors like joe, nano, or gEdit. neovim is probably a lot easier to follow than vim, as well. You could also peruse these github projects: https://github.com/collections/text-editors https://github.com/collections/text-editors
• caconym_ 5y ago
  I wrote a toy text editor for fun a while back. I was aware of the 'rope' and 'gap buffer' data structures, but beyond that I had no knowledge of how "mainstream" text editors are put together. I still don't, really, but I feel that I did get an understanding of many of the core problems. My editor had a modular architecture flexible enough to implement different input modes, including a mostly-complete subset of Vim bindings, and was fast enough to open and edit files on the order of a few hundred megabytes without perceptible slowdown. I'm sure my implementation would have looked insane to anybody who's worked on a real text editor, but I was fairly proud of it myself. Anyway, I guess the point is that it was interesting and rewarding to navigate the core challenges myself. I'm not sure I would have gotten as much out of trying to understand how a massive project like vscode is put together, since the actual text editing functionality is (presumably) a comparatively small part of the software as a whole.
• munificent 5y ago
  I have a little fantasy console project I tinker on that has a built-in text editor. Working on that has been an insightful trip into all of the subtleties of text editing that we intuitively know but don't know we know. Here's one I stumbled onto recently. Say your file looks like this, with the cursor at `|`: 12345|678 123 12345678 Press arrow key down once, and you get: 12345678 123| 12345678 Because the second line's length is too short to preserve the cursor column, the cursor snaps to the end of the line. Note that the cursor is really here. If the user were to press left once, it would take them to: 12|3 Instead of pressing left, say the user presses down again. The cursor is currently on column 4, so you'd expect: 12345678 123 123|45678 But in the text editors I've tested, you get: 12345678 123 12345|678 So the cursor is snapped to the end of the line on line 2, but if the user keeps cursoring past that, the original cursor column is remembered and restored. The text editor has to display where the cursor currently is, but track where it "wants" to be if the line length weren't getting in the way. This is definitely useful behavior. If you arrow down through a bunch of lines of various lengths, it's really annoying if the cursor starts drifting left. But implementing it correctly was much more subtle than I expected. There are all sorts of other edge cases too. If a user presses Command-Up to move the cursor to the beginning of the file, and then presses Down, does the cursor always stay on column one, or does it remember the original column before Command-Up was pressed?
  • dllthomas 5y ago
    This is used as a mechanic in the demo of vim-adventures, incidentally.
• coutego 5y ago
  I was on the same boat many years ago, when I co-founded Monodevelop, a port of #SharpDevelop to Mono. I started by porting the text component, which was the core of the text editor itself. There was a book explaining the implementation of this IDE: https://www.amazon.com/Dissecting-C-Application-Inside-SharpDevelop/dp/1861008171/ref=mp_s_a_1_2?crid=2LD87802QQDY0&keywords=sharpdevelop+dissecting+christian+holm&qid=1641838236&sprefix=sharpdevelop+dissecting+christian+holm%2Caps%2C179&sr=8-2 https://www.amazon.com/Dissecting-C-Application-Inside-Sharp... I think you might find it interesting for what you are trying to do, even though it's a bit old.
• mattferderer 5y ago
  This article on VS Code has been on my read list for a while. Maybe you'll get to it faster than me. Bracket pair colorization 10,000x faster - https://code.visualstudio.com/blogs/2021/09/29/bracket-pair-colorization https://code.visualstudio.com/blogs/2021/09/29/bracket-pair-...
• schemathings 5y ago
  'Challenging projects every programmer should try' - first example is a text editor with some pointers to some data structures to consider, some design patterns to use, and a few links to other resources. https://austinhenley.com/blog/challengingprojects.html https://austinhenley.com/blog/challengingprojects.html
• cdrini 5y ago
  If you're looking for more CS fundamental things--which I take to mean slightly more theoretical--you might be more interested in programming language design. A lot of the features that make an IDE powerful, things like tracking variable references, resolving/unioning types for better autocomplete, dependency graphs, tree parsing, etc., are based on programming language design/theory. I haven't used this, but this seems like an interesting open book on the subject: https://craftinginterpreters.com/ https://craftinginterpreters.com/ If you're interested more in the complexity management--which I would call more software engineering-y--not sure what the best way might be! I know one cool thing about Vs code is the language server provider (LSP). This provides all the IDE-goodies for all the languages Vs code supports in an abstract interface any plugin can implement. The spec has been so popular it's now supported in a bunch of editors! So you could develop an LSP for eg python, and then just have a light wrapper for vim, Vs code, Emacs.
• abagheri43 5y ago
• andrewstuart 5y ago
  >> I tried reading their source code, also of atom, brackets, light table etc. Honestly I don't understand anything. I am not able to make sense of the data flow and the architecture. Even so, this is exactly the right thing to do, except probably you are studying editors that are too big for your purposes - find something smaller in scope. You won't understand without significant effort - you need to put in the work. So, here's what I suggest: 1: keep examining the source code of various editors - however, focus on trying to find small editors that are very focused in what they do. Also look for old editors for operating systems like DOS - they might be smaller in scope and therefore easier to understand. Also look at editors in other languages, such as Pascal. 2: The essence of learning is to implement - so start writing the smallest editor you can. 3: continue searching for and reading any sort of written documentation/blog posts/articles as you are doing. Eventually the light will switch on.
• wnolens 5y ago
  Off topic/meta: I really really love this question, and appreciate so many good replies. <3 HN
• fit2rule 5y ago
• cartesius13 5y ago
  Emacs and vim are way too complicated to just jump directly to the source code. Try something smaller and more manageable. I would personally recommend vis[1]. Also, take a look at https://texteditors.org/ https://texteditors.org/ to discover new editors and resources on design and implementation [1]https://github.com/martanne/vis https://github.com/martanne/vis