14 ms·

Xi-Editor Retrospective

538 points by raphlinus 6y ago

blondin 6y ago
had that feeling... as a side note, i think more and more people are starting to realize that the async paradigm is not a panacea. that paradigm or at least the way we implement it might even mean more trouble than a synchronous counterpart. also, other editors that i feel will be sunset or not worked on at some point are atom and brackets.
- sq_ 6y ago
  Atom already seems to have gone somewhat towards maintenance mode since Microsoft bought Github. Feels like there's a lot less by way of features being released and a lot more just keeping up with ecosystem/OS changes. I could certainly be wrong though.
  - brunoqc 6y ago
    Microsoft probably wants people to use vscode.
    - pjmlp 6y ago
      And given how Office loves React Native, and Microsoft is making it work across all major desktop OSes, with the team continuously bashing Electron's bloat on their talks, I look forward to the way when RN powers VSCode.
  - andrewflnr 6y ago
    Atom competes pretty directly against VS Code doesn't it? It makes sense that Microsoft would consolidate their efforts.
    - sq_ 6y ago
      That's what I thought, too. Hopefully the team that was working on Atom has gotten/will get a chance to bring some of its components/features to VS Code if it doesn't already have them.
      - bhl 6y ago
        There was this one comment when Github Codespaces got released that that's where ex-Atom devs got moved to: https://news.ycombinator.com/item?id=23093150 https://news.ycombinator.com/item?id=23093150
  - ilrwbwrkhv 6y ago
    Atom is open source. The community can always keep working on the project if they care.
  - ComputerGuru 6y ago
    I don’t see anything wrong with that, tbh. While it’s apples and oranges to compare atom to sublime, I think in an extremely fair and objective head-to-head comparison between VS Code and Atom, the latter is just a worse choice on all fronts. It came first, it filled a need, and now there’s something better. (And it and that something better is now both owned/maintained by the same organization, making it even more of a no-brainer to slowly phase the inferior option out.)
- gw 6y ago
  > as a side note, i think more and more people are starting to realize that the async paradigm is not a panacea. This is why libvim (from the oni2 project) is based on vim rather than neovim. Even aside from performance, it is a huge simplification if you can interact with an editor engine in the same process, synchronously. At some point we replaced simple function calls with baroque APIs accessed over localhost...
  - nhooyr 6y ago
    It's not that cut and dry. See https://github.com/onivim/libvim#why-is-libvim-based-on-vim-and-not-neovim https://github.com/onivim/libvim#why-is-libvim-based-on-vim-... It's mostly due to their build system.
    - gw 6y ago
      I've seen that but Bryan said this just a few days ago: > This was a smaller consideration - but more fundamentally, the model Neovim uses for input - queuing it on an event loop and handling it asynchronously - is at odds with what we required - to be able to process the input -> handle updates synchronously. https://news.ycombinator.com/item?id=23628381 https://news.ycombinator.com/item?id=23628381
- The_rationalist 6y ago
  I believe that his async code would have been far more maintainable / readable / expressive if it had used a reactive streams implementation such as colorless kotlin coroutines + flow. Sadly rust doesn't yet have one. What are your thoughts on this topic @raphlinus ?
  - Matthias247 6y ago
    I think this is not about coding style for async code (coroutines vs. callbacks), but about general asynchronity between various processes/threads. If you have any asynchronous background task it means the state of your application can change without the current thread having made any change. And your code needs to account for that possibility, which makes it a lot more complicated.
- CyberDildonics 6y ago
  I don't think you can blame this on an async design. Right off it talks about having the keyboard input as a separate process, not knowing what version of the buffer input should apply to, using timeouts and I guess, doing everything over local IP. It almost reads like a list of what not to do. Why not put the input into the window, put a version number with the buffers and keyboard input and use shared memory? Of course something isn't going to work if you don't confront the problems you have at a fundamental level.
pokstad 6y ago
Is there another modern editor like Xi that strictly separates the GUI front end (view) from the backend (model and controller)?
- sgrove 6y ago
  I think perhaps libvim is like that? https://github.com/onivim/libvim https://github.com/onivim/libvim
- raphlinus 6y ago
  I believe Onivim is in this camp. I don't know how well that's working out for them, or whether there are any others.
  - pokstad 6y ago
    Thanks, this looks amazing. For others looking for more info: https://www.onivim.io/ https://www.onivim.io/
- noobermin 6y ago
  It's saying one thing that you're asking for it when the article explicitly says he believes this was a mistake from the beginning.
  - pokstad 6y ago
    That’s fine. I don’t completely agree with his analysis. Just because he couldn’t make it work doesn’t mean someone else can’t.
    - blub 6y ago
      Maybe someone can make it work, but it's such a terrible idea that everyone else which doesn't use this oddball architecture will be running circles around them.
  - soraminazuki 6y ago
saagarjha 6y ago
Aww, this is really disappointing to hear but an excellent read nonetheless. I always had Xi in the corner of my eye as a very interesting way to make a text editor that had a lot of good things going for it: modularity, native UI, speed…I think if it worked out it would have been really great. It's really sad to hear that it didn't. Thank you, Raph and the rest of the Xi contributors, for working on it for all these years; even if it didn't upend the text editor market I am appreciative of the effort and the lessons you've learned along the way. Some other side conversations I've had about this that might be interesting and relevant: * Native UI is not always the right choice. That might seem surprising coming from someone who pushes native UI extremely strongly in general, and I don't want to be misunderstood as recommending you choose something else for your next project (if you have to ask: don't). But in some very specific contexts, such as performant text editing/rendering, it can be problematic to use a high-level control. In fact, many of the text views I use in are not using the standard platform controls but a custom view doing its own drawing: Terminal.app does its own thing (it doesn't even use NSScrollView!), Xcode has the Pegasus editor written in Swift, iTerm draws text using Metal, and Sublime Text is Skia. Platform controls are meant for 99% of apps that don't need absolute control over performance or text rendering, and sometimes you can push them quite far. But in some cases it's appropriate to abandon them. (I will note that all of the examples handle actual text editing very, very well. If you don't know what the standard shortcuts for editing text are for my platform, you're not going to do it right.) * I love that LSP exists and that it makes my "dumb" editor "smart". But having worked with it a bit, I think it might suffer from some of the same problems: it uses a client-server architecture and JSON to communicate, which means it is not all that performant and needs to keep track of asynchronous state in a coherent way. And not just the state of one document, but the states of many documents, all in different contexts and workspaces, all with their own compiler flags and completion contexts and cursor locations–it is really hard to do this correctly and quickly, although many language servers do an admirably good job. Perhaps it's just "good enough" that it's here to stay, or maybe it has enough support that it can have some of its issues wrinkled out with time.
- raphlinus 6y ago
  Thanks for the kind words! I agree with you that getting shortcuts (and other similar aspects) right is more important than which code is used to implement text editing. We're very much going to try to take that to heart as we implement more text capabilities in Druid. Regarding LSP. Yes, while the functionality is there, it is very difficult to implement a really polished user experience with it. I think a lot of that is in the design of the protocol itself; it's hard to avoid annotating the wrong region when there are concurrent edits. It's interesting to think about how to do better, but I fear you'd be bringing in a lot of complexity, especially if you started doing some form of Operational Transformation. I would highly recommend that if someone were to take this on, they study xi in detail, both for what it got right and what was difficult.
ZeroCool2u 6y ago
I've been watching/rooting for Xi from a distance for a few years now. This is an excellent retrospective. Well written, but also very honest, especially about those all too typical project highs and lows. I remember looking at one of Raph's early presentations and being both convinced by his arguments for and impressed with the design decisions made, the same ones he outlines here. After doing some of my own comparatively rudimentary research, coming to similar conclusions as he did. It left me very excited to watch the project develop over time. One thing I was wondering about is that the Xi website contains a list of front ends similar to the list in the GitHub repo, but includes a Flutter implementation for Fuchsia[1] that is now a dead link and missing from the GitHub list. He mentions it briefly in this retrospective as well. Any ideas what happened to it? [1]: https://xi-editor.io/frontends.html https://xi-editor.io/frontends.html
- raphlinus 6y ago
  It was maintained for a little while after I left Fuchsia (almost two years ago now), but the pace of development on Fuchsia is such that if it's not actively maintained, it bitrots very quickly. And there aren't really any devices running Fuchsia that need text editing at scale.
Firehawke 6y ago
It's disappointing that things didn't work out. I'd looked at Xi about a year ago when I was finally running out of patience with Atom's terrible performance, but the lack of a workable Windows solution pushed me to VSCode in the end.
blub 6y ago
Am I the only one on HN which thought this was a bad idea all around from the beginning? I remember that people were very enthusiastic back then because of the Rust hype and the author's pedigree, so critics were quickly silenced. For me, picking Rust and the JSON-based IPC were huge red flags and this post-mortem confirms that. But what I find odd is that many are still not willing to accept the conclusion that using Rust for a UI-intensive app is a bad idea. And using multiple languages in a project and splitting in back-end vs. front-end or lib + UI is also quite popular around here, unjustifiably in my opinion when one considers the extra complexity involved.
- GolDDranks 6y ago
  I was quite enthusiastic about it, and don't see even now why using Rust for a UI-intensive app would be a bad idea in general. If doing an app with a "monolithic" design, the lack of good frameworks is a problem, but that's improving, so it wouldn't be a good idea if you want to see results right now, but I'm pretty comfident that's gonna change. However, I fail to see how the choice of Rust is relevant in this particular project, as the UI part was meant to be programmed in any language, making Rust an irrelevant part of the equation. On another note, I'm baffled why you are downvoted. I find your comment negative, but not unreasonable.
- okatsu 6y ago
  I don't have any opinion on Rust but reading about microservices, JSON and IPC in the context of a high-performance editor did make me raise an eyebrow.
- bhl 6y ago
  I wasn't there when xi-editor started, but writing a text editor in Rust and splitting the backend from the front-end still seems like a good idea. Why? You could compile Rust to Web-assembly -- getting a more performant text editor than if it was written in Javascript -- and then develop your browser or native frontend in HTML/CSS/JS.
- kllrnohj 6y ago
  > For me, picking Rust and the JSON-based IPC were huge red flags and this post-mortem confirms that. But what I find odd is that many are still not willing to accept the conclusion that using Rust for a UI-intensive app is a bad idea. This postmortem says Rust was, and still is, a great fit for the problem domain for which it was used. Where do you reach the conclusion that Rust is bad for UI-intensive apps? That claim appears to have no relation to the post-mortem as Xi didn't use Rust for the UI in the first place? Things might have been better if Rust was used for the UI if anything, as attempting to be native instead was one of the problem areas.
cjohansson 6y ago
Front-end / interace needs to be synchronously controlled, it doesn’t feel right otherwise. I followed xi-editor but it never was faster than emacs on opening really large files and editing them so it never had a use case for me
- fluffything 6y ago
  I use emacs full time, and I open large files on xi (>300 GB). When I handle files in the 500 MB-10 GB range on emacs, I basically need to disable everything and use "text-mode" exclusively, and yet it still struggles if the file is only a couple of GBs long....
chubot 6y ago
Tangent/clarification: the regex slowness issue is almost certainly due to using Perl-style exponential time regexes (which are required by Sublime syntax definitions) rather than "regular languages". The syntect library mentioned uses fancy-regex, which is explicitly different than Rust's default regex crate, in that it supports regex constructs that require exponential backtracking: https://github.com/fancy-regex/fancy-regex https://github.com/fancy-regex/fancy-regex In particular, it uses backtracking to implement "fancy" features such as look-around and backtracking, which are not supported in purely NFA-based implementations (exemplified by RE2, and implemented in Rust in the regex crate). I think "regexes" have gotten bad name so I call them "regular languages" now. That is, RE2, rust/regex, and libc are "regular language" engines while Perl/Python/PCRE/Java/etc. are "regex" engines. In my experience, lexing with regular languages can beat hand written code. In the case of Oil vs. zsh, it's beating hand-written C code by an order of magnitude (9-10x). http://www.oilshell.org/blog/2020/01/parser-benchmarks.html http://www.oilshell.org/blog/2020/01/parser-benchmarks.html https://lobste.rs/s/77nu3d/oil_s_parser_is_160x_200x_faster_than_it_was https://lobste.rs/s/77nu3d/oil_s_parser_is_160x_200x_faster_... --- First, TextMate / Sublime style syntax highlighting is not really all that great. It is quite slow, largely because it grinds through a lot of regular expressions with captures pedantic: I think the issue is probably more lookahead than captures ? You can do captures quickly in a "regular language" engine. It may be surprising just how much slower regex-based highlighting is than fast parsers. The library that xi uses, syntect, is probably the fastest open source implementation in existence (the one in Sublime is faster but not open source). Even so, it is approximately 2500 times slower for parsing Markdown than pulldown-cmark.
- an_opabinia 6y ago
  The bigger problem I feel is with this: > On the plus side, there is a large and well-curated open source collection of syntax definitions Whenever I tried to get Java highlighting updated in GitHub for “var” my issue is closed minutes later by people who don’t work at GitHub telling me to open an issue with some random unpaid TextMate / Sublime repo. Really the overall problem is not owning the front end. It means you can’t do things like fix syntax highlighting, at least as important to programming as high performance word wrap.
- 6y ago
m0zg 6y ago
This was solving problems nobody really has, so its fate was predictable. Here's what I think could take off like a rocket: a VSCode-like experience that runs completely in terminal, and therefore does not require a "remote" of any kind. Better yet if it takes most of the same plugins, to reuse the immense amount of work people have done there (and are not going to re-do for some hotshot new editor). Note how none of the above mentions "async", "ropes" or "crdt".
- saagarjha 6y ago
  I think it had a lot of goals that matched I'd like from my text editor: * Using native technologies * Fast * Modular and extensible * Open source
  - m0zg 6y ago
    And I'm not disputing that. But one goal eclipses all others: * Usable
    - saagarjha 6y ago
      That is correct, hence why I never used Xi. But if it did end up usable I could imagine myself switching to it.
- Arnavion 6y ago
  I don't want to speak too much for raphlinus here [1], but I don't think it was ever the goal to make an editor that gets a lot of downloads like a startup pitching a product. All the things you've listed are non-goals, or at least lower-priority goals. The goal was to make an editor based on sound technology principles, and that included investigating said technology principles. It turned out some of those principles were bad ideas, and that's that. [1]: My involvement with xi was limited to being part of conversations he had about rendering performance on Windows in the winapi crate's IRC channel.
  - raphlinus 6y ago
taeric 6y ago
The assertion that gpu is required for good text rendering caught me off guard. I can't claim it is wrong, but it does feel like it should be wrong.
- saagarjha 6y ago
  I think the point being made is that it is required for fast text rendering.
  - taeric 6y ago
    Still feels like that should not be the case. What changed so heavily in the last decade? 2d rendering used to be hella fast without the gpu. Right?
    - chubot 6y ago
      Screen resolution / 4k monitors? And maybe font complexity (i.e. lots of web pages seem to be downloading fonts now)
      - taeric 6y ago
        I was curious if font complexity could be it. Does feel like a good guess. Resolution could also matter, but we did have higher resolution displays back in the day, as well. Feels like 2d acceleration somehow regressed.
    - kllrnohj 6y ago
      Rendering models & expectations changed. Once upon a time (a decade ago) it was OK for apps to directly draw to the front window, compositing wasn't a super established thing. This saves on precious memory bandwidth, which CPUs didn't really get all that much more of over the last decade. However now that GPU composition has to happen, GPUs actually don't super like the linear formats that CPUs write to. They want swizzled textures, and they keep those formats private. And on not-heterogeneous memory systems, the buffer the CPU writes into needs to be sent over to the GPU as well. Meanwhile screen resolutions & pixel counts skyrocket, as did UI visual & animation expectations. Being responsive is a lot easier than being fluid. Especially if you're trying to be fluid on a high-resolution and/or high-refresh rate display. And this is all while ignoring things like Apple's "high DPI" handling, which is to just say fuck it and downscale instead. Which means you're pushing resolutions far higher than the display's actual resolution quite commonly.
mwcampbell 6y ago
> There is no such thing as native GUI I understand why you were disillusioned with the performance of native GUIs. But in other areas, such as accessibility, nativeness absolutely does help. I'm guessing you know this; I just want to make sure other developers get this message. Here is a quite comprehensive list of the benefits of a native GUI for accessibility, particularly on macOS: https://news.ycombinator.com/item?id=18585443 https://news.ycombinator.com/item?id=18585443
- microcolonel 6y ago
  The accessibility APIs can be programmed against by anyone. If there is a way to make something like this accessible to screen readers in some useful way, it would likely involve specific accessibility work either way.
  - saagarjha 6y ago
    Platform widgets do a lot of the accessibility work for you. In some cases you can get away with doing zero* work. *Or very close to zero
  - mwcampbell 6y ago
    > If there is a way to make something like this accessible to screen readers in some useful way If by "something like this" you mean a text editor, then there's no question; they can be made accessible with a screen reader, and some complex ones (e.g. Visual Studio Code, Visual Studio, and Xcode) are accessible already. > it would likely involve specific accessibility work either way. For the main UI of a full-featured programmer's editor or word processor, that's definitely true. What I want to avoid is developers using a custom GUI toolkit where the platform's native toolkit would do, because they want the best possible performance (regardless of whether the platform's toolkit is good enough), and forfeiting all of the accessibility benefits that the platform's native toolkit brings more or less automatically.
    - notriddle 6y ago
      One of the cliches of modern business is: don't outsource your core competencies. If you are developing a text editor, you should put the time into your text editor. It's the thing that distinguishes you from your competitors. And it's the part that users interact with the most. Your users are likely to ask for features, like vi keybindings, that you can't easily build with the OS-native text field anyway. Incidental forms, like your preferences screen? Unless you have a lot of time and money to burn, the OS widgets will probably be better than yours.
stephc_int13 6y ago
Too much over-engineering here. It is always better to start with simple, dumb and reliable code, with minimal architecture, and then grow organically. With experience, I tend to write the less smart code possible, I felt in love with the power of brute-forcing everything.
- magicalist 6y ago
  > With experience, I tend to write the less smart code possible I'm always amused by the hn threads patting ourselves on the backs for in depth technical discussions[1], when so often the top comments are basically non-specific thought leader tweets. For those that got this far, the article is well worth reading about a design space with difficult tradeoffs, and there are comments actually engaging with the content below this. [1] today brings us multiple instances in https://news.ycombinator.com/item?id=23664067 https://news.ycombinator.com/item?id=23664067
  - saagarjha 6y ago
    I found some interesting ones in this thread!
- microcolonel 6y ago
  The thing is, that is the story of the existing big editors, and we know where that story goes.
  - eeereerews 6y ago
    No, where does it go? Are the big editors unsuccessful?
    - microcolonel 6y ago
      They are unscalable. I love Emacs, and it continues to be my editor of choice, but it doesn't seem the design is amenable to adding good threading primitives. NeoVIM seems like a success story in this regard, though VIMScript is a much less pleasant extension language than Elisp, which is not perfect, but is at least a real programming language. There may be a way forward for Emacs, building a sandbox that looks like a full Emacs instance to existing Elisp, and slowly factoring out the whole-editor blocking issues; but that might be almost as complicated as starting fresh, with fewer benefits.
akhilcacharya 6y ago
I was personally really intrigued by the ambitions and design decisions in Xi, but it somehow lacked the one thing I can’t live without in a non-terminal editor - a tree view, at least when I first tried the OSX version. Perhaps that changed, but you’d think that would take higher feature precedence over something like collaborative editing right?
bhl 6y ago
> And it [syntax highlighting] basically doesn’t fit nicely in the CRDT model at all, as that requires the ability to resolve arbitrarily divergent edits between the different processes (imagine that one goes offline for a bit, types a bit, then the language server comes back online and applies indentation). What's a rationale for having syntax highlighting server-side as opposed to client-side? I'm working on side project that uses ProseMirror and CRDTs through yjs, and the idea of having a server for syntax highlighting for editable text never occurred to me. > Even so, xray was a bit of a wake-up call for me. It was evidence that the vision I had set out for xi was not quite compelling enough that people would want to join forces. I think in hindsight this was a virtue. If you look at the xray project, it died after the Microsoft acquisition of Github: https://github.com/atom-archive/xray/issues/177 https://github.com/atom-archive/xray/issues/177. By not combining projects, it allowed for redundancy.
- saagarjha 6y ago
  Your server generally has a better idea about the language than you do, presumably.
  - bhl 6y ago
    I would only think that's the case when the language of the file you're editing is unknown, so you would have to do some server-side analysis of the entire file and make an educated guess. For example, https://github.com/github/linguist https://github.com/github/linguist. But if the language is known in advance, then you could send the grammar rules to the client.
    - saagarjha 6y ago
      A server also means it still works when the language is not simple enough to condense into simple grammar rules :)
- raphlinus 6y ago
  So, just syntax highlighting is not compelling to move out to a server. But part of my idea is that if you have async to hide server latency, so it's not slowing down typing, maybe you can do a much deeper analysis than just surface syntax. Right now you get basically instant coloring and then much slower but more accurate feedback that's generally comparable to an incremental recompile. I was thinking that a language server might be able to give you quick feedback on some aspects of your program before having to do all of it. Perhaps an even better motivation is indentation. When I use editors with regex-based indentation (the norm), I'm regularly annoyed when it gets it wrong. Being able to budget a few milliseconds or tends of milliseconds to do indentation that exactly matches what rustfmt or gofmt would recommend would be worth it, imho.
ncmncm 6y ago
> it might be able to safely save the file, but you can also do that by frequently checkpointing It is always amazing to discover niches where the conceptual and practical power of logging has not yet penetrated.
namelosw 6y ago
This is an excellent read. I was always wondering if CRDT would really achieve encapsulation, but this kind of things are very hard to know without trying. Thank you for you works!
mov_ 6y ago
On the github page https://github.com/xi-editor/xi-editor https://github.com/xi-editor/xi-editor: JSON. The protocol for front-end / back-end communication, as well as between the back-end and plug-ins, is based on simple JSON messages. I considered binary formats, but the actual improvement in performance would be completely in the noise. Using JSON considerably lowers friction for developing plug-ins, as it’s available out of the box for most modern languages, and there are plenty of the libraries available for the other ones. 4 years later: The choice of JSON was controversial from the start. It did end up being a source of friction, but for surprising reasons. For one, JSON in Swift is shockingly slow. Surprising reasons?!
- raphlinus 6y ago
  Yes. In this discussion and on Reddit, people still talk about binary vs textual as the source of the problem, but I've argued (based on empirical data) that the lexical details are not the reason for the performance problems. Also, Swift is marketed as a fast language (also based on LLVM), yet in my measurements it's 20x to 50x slower than Rust for JSON processing. I found that surprising. Would you not?
  - mov_ 6y ago
    I agree with you, it's weird that Swift JSON is so slow and binary isn't necessarily faster. My exasperation is you were surprised by JSON becoming an issue, when this was the most controversial design decision about the project. To me, it's completely unsurprising.
    - __s 6y ago
      They're saying the reasons were surprising, ie they expected it to be controversial for different reasons
  - shakna 6y ago
    It wouldn't be especially surprising to me. JSON's parsing performance is difficult, whether or not the language doing the parsing is fast or not. There's a reason that there are almost always libraries claiming faster JSON performance, regardless of the language. Go is marketed as a fast language, and they're still trying to build a high performance JSON parser [0]. Parsing JSON will always be slower than most of the alternatives. [0] https://dave.cheney.net/high-performance-json.html https://dave.cheney.net/high-performance-json.html
dilandau 6y ago
Honest and thoughtful retrospective, obviously with no hidden agenda, and written with humility. Fantastic stuff.
nickpinkston 6y ago
This is how you write Xi Thought, and if you contribute enough code you can become the Xi Dada :-)
rl3 6y ago
>One of those was learning about the xray project. I probably shouldn’t have taken this personally, ... I've run into that phenomenon a lot with various startup projects over the years. It's soul crushing, and often an immediate reaction is to assume certain projects are direct competition, rendering your project fruitless. In retrospect, this ends up almost never being the case. In a way, you become your own worst enemy. The correct solution is to simply ignore it and press on, but this is certainly easier said than done—especially when morale is already low, and without the benefit of hindsight. >Perhaps more important, because these projects are more ambitious than one person could really take on, the community started around xi-editor is evolving into one that can sustain GUI in Rust. In the end, I still think that counts as a success. Perhaps it will prove to be an even greater success than the original goal. Thanks for all these write-ups, they're excellent. Like other commenters, I've been watching at a distance for years now. Your work is much appreciated.
- smichel17 6y ago
  It's not a startup and the alternatives are not competitors, but I've had a similar experience affecting my motivation to work on Red Moon. There's two other screen filter apps on F-Droid, and at one point I proposed that we [work together], either merging our apps or creating shared libraries. While the other devs initially expressed some interest, they weren't willing/able to put in the effort and nothing came of it. This has killed my motivation to work on Red Moon. It feels like a waste of time, duplicating other people's efforts. That's not intrinsic to other people working separately to solve the same problem. If Red Moon were an educational project, or if it took a unique approach to screen filtering (like the other project I'm part of and [crowdfunding]), then I'd feel differently. But my main motivation was creating a FLO screen filter app, since previously I depended on the proprietary Twilight. And while there's a handful of valid approaches to screen filtering, the bulk of the effort is on UX, state transitions, and working around the kinks of Android, all of which are shared. Note: I don't bear any hard feelings towards the other devs, genuinely -- the most frustrating bit was [lack of communication], and even that is totally understandable; it's hard to say no to something you like the idea of, even to yourself (eg, acknowledging you'll never get to that project near the bottom of your list). [work together]: https://github.com/LibreShift/red-moon/issues/222 https://github.com/LibreShift/red-moon/issues/222 [lack of communication]: https://github.com/LibreShift/TransitionScheduler/issues/14 https://github.com/LibreShift/TransitionScheduler/issues/14 [crowdfunding]: https://wiki.snowdrift.coop/market-research/other-crowdfunding https://wiki.snowdrift.coop/market-research/other-crowdfundi...
deleted 6y ago
[deleted]
ryuukk_ 6y ago
tldr: rust sucks
jussij 6y ago
> First, TextMate / Sublime style syntax highlighting is not really all that great. It is quite slow, largely because it grinds through a lot of regular expressions with captures, and it is also not very precise. The tree-sitter framework provides a pretty good engine for syntax highlight: https://github.com/tree-sitter/tree-sitter https://github.com/tree-sitter/tree-sitter
fractalb 6y ago
What is the successful text-editor project in Rust now? Xray seems to have been abandoned too.
taooftea 6y ago
This editor will be blocked in China, obviously
signa11 6y ago
one of the design goals is > CRDT as a mechanism for concurrent modification. i am genuinely wondering if concurrent modification is really, truly needed ?
amedvednikov 6y ago
> When doing some performance work on xi, I found to my great disappointment that performance of these so-called “native” UI toolkits was often pretty poor, even for what you’d think of as the relatively simple task of displaying a screenful of text. The same thing I experienced. Text rendering performance is terrible with Cocoa, and even worse with Win32. Calculating text width is an order of magnitude slower than with FreeType. That's why for my UI framework I also went from using "native" APIs to custom GPU rendering.
- mwcampbell 6y ago
  If your framework has a broad audience, please make sure it accounts for the full complexity of text rendering [1] and editing [2], particularly when it comes to internationalization. You may find that there are good reasons for Cocoa to be as slow as it is in the measurement phase, if not the rendering phase. And of course, please don't forget about accessibility. [1]: https://gankra.github.io/blah/text-hates-you/ https://gankra.github.io/blah/text-hates-you/ [2]: https://lord.io/blog/2019/text-editing-hates-you-too/ https://lord.io/blog/2019/text-editing-hates-you-too/
jfkebwjsbx 6y ago
I didn't know about Xi, but JSON, RPC, multiprocess, microservices, Async... are all red flags.
fluffything 6y ago
Maybe the time is just not right for Xi yet. I wish that after druid becomes production ready you'll pick it up again. The coolest thing about Xi is that some have attempted to make it a vim replacement, others an emacs replacement, etc. I wish it becomes a framework that can be used to build better text editors in general.
- raphlinus 6y ago
  I also hope to be able to pick up text editing again, but make no promises. If not, then I hope Druid provides a solid foundation for others to build the dream editor I envisioned.
Myrmornis 6y ago
I (naively) would have thought that the IPC / multi-process model strongly conflicted with the extremely high performance goals (e.g. time between keypress and painting). Can anyone explain why my instinct there is wrong?
hyperion2010 6y ago
> Looking back, I see much of the promise of modular software as addressing goals related to project management, not technical excellence. I couldn't find this point discussed elsewhere in the comments. I think this is one of the most important sentences in the whole article. I know this is essentially just another perspective on Conway's law, but I think it is important to identify factors like this early in a project if at all possible so that human factors around development can be factored out of engineering decisions. I think that this is especially critical for open projects that want to achieve long term sustainability. Open communities don't have the resources to maintain all the packages, even if they use the same build system and follow the same pattern, even if most of the maintenance can be automated, some human being still has to deal each of those systems separately because they are part of the environment surrounding that individual part of the project.
archagon 6y ago
Thank you for these fantastically in-depth and introspective blog posts! It’s a pleasure to observe how a single, experimental project can spawn a dozen different research branches, and it’s something to aspire to in my own work. (I have probably missed many such opportunities due to laziness and lack of curiosity.)
- raphlinus 6y ago
  Glad you enjoy them! I certainly plan to continue writing.

538 points by raphlinus 6y ago

had that feeling... as a side note, i think more and more people are starting to realize that the async paradigm is not a panacea. that paradigm or at least the way we implement it might even mean more trouble than a synchronous counterpart. also, other editors that i feel will be sunset or not worked on at some point are atom and brackets.

sq_ 6y ago
Atom already seems to have gone somewhat towards maintenance mode since Microsoft bought Github. Feels like there's a lot less by way of features being released and a lot more just keeping up with ecosystem/OS changes. I could certainly be wrong though.
- brunoqc 6y ago
  Microsoft probably wants people to use vscode.
  - pjmlp 6y ago
    And given how Office loves React Native, and Microsoft is making it work across all major desktop OSes, with the team continuously bashing Electron's bloat on their talks, I look forward to the way when RN powers VSCode.
- andrewflnr 6y ago
  Atom competes pretty directly against VS Code doesn't it? It makes sense that Microsoft would consolidate their efforts.
  - sq_ 6y ago
    That's what I thought, too. Hopefully the team that was working on Atom has gotten/will get a chance to bring some of its components/features to VS Code if it doesn't already have them.
    - bhl 6y ago
      There was this one comment when Github Codespaces got released that that's where ex-Atom devs got moved to: https://news.ycombinator.com/item?id=23093150 https://news.ycombinator.com/item?id=23093150
- ilrwbwrkhv 6y ago
  Atom is open source. The community can always keep working on the project if they care.
- ComputerGuru 6y ago
  I don’t see anything wrong with that, tbh. While it’s apples and oranges to compare atom to sublime, I think in an extremely fair and objective head-to-head comparison between VS Code and Atom, the latter is just a worse choice on all fronts. It came first, it filled a need, and now there’s something better. (And it and that something better is now both owned/maintained by the same organization, making it even more of a no-brainer to slowly phase the inferior option out.)
gw 6y ago
> as a side note, i think more and more people are starting to realize that the async paradigm is not a panacea. This is why libvim (from the oni2 project) is based on vim rather than neovim. Even aside from performance, it is a huge simplification if you can interact with an editor engine in the same process, synchronously. At some point we replaced simple function calls with baroque APIs accessed over localhost...
- nhooyr 6y ago
  It's not that cut and dry. See https://github.com/onivim/libvim#why-is-libvim-based-on-vim-and-not-neovim https://github.com/onivim/libvim#why-is-libvim-based-on-vim-... It's mostly due to their build system.
  - gw 6y ago
    I've seen that but Bryan said this just a few days ago: > This was a smaller consideration - but more fundamentally, the model Neovim uses for input - queuing it on an event loop and handling it asynchronously - is at odds with what we required - to be able to process the input -> handle updates synchronously. https://news.ycombinator.com/item?id=23628381 https://news.ycombinator.com/item?id=23628381
The_rationalist 6y ago
I believe that his async code would have been far more maintainable / readable / expressive if it had used a reactive streams implementation such as colorless kotlin coroutines + flow. Sadly rust doesn't yet have one. What are your thoughts on this topic @raphlinus ?
- Matthias247 6y ago
  I think this is not about coding style for async code (coroutines vs. callbacks), but about general asynchronity between various processes/threads. If you have any asynchronous background task it means the state of your application can change without the current thread having made any change. And your code needs to account for that possibility, which makes it a lot more complicated.
CyberDildonics 6y ago
I don't think you can blame this on an async design. Right off it talks about having the keyboard input as a separate process, not knowing what version of the buffer input should apply to, using timeouts and I guess, doing everything over local IP. It almost reads like a list of what not to do. Why not put the input into the window, put a version number with the buffers and keyboard input and use shared memory? Of course something isn't going to work if you don't confront the problems you have at a fundamental level.

pokstad 6y ago

Is there another modern editor like Xi that strictly separates the GUI front end (view) from the backend (model and controller)?

sgrove 6y ago
I think perhaps libvim is like that? https://github.com/onivim/libvim https://github.com/onivim/libvim
raphlinus 6y ago
I believe Onivim is in this camp. I don't know how well that's working out for them, or whether there are any others.
- pokstad 6y ago
  Thanks, this looks amazing. For others looking for more info: https://www.onivim.io/ https://www.onivim.io/
noobermin 6y ago
It's saying one thing that you're asking for it when the article explicitly says he believes this was a mistake from the beginning.
- pokstad 6y ago
  That’s fine. I don’t completely agree with his analysis. Just because he couldn’t make it work doesn’t mean someone else can’t.
  - blub 6y ago
    Maybe someone can make it work, but it's such a terrible idea that everyone else which doesn't use this oddball architecture will be running circles around them.
- soraminazuki 6y ago

saagarjha 6y ago

Aww, this is really disappointing to hear but an excellent read nonetheless. I always had Xi in the corner of my eye as a very interesting way to make a text editor that had a lot of good things going for it: modularity, native UI, speed…I think if it worked out it would have been really great. It's really sad to hear that it didn't. Thank you, Raph and the rest of the Xi contributors, for working on it for all these years; even if it didn't upend the text editor market I am appreciative of the effort and the lessons you've learned along the way. Some other side conversations I've had about this that might be interesting and relevant: * Native UI is not always the right choice. That might seem surprising coming from someone who pushes native UI extremely strongly in general, and I don't want to be misunderstood as recommending you choose something else for your next project (if you have to ask: don't). But in some very specific contexts, such as performant text editing/rendering, it can be problematic to use a high-level control. In fact, many of the text views I use in are not using the standard platform controls but a custom view doing its own drawing: Terminal.app does its own thing (it doesn't even use NSScrollView!), Xcode has the Pegasus editor written in Swift, iTerm draws text using Metal, and Sublime Text is Skia. Platform controls are meant for 99% of apps that don't need absolute control over performance or text rendering, and sometimes you can push them quite far. But in some cases it's appropriate to abandon them. (I will note that all of the examples handle actual text editing very, very well. If you don't know what the standard shortcuts for editing text are for my platform, you're not going to do it right.) * I love that LSP exists and that it makes my "dumb" editor "smart". But having worked with it a bit, I think it might suffer from some of the same problems: it uses a client-server architecture and JSON to communicate, which means it is not all that performant and needs to keep track of asynchronous state in a coherent way. And not just the state of one document, but the states of many documents, all in different contexts and workspaces, all with their own compiler flags and completion contexts and cursor locations–it is really hard to do this correctly and quickly, although many language servers do an admirably good job. Perhaps it's just "good enough" that it's here to stay, or maybe it has enough support that it can have some of its issues wrinkled out with time.

raphlinus 6y ago
Thanks for the kind words! I agree with you that getting shortcuts (and other similar aspects) right is more important than which code is used to implement text editing. We're very much going to try to take that to heart as we implement more text capabilities in Druid. Regarding LSP. Yes, while the functionality is there, it is very difficult to implement a really polished user experience with it. I think a lot of that is in the design of the protocol itself; it's hard to avoid annotating the wrong region when there are concurrent edits. It's interesting to think about how to do better, but I fear you'd be bringing in a lot of complexity, especially if you started doing some form of Operational Transformation. I would highly recommend that if someone were to take this on, they study xi in detail, both for what it got right and what was difficult.

ZeroCool2u 6y ago

I've been watching/rooting for Xi from a distance for a few years now. This is an excellent retrospective. Well written, but also very honest, especially about those all too typical project highs and lows. I remember looking at one of Raph's early presentations and being both convinced by his arguments for and impressed with the design decisions made, the same ones he outlines here. After doing some of my own comparatively rudimentary research, coming to similar conclusions as he did. It left me very excited to watch the project develop over time. One thing I was wondering about is that the Xi website contains a list of front ends similar to the list in the GitHub repo, but includes a Flutter implementation for Fuchsia[1] that is now a dead link and missing from the GitHub list. He mentions it briefly in this retrospective as well. Any ideas what happened to it? [1]: https://xi-editor.io/frontends.html https://xi-editor.io/frontends.html

raphlinus 6y ago
It was maintained for a little while after I left Fuchsia (almost two years ago now), but the pace of development on Fuchsia is such that if it's not actively maintained, it bitrots very quickly. And there aren't really any devices running Fuchsia that need text editing at scale.

Firehawke 6y ago

It's disappointing that things didn't work out. I'd looked at Xi about a year ago when I was finally running out of patience with Atom's terrible performance, but the lack of a workable Windows solution pushed me to VSCode in the end.

blub 6y ago

Am I the only one on HN which thought this was a bad idea all around from the beginning? I remember that people were very enthusiastic back then because of the Rust hype and the author's pedigree, so critics were quickly silenced. For me, picking Rust and the JSON-based IPC were huge red flags and this post-mortem confirms that. But what I find odd is that many are still not willing to accept the conclusion that using Rust for a UI-intensive app is a bad idea. And using multiple languages in a project and splitting in back-end vs. front-end or lib + UI is also quite popular around here, unjustifiably in my opinion when one considers the extra complexity involved.

GolDDranks 6y ago
I was quite enthusiastic about it, and don't see even now why using Rust for a UI-intensive app would be a bad idea in general. If doing an app with a "monolithic" design, the lack of good frameworks is a problem, but that's improving, so it wouldn't be a good idea if you want to see results right now, but I'm pretty comfident that's gonna change. However, I fail to see how the choice of Rust is relevant in this particular project, as the UI part was meant to be programmed in any language, making Rust an irrelevant part of the equation. On another note, I'm baffled why you are downvoted. I find your comment negative, but not unreasonable.
okatsu 6y ago
I don't have any opinion on Rust but reading about microservices, JSON and IPC in the context of a high-performance editor did make me raise an eyebrow.
bhl 6y ago
I wasn't there when xi-editor started, but writing a text editor in Rust and splitting the backend from the front-end still seems like a good idea. Why? You could compile Rust to Web-assembly -- getting a more performant text editor than if it was written in Javascript -- and then develop your browser or native frontend in HTML/CSS/JS.
kllrnohj 6y ago
> For me, picking Rust and the JSON-based IPC were huge red flags and this post-mortem confirms that. But what I find odd is that many are still not willing to accept the conclusion that using Rust for a UI-intensive app is a bad idea. This postmortem says Rust was, and still is, a great fit for the problem domain for which it was used. Where do you reach the conclusion that Rust is bad for UI-intensive apps? That claim appears to have no relation to the post-mortem as Xi didn't use Rust for the UI in the first place? Things might have been better if Rust was used for the UI if anything, as attempting to be native instead was one of the problem areas.

cjohansson 6y ago

Front-end / interace needs to be synchronously controlled, it doesn’t feel right otherwise. I followed xi-editor but it never was faster than emacs on opening really large files and editing them so it never had a use case for me

fluffything 6y ago
I use emacs full time, and I open large files on xi (>300 GB). When I handle files in the 500 MB-10 GB range on emacs, I basically need to disable everything and use "text-mode" exclusively, and yet it still struggles if the file is only a couple of GBs long....

chubot 6y ago

Tangent/clarification: the regex slowness issue is almost certainly due to using Perl-style exponential time regexes (which are required by Sublime syntax definitions) rather than "regular languages". The syntect library mentioned uses fancy-regex, which is explicitly different than Rust's default regex crate, in that it supports regex constructs that require exponential backtracking: https://github.com/fancy-regex/fancy-regex https://github.com/fancy-regex/fancy-regex In particular, it uses backtracking to implement "fancy" features such as look-around and backtracking, which are not supported in purely NFA-based implementations (exemplified by RE2, and implemented in Rust in the regex crate). I think "regexes" have gotten bad name so I call them "regular languages" now. That is, RE2, rust/regex, and libc are "regular language" engines while Perl/Python/PCRE/Java/etc. are "regex" engines. In my experience, lexing with regular languages can beat hand written code. In the case of Oil vs. zsh, it's beating hand-written C code by an order of magnitude (9-10x). http://www.oilshell.org/blog/2020/01/parser-benchmarks.html http://www.oilshell.org/blog/2020/01/parser-benchmarks.html https://lobste.rs/s/77nu3d/oil_s_parser_is_160x_200x_faster_than_it_was https://lobste.rs/s/77nu3d/oil_s_parser_is_160x_200x_faster_... --- First, TextMate / Sublime style syntax highlighting is not really all that great. It is quite slow, largely because it grinds through a lot of regular expressions with captures pedantic: I think the issue is probably more lookahead than captures ? You can do captures quickly in a "regular language" engine. It may be surprising just how much slower regex-based highlighting is than fast parsers. The library that xi uses, syntect, is probably the fastest open source implementation in existence (the one in Sublime is faster but not open source). Even so, it is approximately 2500 times slower for parsing Markdown than pulldown-cmark.

an_opabinia 6y ago
The bigger problem I feel is with this: > On the plus side, there is a large and well-curated open source collection of syntax definitions Whenever I tried to get Java highlighting updated in GitHub for “var” my issue is closed minutes later by people who don’t work at GitHub telling me to open an issue with some random unpaid TextMate / Sublime repo. Really the overall problem is not owning the front end. It means you can’t do things like fix syntax highlighting, at least as important to programming as high performance word wrap.
6y ago

m0zg 6y ago

This was solving problems nobody really has, so its fate was predictable. Here's what I think could take off like a rocket: a VSCode-like experience that runs completely in terminal, and therefore does not require a "remote" of any kind. Better yet if it takes most of the same plugins, to reuse the immense amount of work people have done there (and are not going to re-do for some hotshot new editor). Note how none of the above mentions "async", "ropes" or "crdt".

saagarjha 6y ago
I think it had a lot of goals that matched I'd like from my text editor: * Using native technologies * Fast * Modular and extensible * Open source
- m0zg 6y ago
  And I'm not disputing that. But one goal eclipses all others: * Usable
  - saagarjha 6y ago
    That is correct, hence why I never used Xi. But if it did end up usable I could imagine myself switching to it.
Arnavion 6y ago
I don't want to speak too much for raphlinus here [1], but I don't think it was ever the goal to make an editor that gets a lot of downloads like a startup pitching a product. All the things you've listed are non-goals, or at least lower-priority goals. The goal was to make an editor based on sound technology principles, and that included investigating said technology principles. It turned out some of those principles were bad ideas, and that's that. [1]: My involvement with xi was limited to being part of conversations he had about rendering performance on Windows in the winapi crate's IRC channel.
- raphlinus 6y ago

taeric 6y ago

The assertion that gpu is required for good text rendering caught me off guard. I can't claim it is wrong, but it does feel like it should be wrong.

saagarjha 6y ago
I think the point being made is that it is required for fast text rendering.
- taeric 6y ago
  Still feels like that should not be the case. What changed so heavily in the last decade? 2d rendering used to be hella fast without the gpu. Right?
  - chubot 6y ago
    Screen resolution / 4k monitors? And maybe font complexity (i.e. lots of web pages seem to be downloading fonts now)
    - taeric 6y ago
      I was curious if font complexity could be it. Does feel like a good guess. Resolution could also matter, but we did have higher resolution displays back in the day, as well. Feels like 2d acceleration somehow regressed.
  - kllrnohj 6y ago
    Rendering models & expectations changed. Once upon a time (a decade ago) it was OK for apps to directly draw to the front window, compositing wasn't a super established thing. This saves on precious memory bandwidth, which CPUs didn't really get all that much more of over the last decade. However now that GPU composition has to happen, GPUs actually don't super like the linear formats that CPUs write to. They want swizzled textures, and they keep those formats private. And on not-heterogeneous memory systems, the buffer the CPU writes into needs to be sent over to the GPU as well. Meanwhile screen resolutions & pixel counts skyrocket, as did UI visual & animation expectations. Being responsive is a lot easier than being fluid. Especially if you're trying to be fluid on a high-resolution and/or high-refresh rate display. And this is all while ignoring things like Apple's "high DPI" handling, which is to just say fuck it and downscale instead. Which means you're pushing resolutions far higher than the display's actual resolution quite commonly.

mwcampbell 6y ago

> There is no such thing as native GUI I understand why you were disillusioned with the performance of native GUIs. But in other areas, such as accessibility, nativeness absolutely does help. I'm guessing you know this; I just want to make sure other developers get this message. Here is a quite comprehensive list of the benefits of a native GUI for accessibility, particularly on macOS: https://news.ycombinator.com/item?id=18585443 https://news.ycombinator.com/item?id=18585443

microcolonel 6y ago
The accessibility APIs can be programmed against by anyone. If there is a way to make something like this accessible to screen readers in some useful way, it would likely involve specific accessibility work either way.
- saagarjha 6y ago
  Platform widgets do a lot of the accessibility work for you. In some cases you can get away with doing zero* work. *Or very close to zero
- mwcampbell 6y ago
  > If there is a way to make something like this accessible to screen readers in some useful way If by "something like this" you mean a text editor, then there's no question; they can be made accessible with a screen reader, and some complex ones (e.g. Visual Studio Code, Visual Studio, and Xcode) are accessible already. > it would likely involve specific accessibility work either way. For the main UI of a full-featured programmer's editor or word processor, that's definitely true. What I want to avoid is developers using a custom GUI toolkit where the platform's native toolkit would do, because they want the best possible performance (regardless of whether the platform's toolkit is good enough), and forfeiting all of the accessibility benefits that the platform's native toolkit brings more or less automatically.
  - notriddle 6y ago
    One of the cliches of modern business is: don't outsource your core competencies. If you are developing a text editor, you should put the time into your text editor. It's the thing that distinguishes you from your competitors. And it's the part that users interact with the most. Your users are likely to ask for features, like vi keybindings, that you can't easily build with the OS-native text field anyway. Incidental forms, like your preferences screen? Unless you have a lot of time and money to burn, the OS widgets will probably be better than yours.

stephc_int13 6y ago

Too much over-engineering here. It is always better to start with simple, dumb and reliable code, with minimal architecture, and then grow organically. With experience, I tend to write the less smart code possible, I felt in love with the power of brute-forcing everything.

magicalist 6y ago
> With experience, I tend to write the less smart code possible I'm always amused by the hn threads patting ourselves on the backs for in depth technical discussions[1], when so often the top comments are basically non-specific thought leader tweets. For those that got this far, the article is well worth reading about a design space with difficult tradeoffs, and there are comments actually engaging with the content below this. [1] today brings us multiple instances in https://news.ycombinator.com/item?id=23664067 https://news.ycombinator.com/item?id=23664067
- saagarjha 6y ago
  I found some interesting ones in this thread!
microcolonel 6y ago
The thing is, that is the story of the existing big editors, and we know where that story goes.
- eeereerews 6y ago
  No, where does it go? Are the big editors unsuccessful?
  - microcolonel 6y ago
    They are unscalable. I love Emacs, and it continues to be my editor of choice, but it doesn't seem the design is amenable to adding good threading primitives. NeoVIM seems like a success story in this regard, though VIMScript is a much less pleasant extension language than Elisp, which is not perfect, but is at least a real programming language. There may be a way forward for Emacs, building a sandbox that looks like a full Emacs instance to existing Elisp, and slowly factoring out the whole-editor blocking issues; but that might be almost as complicated as starting fresh, with fewer benefits.

akhilcacharya 6y ago

I was personally really intrigued by the ambitions and design decisions in Xi, but it somehow lacked the one thing I can’t live without in a non-terminal editor - a tree view, at least when I first tried the OSX version. Perhaps that changed, but you’d think that would take higher feature precedence over something like collaborative editing right?

bhl 6y ago

> And it [syntax highlighting] basically doesn’t fit nicely in the CRDT model at all, as that requires the ability to resolve arbitrarily divergent edits between the different processes (imagine that one goes offline for a bit, types a bit, then the language server comes back online and applies indentation). What's a rationale for having syntax highlighting server-side as opposed to client-side? I'm working on side project that uses ProseMirror and CRDTs through yjs, and the idea of having a server for syntax highlighting for editable text never occurred to me. > Even so, xray was a bit of a wake-up call for me. It was evidence that the vision I had set out for xi was not quite compelling enough that people would want to join forces. I think in hindsight this was a virtue. If you look at the xray project, it died after the Microsoft acquisition of Github: https://github.com/atom-archive/xray/issues/177 https://github.com/atom-archive/xray/issues/177. By not combining projects, it allowed for redundancy.

saagarjha 6y ago
Your server generally has a better idea about the language than you do, presumably.
- bhl 6y ago
  I would only think that's the case when the language of the file you're editing is unknown, so you would have to do some server-side analysis of the entire file and make an educated guess. For example, https://github.com/github/linguist https://github.com/github/linguist. But if the language is known in advance, then you could send the grammar rules to the client.
  - saagarjha 6y ago
    A server also means it still works when the language is not simple enough to condense into simple grammar rules :)
raphlinus 6y ago
So, just syntax highlighting is not compelling to move out to a server. But part of my idea is that if you have async to hide server latency, so it's not slowing down typing, maybe you can do a much deeper analysis than just surface syntax. Right now you get basically instant coloring and then much slower but more accurate feedback that's generally comparable to an incremental recompile. I was thinking that a language server might be able to give you quick feedback on some aspects of your program before having to do all of it. Perhaps an even better motivation is indentation. When I use editors with regex-based indentation (the norm), I'm regularly annoyed when it gets it wrong. Being able to budget a few milliseconds or tends of milliseconds to do indentation that exactly matches what rustfmt or gofmt would recommend would be worth it, imho.

ncmncm 6y ago

> it might be able to safely save the file, but you can also do that by frequently checkpointing It is always amazing to discover niches where the conceptual and practical power of logging has not yet penetrated.

namelosw 6y ago

This is an excellent read. I was always wondering if CRDT would really achieve encapsulation, but this kind of things are very hard to know without trying. Thank you for you works!

mov_ 6y ago

On the github page https://github.com/xi-editor/xi-editor https://github.com/xi-editor/xi-editor: JSON. The protocol for front-end / back-end communication, as well as between the back-end and plug-ins, is based on simple JSON messages. I considered binary formats, but the actual improvement in performance would be completely in the noise. Using JSON considerably lowers friction for developing plug-ins, as it’s available out of the box for most modern languages, and there are plenty of the libraries available for the other ones. 4 years later: The choice of JSON was controversial from the start. It did end up being a source of friction, but for surprising reasons. For one, JSON in Swift is shockingly slow. Surprising reasons?!

raphlinus 6y ago
Yes. In this discussion and on Reddit, people still talk about binary vs textual as the source of the problem, but I've argued (based on empirical data) that the lexical details are not the reason for the performance problems. Also, Swift is marketed as a fast language (also based on LLVM), yet in my measurements it's 20x to 50x slower than Rust for JSON processing. I found that surprising. Would you not?
- mov_ 6y ago
  I agree with you, it's weird that Swift JSON is so slow and binary isn't necessarily faster. My exasperation is you were surprised by JSON becoming an issue, when this was the most controversial design decision about the project. To me, it's completely unsurprising.
  - __s 6y ago
    They're saying the reasons were surprising, ie they expected it to be controversial for different reasons
- shakna 6y ago
  It wouldn't be especially surprising to me. JSON's parsing performance is difficult, whether or not the language doing the parsing is fast or not. There's a reason that there are almost always libraries claiming faster JSON performance, regardless of the language. Go is marketed as a fast language, and they're still trying to build a high performance JSON parser [0]. Parsing JSON will always be slower than most of the alternatives. [0] https://dave.cheney.net/high-performance-json.html https://dave.cheney.net/high-performance-json.html

dilandau 6y ago

Honest and thoughtful retrospective, obviously with no hidden agenda, and written with humility. Fantastic stuff.

nickpinkston 6y ago

This is how you write Xi Thought, and if you contribute enough code you can become the Xi Dada :-)

rl3 6y ago

>One of those was learning about the xray project. I probably shouldn’t have taken this personally, ... I've run into that phenomenon a lot with various startup projects over the years. It's soul crushing, and often an immediate reaction is to assume certain projects are direct competition, rendering your project fruitless. In retrospect, this ends up almost never being the case. In a way, you become your own worst enemy. The correct solution is to simply ignore it and press on, but this is certainly easier said than done—especially when morale is already low, and without the benefit of hindsight. >Perhaps more important, because these projects are more ambitious than one person could really take on, the community started around xi-editor is evolving into one that can sustain GUI in Rust. In the end, I still think that counts as a success. Perhaps it will prove to be an even greater success than the original goal. Thanks for all these write-ups, they're excellent. Like other commenters, I've been watching at a distance for years now. Your work is much appreciated.

smichel17 6y ago
It's not a startup and the alternatives are not competitors, but I've had a similar experience affecting my motivation to work on Red Moon. There's two other screen filter apps on F-Droid, and at one point I proposed that we [work together], either merging our apps or creating shared libraries. While the other devs initially expressed some interest, they weren't willing/able to put in the effort and nothing came of it. This has killed my motivation to work on Red Moon. It feels like a waste of time, duplicating other people's efforts. That's not intrinsic to other people working separately to solve the same problem. If Red Moon were an educational project, or if it took a unique approach to screen filtering (like the other project I'm part of and [crowdfunding]), then I'd feel differently. But my main motivation was creating a FLO screen filter app, since previously I depended on the proprietary Twilight. And while there's a handful of valid approaches to screen filtering, the bulk of the effort is on UX, state transitions, and working around the kinks of Android, all of which are shared. Note: I don't bear any hard feelings towards the other devs, genuinely -- the most frustrating bit was [lack of communication], and even that is totally understandable; it's hard to say no to something you like the idea of, even to yourself (eg, acknowledging you'll never get to that project near the bottom of your list). [work together]: https://github.com/LibreShift/red-moon/issues/222 https://github.com/LibreShift/red-moon/issues/222 [lack of communication]: https://github.com/LibreShift/TransitionScheduler/issues/14 https://github.com/LibreShift/TransitionScheduler/issues/14 [crowdfunding]: https://wiki.snowdrift.coop/market-research/other-crowdfunding https://wiki.snowdrift.coop/market-research/other-crowdfundi...

deleted 6y ago

[deleted]

ryuukk_ 6y ago

tldr: rust sucks

jussij 6y ago

> First, TextMate / Sublime style syntax highlighting is not really all that great. It is quite slow, largely because it grinds through a lot of regular expressions with captures, and it is also not very precise. The tree-sitter framework provides a pretty good engine for syntax highlight: https://github.com/tree-sitter/tree-sitter https://github.com/tree-sitter/tree-sitter

fractalb 6y ago

What is the successful text-editor project in Rust now? Xray seems to have been abandoned too.

taooftea 6y ago

This editor will be blocked in China, obviously

signa11 6y ago

one of the design goals is > CRDT as a mechanism for concurrent modification. i am genuinely wondering if concurrent modification is really, truly needed ?

amedvednikov 6y ago

> When doing some performance work on xi, I found to my great disappointment that performance of these so-called “native” UI toolkits was often pretty poor, even for what you’d think of as the relatively simple task of displaying a screenful of text. The same thing I experienced. Text rendering performance is terrible with Cocoa, and even worse with Win32. Calculating text width is an order of magnitude slower than with FreeType. That's why for my UI framework I also went from using "native" APIs to custom GPU rendering.

mwcampbell 6y ago
If your framework has a broad audience, please make sure it accounts for the full complexity of text rendering [1] and editing [2], particularly when it comes to internationalization. You may find that there are good reasons for Cocoa to be as slow as it is in the measurement phase, if not the rendering phase. And of course, please don't forget about accessibility. [1]: https://gankra.github.io/blah/text-hates-you/ https://gankra.github.io/blah/text-hates-you/ [2]: https://lord.io/blog/2019/text-editing-hates-you-too/ https://lord.io/blog/2019/text-editing-hates-you-too/

jfkebwjsbx 6y ago

I didn't know about Xi, but JSON, RPC, multiprocess, microservices, Async... are all red flags.

fluffything 6y ago

Maybe the time is just not right for Xi yet. I wish that after druid becomes production ready you'll pick it up again. The coolest thing about Xi is that some have attempted to make it a vim replacement, others an emacs replacement, etc. I wish it becomes a framework that can be used to build better text editors in general.

raphlinus 6y ago
I also hope to be able to pick up text editing again, but make no promises. If not, then I hope Druid provides a solid foundation for others to build the dream editor I envisioned.

Myrmornis 6y ago

I (naively) would have thought that the IPC / multi-process model strongly conflicted with the extremely high performance goals (e.g. time between keypress and painting). Can anyone explain why my instinct there is wrong?

hyperion2010 6y ago

> Looking back, I see much of the promise of modular software as addressing goals related to project management, not technical excellence. I couldn't find this point discussed elsewhere in the comments. I think this is one of the most important sentences in the whole article. I know this is essentially just another perspective on Conway's law, but I think it is important to identify factors like this early in a project if at all possible so that human factors around development can be factored out of engineering decisions. I think that this is especially critical for open projects that want to achieve long term sustainability. Open communities don't have the resources to maintain all the packages, even if they use the same build system and follow the same pattern, even if most of the maintenance can be automated, some human being still has to deal each of those systems separately because they are part of the environment surrounding that individual part of the project.

archagon 6y ago

Thank you for these fantastically in-depth and introspective blog posts! It’s a pleasure to observe how a single, experimental project can spawn a dozen different research branches, and it’s something to aspire to in my own work. (I have probably missed many such opportunities due to laziness and lack of curiosity.)

raphlinus 6y ago
Glad you enjoy them! I certainly plan to continue writing.