8 ms·
Ropey – A UTF8 text rope for manipulating and editing large text
- javier2 2y agoAny editors using this?
- recov 2y agoNot that library in particular, but https://zed.dev/blog/zed-decoded-rope-sumtree https://zed.dev/blog/zed-decoded-rope-sumtree
- cschmidt 2y agoIt seems like Helix is using it https://github.com/helix-editor/helix/blob/master/docs/architecture.md https://github.com/helix-editor/helix/blob/master/docs/archi...
- abound 2y agoFWIW, I use Helix as my main editor and every time it has crashed (probably a few dozen times over a year or two, I've filed issues), it's related to bad text position stuff, where it effectively goes "out of bounds" on the text data structure. I think its mostly due to multiple buffers showing the same content, as opposed to this Ropey library directly.
- polyaniline 2y agoThat's unexpected. I've used it for over a year and haven't had a single crash. I've used it for data files millions of lines long, and often open the same file in buffers side by side too. Could you share the issue you opened?
- abound 2y ago- https://github.com/helix-editor/helix/issues/11677 https://github.com/helix-editor/helix/issues/11677 - https://github.com/helix-editor/helix/issues/6752 https://github.com/helix-editor/helix/issues/6752
- rdimartino 2y agoI hadn't heard of rope data structures until I read about the xi editor (also written in Rust) a few years ago, but it looks like that's been discontinued. https://github.com/xi-editor/xi-editor https://github.com/xi-editor/xi-editor
- kibwen 2y agoThe authors of Xi are currently working on Xilem, an experimental reactive UI framework for Rust: https://github.com/linebender/xilem https://github.com/linebender/xilem In the announcement post, they mention that work on Xi is considered "on hold" rather than strictly discontinued: https://raphlinus.github.io/rust/gui/2022/05/07/ui-architecture.html https://raphlinus.github.io/rust/gui/2022/05/07/ui-architect...
- infogulch 2y agoLegendary-tier yak shaving. "I want to build an editor, but first I must solve rendering 2D graphics purely on the GPU, invent a parallelizable path solver, and code a human perception-based color value manipulation library."
- PoignardAzur 2y agoYou have no idea. I think we're at five or six levels of yaks by now. (xi -> xilem -> masonry -> vello -> peniko -> color)
- bbkane 2y agoIt's a lot of fun to follow, especially as its so different than my developmental expertise. You can see the current projects (13 active) on https://linebender.org https://linebender.org , and several members post interesting checkins in https://xi.zulipchat.com/ https://xi.zulipchat.com/
- kstrauser 2y agoThis is the path to Enlightenment (17).
- deleted 2y ago[deleted]
- neilv 2y agoHow would you associate non-character data with ranges of characters, such as for syntax coloring, semantic links, and references to points in the text? (I couldn't find a mention of this in the README, design.md, or examples.) In Emacs buffers, the concepts include text properties, overlays, and markers.
- filcuk 2y agoThat would depend on your editor's implementation.
- neilv 2y agoBut, within this API, is there any support for the associations with non-character data? For example, if you delete some text these Ropey data structure, does Ropey have facilities to update the associated non-character data (such as deleting all or part of one or more chunks of the non-character data, and/or updating positional information)? Or do you have to do that separately outside of Ropey?
- zaphar 2y agoA rope is only concerned with manipulating a string with very low cpu overhead while maintaining the illusion of a sequence of characters or bytes. It doesn't really care or maintain any other text decoration you might be maintaining. That is the concern of the consumer of the rope and I'm not sure there is a good common interface for that.
- neilv 2y agoThanks. I was a little confused, because the lede sentence was "Ropey is a utf8 text rope for Rust, designed to be the backing text-buffer for applications such as text editors." Pretty much all text editors are expected to implement decorations and references, somehow, and some popular text buffer APIs support those.
- 2y ago
- ComputerGuru 2y agoRust is missing an abstraction over non-contiguous chunks of contiguous allocations of data that would make handling ropes seamless and more natural even for smaller sizes. C# has the concept of “Sequences” which is basically a generalization of a deque with associated classes and apis such as ReadOnlySequence and SequenceReader to encourage reduced allocations, reuse of existing buffers/slices even for composition, etc Knowing the rust community, I wouldn’t be surprised if there’s already an RFC for something like this.
- jzelinskie 2y agoIs buf-list[0] what you're describing? [0]: https://crates.io/crates/buf-list https://crates.io/crates/buf-list
- gpm 2y agoI think you might be looking for the bytes crate, which is pretty widely used in networking code: https://docs.rs/bytes/latest/bytes/index.html https://docs.rs/bytes/latest/bytes/index.html In general this sort of structure is the sort of thing I'd expect to see in an external crate in rust, not the standard library. So it's unlikely there's any RFCs, and more likely there's a few competing implementations lying around.
- deleted 2y ago[deleted]
- zamalek 2y agoBytes is essentially multiple slices over a optimistically single contiguous arc buffer. It's basically the inverse of what the root comment is after (an array of buffers). It's a rather strange crate because network IO doesn't actually need contiguous memory. std does actually have a vague version of what the root comment wants: https://doc.rust-lang.org/std/io/struct.IoSlice.html https://doc.rust-lang.org/std/io/struct.IoSlice.html and its sibling IoSliceMut (slicing, appending, inserting, etc. is out of scope for both - so not usable for rope stuff)
- 2y ago
- pixelpoet 2y agoAuthor is perhaps better known for his really great path tracer (and attendant blog), Psychopath: https://github.com/cessen/psychopath https://github.com/cessen/psychopath Also, I have to wonder when this fad of loudly announcing when something is written in Rust will finally come to pass. Maybe software written in other languages should loudly announce it in every second sentence? To me at least it's become as self-aggrandizingly cringe as "Sent from my iPhone" at the end of every email...
- mattdw 2y agoWhen it’s a library of code, the language it is written in is pretty pertinent information as that’s the language it has to be consumed from…
- pixelpoet 2y agoI get that of course, but on the other hand I'm sure you know what I'm getting at too: users of certain languages, platforms etc feel the need to announce it as a point of pride, or feature in itself, and frequently. Every language will have this to some degree (with the possible exception of COBOL, lol), but there are definite outliers.
- itishappy 2y agoI believe HN actually had an article for a Minecraft server written in COBOL about a month ago, lol. https://news.ycombinator.com/item?id=42513022 https://news.ycombinator.com/item?id=42513022
- uecker 2y agoMaybe it is already considered an achievement if someone manages to write a program in Rust.
- ricciardo 2y agoThink about the implications of the language though. When something is written in rust, management of memory is safer, multi threaded applications are safer, etc... (due to the nature of the language). If something is written in C++, the developer might be more inclined to review the code and tests to ensure proper handling of memory as well as determining if (when there is) non-deterministic behavior is safe. Hence, when highlighting something is written in Rust, it might not be just for the buzzword but also for something like developer confidence.
- qup 2y agoWhat a perfect readme. Kudos to the author.
- mathfailure 2y agoNo need for sarcasm, maybe target auditory already knows what 'rope' is.
- rileytg 2y agoi hope someone can use this to create an editor similar to notepad++ that is cross platform. I have not found an editor that can handle large files as well as notepad++ on non windows systems. Last I looked into this, the issue was lack of low level libraries to handle large files.
- ori_b 2y agoYou're someone. Start typing.
- nicoburns 2y agoIt's not open source, but sublime text does well with large files (depending on your definition of large, but several GB is fine)
- polyaniline 2y agoHelix is on Windows, right?,
- RockRobotRock 2y agobuilt in rust btw
- ggregoire 2y agoDidn't know about this data structure. What are some use cases other than text editors? The article on wikipedia [1] doesn't expand much on this. [1] https://en.wikipedia.org/wiki/Rope_(data_structure) https://en.wikipedia.org/wiki/Rope_(data_structure)
- josephg 2y agoWe use them pretty heavily in realtime collaborative editing libraries for text. Ie, text CRDTs. In diamond types, merging a branch into another branch requires (essentially) replaying a whole lot of edit operations. Using a rope makes that fast.
- zahlman 2y agoI imagine that text template renders could benefit. The general use case is any time you need to repeatedly splice (take slices, insert data in the middle etc) a text of significant length and performance is important. (I heard of this concept a long time ago, but rarely think about it.)
- devit 2y ago"Handling texts that are larger than available memory. Ropey is an in-memory data structure." That seems to make it of dubious use, not really suitable for a well-engineered text editor. The fact that it's UTF-8 only is also a serious problem since files can contain arbitrary byte sequences.
- TheRealPomax 2y agoWhat text editor do you use where loading a large text file doesn't put that entire text file in memory? In a world where even sub-$100 devices have gigabytes of memory, this doesn't seem even remotely problematic.
- eviks 2y agothose gigabytes of memory on sub-$100 devices are usually occupied by other apps taking the same careless approach to memory, so it's very problematic
- TheRealPomax 2y agoDevices with lots of ram can fit huge files, devices with very little ram can still fit very large, but not gigabytes-huge files. You're not going to fire up VS Code on pi 2 and load a 5GB log file, but you are going to fire up a simpler editor and open a 100MB log file. So as long as you can fit files that make sense for your device in memory, so why would "it loads files into memory" be a problem?
- eviks 2y agoYou're just making up numbers to fit the conclusion. For example, the only alternative is either a 5G or 100M file. But in reality, just like you have many tabs in a browser, each eating their 100M away, you'll be opening multiple log files. So those cool sounding "several gigs" is actually very constrained, for example: 3G is "several gigs", but on Windows you will be almost always memory-constrained with a few browsers running (web, chat, etc) and you won't have much to open a few of those 100M log files
- Validark 2y agoFrom the Readme: "Unsafe code Ropey uses unsafe code to help achieve some of its space and performance characteristics. Although effort has been put into keeping the unsafe code compartmentalized and making it correct, please be cautious about using Ropey in software that may face adversarial conditions. Auditing, fuzzing, etc. of the unsafe code in Ropey is extremely welcome. If you find any unsoundness, please file an issue! Also welcome are recommendations for how to remove any of the unsafe code without introducing significant space or performance regressions, or how to compartmentalize the unsafe code even better."
- p-e-w 2y agoThat's such a weird disclaimer, considering that an overwhelming majority of mission-critical software is written entirely in "unsafe" code (that is, C/C++). "Please be cautious about using Linux/macOS/Windows/Firefox/Chrome/Safari in adversarial conditions." I've never read a statement like that, even though it would be more warranted than in this case. And even unsafe Rust is far safer than C and C++. It still provides automatic memory management by default, the thread safety guarantees that come with ownership, and abstraction mechanics that make it harder to commit blunders that can lead to unsafety.
- oguz-ismail 2y agoThose are well-established languages. Rust's only selling point is its alleged safety
- p-e-w 2y agoRust is pretty well-established now, being used in production by companies like Amazon. Safety is most certainly not its "only selling point". And the underlying mechanisms have been evaluated in detail by many researchers, both theoretically and practically, so labeling it as "alleged safety" is disingenuous.
- johannes1234321 2y agoMarking code as unsafe in Rust is the escape hatch to do optimisations and reaching out to other systems. It draws the attention to that area for audits and allows building safer stuff atop. In another language, like C, you can have a good structure and well organized attractions, but you have your "unsafe" potentially sprinkled all over.
- cryptonector 2y agoReminds me of xi.
- bjoli 2y agoCan someone explain to me why ropes are better than RRB trees? I havent implemented ropes in a long time, but I remember that there was very little performance benefits over gap buffers even for things like multiple cursors until the document became 10+ mb. I have always thought that a text editor using rrb-trees would probably be the easiest option that would ensure decent performance for small files, and good performance for large files while also being great for random access or linear search.
- seiferteric 2y agoI am interested in this. At my job we have shared systems where engineers often open very large files (10+GB) using vim/gvim and it loads the entire thing into memory and often means these servers are memory starved. Would ropey help in these situations?
- CodesInChaos 2y agoNo: > On the other hand, Ropey is not good at: > Handling texts that are larger than available memory. Ropey is an in-memory data structure. Also, this is a rust library, not an editor application.
- runeblaze 2y agoI am literally guessing at this point — but can they mmap those files? (Or I mean can you shard the files/store the files more efficiently)
- seiferteric 2y agoI am not sure, I have looked into whether vim mem maps files or not over the years and get conflicting answers. Some say no, some say newer versions do, or it depends file type (binary vs text) or it depends on what plugins are enabled etc. All I know is I often see gvim processes using tons of rss memory.