11 ms·

I just loaded a 500MB file in NotepadNext. Windows shows Notepad Next using is 1397MB. Meanwhile if I use Notepad++ and load the same file windows shows it is

I just loaded a 500MB file in NotepadNext. Windows shows Notepad Next using is 1397MB. Meanwhile if I use Notepad++ and load the same file windows shows it is using 586MB. Any ideas why such high memory usage if this is a direct port to QT?

john_moscow 5y ago
Efficiently handling large text files requires extra special care. ~2x memory looks like a naive implementation of just allocating an std::string from heap for each line. Due to heap fragmentation and various overhead it would quickly blow up. ~1x memory looks like just reading the entire file into RAM (that would still be slow). A truly efficient implementation would never need to load the entire original file in RAM. It would just need to remember the binary offset of each line in a way that combines random access and reasonably fast insertion/deletion (e.g. K-fold trees). You can even keep everything beyond the top-level directory in an temporary on-disk file, so your RAM usage could be less than 1MB with nearly instant performance. The efficient implementation is tricky, error-prone, and involves handling a solid amount of corner cases, which is beyond the amount of hassle a typical hobbyist developer is willing to go through.
- erwincoumans 5y ago
  Such truly efficient implementation sounds really nice, with offsets and mmaps. Is there any open source text editor doing this?
- VWWHFSfQ 5y ago
  > Due to heap fragmentation and various overhead it would quickly blow up I think this is mostly likely the culprit. There almost definitely isn't that much contiguous memory available for a large file like this, so there are a lot of wasted pages (maybe 2-3x) which is causing that footprint to balloon.
  - johncolanduoni 5y ago
    For 500MB there should easily be enough contiguous virtual address space, and if there isn’t the allocation would fail. I don’t see how it could take up 2-3x extra pages unless it’s being stored in a really poorly designed segmented memory structure.
    - VWWHFSfQ 5y ago
      Yeah without knowing anything about the implementation when I see "I opened a 500MB file and it's taking 1.3GB of memory" that's just my first guess. Maybe it's wrong.
- kzrdude 5y ago
  Does anything implement it like that?
  - nly 5y ago
    Wouldn't mind betting Sublime does a good job
    - OneLeggedCat 5y ago
      It does. I loaded a 14GB file in it once.
  - _hl_ 5y ago
    xi-editor has some reasonably sophisticated handling for large files
  - mftb 5y ago
    It's vaguely reminiscent of Vim's implementation, in the sense that I think, for large files, in it's default config, Vim keeps most of the text in it's swap file and pages it in as needed. Vim's implementation also feels somewhat line-oriented in that if you load a large JSON file that has everything on 1 line and you try to edit that line, uh... it will be sluggish, but if you do, like %!jq '.' and format it, you can them move around the file a little easier. Edit - What the heck while I'm at it: Emacs - Historically used a Gap-Buffer which optimized more for locality of edits than loading large files. It doesn't suffer though as much from heap fragmentation though as the linked-list of strings type approach. Monaco - The editor in VS Code recently went from a linked-list of strings to a PieceTree buffer. Which basically loads a the whole file into an immutable buffer and then uses the PieceTree to manage the edits. Others - Lately people keep talking about Ropes, which is another tree type deal with extra-smarts specifically for text editing. I don't know of a game-changing editor like the others mentioned that uses it tho.
    - jll29 5y ago
      Ropes were introduced in this paper in the journal Software, Pracise and Experience: https://doi.org/10.1002/spe.4380251203 https://doi.org/10.1002/spe.4380251203 (best accessed from a university domain) An example for an editor that uses them to manage large text files is Xi-Editor: https://github.com/xi-editor/xi-editor https://github.com/xi-editor/xi-editor (edit: written in Rust).
- rndgermandude 5y ago
  Or it might be that one is representing the character data as 16-bit wchars in memory, and the other treats them as 8-bit chars.
- gnu8 5y ago
  I’m afraid to ask, but how does emacs do it?
  - DiggyJohnson 5y ago
    grabs popcorn
  - mftb 5y ago
    Never fear, Emacs approach is pretty great. It uses a gap-buffer, which is conceptually an array with a space that moves around with the insertion point. It's pretty cool approach because, it's simple, there's very little bookkeeping overhead and moving around the gap is gonna generally be cheaper than shifting around the whole array and minimize complete re-allocations. In other words it's good for what we do the vast majority of the time with a text editor which is jump around reasonably size files making small edits. One downside, relative to this conversation, is it doesn't make working with pathologically large files a lot better. Without further optimization a gap buffer could push your box into swap if you open a large enough file.
    - girvo 5y ago
      I love gap buffers because it was super simple to implement. I struggled with ropes and certain tree based algos when I wrote a terminal based text editor in Nim
- tenebrisalietum 5y ago
  Why not just use mmap() or the Windows equivalent?
  - Taywee 5y ago
    Because then all your edits have to go directly into the file, so you have no real "save" flow unless you make a swap file for every file and then mmap that (which can be an appropriate approach, to a degree). Then all your caching and memory use subject to OS I/O and filesystem concerns, and you can't optimize for specific use. And inserting or deleting the middle of the file means shifting all the bytes after it. I could think of a reasonable implementation for Linux that would use mmap and fallocate, but it wouldn't "just" be an mmap, as the swap file would still need to represent a rope or a gap buffer or something else of the sort for efficient editing.
    - jhgb 5y ago
      Aren't edits of a huge file expected to be comparatively small? If you're editing an mmap'd file, I'd expect the editor to maintain a display combining the original mmap'd file with a list of changes that have been performed so far.
  - deleted 5y ago
    [deleted]
  - AnotherGoodName 5y ago
    Let me rant for a bit because I've dealt with the following: Another downside is you'll get people complaining about memory usage despite mmap only paging in what's needed. It's frustrating but users don't understand how to read memory usage at all. There's even guides out there quite wrongly telling users to look at virtual memory usage for each app. Mmap is a brilliant way to have the os handle which parts of the file actually reside in memory but you'll seriously have to deal with users that know only enough to be dangerous complaining that your app uses gb of ram when it's merely mapping a file into virtual address space to allow the os to page as it sees fit. In fact I wouldn't at all be surprised if the memory measurements in this thread were judging virtual allocation rather than physical memory used.
- abalaji 5y ago
  This is why I will always pay for the latest version of Sublime Text. First to support indie developers, but second for the marvelous piece of software that allows me to browse gigabytes of data in a GUI.
  - Jonovono 5y ago
    Dammm havn't heard that word in awhile - had to look at my calendar to double check what year it was. Great to hear they are still rocking!
    - WesleyHale 5y ago
      Python Crash Course by Eric Mathes is updated up to Python 3.7 and uses Sublime as the editor of choice also.
    - lofatdairy 5y ago
      I thought Sublime's popularity has grown in recent years? Was it even more popular in the past or am I mistaken? It's just that compared to the standard vim vs emacs debate, I now always hear sublime thrown in for good measure when discussing pure text editors.
      - AlchemistCamp 5y ago
        I last used Sublime Text in early 2014 and the same is true for most of my friends. It is a lot more performant than VS Code, but VS Code seems to have won on the sheer quantity of “good enough” plugins. Its pair programming features are a big deal as well. For the most part I use VIM for quick things and VS Code when I need a plugin or to work with others. SBT kinda sits between those two extremes. The pair programming feature from VSC is the only one I’d really miss from that side if I were to go back to SBT. Not sure about from the VIM side because I’m sure the SBT VIM plugins have improved over the past 8 years.
      - wtetzner 5y ago
        I think it was a joke about the band.
- jmull 5y ago
  > A truly efficient implementation would never need to load the entire original file in RAM. It would just need to remember the binary offset of each line in a way that combines random access and reasonably fast insertion/deletion (e.g. K-fold trees). To know the offset of each line, you'll need to load the entire file from disk. So you're talking about loading it, processing it, and throwing it out again. You should qualify calling that "efficient" since you will need to re-load portions of the file, as needed. Also, an OS's virtual memory systems may work well for you with minor tuning, so it's work looking into that before you spend a lot of time essentially writing your own. I think unless you have a good reason not to, your best bet for a text editor on a modern OS is to read the entire file into memory and leave it there. (But don't make multiple copies of it or use two bytes to store each character as the current version of NotepadNext is apparently doing.)
  - jmmv 5y ago
    > To know the offset of each line, you'll need to load the entire file from disk. No. You will need to read the file, not load it all at once in memory. And for a sequential scan, you need very little memory at any one time. And if you are doing that, you can start displaying the file as soon as you have scanned enough to fill the view. The rest could happen in the background.
    - jmull 5y ago
      I didn’t say you had to keep it all in memory to process the line ending. Of course you don’t. You need only two characters of memory (assuming you’re handling windows line breaks).
  - nickduggets 5y ago
    With a 64-bit address space, you're probably better off just mapping the entire file into memory, not specifically loading it all. Let the OS swap in and out the pieces you need as you access it (and build your index of line offsets or whatever other structures you need).
    - 5y ago
- teraflop 5y ago
  > A truly efficient implementation would never need to load the entire original file in RAM. It would just need to remember the binary offset of each line Except that this would most likely result in unpredictable corruption if another process modified the file while you had it open, which is contrary to the way basically every text editor works. It would be nice if there was a way to mmap a file in such a way that the OS would give you a snapshot view, but AFAIK that doesn't exist, at least on Linux in a filesystem-agnostic way (not sure about other operating systems).
  - dataflow 5y ago
    I think every platform supports FS notifications at least right? The app could listen for changes to the file and reload/discard when it goes out of sync. In fact Notepad++ already does this. On Windows in particular you also have a ton of other facilities that could help with this (opportunistic locks, transactions, etc.) but notification is usually good enough.
    - chongli 5y ago
      My main editor is vim. It notifies the user when a file you have open in a buffer gets modified on disk by another process. It explicitly gives you the option to load the file from disk at this time. I would not use a text editor which silently reloads a file because this could result in a loss of work.
      - dataflow 5y ago
        Yeah Notepad++ asks you too. I don't know of any editors that silently discard changes by default.
      - workerdrone451 5y ago
        While many editors will come and go, vi in some form will always remain. Once you "get" it, using anything else starts to become alien.
  - phendrenad2 5y ago
- quickthrower2 5y ago
  Also if the file is 500Mb, probably a text editor isn't the best choice of app to work with it. That said it is often the most convenient. But a 'non loading' text editor would be interesting for this. E.g. it doesn't show the file, but you can still do find in file etc. operations. Maybe basically a thin UI around grep etc.
- eternityforest 5y ago
  Seems like anything past a few megabytes would really be better done with dedicated tools for that amount of data. I think I'd be a bit unhappy if I was regularly loading megabyte blobs of text into editors, and start looking at other representations or tools. The only files that big that I know of are log files, and very occasionally a CSV or two as some kind of data dump, and there's usually dedicated tools for those cases.
- DeathArrow 5y ago
  How it can be nearly instant performance if you have to load data from disk on demand? If users jumps a few pages, or he performs a search or replace, things will get real slow. We wouldn't use large quantities of RAM today if there were any techniques that could make disk acces "nearly instant".
- userbinator 5y ago
  ~1x memory looks like just reading the entire file into RAM (that would still be slow). It's worth keeping in mind that typical CPUs from the past few years to today have memory bandwidths of several GB/s. Disks have sequential read speeds of ~100MB/s; SSDs can easily do 1GB/s and above.
  - brokenmachine 5y ago
    It's amazing to me how slow some stuff is even on modern hardware. Lazy programming abounds.
- _pmf_ 5y ago
  > 2x memory looks like a naive implementation of just allocating an std::string from heap for each line. Due to heap fragmentation and various overhead it would quickly blow up. Looks more line Qt's internal UTF-16 encoding in this case.
TonyTrapp 5y ago
If it uses QString as backing storage, that would be UTF-16 internally and would explain the doubling in size.
- danhau 5y ago
  I would be surprised if they use straight QStrings. Text editors typically use more sophisticated data structures for their text buffers - Ropes, Piece Tables, Gap Buffers, etc. - tuned for efficient text operations. Maybe their implementation has memory overhead? Wouldn't Notepad++ use UTF-16 internally too, with its Windows heritage?
  - TonyTrapp 5y ago
    > I would be surprised if they use straight QStrings. Text editors typically use more sophisticated data structures for their text buffers - Ropes, Piece Tables, Gap Buffers, etc. - tuned for efficient text operations. Sure, I didn't mean to imply that the whole document would be one giant QString. But those data structures might still use a QString as backing memory in the new implementation. I tried to have a look around, but didn't have too much time on hand to dive deep. I could see some QString usage but couldn't confirm if the document itself uses it for storage. > Wouldn't Notepad++ use UTF-16 internally too, with its Windows heritage Not necessarily, as the memory consumption of 586MB for a 500MB file shows.
    - arka2147483647 5y ago
      I would suspect that it supports multiple encodings, and handle each of them natively
      - jraph 5y ago
        It could also handle one encoding internally and do the conversion when opening and saving the file.
asddubs 5y ago
it's not a port, is it? I thought notepad++ was closed source?
- dindresto 5y ago
  https://github.com/notepad-plus-plus/notepad-plus-plus https://github.com/notepad-plus-plus/notepad-plus-plus
  - asddubs 5y ago
    huh, I never knew.
- Delk 5y ago
  Notepad++ is GPL-licensed.
pkrumins 5y ago
Why are you concerned about the memory usage? Computers have a lot of memory.
- tom_ 5y ago
  But a lot is still a finite amount!
fernly 5y ago
I'm having trouble finding anything in the source tree, but searches on "QTextEdit" and "QPlainText" fail, so it's maybe not using Qt's very competent text edit classes? The Readme indicates Qt is used, it would be odd not to use the built-in (but highly customizable) editor which supports a syntax analyzer and lots of other things you would think this project needs.
- jcelerier 5y ago
  I checked with heaptrack, the big allocation seems to be within scintilla data structures (which should be the same on notepad++ if I'm not mistaken ?). So likely some suboptimal thing here like having some cache somewhere which does not matter for the average text file.
- self_awareness 5y ago
  The screenshot mentions Scintilla, which is a popular text-editing widget (https://www.scintilla.org/ https://www.scintilla.org/).
crispyambulance 5y ago
Dealing with large files or huge number of open files is something where Notepad++ really shines, it's basically the reason you would use it for text files instead of VS Code, IMHO. I've had Notepad++ open many hundreds of files in single sittings to do file exploration, data-cleaning and really aggressive regex search/replace. It's obviously smart about how to deal with this without locking up the application. Definitely my goto editor for non-code files. I've wondered if emacs can do the same, but when I've asked an emacs user responded with "Why would you want to open 800 files?" I take that as a "no"?
- arbitrage 5y ago
  They may just be trying really hard to avoid saying "I don't know", or perhaps "It never occurred to me to try". Emacs users aren't accustomed to getting questions they don't know the answer to, IMHO.
- teddyh 5y ago
  > I've wondered if emacs can [deal with large files or huge number of open files] […] I take that as a "no"? Emacs handles it fine. Two years ago¹, I wrote this: “Just last week I opened, edited, and saved, a 4 Gigabyte SQL dump file, in Emacs. It was a bit slower than usual, but still well usable, and certainly no crashes.” 1. https://news.ycombinator.com/item?id=22880333 https://news.ycombinator.com/item?id=22880333
  - andai 5y ago
    That sounds like a lot, but most (power user) machines have more RAM than that these days. Are there any editors that can deal with files larger than available memory?
    - teddyh 5y ago
      > Are there any editors that can deal with files larger than available memory? Emacs: https://elpa.gnu.org/packages/vlf.html https://elpa.gnu.org/packages/vlf.html

john_moscow 5y ago

Efficiently handling large text files requires extra special care. ~2x memory looks like a naive implementation of just allocating an std::string from heap for each line. Due to heap fragmentation and various overhead it would quickly blow up. ~1x memory looks like just reading the entire file into RAM (that would still be slow). A truly efficient implementation would never need to load the entire original file in RAM. It would just need to remember the binary offset of each line in a way that combines random access and reasonably fast insertion/deletion (e.g. K-fold trees). You can even keep everything beyond the top-level directory in an temporary on-disk file, so your RAM usage could be less than 1MB with nearly instant performance. The efficient implementation is tricky, error-prone, and involves handling a solid amount of corner cases, which is beyond the amount of hassle a typical hobbyist developer is willing to go through.

erwincoumans 5y ago
Such truly efficient implementation sounds really nice, with offsets and mmaps. Is there any open source text editor doing this?
VWWHFSfQ 5y ago
> Due to heap fragmentation and various overhead it would quickly blow up I think this is mostly likely the culprit. There almost definitely isn't that much contiguous memory available for a large file like this, so there are a lot of wasted pages (maybe 2-3x) which is causing that footprint to balloon.
- johncolanduoni 5y ago
  For 500MB there should easily be enough contiguous virtual address space, and if there isn’t the allocation would fail. I don’t see how it could take up 2-3x extra pages unless it’s being stored in a really poorly designed segmented memory structure.
  - VWWHFSfQ 5y ago
    Yeah without knowing anything about the implementation when I see "I opened a 500MB file and it's taking 1.3GB of memory" that's just my first guess. Maybe it's wrong.
kzrdude 5y ago
Does anything implement it like that?
- nly 5y ago
  Wouldn't mind betting Sublime does a good job
  - OneLeggedCat 5y ago
    It does. I loaded a 14GB file in it once.
- _hl_ 5y ago
  xi-editor has some reasonably sophisticated handling for large files
- mftb 5y ago
  It's vaguely reminiscent of Vim's implementation, in the sense that I think, for large files, in it's default config, Vim keeps most of the text in it's swap file and pages it in as needed. Vim's implementation also feels somewhat line-oriented in that if you load a large JSON file that has everything on 1 line and you try to edit that line, uh... it will be sluggish, but if you do, like %!jq '.' and format it, you can them move around the file a little easier. Edit - What the heck while I'm at it: Emacs - Historically used a Gap-Buffer which optimized more for locality of edits than loading large files. It doesn't suffer though as much from heap fragmentation though as the linked-list of strings type approach. Monaco - The editor in VS Code recently went from a linked-list of strings to a PieceTree buffer. Which basically loads a the whole file into an immutable buffer and then uses the PieceTree to manage the edits. Others - Lately people keep talking about Ropes, which is another tree type deal with extra-smarts specifically for text editing. I don't know of a game-changing editor like the others mentioned that uses it tho.
  - jll29 5y ago
    Ropes were introduced in this paper in the journal Software, Pracise and Experience: https://doi.org/10.1002/spe.4380251203 https://doi.org/10.1002/spe.4380251203 (best accessed from a university domain) An example for an editor that uses them to manage large text files is Xi-Editor: https://github.com/xi-editor/xi-editor https://github.com/xi-editor/xi-editor (edit: written in Rust).
rndgermandude 5y ago
Or it might be that one is representing the character data as 16-bit wchars in memory, and the other treats them as 8-bit chars.
gnu8 5y ago
I’m afraid to ask, but how does emacs do it?
- DiggyJohnson 5y ago
  grabs popcorn
- mftb 5y ago
  Never fear, Emacs approach is pretty great. It uses a gap-buffer, which is conceptually an array with a space that moves around with the insertion point. It's pretty cool approach because, it's simple, there's very little bookkeeping overhead and moving around the gap is gonna generally be cheaper than shifting around the whole array and minimize complete re-allocations. In other words it's good for what we do the vast majority of the time with a text editor which is jump around reasonably size files making small edits. One downside, relative to this conversation, is it doesn't make working with pathologically large files a lot better. Without further optimization a gap buffer could push your box into swap if you open a large enough file.
  - girvo 5y ago
    I love gap buffers because it was super simple to implement. I struggled with ropes and certain tree based algos when I wrote a terminal based text editor in Nim
tenebrisalietum 5y ago
Why not just use mmap() or the Windows equivalent?
- Taywee 5y ago
  Because then all your edits have to go directly into the file, so you have no real "save" flow unless you make a swap file for every file and then mmap that (which can be an appropriate approach, to a degree). Then all your caching and memory use subject to OS I/O and filesystem concerns, and you can't optimize for specific use. And inserting or deleting the middle of the file means shifting all the bytes after it. I could think of a reasonable implementation for Linux that would use mmap and fallocate, but it wouldn't "just" be an mmap, as the swap file would still need to represent a rope or a gap buffer or something else of the sort for efficient editing.
  - jhgb 5y ago
    Aren't edits of a huge file expected to be comparatively small? If you're editing an mmap'd file, I'd expect the editor to maintain a display combining the original mmap'd file with a list of changes that have been performed so far.
- deleted 5y ago
  [deleted]
- AnotherGoodName 5y ago
  Let me rant for a bit because I've dealt with the following: Another downside is you'll get people complaining about memory usage despite mmap only paging in what's needed. It's frustrating but users don't understand how to read memory usage at all. There's even guides out there quite wrongly telling users to look at virtual memory usage for each app. Mmap is a brilliant way to have the os handle which parts of the file actually reside in memory but you'll seriously have to deal with users that know only enough to be dangerous complaining that your app uses gb of ram when it's merely mapping a file into virtual address space to allow the os to page as it sees fit. In fact I wouldn't at all be surprised if the memory measurements in this thread were judging virtual allocation rather than physical memory used.
abalaji 5y ago
This is why I will always pay for the latest version of Sublime Text. First to support indie developers, but second for the marvelous piece of software that allows me to browse gigabytes of data in a GUI.
- Jonovono 5y ago
  Dammm havn't heard that word in awhile - had to look at my calendar to double check what year it was. Great to hear they are still rocking!
  - WesleyHale 5y ago
    Python Crash Course by Eric Mathes is updated up to Python 3.7 and uses Sublime as the editor of choice also.
  - lofatdairy 5y ago
    I thought Sublime's popularity has grown in recent years? Was it even more popular in the past or am I mistaken? It's just that compared to the standard vim vs emacs debate, I now always hear sublime thrown in for good measure when discussing pure text editors.
    - AlchemistCamp 5y ago
      I last used Sublime Text in early 2014 and the same is true for most of my friends. It is a lot more performant than VS Code, but VS Code seems to have won on the sheer quantity of “good enough” plugins. Its pair programming features are a big deal as well. For the most part I use VIM for quick things and VS Code when I need a plugin or to work with others. SBT kinda sits between those two extremes. The pair programming feature from VSC is the only one I’d really miss from that side if I were to go back to SBT. Not sure about from the VIM side because I’m sure the SBT VIM plugins have improved over the past 8 years.
    - wtetzner 5y ago
      I think it was a joke about the band.
jmull 5y ago
> A truly efficient implementation would never need to load the entire original file in RAM. It would just need to remember the binary offset of each line in a way that combines random access and reasonably fast insertion/deletion (e.g. K-fold trees). To know the offset of each line, you'll need to load the entire file from disk. So you're talking about loading it, processing it, and throwing it out again. You should qualify calling that "efficient" since you will need to re-load portions of the file, as needed. Also, an OS's virtual memory systems may work well for you with minor tuning, so it's work looking into that before you spend a lot of time essentially writing your own. I think unless you have a good reason not to, your best bet for a text editor on a modern OS is to read the entire file into memory and leave it there. (But don't make multiple copies of it or use two bytes to store each character as the current version of NotepadNext is apparently doing.)
- jmmv 5y ago
  > To know the offset of each line, you'll need to load the entire file from disk. No. You will need to read the file, not load it all at once in memory. And for a sequential scan, you need very little memory at any one time. And if you are doing that, you can start displaying the file as soon as you have scanned enough to fill the view. The rest could happen in the background.
  - jmull 5y ago
    I didn’t say you had to keep it all in memory to process the line ending. Of course you don’t. You need only two characters of memory (assuming you’re handling windows line breaks).
- nickduggets 5y ago
  With a 64-bit address space, you're probably better off just mapping the entire file into memory, not specifically loading it all. Let the OS swap in and out the pieces you need as you access it (and build your index of line offsets or whatever other structures you need).
  - 5y ago
teraflop 5y ago
> A truly efficient implementation would never need to load the entire original file in RAM. It would just need to remember the binary offset of each line Except that this would most likely result in unpredictable corruption if another process modified the file while you had it open, which is contrary to the way basically every text editor works. It would be nice if there was a way to mmap a file in such a way that the OS would give you a snapshot view, but AFAIK that doesn't exist, at least on Linux in a filesystem-agnostic way (not sure about other operating systems).
- dataflow 5y ago
  I think every platform supports FS notifications at least right? The app could listen for changes to the file and reload/discard when it goes out of sync. In fact Notepad++ already does this. On Windows in particular you also have a ton of other facilities that could help with this (opportunistic locks, transactions, etc.) but notification is usually good enough.
  - chongli 5y ago
    My main editor is vim. It notifies the user when a file you have open in a buffer gets modified on disk by another process. It explicitly gives you the option to load the file from disk at this time. I would not use a text editor which silently reloads a file because this could result in a loss of work.
    - dataflow 5y ago
      Yeah Notepad++ asks you too. I don't know of any editors that silently discard changes by default.
    - workerdrone451 5y ago
      While many editors will come and go, vi in some form will always remain. Once you "get" it, using anything else starts to become alien.
- phendrenad2 5y ago
quickthrower2 5y ago
Also if the file is 500Mb, probably a text editor isn't the best choice of app to work with it. That said it is often the most convenient. But a 'non loading' text editor would be interesting for this. E.g. it doesn't show the file, but you can still do find in file etc. operations. Maybe basically a thin UI around grep etc.
eternityforest 5y ago
Seems like anything past a few megabytes would really be better done with dedicated tools for that amount of data. I think I'd be a bit unhappy if I was regularly loading megabyte blobs of text into editors, and start looking at other representations or tools. The only files that big that I know of are log files, and very occasionally a CSV or two as some kind of data dump, and there's usually dedicated tools for those cases.
DeathArrow 5y ago
How it can be nearly instant performance if you have to load data from disk on demand? If users jumps a few pages, or he performs a search or replace, things will get real slow. We wouldn't use large quantities of RAM today if there were any techniques that could make disk acces "nearly instant".
userbinator 5y ago
~1x memory looks like just reading the entire file into RAM (that would still be slow). It's worth keeping in mind that typical CPUs from the past few years to today have memory bandwidths of several GB/s. Disks have sequential read speeds of ~100MB/s; SSDs can easily do 1GB/s and above.
- brokenmachine 5y ago
  It's amazing to me how slow some stuff is even on modern hardware. Lazy programming abounds.
_pmf_ 5y ago
> 2x memory looks like a naive implementation of just allocating an std::string from heap for each line. Due to heap fragmentation and various overhead it would quickly blow up. Looks more line Qt's internal UTF-16 encoding in this case.

TonyTrapp 5y ago

If it uses QString as backing storage, that would be UTF-16 internally and would explain the doubling in size.

danhau 5y ago
I would be surprised if they use straight QStrings. Text editors typically use more sophisticated data structures for their text buffers - Ropes, Piece Tables, Gap Buffers, etc. - tuned for efficient text operations. Maybe their implementation has memory overhead? Wouldn't Notepad++ use UTF-16 internally too, with its Windows heritage?
- TonyTrapp 5y ago
  > I would be surprised if they use straight QStrings. Text editors typically use more sophisticated data structures for their text buffers - Ropes, Piece Tables, Gap Buffers, etc. - tuned for efficient text operations. Sure, I didn't mean to imply that the whole document would be one giant QString. But those data structures might still use a QString as backing memory in the new implementation. I tried to have a look around, but didn't have too much time on hand to dive deep. I could see some QString usage but couldn't confirm if the document itself uses it for storage. > Wouldn't Notepad++ use UTF-16 internally too, with its Windows heritage Not necessarily, as the memory consumption of 586MB for a 500MB file shows.
  - arka2147483647 5y ago
    I would suspect that it supports multiple encodings, and handle each of them natively
    - jraph 5y ago
      It could also handle one encoding internally and do the conversion when opening and saving the file.

asddubs 5y ago

it's not a port, is it? I thought notepad++ was closed source?

dindresto 5y ago
https://github.com/notepad-plus-plus/notepad-plus-plus https://github.com/notepad-plus-plus/notepad-plus-plus
- asddubs 5y ago
  huh, I never knew.
Delk 5y ago
Notepad++ is GPL-licensed.

pkrumins 5y ago

Why are you concerned about the memory usage? Computers have a lot of memory.

tom_ 5y ago
But a lot is still a finite amount!

fernly 5y ago

I'm having trouble finding anything in the source tree, but searches on "QTextEdit" and "QPlainText" fail, so it's maybe not using Qt's very competent text edit classes? The Readme indicates Qt is used, it would be odd not to use the built-in (but highly customizable) editor which supports a syntax analyzer and lots of other things you would think this project needs.

jcelerier 5y ago
I checked with heaptrack, the big allocation seems to be within scintilla data structures (which should be the same on notepad++ if I'm not mistaken ?). So likely some suboptimal thing here like having some cache somewhere which does not matter for the average text file.
self_awareness 5y ago
The screenshot mentions Scintilla, which is a popular text-editing widget (https://www.scintilla.org/ https://www.scintilla.org/).

crispyambulance 5y ago

Dealing with large files or huge number of open files is something where Notepad++ really shines, it's basically the reason you would use it for text files instead of VS Code, IMHO. I've had Notepad++ open many hundreds of files in single sittings to do file exploration, data-cleaning and really aggressive regex search/replace. It's obviously smart about how to deal with this without locking up the application. Definitely my goto editor for non-code files. I've wondered if emacs can do the same, but when I've asked an emacs user responded with "Why would you want to open 800 files?" I take that as a "no"?

arbitrage 5y ago
They may just be trying really hard to avoid saying "I don't know", or perhaps "It never occurred to me to try". Emacs users aren't accustomed to getting questions they don't know the answer to, IMHO.
teddyh 5y ago
> I've wondered if emacs can [deal with large files or huge number of open files] […] I take that as a "no"? Emacs handles it fine. Two years ago¹, I wrote this: “Just last week I opened, edited, and saved, a 4 Gigabyte SQL dump file, in Emacs. It was a bit slower than usual, but still well usable, and certainly no crashes.” 1. https://news.ycombinator.com/item?id=22880333 https://news.ycombinator.com/item?id=22880333
- andai 5y ago
  That sounds like a lot, but most (power user) machines have more RAM than that these days. Are there any editors that can deal with files larger than available memory?
  - teddyh 5y ago
    > Are there any editors that can deal with files larger than available memory? Emacs: https://elpa.gnu.org/packages/vlf.html https://elpa.gnu.org/packages/vlf.html