3 ms·

This piqued my interest: > handle arbitrary files (this includes large ones, think >100M SQL-dumps) My old hex editor Hex Fiend was a serious attempt to handl

This piqued my interest: > handle arbitrary files (this includes large ones, think >100M SQL-dumps) My old hex editor Hex Fiend was a serious attempt to handle arbitrary-sized files correctly. It's hard! In particular, operations which are usually instantaneous (e.g. Find) now may take a long time: they need progress reporting and cancellation, and ideally should not be modal. A text editor makes that even harder, because now simple operations like "go to beginning of line" may take a long time if you have to find the beginning of the line. There's probably some conditions that you could impose (e.g. handles large files but not long lines); it would be interesting to see what those are. > Loading a file from disk is as simple as mmap This is a sketchy decision. For one thing, it means you cannot work with files larger than maybe 3 GB, or even 3 1 GB files, in a 32 bit process. I'd also be uncomfortable relying on mmap over NFS. Hex Fiend handled this by not mapping files, but reading them (via pread) on demand. > Since the buffers are append only and the spans/pieces are never destroyed undo/redo functionality is implemented by swapping the required spans/pieces back in. Saving is what makes this tricky. Say I open a 10 GB file, delete it all, and save it. Can I now undo that delete? (Hex Fiend initially could not, and users were unhappy). If so, where does that 10 GB data live? For that matter, how DO files get saved? Say I append 1 byte to the end of a file: is the entire file rewritten? Say I delete 1 byte from the front of the file: does it require twice the disk space to save it? How about copy and paste? Say I open a 1 GB file, copy it, and paste it into another. Does that 1 GB of data get copied into memory, or is it just referenced in the original file? If it's referenced, what happens if I now edit that original file - does the data get copied at that point? Anyways this is really tricky (but fun) stuff, and I hope the author succeeds since I do want a fast text editor that can operate on arbitrarily sized files.

rtpg 12y ago
>For one thing, it means you cannot work with files larger than maybe 3 GB, or even 3 1 GB files, in a 32 bit process. Serious question : is it bad to assume a 64-bit capable running environment for developer's computers nowadays? I feel like it's better to run with something simple (and way less error prone) if the cost is some 32-bit issues like stated. Granted, vim is a special case, since we're running this through ssh on servers or whatnot. Lots of really good cases mentioned in this post, though, bet you make a good debugger
- ridiculous_fish 12y ago
  There's definitely still new Linux systems that are 32 bit (Raspberry Pi, etc) though probably not many as a developer computer. Maybe Crouton on a Chromebook? Anyways you have a point about the simplicity of mmap. Reading up on the suckless philosophy, it does seem to fit.
  - mjcohen 12y ago
    I think Crouton installs 64-bit Linux.
- chriswarbo 12y ago
  > Serious question : is it bad to assume a 64-bit capable running environment for developer's computers nowadays? There's at least one counter-example, since all my machines are 32bit. Is there anything in particular about software developers which makes you think they're more likely to grind on the hardware upgrade treadmill? I would have thought the opposite. The software I use (linux + xmonad + bash + text editor) has far lower requirements than that my family and friends use (various versions of Windows, online video players, antivirus, games, etc.) I tend to give my hardware more respect than my friends and family too, who are much more likely to drop machines, drop/spill things on to machines, rotate them with hard disks spinning, put stress on hinges or weak points (eg. carrying a laptop by the screen), insert/remove peripherals in mechanically stressful ways, etc. I'm also more able to investigate and fix problems, rather than eg. buying a new computer when the hard drive gets full. My current machines all use AMD K7 chips (Athlon/Duron), so I've built up a collection of compatible CPUs, RAM, IDE cables/drives etc. from dead machines to fix any issues I get with the remaining ones. I'm perfectly happy with my hardware, so I see no reason to "upgrade"; rather, I've seen news feeds full of compatibility wars, eg. amd64/IA64, IDEx/eIDE, SATA/PATA, HDMI/Thunderbolt/DisplayPort/whatever, BluRay/HDDVD, etc. none of which I've had to care about.
  - rtpg 12y ago
    It's been 11 years since the widespread introduction of 64-bit processors in consumer hardware. I think it's a pretty safe assumption to make that a vast majority of people are using 64-bit-capable machines. There are advantages to the "hardware upgrade" treadmill if you're on a laptop (not many laptops got 9 hours of battery life in 2003), and if you open an IDE from time to time, or do heavy consumer-facing web development, having a computer from the last decade is useful. Also easier to run a high-res multi-monitor setup if you have some good speed behind that. It's possible to be good and capable on the machines you're citing, but the overwhelming majority of software developers (even the ones using vim/emacs) are not of the linux + xmonad type.
marvy 12y ago
Um, how on earth do you deal with the 10GB delete? You can't store it in RAM, since there may not be enough, especially if you want 32 bit support. Store it on disk? You can't store it in the original file, since you have no flexibility over its contents after a save: a hex editor must write the bytes asked for, unlike a word processor which controls the file format. Ok, maybe we store it in some other file, call it "temp". That almost works, but what if the point was to save disk space? "Please restart the editor"? And if the power goes out one hour later, the user finds out at the next boot that clearing out a 10 gig file did not save any disk space? I'm out of ideas. What does Hex Fiend do?
- ridiculous_fish 12y ago
  The short answer is that it preserves the undo stack, if it can do so without using more than 16 MB of additional memory. If it exceeds that threshold it drops the stack. (Hey, I didn't claim to have solved it well!) The longer answer: when a file is going to be saved, HF identifies the ranges of the file that will be modified, and then identifies every other "byte array" (what vis calls a "piece chain") that references one of those file ranges. This includes data that may have been copied and pasted from the file, as well as the undo stack. It then attempts to "break dependencies" on the about-to-be-overwritten ranges in the file, by copying the referenced bytes into memory first. But if that copying would exceed that 16MB threshold it gives up. Giving up in the case of the undo stack means silently dropping it, and in the case of pasting into another document, it prompts the user: http://i.imgur.com/6Vq7VyQ.png http://i.imgur.com/6Vq7VyQ.png The guiding UI principle is to avoid surprise allocations of large amounts of memory or disk space.
  - p0nce 12y ago
    Interesting, I've always wondered how hex editors did it.
  - marvy 12y ago
    You may not have solved it perfectly, but better than a version of Excel I once used. Every time you save, it drops the undo stack. No matter what. I think the latest versions have fixed this by now.
shrizza 12y ago
You mentioned Hex Fiend; hed is also a decent hex editor for large files. I've edited 5G files in it. Edit: Ah, just noticed you're the author. Hi.
101914 12y ago
"... I do want a fast text editor that can operate on arbitrarily sized files." Ever try radare (now radare2)?

rtpg 12y ago

>For one thing, it means you cannot work with files larger than maybe 3 GB, or even 3 1 GB files, in a 32 bit process. Serious question : is it bad to assume a 64-bit capable running environment for developer's computers nowadays? I feel like it's better to run with something simple (and way less error prone) if the cost is some 32-bit issues like stated. Granted, vim is a special case, since we're running this through ssh on servers or whatnot. Lots of really good cases mentioned in this post, though, bet you make a good debugger

ridiculous_fish 12y ago
There's definitely still new Linux systems that are 32 bit (Raspberry Pi, etc) though probably not many as a developer computer. Maybe Crouton on a Chromebook? Anyways you have a point about the simplicity of mmap. Reading up on the suckless philosophy, it does seem to fit.
- mjcohen 12y ago
  I think Crouton installs 64-bit Linux.
chriswarbo 12y ago
> Serious question : is it bad to assume a 64-bit capable running environment for developer's computers nowadays? There's at least one counter-example, since all my machines are 32bit. Is there anything in particular about software developers which makes you think they're more likely to grind on the hardware upgrade treadmill? I would have thought the opposite. The software I use (linux + xmonad + bash + text editor) has far lower requirements than that my family and friends use (various versions of Windows, online video players, antivirus, games, etc.) I tend to give my hardware more respect than my friends and family too, who are much more likely to drop machines, drop/spill things on to machines, rotate them with hard disks spinning, put stress on hinges or weak points (eg. carrying a laptop by the screen), insert/remove peripherals in mechanically stressful ways, etc. I'm also more able to investigate and fix problems, rather than eg. buying a new computer when the hard drive gets full. My current machines all use AMD K7 chips (Athlon/Duron), so I've built up a collection of compatible CPUs, RAM, IDE cables/drives etc. from dead machines to fix any issues I get with the remaining ones. I'm perfectly happy with my hardware, so I see no reason to "upgrade"; rather, I've seen news feeds full of compatibility wars, eg. amd64/IA64, IDEx/eIDE, SATA/PATA, HDMI/Thunderbolt/DisplayPort/whatever, BluRay/HDDVD, etc. none of which I've had to care about.
- rtpg 12y ago
  It's been 11 years since the widespread introduction of 64-bit processors in consumer hardware. I think it's a pretty safe assumption to make that a vast majority of people are using 64-bit-capable machines. There are advantages to the "hardware upgrade" treadmill if you're on a laptop (not many laptops got 9 hours of battery life in 2003), and if you open an IDE from time to time, or do heavy consumer-facing web development, having a computer from the last decade is useful. Also easier to run a high-res multi-monitor setup if you have some good speed behind that. It's possible to be good and capable on the machines you're citing, but the overwhelming majority of software developers (even the ones using vim/emacs) are not of the linux + xmonad type.

marvy 12y ago

Um, how on earth do you deal with the 10GB delete? You can't store it in RAM, since there may not be enough, especially if you want 32 bit support. Store it on disk? You can't store it in the original file, since you have no flexibility over its contents after a save: a hex editor must write the bytes asked for, unlike a word processor which controls the file format. Ok, maybe we store it in some other file, call it "temp". That almost works, but what if the point was to save disk space? "Please restart the editor"? And if the power goes out one hour later, the user finds out at the next boot that clearing out a 10 gig file did not save any disk space? I'm out of ideas. What does Hex Fiend do?

ridiculous_fish 12y ago
The short answer is that it preserves the undo stack, if it can do so without using more than 16 MB of additional memory. If it exceeds that threshold it drops the stack. (Hey, I didn't claim to have solved it well!) The longer answer: when a file is going to be saved, HF identifies the ranges of the file that will be modified, and then identifies every other "byte array" (what vis calls a "piece chain") that references one of those file ranges. This includes data that may have been copied and pasted from the file, as well as the undo stack. It then attempts to "break dependencies" on the about-to-be-overwritten ranges in the file, by copying the referenced bytes into memory first. But if that copying would exceed that 16MB threshold it gives up. Giving up in the case of the undo stack means silently dropping it, and in the case of pasting into another document, it prompts the user: http://i.imgur.com/6Vq7VyQ.png http://i.imgur.com/6Vq7VyQ.png The guiding UI principle is to avoid surprise allocations of large amounts of memory or disk space.
- p0nce 12y ago
  Interesting, I've always wondered how hex editors did it.
- marvy 12y ago
  You may not have solved it perfectly, but better than a version of Excel I once used. Every time you save, it drops the undo stack. No matter what. I think the latest versions have fixed this by now.

shrizza 12y ago

You mentioned Hex Fiend; hed is also a decent hex editor for large files. I've edited 5G files in it. Edit: Ah, just noticed you're the author. Hi.

101914 12y ago

"... I do want a fast text editor that can operate on arbitrarily sized files." Ever try radare (now radare2)?