7 ms·

Finding and fixing Ghostty's largest memory leak

632 points by thorel 9mo ago

quantummagic 9mo ago
This is great news! Well done to everyone who helped sort it out. It was a problem noted by users in a thread here just last week, https://news.ycombinator.com/item?id=46460319 https://news.ycombinator.com/item?id=46460319 While Claude Code might have been the reason this bug became triggered by more people, there are some of us who were hitting it without ever having used Claude Code at all. Maybe the assumption about what makes a page non-standard, isn't as black-and-white as presumed. And I wonder if the leak would have been triggered more often for people who use scrollback-limit = 0, or something very small. Probably not a huge deal, but it does seem the fix will needlessly delete and recreate non-standard pages in the case where the new page needs to be non-standard, and the oldest one (that needs to be pruned) already is non-standard and could be reused.
- mitchellh 9mo ago
  > Probably not a huge deal, but it does seem the fix will needlessly delete and recreate non-standard pages in the case where the new page needs to be non-standard, and the oldest one (that needs to be pruned) already is non-standard and could be reused. This is addressed in the blog post. It is how the PageList has always worked, and also how it worked before with the bug, because during capacity adjustment we would see the wrong size. This shouldn't change any perceived performance. And as I note in the blog post, there are alternative approaches such as the one you suggested, but we don't have enough empirical data to support changing our viewpoint on that whereas our current viewpoint (standard sizes are common) is well supported by known benchmarks. I'm open to changing my mind here, but I didn't want to change worldviews AND fix the leak in the same go.
  - fragmede 9mo ago
    Of all the things to be impressed by you about, your patience is commendable. I'd be losing my shit if someone couldn't be bothered to read what I wrote and just spout off about something I'd addressed in my writing, but I suppose that's why your bank account has two commas and a bunch more. Thank you for everything. Can we go flying sometime?
    - ATMLOTTOBEER 9mo ago
      [flagged]
      - fragmede 9mo ago
        Hell yeah! You don't got any heros? No body you look up to or respect? Not even a little bit?
    - Aurornis 9mo ago
      > I'd be losing my shit if someone couldn't be bothered to read what I wrote and just spout off about something I'd addressed in my writing In my experience that’s a universal feature of comment sections everywhere, and HN is not an exception. This is very common in HN comments which is why it’s important to always read the article, not just the comments.
  - fartfeatures 9mo ago
    How come this isn't released as a hotfix / out of band patch but will follow the standard release cycle in March?
- macote 9mo ago
  The thread about memory leak is here: https://news.ycombinator.com/item?id=46461061 https://news.ycombinator.com/item?id=46461061
  - Maxious 9mo ago
    And the same diagnosis in the blog post was reported by a user in discussions a month ago but ignored https://github.com/ghostty-org/ghostty/discussions/9786#discussioncomment-15179217 https://github.com/ghostty-org/ghostty/discussions/9786#disc...
    - julien_p 9mo ago
      That doesn't sound like the actual issue, or am I not understanding it correctly?
      - dkdcio 9mo ago
        I think you’re correct. the reproduction isn’t very precise and the solution doesn’t seem right (I’m not seeing anything about the non-standard pages not being freed). I’d guess this was ignored because it was wrong…
- commandersaki 9mo ago
  > Well done to everyone who helped sort it out. It was a problem noted by users in a thread here just last week I'm feeling a bit lucky I was able to sneak in an issue during the beta phase, but it was a real reproducible one that led to a segfault.
- larodi 9mo ago
  As a side note - Claude Code is making the CLI attractive in a renewed fashion - more than anything else did it last 20years.
kepano 9mo ago
Reliable reproductions are so valuable.
LgWoodenBadger 9mo ago
The contrast between the attitude here https://news.ycombinator.com/item?id=46461860 https://news.ycombinator.com/item?id=46461860 and in this story is a bit wacky to me.
- resonious 9mo ago
  I think there's only a perceptible "attitude" difference if you are fired up by the fact that they are conservative about using the "issues" tab.
- mitchellh 9mo ago
  What contrast? I stand by what I said there. I just re-read every point and I would say the same thing today and I don't think my blog post contradicts any of that? A user came along and provided a reliable reproduction for me (last night) that allowed me to find and fix the issue. Simultaneously they found the same thing and produced a similar fix, which also helped validate both our approaches. So, we were able to move forward. I said in the linked comment that I believed the leak existed, just couldn't find it. It also was fairly limited in impact. As far as Ghostty bugs go, the number of upvotes the bug report had (9) is very small. The "largest" in the title is with regards to the size of the leak in bytes, not the size of the leak in terms of reach. As extra data to support this, this bug has existed for at least 3 years (since the introduction of this data structure in Ghostty during the private beta). The first time I even heard about it in a way where I can confidently say it was this was maybe 3 or 4 months ago. It was extremely rare. I think the recent rise in popularity of Claude Code in particular was bringing this to the surface more often, but never to the point it rose to a massively reported issue.
  - 1a527dd5 9mo ago
    [flagged]
    - mitchellh 9mo ago
      Discussion upvotes, discussion activity, and Discord reorts. I read every discussion and have been doing this project specifically for a few years now. There is a stark difference between a widespread and common bug and something like this. Like I said, this bug has existed for 3 years at this point and Ghostty is likely used by hundreds of thousands if not a million+ people daily (we don't have any analytics at all but have some side signals based on terminal reports from 3rd party CLIs). Trust me when I say that when there is a widespread issue, we hear it MUCH more loudly. :)
hotpotat 9mo ago
@mitchellh what did you use for the memory visualizations? Looks nice, and the website plays well with mobile. Whats the stack?
- mitchellh 9mo ago
  Static HTML/CSS generated by Opus 4.5. I like using AI for visualizations because it is one-time use throwaway code, so the quality doesn't matter at all (above not being TOTALLY stupid), it doesn't need to be maintained. I review the end result carefully for correctness because it's on a topic I'm an expert of. I produce non-reusable diagrams namespaced by blog post (so they're never used by any other post). I just sanity check that the implementation isn't like... mining bitcoin or leaking secrets (my personal site has no secrets to build) or something. After that, I don't care at all about that quality. The information is conveys is the critical part, and diagrams like this make it so much more consumable for people.
  - hotpotat 9mo ago
    That’s reasonable, thanks!
  - 63 9mo ago
    That's really cool. I was looking at them and thinking "I could probably make these with vanilla html/css but it'd be pretty tedious." Perfect use case for AI. I need to work on developing a reflex for it.
    - mjn 9mo ago
      I've also started doing this, and it's surprisingly enjoyable to both do and even to read. The end result is often more readable to me than using a 3rd-party JS visualization library, because I only need to know standard HTML/CSS concepts to understand what's going on. And a side benefit is smaller pages with less bitrot due to being able to skip the dependencies.
hotpotat 9mo ago
speaking of claude code in Ghostty, I’ve noticed I can’t drag and drop images into the prompt when the session is within a tmux pane. I miss that, coming from the mac terminal app, which allowed me to do so. I’d be willing to look into this myself, but mention it in case someone already knows where to start looking.
- deleted 9mo ago
  [deleted]
bryancoxwell 9mo ago
Super accessible write up as someone unfamiliar with Ghostty and terminal emulators in general. Thanks!
jrpelkonen 9mo ago
Great write-up. And, thanks mitchellh for Ghostty, I switched to it last year, and have not regretted it. However, I am a somewhat surprised that the fix is reserved for a feature release in a couple of months. I would have expected this to be included in a bug fix release.
- msephton 9mo ago
  It's already released in the latest nightly build.
  - DrammBA 9mo ago
    Are the nightly releases the expected way to get timely bugfixes?
    - amazingman 9mo ago
      That is how software releases generally work. AFAICT this is not a bug with broad impact or security implications.
      - fartfeatures 9mo ago
        I guess thats arguable, a memory leak can make a system unpleasant to use although I accept it can be solved by repeatedly restarting the offending app.
    - msephton 9mo ago
      Yes, if you want fixes as soon as they're committed, rather than waiting for a more regular release that might be tested and more stable.
drob518 9mo ago
Why not just use a circular buffer for the scroll back? Why use blocks at all if you’re just going to recycle them anyway? That said, great write-up.
- mitchellh 9mo ago
  It started that way, and that's a common way to do this. One of the reasons is to avoid large pre-allocations OR large copies. A few other notes over on lobsters: https://lobste.rs/s/vlzg2m/finding_fixing_ghostty_s_largest_memory#c_c9i1yj https://lobste.rs/s/vlzg2m/finding_fixing_ghostty_s_largest_...
  - drob518 9mo ago
    Cool, thanks for the link.
neobrain 9mo ago
Funny timing, I moved to Ghostty this week and just today I ran into OOM crashes in Ghostty while developing a terminal UI app. Coincidentally this TUI has a tab bar that looks like this, where UTF8 icons are used for recognizability and activity indicators (using © and € as placeholders here): 1|Flakes © 2|Installed © 3|Store © € 4|Security © € ────────────────────────────────────────────────────────────── This works fine normally, but resizing the terminal would quickly trigger the crash - easy to avoid but still annoying! I was already preparing myself to file a bug report with the easy repro, but this sounds suspiciously close to what the blog post is describing. Fingers crossed :) (EDIT: HN filters unicode, booo :( )
- smoyer 9mo ago
  Why would I move to GhosTTY versus the terminal emulator that comes with my OS as it's not clear to me from the documentation?
  - neobrain 9mo ago
    I don't think I can do a better overview than https://ghostty.org/docs/about https://ghostty.org/docs/about . It's not world-changing but simply a very polished, well-executed terminal. GPU rendering virtually eliminates typing latency. Most terminals that have it don't support native content like tabs, but Ghostty gets minimal latency without having to compromise on essentials since it uses native toolkits under the hood. The modern TTY has lots of protocol extensions that allow your CLI tools to do things like display high-resolution images. There's tons of good-quality color themes out-of-the-box (with a built-in browser for preview). Configuration is highly customizable but the defaults are good enough that you barely need it.
    - smoyer 9mo ago
      I wish a couple of those paragraphs were on the home page!
  - geon 9mo ago
    I moved because Ghostty feels just like the native terminal but allows me to set the color scheme. I have it set to match the vscode Monokai theme. No, macos Terminal will not let you use whatever colors you like. It will helpfully adjust the colors you select to increase contrast. And it can't be disabled. It bugged me for years.
Neywiny 9mo ago
Edit: I'm getting a lot of down votes for this but nobody is saying why I'm wrong. If you think I'm wrong enough to down vote, please reply why. I don't understand why that is the preferred fix. I would have solved it other ways: 1. When resizing the page, leave some flag of how it was allocated. This tagging is commonly done as the always 0 bits in size or address fields to save space. 2. Since the pool is a known size of contiguous memory, check if the memory to be freed is within that range 3. Make the size immutable. If you want to realloc, go for it, and have the memory manager handle that boundary for you. Both of those not only maintain functionality which seems to have been lost with the feature reduction but also are more future proof to any other changes in size.
- hotpotat 9mo ago
  I upvoted you because I would like to know the response to these approaches
  - Neywiny 9mo ago
    Thank you. Sometimes I get to like -4 or even -7 before it starts going up. It might be nice to graph it at some point to see my most varied comments. I'm at -2 right now 23 minutes later I'm at +2 6 minutes after, +5 +4min now +6, another 20 minutes +8. I think I'm in the clear
    - yakaccount4 9mo ago
      I just stopped caring about votes. It's often driven by inertia, and it can't differentiate a vote from someone who doesn't know anything vs a domain expert. Life is better once you stop caring about karma points.
      - Neywiny 9mo ago
        While very true and sound advice, fake internet points make dopamine go brrrrr
    - deleted 9mo ago
dangoodmanUT 9mo ago
waiting for someone to say "this wouldn't have happen if you chose rust"
- woodruffw 9mo ago
  You’ll probably be waiting a long time, since Rust very explicitly doesn’t have “leak safety” as a constructive property. Safe Rust programs are allowed to leak memory, because memory leaks themselves don’t cause safety issues. There’s even a standard, non-unsafe API for leaking memory[1]. (What Rust does do is make it harder to construct programs that leak memory unintentionally. It’s possible but not guaranteed that a similar leak would be difficult to express idiomatically in Rust.) [1]: https://doc.rust-lang.org/std/boxed/struct.Box.html#method.leak https://doc.rust-lang.org/std/boxed/struct.Box.html#method.l...
  - tialaramex 9mo ago
    The specific language feature you want if you insist that you don't want this kind of leak is Linear Types. Rust has Affine Types. This means Rust cares that for any value V of type T, Rust can see that we did not destroy V twice (or more often). With Linear Types the compiler checks that you destroyed V exactly once, not less and not more. However, one reason I don't end up caring about Leak Safety of this sort is that in fact users do not care that you didn't "leak" data in this nerd sense. In this nerd sense what matters is only leaks where we lost all reference to the heap data. But from a user's perspective it's just as bad if we did have the reference but we forgot - or even decided explicitly not - to throw it away and get back the RAM. The obvious way to make this mistake "by accident" in Rust is to have two things which keep each other alive via reference counting and yet have been disconnected and forgotten by the rest of the system. A typical garbage collected language would notice that these are garbage and destroy them both, but Rust isn't a GC language of course. Calling Box::leak isn't likely to happen by accident (though you might mistakenly believe you will call it only once but actually use it much more often) I think the main part of Ghostty's design mentioned here that - as a Rust programmer - I think is probably a mistake is the choice to use a linked list. To me this looks exactly like it needs VecDeque, a circular buffer backed by a growable array type. Their "clever" typical case where you emit more text and so your oldest page is scrapped and re-used to form your newest page, works very nicely in VecDeque, and it seems like they never want the esoteric fast things a linked list can do, nor do they need multi-writer concurrency like the guts of an OS kernel, they want O(1) pop & push from opposite ends. Zig's Deque is probably that same thing but in Zig.
cyberax 9mo ago
Ugh. Is it just me, or is anyone else feeling a tad uncomfortable that their terminal app needs a custom memory allocator that mucks with low-level page tags?
- flumpcakes 9mo ago
  I am not sure on what your commented is based on, but in short: No? High performance software needs to deal with memory, and optimisations often will need some kind of direct control - as in this example where re-using memory is more performant than constantly churning with mmap.
  - sequin 9mo ago
    I honestly don't understand why a terminal emulator needs to be performant. Seems like peak bikeshedding to me.
    - syntheticnature 9mo ago
      You've missed all the posts where people complain about a terminal emulator taking 1ms longer to respond to a keystroke than their preferred one, haven't you?
    - homebrewer 9mo ago
      Scrolling and searching through megabytes of output is often useful. Sometimes you don't expect it and can't prepare for it in advance.
    - RickHull 9mo ago
      https://ghostty.org/docs/about https://ghostty.org/docs/about > Ghostty is a terminal emulator that differentiates itself by being fast, feature-rich, and native. While there are many excellent terminal emulators available, they all force you to choose between speed, features, or native UIs. Ghostty provides all three. > In all categories, I am not trying to claim that Ghostty is the best (i.e. the fastest, most feature-rich, or most native). But when I set out to create Ghostty, I felt all terminals made you choose at most two of these categories. I wanted to create a terminal that was competitive in all three categories and I believe Ghostty achieves that goal. > Before diving into the details, I also want to note that Ghostty is a passion project started by Mitchell Hashimoto (that's me!). It's something I work on in my free time and is a labor of love. Please don't forget this when interacting with the project. I'm doing my best to make something great along with the lovely contributors, but it's not a full-time job for any of us.
vegabook 9mo ago
[flagged]
rvz 9mo ago
[flagged]
stephc_int13 9mo ago
I've been following the development of Ghostty for a while and while I have the feeling that there is a bit of over-engineering in this project, I find this kind of bug post mortem to be extremely valuable for anyone in love with the craft.
- trevorhinesley 9mo ago
  Over-engineered in what way?
  - nesarkvechnep 9mo ago
    It’s just a feeling, man.
  - cbmuser 9mo ago
    Having to introduce a new language stack to distributioms just to be able to build a terminal emulator is what I would consider over-engineering.
    - surajrmal 9mo ago
      So anything that uses a less popular language is considered over engineering? Distros support lots of different languages already and there are likely other packages built with zig already.
    - weebull 9mo ago
      A 50-ish MB build time dependency that doesn't need any special privileges or installation to run? That's over engineering? A binary release of just CMake is bigger than all of Zig.
reactordev 9mo ago
The moment you started talking about pages, I was like: “Ok, obviously memory pooled” and yup, it is. Then I said “obviously ring buffered” and yeah, essentially your scroll back reuse. Then I knew exactly where the bug was before getting to that part, not freeing the pages memory properly and sure enough - bingo! With some great looking diagrams of memory space alignment. Kudos, that was a good read. Just remember that every time you do something novel, there’s potential for leaks :D
deleted 9mo ago
[deleted]
sean_pedersen 9mo ago
Would this kind of bug have been catched by the Rust compiler?
- autarch 9mo ago
  I was wondering about this myself. My guess is no, since AFAIK the only way to do this sort manual memory management is to use unsafe code. But there's also things like the (bumpalo)[https://docs.rs/bumpalo/latest/bumpalo https://docs.rs/bumpalo/latest/bumpalo] crate in Rust, so maybe you wouldn't need to do this sort of thing by hand, in which case you're as leak-free as the bumpalo crate.
KaoruAoiShiho 9mo ago
What's the best claude code terminal? I'm not sure if ghostty is it, which one can sync to iphone / android tablet for remote use of the same session?
- surajrmal 9mo ago
  Sharing a session is independent of the terminal emulator itself. Use tmux for that. There are a handful of good terminal emulators. Weztern, alacritty, and kitty are popular. I use. Tiling window manager so I prefer to avoid tabs and use alacritty for that reason.
ComputerGuru 9mo ago
The number of people here on HN gaslighting those that said they ran into this bug an challenging them to prove it was real..
- mariusor 9mo ago
  As you could see from TFA, getting a reliable reproduction case was the tricky part of fixing this bug, so "asking to prove it's real" is just a mean way of saying asking for reproduction steps, not gaslighting.
  - Maxious 9mo ago
    It only took using claude code or other emoji heavy apps to reproduce and the memory grows linearly over time https://github.com/ghostty-org/ghostty/discussions/9786 https://github.com/ghostty-org/ghostty/discussions/9786
    - mariusor 9mo ago
      "only"... I don't think that means what you think it means in this context.
cyh555 9mo ago
I wonder how a Rust-based terminal implements this without sacrificing performance.
- 0xbrayo 9mo ago
  Someone check out alacritty source code and answer it for us
  - rrgok 9mo ago
    Ask Claude Code about it using Ghostty without this fix ;)
jhhh 9mo ago
This feels like a case of guessing at something you could know. There are two types of allocations that each have a size and free method. The free method is polymorphic over the allocations type. Instead of using a tag to know absolutely which type an object it is you guess based on some other factor, in this case a size invariant which was violated. It also doesn't seem like this invariant was ever codified otherwise the first time a large alloc was modified to a standard size it would've blown up. It's worth asking yourself if your distinguishing factor is the best you can use or perhaps there is a better test. Maybe in this case a tag would've been too expensive.
llmslave3 9mo ago
I hate to say it, but this probably would not have happened in a garbage collected language. GC languages are fast these days. If you don't want a runtime like C# (which has excellent performance) a language like Go would have worked just fine here, compiling to a small native binary but with a GC. I don't really understand the aversion to GC's. In memory constrained scenarios or where performance is an absolute top priority, I understand wanting manual control. But that seems like a very rare scenario in user space.
- p-e-w 9mo ago
  I agree that garbage collection is fine and Go indeed has an amazing garbage collector. Unfortunately, it also has the worst type system of all mainstream languages created in the 21st century, so the benefits are rarely worth the drawbacks.
  - llmslave3 9mo ago
    Go's type system is fine. This kind of comment is just pointless and goes against HN rules.
- surajrmal 9mo ago
  Why do you think trippling the memory usage of a program is an acceptable tradeoff? It's not just GC pauses that are problematic with gc languages. Some software wants to run on systems with less than 4GiB of RAM.
  - llmslave3 9mo ago
    [flagged]
gfyhthgyrfg 9mo ago
[dead]
deleted 9mo ago
[deleted]
gethly 9mo ago
Should have used Odin instead of Zig.
- tialaramex 9mo ago
  How exactly would using a different unfinished programming language have helped?
deleted 9mo ago
[deleted]
liveoneggs 9mo ago
claude code also has a weird thing in ghostty where it breaks copy-paste after exiting. `reset` fixes it but it's annoying
bschwarz 9mo ago
How little guidance can you give Claude Code to a) find and b) fix this memory leak? Summoning @simonw
tk90 9mo ago
A couple weeks ago my Ghostty session crashed and found that it was using 40GB of RAM(!) - glad this was resolved!
andrewaylett 9mo ago
Let me see if I can understand this properly: There's a linear buffer of pages, most of which come from the pool. It's not clear to me under what conditions these are returned to the pool? Is it when the specific session terminates? When a non-standard page reaches the point of being recycled, it'll instead be re-added to the list but with a standard size. That effectively leaks the extra space above the standard size. But when the buffer is released (because the session ends?) the pool is also released, which releases all the standard sized pages but leaks the custom-sized ones? Which suggests that the issue may be even rarer than it initially looked to me: I tend to open a small number of sessions and then use them continuously, rather than starting new sessions during the lifetime of the process. If I never terminated a session, I would never fully leak the memory?
deleted 9mo ago
[deleted]

632 points by thorel 9mo ago

This is great news! Well done to everyone who helped sort it out. It was a problem noted by users in a thread here just last week, https://news.ycombinator.com/item?id=46460319 https://news.ycombinator.com/item?id=46460319 While Claude Code might have been the reason this bug became triggered by more people, there are some of us who were hitting it without ever having used Claude Code at all. Maybe the assumption about what makes a page non-standard, isn't as black-and-white as presumed. And I wonder if the leak would have been triggered more often for people who use scrollback-limit = 0, or something very small. Probably not a huge deal, but it does seem the fix will needlessly delete and recreate non-standard pages in the case where the new page needs to be non-standard, and the oldest one (that needs to be pruned) already is non-standard and could be reused.

mitchellh 9mo ago
> Probably not a huge deal, but it does seem the fix will needlessly delete and recreate non-standard pages in the case where the new page needs to be non-standard, and the oldest one (that needs to be pruned) already is non-standard and could be reused. This is addressed in the blog post. It is how the PageList has always worked, and also how it worked before with the bug, because during capacity adjustment we would see the wrong size. This shouldn't change any perceived performance. And as I note in the blog post, there are alternative approaches such as the one you suggested, but we don't have enough empirical data to support changing our viewpoint on that whereas our current viewpoint (standard sizes are common) is well supported by known benchmarks. I'm open to changing my mind here, but I didn't want to change worldviews AND fix the leak in the same go.
- fragmede 9mo ago
  Of all the things to be impressed by you about, your patience is commendable. I'd be losing my shit if someone couldn't be bothered to read what I wrote and just spout off about something I'd addressed in my writing, but I suppose that's why your bank account has two commas and a bunch more. Thank you for everything. Can we go flying sometime?
  - ATMLOTTOBEER 9mo ago
    [flagged]
    - fragmede 9mo ago
      Hell yeah! You don't got any heros? No body you look up to or respect? Not even a little bit?
  - Aurornis 9mo ago
    > I'd be losing my shit if someone couldn't be bothered to read what I wrote and just spout off about something I'd addressed in my writing In my experience that’s a universal feature of comment sections everywhere, and HN is not an exception. This is very common in HN comments which is why it’s important to always read the article, not just the comments.
- fartfeatures 9mo ago
  How come this isn't released as a hotfix / out of band patch but will follow the standard release cycle in March?
macote 9mo ago
The thread about memory leak is here: https://news.ycombinator.com/item?id=46461061 https://news.ycombinator.com/item?id=46461061
- Maxious 9mo ago
  And the same diagnosis in the blog post was reported by a user in discussions a month ago but ignored https://github.com/ghostty-org/ghostty/discussions/9786#discussioncomment-15179217 https://github.com/ghostty-org/ghostty/discussions/9786#disc...
  - julien_p 9mo ago
    That doesn't sound like the actual issue, or am I not understanding it correctly?
    - dkdcio 9mo ago
      I think you’re correct. the reproduction isn’t very precise and the solution doesn’t seem right (I’m not seeing anything about the non-standard pages not being freed). I’d guess this was ignored because it was wrong…
commandersaki 9mo ago
> Well done to everyone who helped sort it out. It was a problem noted by users in a thread here just last week I'm feeling a bit lucky I was able to sneak in an issue during the beta phase, but it was a real reproducible one that led to a segfault.
larodi 9mo ago
As a side note - Claude Code is making the CLI attractive in a renewed fashion - more than anything else did it last 20years.

kepano 9mo ago

Reliable reproductions are so valuable.

LgWoodenBadger 9mo ago

The contrast between the attitude here https://news.ycombinator.com/item?id=46461860 https://news.ycombinator.com/item?id=46461860 and in this story is a bit wacky to me.

resonious 9mo ago
I think there's only a perceptible "attitude" difference if you are fired up by the fact that they are conservative about using the "issues" tab.
mitchellh 9mo ago
What contrast? I stand by what I said there. I just re-read every point and I would say the same thing today and I don't think my blog post contradicts any of that? A user came along and provided a reliable reproduction for me (last night) that allowed me to find and fix the issue. Simultaneously they found the same thing and produced a similar fix, which also helped validate both our approaches. So, we were able to move forward. I said in the linked comment that I believed the leak existed, just couldn't find it. It also was fairly limited in impact. As far as Ghostty bugs go, the number of upvotes the bug report had (9) is very small. The "largest" in the title is with regards to the size of the leak in bytes, not the size of the leak in terms of reach. As extra data to support this, this bug has existed for at least 3 years (since the introduction of this data structure in Ghostty during the private beta). The first time I even heard about it in a way where I can confidently say it was this was maybe 3 or 4 months ago. It was extremely rare. I think the recent rise in popularity of Claude Code in particular was bringing this to the surface more often, but never to the point it rose to a massively reported issue.
- 1a527dd5 9mo ago
  [flagged]
  - mitchellh 9mo ago
    Discussion upvotes, discussion activity, and Discord reorts. I read every discussion and have been doing this project specifically for a few years now. There is a stark difference between a widespread and common bug and something like this. Like I said, this bug has existed for 3 years at this point and Ghostty is likely used by hundreds of thousands if not a million+ people daily (we don't have any analytics at all but have some side signals based on terminal reports from 3rd party CLIs). Trust me when I say that when there is a widespread issue, we hear it MUCH more loudly. :)

hotpotat 9mo ago

@mitchellh what did you use for the memory visualizations? Looks nice, and the website plays well with mobile. Whats the stack?

mitchellh 9mo ago
Static HTML/CSS generated by Opus 4.5. I like using AI for visualizations because it is one-time use throwaway code, so the quality doesn't matter at all (above not being TOTALLY stupid), it doesn't need to be maintained. I review the end result carefully for correctness because it's on a topic I'm an expert of. I produce non-reusable diagrams namespaced by blog post (so they're never used by any other post). I just sanity check that the implementation isn't like... mining bitcoin or leaking secrets (my personal site has no secrets to build) or something. After that, I don't care at all about that quality. The information is conveys is the critical part, and diagrams like this make it so much more consumable for people.
- hotpotat 9mo ago
  That’s reasonable, thanks!
- 63 9mo ago
  That's really cool. I was looking at them and thinking "I could probably make these with vanilla html/css but it'd be pretty tedious." Perfect use case for AI. I need to work on developing a reflex for it.
  - mjn 9mo ago
    I've also started doing this, and it's surprisingly enjoyable to both do and even to read. The end result is often more readable to me than using a 3rd-party JS visualization library, because I only need to know standard HTML/CSS concepts to understand what's going on. And a side benefit is smaller pages with less bitrot due to being able to skip the dependencies.

hotpotat 9mo ago

speaking of claude code in Ghostty, I’ve noticed I can’t drag and drop images into the prompt when the session is within a tmux pane. I miss that, coming from the mac terminal app, which allowed me to do so. I’d be willing to look into this myself, but mention it in case someone already knows where to start looking.

deleted 9mo ago
[deleted]

bryancoxwell 9mo ago

Super accessible write up as someone unfamiliar with Ghostty and terminal emulators in general. Thanks!

jrpelkonen 9mo ago

Great write-up. And, thanks mitchellh for Ghostty, I switched to it last year, and have not regretted it. However, I am a somewhat surprised that the fix is reserved for a feature release in a couple of months. I would have expected this to be included in a bug fix release.

msephton 9mo ago
It's already released in the latest nightly build.
- DrammBA 9mo ago
  Are the nightly releases the expected way to get timely bugfixes?
  - amazingman 9mo ago
    That is how software releases generally work. AFAICT this is not a bug with broad impact or security implications.
    - fartfeatures 9mo ago
      I guess thats arguable, a memory leak can make a system unpleasant to use although I accept it can be solved by repeatedly restarting the offending app.
  - msephton 9mo ago
    Yes, if you want fixes as soon as they're committed, rather than waiting for a more regular release that might be tested and more stable.

drob518 9mo ago

Why not just use a circular buffer for the scroll back? Why use blocks at all if you’re just going to recycle them anyway? That said, great write-up.

mitchellh 9mo ago
It started that way, and that's a common way to do this. One of the reasons is to avoid large pre-allocations OR large copies. A few other notes over on lobsters: https://lobste.rs/s/vlzg2m/finding_fixing_ghostty_s_largest_memory#c_c9i1yj https://lobste.rs/s/vlzg2m/finding_fixing_ghostty_s_largest_...
- drob518 9mo ago
  Cool, thanks for the link.

neobrain 9mo ago

Funny timing, I moved to Ghostty this week and just today I ran into OOM crashes in Ghostty while developing a terminal UI app. Coincidentally this TUI has a tab bar that looks like this, where UTF8 icons are used for recognizability and activity indicators (using © and € as placeholders here): 1|Flakes © 2|Installed © 3|Store © € 4|Security © € ────────────────────────────────────────────────────────────── This works fine normally, but resizing the terminal would quickly trigger the crash - easy to avoid but still annoying! I was already preparing myself to file a bug report with the easy repro, but this sounds suspiciously close to what the blog post is describing. Fingers crossed :) (EDIT: HN filters unicode, booo :( )

smoyer 9mo ago
Why would I move to GhosTTY versus the terminal emulator that comes with my OS as it's not clear to me from the documentation?
- neobrain 9mo ago
  I don't think I can do a better overview than https://ghostty.org/docs/about https://ghostty.org/docs/about . It's not world-changing but simply a very polished, well-executed terminal. GPU rendering virtually eliminates typing latency. Most terminals that have it don't support native content like tabs, but Ghostty gets minimal latency without having to compromise on essentials since it uses native toolkits under the hood. The modern TTY has lots of protocol extensions that allow your CLI tools to do things like display high-resolution images. There's tons of good-quality color themes out-of-the-box (with a built-in browser for preview). Configuration is highly customizable but the defaults are good enough that you barely need it.
  - smoyer 9mo ago
    I wish a couple of those paragraphs were on the home page!
- geon 9mo ago
  I moved because Ghostty feels just like the native terminal but allows me to set the color scheme. I have it set to match the vscode Monokai theme. No, macos Terminal will not let you use whatever colors you like. It will helpfully adjust the colors you select to increase contrast. And it can't be disabled. It bugged me for years.

Neywiny 9mo ago

Edit: I'm getting a lot of down votes for this but nobody is saying why I'm wrong. If you think I'm wrong enough to down vote, please reply why. I don't understand why that is the preferred fix. I would have solved it other ways: 1. When resizing the page, leave some flag of how it was allocated. This tagging is commonly done as the always 0 bits in size or address fields to save space. 2. Since the pool is a known size of contiguous memory, check if the memory to be freed is within that range 3. Make the size immutable. If you want to realloc, go for it, and have the memory manager handle that boundary for you. Both of those not only maintain functionality which seems to have been lost with the feature reduction but also are more future proof to any other changes in size.

hotpotat 9mo ago
I upvoted you because I would like to know the response to these approaches
- Neywiny 9mo ago
  Thank you. Sometimes I get to like -4 or even -7 before it starts going up. It might be nice to graph it at some point to see my most varied comments. I'm at -2 right now 23 minutes later I'm at +2 6 minutes after, +5 +4min now +6, another 20 minutes +8. I think I'm in the clear
  - yakaccount4 9mo ago
    I just stopped caring about votes. It's often driven by inertia, and it can't differentiate a vote from someone who doesn't know anything vs a domain expert. Life is better once you stop caring about karma points.
    - Neywiny 9mo ago
      While very true and sound advice, fake internet points make dopamine go brrrrr
  - deleted 9mo ago

dangoodmanUT 9mo ago

waiting for someone to say "this wouldn't have happen if you chose rust"

woodruffw 9mo ago
You’ll probably be waiting a long time, since Rust very explicitly doesn’t have “leak safety” as a constructive property. Safe Rust programs are allowed to leak memory, because memory leaks themselves don’t cause safety issues. There’s even a standard, non-unsafe API for leaking memory[1]. (What Rust does do is make it harder to construct programs that leak memory unintentionally. It’s possible but not guaranteed that a similar leak would be difficult to express idiomatically in Rust.) [1]: https://doc.rust-lang.org/std/boxed/struct.Box.html#method.leak https://doc.rust-lang.org/std/boxed/struct.Box.html#method.l...
- tialaramex 9mo ago
  The specific language feature you want if you insist that you don't want this kind of leak is Linear Types. Rust has Affine Types. This means Rust cares that for any value V of type T, Rust can see that we did not destroy V twice (or more often). With Linear Types the compiler checks that you destroyed V exactly once, not less and not more. However, one reason I don't end up caring about Leak Safety of this sort is that in fact users do not care that you didn't "leak" data in this nerd sense. In this nerd sense what matters is only leaks where we lost all reference to the heap data. But from a user's perspective it's just as bad if we did have the reference but we forgot - or even decided explicitly not - to throw it away and get back the RAM. The obvious way to make this mistake "by accident" in Rust is to have two things which keep each other alive via reference counting and yet have been disconnected and forgotten by the rest of the system. A typical garbage collected language would notice that these are garbage and destroy them both, but Rust isn't a GC language of course. Calling Box::leak isn't likely to happen by accident (though you might mistakenly believe you will call it only once but actually use it much more often) I think the main part of Ghostty's design mentioned here that - as a Rust programmer - I think is probably a mistake is the choice to use a linked list. To me this looks exactly like it needs VecDeque, a circular buffer backed by a growable array type. Their "clever" typical case where you emit more text and so your oldest page is scrapped and re-used to form your newest page, works very nicely in VecDeque, and it seems like they never want the esoteric fast things a linked list can do, nor do they need multi-writer concurrency like the guts of an OS kernel, they want O(1) pop & push from opposite ends. Zig's Deque is probably that same thing but in Zig.

cyberax 9mo ago

Ugh. Is it just me, or is anyone else feeling a tad uncomfortable that their terminal app needs a custom memory allocator that mucks with low-level page tags?

flumpcakes 9mo ago
I am not sure on what your commented is based on, but in short: No? High performance software needs to deal with memory, and optimisations often will need some kind of direct control - as in this example where re-using memory is more performant than constantly churning with mmap.
- sequin 9mo ago
  I honestly don't understand why a terminal emulator needs to be performant. Seems like peak bikeshedding to me.
  - syntheticnature 9mo ago
    You've missed all the posts where people complain about a terminal emulator taking 1ms longer to respond to a keystroke than their preferred one, haven't you?
  - homebrewer 9mo ago
    Scrolling and searching through megabytes of output is often useful. Sometimes you don't expect it and can't prepare for it in advance.
  - RickHull 9mo ago
    https://ghostty.org/docs/about https://ghostty.org/docs/about > Ghostty is a terminal emulator that differentiates itself by being fast, feature-rich, and native. While there are many excellent terminal emulators available, they all force you to choose between speed, features, or native UIs. Ghostty provides all three. > In all categories, I am not trying to claim that Ghostty is the best (i.e. the fastest, most feature-rich, or most native). But when I set out to create Ghostty, I felt all terminals made you choose at most two of these categories. I wanted to create a terminal that was competitive in all three categories and I believe Ghostty achieves that goal. > Before diving into the details, I also want to note that Ghostty is a passion project started by Mitchell Hashimoto (that's me!). It's something I work on in my free time and is a labor of love. Please don't forget this when interacting with the project. I'm doing my best to make something great along with the lovely contributors, but it's not a full-time job for any of us.

vegabook 9mo ago

[flagged]

rvz 9mo ago

[flagged]

stephc_int13 9mo ago

I've been following the development of Ghostty for a while and while I have the feeling that there is a bit of over-engineering in this project, I find this kind of bug post mortem to be extremely valuable for anyone in love with the craft.

trevorhinesley 9mo ago
Over-engineered in what way?
- nesarkvechnep 9mo ago
  It’s just a feeling, man.
- cbmuser 9mo ago
  Having to introduce a new language stack to distributioms just to be able to build a terminal emulator is what I would consider over-engineering.
  - surajrmal 9mo ago
    So anything that uses a less popular language is considered over engineering? Distros support lots of different languages already and there are likely other packages built with zig already.
  - weebull 9mo ago
    A 50-ish MB build time dependency that doesn't need any special privileges or installation to run? That's over engineering? A binary release of just CMake is bigger than all of Zig.

reactordev 9mo ago

The moment you started talking about pages, I was like: “Ok, obviously memory pooled” and yup, it is. Then I said “obviously ring buffered” and yeah, essentially your scroll back reuse. Then I knew exactly where the bug was before getting to that part, not freeing the pages memory properly and sure enough - bingo! With some great looking diagrams of memory space alignment. Kudos, that was a good read. Just remember that every time you do something novel, there’s potential for leaks :D

deleted 9mo ago

[deleted]

sean_pedersen 9mo ago

Would this kind of bug have been catched by the Rust compiler?

autarch 9mo ago
I was wondering about this myself. My guess is no, since AFAIK the only way to do this sort manual memory management is to use unsafe code. But there's also things like the (bumpalo)[https://docs.rs/bumpalo/latest/bumpalo https://docs.rs/bumpalo/latest/bumpalo] crate in Rust, so maybe you wouldn't need to do this sort of thing by hand, in which case you're as leak-free as the bumpalo crate.

KaoruAoiShiho 9mo ago

What's the best claude code terminal? I'm not sure if ghostty is it, which one can sync to iphone / android tablet for remote use of the same session?

surajrmal 9mo ago
Sharing a session is independent of the terminal emulator itself. Use tmux for that. There are a handful of good terminal emulators. Weztern, alacritty, and kitty are popular. I use. Tiling window manager so I prefer to avoid tabs and use alacritty for that reason.

ComputerGuru 9mo ago

The number of people here on HN gaslighting those that said they ran into this bug an challenging them to prove it was real..

mariusor 9mo ago
As you could see from TFA, getting a reliable reproduction case was the tricky part of fixing this bug, so "asking to prove it's real" is just a mean way of saying asking for reproduction steps, not gaslighting.
- Maxious 9mo ago
  It only took using claude code or other emoji heavy apps to reproduce and the memory grows linearly over time https://github.com/ghostty-org/ghostty/discussions/9786 https://github.com/ghostty-org/ghostty/discussions/9786
  - mariusor 9mo ago
    "only"... I don't think that means what you think it means in this context.

cyh555 9mo ago

I wonder how a Rust-based terminal implements this without sacrificing performance.

0xbrayo 9mo ago
Someone check out alacritty source code and answer it for us
- rrgok 9mo ago
  Ask Claude Code about it using Ghostty without this fix ;)

jhhh 9mo ago

This feels like a case of guessing at something you could know. There are two types of allocations that each have a size and free method. The free method is polymorphic over the allocations type. Instead of using a tag to know absolutely which type an object it is you guess based on some other factor, in this case a size invariant which was violated. It also doesn't seem like this invariant was ever codified otherwise the first time a large alloc was modified to a standard size it would've blown up. It's worth asking yourself if your distinguishing factor is the best you can use or perhaps there is a better test. Maybe in this case a tag would've been too expensive.

llmslave3 9mo ago

I hate to say it, but this probably would not have happened in a garbage collected language. GC languages are fast these days. If you don't want a runtime like C# (which has excellent performance) a language like Go would have worked just fine here, compiling to a small native binary but with a GC. I don't really understand the aversion to GC's. In memory constrained scenarios or where performance is an absolute top priority, I understand wanting manual control. But that seems like a very rare scenario in user space.

p-e-w 9mo ago
I agree that garbage collection is fine and Go indeed has an amazing garbage collector. Unfortunately, it also has the worst type system of all mainstream languages created in the 21st century, so the benefits are rarely worth the drawbacks.
- llmslave3 9mo ago
  Go's type system is fine. This kind of comment is just pointless and goes against HN rules.
surajrmal 9mo ago
Why do you think trippling the memory usage of a program is an acceptable tradeoff? It's not just GC pauses that are problematic with gc languages. Some software wants to run on systems with less than 4GiB of RAM.
- llmslave3 9mo ago
  [flagged]

gfyhthgyrfg 9mo ago

[dead]

deleted 9mo ago

[deleted]

gethly 9mo ago

Should have used Odin instead of Zig.

tialaramex 9mo ago
How exactly would using a different unfinished programming language have helped?

deleted 9mo ago

[deleted]

liveoneggs 9mo ago

claude code also has a weird thing in ghostty where it breaks copy-paste after exiting. `reset` fixes it but it's annoying

bschwarz 9mo ago

How little guidance can you give Claude Code to a) find and b) fix this memory leak? Summoning @simonw

tk90 9mo ago

A couple weeks ago my Ghostty session crashed and found that it was using 40GB of RAM(!) - glad this was resolved!

andrewaylett 9mo ago

Let me see if I can understand this properly: There's a linear buffer of pages, most of which come from the pool. It's not clear to me under what conditions these are returned to the pool? Is it when the specific session terminates? When a non-standard page reaches the point of being recycled, it'll instead be re-added to the list but with a standard size. That effectively leaks the extra space above the standard size. But when the buffer is released (because the session ends?) the pool is also released, which releases all the standard sized pages but leaks the custom-sized ones? Which suggests that the issue may be even rarer than it initially looked to me: I tend to open a small number of sessions and then use them continuously, rather than starting new sessions during the lifetime of the process. If I never terminated a session, I would never fully leak the memory?

deleted 9mo ago

[deleted]