11 ms·
Animated GIF uses over 35GB RAM in Acorn on M1 Mac, likely due to memory leak
- BoorishBears 5y agoThis is not a gif that somehow causes excessive allocations when rendered, the gif is being opened in a specific editor. The gif has nothing to do with it if anything, opening 700+ frames in an editor is causing a problem. It is a clickbait title trying to take advantage of the fact people will think this was like gif bombs of the past. - Downvoting me won't make your uninteresting bug in the systems layer of a new arch interesting, or your title any less clickbaity, nor will your quoting rules about mentioning downvotes like a bot :) Clickbait is dirty stuff that shouldn't be allowed to ruin the usefulness of an aggregator by turning every post into a headline writing competition
- stephen_g 5y agoSeems to just be a memory leak in a framework the editor is using, that seems to only affect M1. Which is mildly interesting but a bit of an anticlimax...
- NobodyNada 5y agoThe framework is part of Apple’s UI stack and indicates a window system/graphics driver bug somewhere in macOS. So it’s probably more than just “an application has a bug”, but still not very interesting until we learn more.
- BoorishBears 5y agoMy point was nowhere nearly as interesting as if a specially crafted gif somehow triggered a memory leak in a driver when rendered. The title doesn't make it clear individually opening 730 frames in an the editor is the userspace action exposing an underlying system bug
- jiofih 5y agoWhere do you see anything about individually opening 730 frames, and why would it be unsurprising for that tiny load to cause a bug? Have you heard of video editors? The same machine can edit four 4K@60fps streams.
- BoorishBears 5y agoThanks for making my point, the program is indeed opening individual frames. And what are you on about after that? No one said it's unsurprising for that to cause a bug. It's saying that the gif is not cause here, an editor is coming across a framework call that's blowing up Call it an editor bug, or a framework bug, it's not an image using the memory, it's the editor
- jiofih 5y agoOh my god. I get it. You’re reading the headline as “this GIF uses xx memory”. That’s not what is says. It’s “GIF uses X memory on x86, X*10 memory on M1”. You’re picking a fight based on your own misreading.
- klibertp 5y ago> My point was nowhere nearly as interesting as if a specially crafted gif somehow triggered a memory leak in a driver when rendered. There's absolutely no mention of "specially crafted" GIF or "when rendered" in the title. You're projecting your expectations which have nothing to do with the story and are not even hinted at by the title, and choose to be offended that they get betrayed. There are many things I'd like to retort here, but the most important is that it's against the HN guidelines, so it would be good if you stopped.
- BoorishBears 5y agoWhat's a GIF? An image format. What's done with image formats? They are viewed. The default thought if someone says they opened a gif isn't "they splayed open the hundreds of frames for individual manipulation" anymore than someone saying "Car moved from point A to point B" would mean "Car disassembled and moved piece by piece from point A to point B" And I don't recall the rule asking you to backseat mod :) Nothing against the rules in explaining simple nuances of written word
- jbverschoor 5y agoHow is this not just another bug? Is exactly that, just another bug
- samatman 5y agoYes, probably a system bug, not an application bug. As the short post you replied to made perfectly clear.
- BoorishBears 5y agoClickbait titles and roundabout Twitter threads elevate it to HN darling status apparently.
- Dylan16807 5y ago> gif bombs of the past. Which would also require opening it in a specific program. This is a strange complaint.
- BoorishBears 5y agoThey would require *rendering* in a specific program This program is an image editor essentially opening 700 tabs, one for each frame. It's not a "bomb" when you intentionally blow it up...
- nieve 5y agoSeveral programs including Firefox Nightly have been reported to blow up, so no it's not limited to a "specific program".
- BoorishBears 5y agoIf you're actually following it proves exactly what I'm saying, the gif has nothing to do with it, Firefox isn't blowing up opening that gif, it's hitting whatever framework leak exists. This is like saying "DOCX bluescreens Windows" (which implies something that could possibly be exploited in some pretty scary ways) when in reality Word just happens to make some syscall that bluescreens Windows no matter who calls it
- jiofih 5y agoI don’t know why you made up your mind on this without having any details. The person reporting is the author of a graphics editor, he damn well knows the difference between a bug in his own editor and system frameworks. The issue seems to be with opening the GIF via default system frameworks, not anything special the editor is doing. Chances are it will affect any other programs using the same frameworks, maybe even GIF viewers.
- BoorishBears 5y ago
- yreg 5y agoThe maintainer in that thread believes that the memory reporting is wrong.
- coldcode 5y agoI would find that highly likely, for some reason Activity Monitor is often flaky.
- danielparks 5y agoThere’s more information, including a copy of the image, in the forum discussion. The forum seems a bit pokey for me at the moment, though. https://forums.flyingmeat.com/t/memory-consumption-when-opening-an-animated-gif/706 https://forums.flyingmeat.com/t/memory-consumption-when-open... Gus Mueller (the author of the linked tweet) is the author of the Acorn image editor.
- rnestler 5y agoThanks for the link. The issue seems to be the IOAccelerator framework: > I’ve narrowed it down quite a bit (and submitted it to Apple as FB9112835). > What’s going on is that the IOAccelerator framework has some sort of massive leak in it, where it’s using up 35GB of ram, 25 of which is going to swap (which is why you’re seeing kernel_task flake out). > On intel, the same image only uses 1480K from IOAccelerator.
- SXX 5y agoWow. This can explain some of excesive swap usage on M1.
- carlhjerpe 5y agoAnd yet it's relatively performant, says a bit about the quality of components in apple hardware (SSD in this case)!
- saagarjha 5y agoSeems like a poor conclusion to draw with this little information.
- zarzavat 5y agoPeople have reported their SSDs filling up (in terms of total writes) much faster on Apple Silicon machines. If IOAccelerator can leak like this then it would definitely explain it. 25GB swap for one image is absurd. Multiply that by a few months of usage. It may not be a smoking gun but it is a fingerprint in the pool of blood.
- est 5y agoI remember gif bomb few years back on forums.
- foobar1962 5y agoThere was a zip bomb too...
- bangonkeyboard 5y agoCould this be related to the earlier reports of excessive SSD wear on M1 Macs?
- TimSchumann 5y agoPossibly, I’m assuming all those extra GB’s of space end up being swap.
- DenverCode 5y agoJust purchased my first Mac after using Linux for most of my adult life. How big of a concern is this? I am seeing mixed reports when looking around - should I be questioning my purchase if I was expecting my M1 Air to last a few years?
- cerved 5y agoAFAIK SSDs are generally replaceable, even in MacBooks. If you can replace it, I wouldn't worry too much
- sk0g 5y agoFrom what I could understand, it's related to how much of your workflow runs over the RAM capacity, at which point the memory gets shuffled off to the SSD, and loaded back when it's needed. I got the 8GB RAM version, and been mainly using it for Unity + JetBrains Rider, both demanding around 4-6GB by themselves. Doesn't help neither of them are ARM native, I'd imagine. So I'm in big trouble :)
- techrat 5y agoEven 360MB seemed a bit high to me so I had to open it up in a few apps to get some perspective. Ristretto: 48.8MB. EOG: 51.3MB. ImageMagick: 169MB. Firefox: 305.1MB. Vivaldi: 185.2MB. Edge: 196.5MB. GIMP: 244.8MB. OS memory usage remained relatively flat once app exited. xUbuntu 20.04.
- thanatos519 5y agoMy GIMP process reached 278MB (93MB of which was shared). According to the Dashboard tab, 60MB of that was cache, including layer thumbnails and mipmaps. It's definitely not inflating the memory footprint more than expected. It's hard to count the total pixels, since every frame has different dimensions.
- marcan_42 5y agoTry this one: https://mrcn.st/t/bomb.gif https://mrcn.st/t/bomb.gif It at least causes ImageMagick to explode, and generally anything that tries to flatten out the GIF into full canvas-sized frames.
- Gaelan 5y agoiOS Safari seems to render it with no performance issues. Interestingly, it can’t seem to decide whether the background is black or green—it’s different each time I load the gif.
- marcan_42 5y agoBrowsers and other players will generally be fine, as they render the image progressively. However, editors /processing tools which attempt to load it as a series of frames or a video will usually break, as memory consumption explodes, unless they have a smart disk buffer system to handle it. I wouldn't be surprised if some poorly implemented players were built on top of abstraction layers that end up flattening the whole thing too and also break, but browsers at least generally do it right.
- 5y ago
- deleted 5y ago[deleted]
- chirau 5y agoDoes Mac have swap? If so, how does it allocate it by default?
- kzrdude 5y agoMy macos knowledge is dated (didn't use it seriously since 10.5), but it used to be a dynamically growing collection of swapfiles, usually covering all of the RAM (probably can't do that anymore).
- umed 5y agoMacOS Big Sur: recently implemented an algorithm that leaked and left it running for quite some time. Got notification that there is not enough RAM and suggestion to close some apps once size of swap grown up to 64GB (or it was size of RAM + swap, don't remember).
- Toutouxc 5y agoYes, macOS creates a series of 1 GB swapfiles on a special disk volume called "VM".
- vaylian 5y agoWhat does VM stand for?
- bufferoverflow 5y agoVirtual memory, guessing here.
- JKCalhoun 5y agoMy first thought was that each "frame" of the GIF is being expanded/rendered to its own backingstore. GIFs are pretty optimized files — where each "frame" can be a diff from the previous. "De-diffing", converting palette-based pixels to full 24 or 32-bit RGB could really blow up fast.
- okamiueru 5y agoFrom a codec perspective gifs would be the first thing that came to mind for a poorly optimized format. The only thing it has going for it is simplicity. This is a bit besides the point you're making, but I did a double take at seeing the words "gif" and "pretty optimized" together. I'm also curious as to what this has to do with M1 in particular. Looks to be like a memory allocation issue in the decoder library. But hard to tell given the amount of information given. Memory leaks tend to be boundless, but increasing, so the op statement of a fixed memory usage kinda sounds more like poorly optimized allocations, rather than non-properly-cleared up allocations. Though, could also be a mix of everything.
- eduo 5y ago"Optimized" in this sense meant that animated gifs can have a frame reference only three pixels of the original image. So an image of 300K with only small movement (think cinemagraphs) wouldn't be much larger. This is a given for movie formats, but at the time the animated GIF came up it was revolutionary. I think the proper phrase should be "animated GIFs can be pretty optimized, taking into account how inefficient the algorithm is, when compared with other animation algorithms of the time". I also think there's an interpretation that applies here: When you see an animated gif, even if it's a frame that changes three pixels and nothing else, internally the renderer may be expanding it into a full movie (that is, uncompressing each resulting "frame"). This usually makes GIFs (regardless of how large or small the GIF actually is) take much more memory than common sense would tell you.
- TomSwirly 5y agoThat's "compressed," not "optimized".
- deleted 5y ago[deleted]
- ulzeraj 5y agoWho else is excited that we are might revive the 80s Cambrian explosion of different system system and architectures? Back then there were so many options.
- SomeHacker44 5y agoI would love to see a CPU Renaissance like this. Back then we had tons of variety, 680x0, x86, Rx000, various lisp machines, Vector computers, VLIW and Multiflow, Sparc, VAX, early ARM, message passing machines, 1-bit multiprocessors, Hypercubes, WD CPUs, and later an explosion of interesting RISC architectures... It was really interesting and enjoyable era.
- TomSwirly 5y agoAs someone who programmed at that time, it was also very hard to write even small production programs. Today I do things in a half-an-hour with Python that would have taken me days - maybe weeks! - to accomplish in 1978. Each little vendor had their own janky tooling. Compilers cost hundreds of 1970s dollars (until Borland's $49 Turbo Pascal, over $150 in today's money). Don't get me wrong. I was very unhappy when Intel dominated everything. The fact that ARM, an open-source architecture, is now eating Intel's lunch makes me happy. But I'd honestly be glad if everyone just settled on ARM and were done with it. It was fun messing with all these weird processors (my first team leader job was writing an operating system for a pocket computer running the 65816 processor!) but it meant that actually generating work was very slow.
- xoa 5y agoI mostly agree with your overall argument, but the "mostly" qualification goes along with a small but important correction: >The fact that ARM, an open-source architecture ARM is in no way open, it's fully proprietary. Unlike x86 it is not vertically integrated and is available for anyone to license all the way to the architectural level, and that's huge. But said licenses certainly are not free either, nor Free. There are promising actual open architectures, in particular OpenPOWER and RISC-V come to mind as interesting with a lot of solid work behind them. So that's one small remaining opening IMO, even if it's more work on the dev side I wouldn't mind having those stick around and get more competitive.
- williesleg 5y agoThey hide a lot of os stuff from standard os tools. It's not Linux or Unix, it's Tim apple os.
- BlueTemplar 5y agoAFAIK GIFs for actual video content (rather than small animated (emot)icons) went obsolete in about 1995, when the first RealPlayer video plugin was released. Can someone explain to me why people are still using them for that 26 years later, especially now that we have much better formats like webm, mp4, or SVG, and will hopefully have AV1 hardware-supported in a few years ?
- andrewingram 5y agoFrom personal experience, it's because not everywhere I want to share a video inline supports proper video files, but they more-often-than-not support gifs. Github was once such example; but now that they support rendering videos in PRs, I've stopped uploading gifs.
- TazeTSchnitzel 5y agoIt's the only form of video which 1) browsers universally treat as an image file and which automatically, reliably and seamlessly loops and 2) also has wide support outside browsers and user familiarity.
- BlueTemplar 5y agoThe auto-start and auto-loop might or might not be wanted. The main issue is just how freaking inefficiently huge gifs are for this use case : This whole thread, but also I was seeing an order of magnitude difference in a recent use-case I had, compared to mp4, which would have made sending the document by e-mail otherwise not viable. (And imagine still living somewhere where a Mo of data doesn't have a negligible cost of transfer...)
- TazeTSchnitzel 5y agoI know that GIFs are very inefficient for what they're commonly used for, but your question was why people still use them. The fact they are “images” that are animated and loop is the whole point.
- DaniloDias 5y agoThat gif deserves an NFT.
- gnoll_of_gozag 5y agoanti-miner defense cross
- p0nce 5y agoBack when the DTK was available I've found that CIImage allocated with imageWithBitmapData was leaking, and moved to CGImageCreate and CGImage instead.
- selimnairb 5y agoThis makes me want to make good on a joke/threat to make a Linux distribution that is incapable of displaying animated GIFs. It will also block emojis.
- Our_Dream 5y agoSomebody tested it on rpi4? If not I will do it tomorrow.
- ok123456 5y agoThe Pillow library in Python detects for these sorts of attacks and just throws an error if an image tries to decode into something that's about 10x the size of a UHD image. You run into these foils if you start dealing with scientific data and have to override these protections.
- m463 5y ago...and here we are, stuck with limited soldered-down memory. :)
- cyanydeez 5y agoprobably an unplannrd early obscelence bug
- VictorDucs 5y agobecause M1 has memory leak https://www.apps-mac.com/issue-with-the-m1-mac-ssd/ https://www.apps-mac.com/issue-with-the-m1-mac-ssd/
- rvr_ 5y agoLooks like apple is using nvme as an infinity swap. This helps explain how 8GB feels enough and will probably be cause of future pain when those chips begin to wear out.