11 ms·
Where has my disk space gone? Flame graphs for file systems
- dewey 10y agoI usually always recommend https://dev.yorhel.nl/ncdu/scr https://dev.yorhel.nl/ncdu/scr for this purpose. Doesn't look as colorful but the output is basically the same. The SVG output is neat though!
- striking 10y agoAgreed. This is usually the first thing I try when attempting to reclaim disk space. It's incredible where you find a spare gigabyte or five.
- brendangregg 10y agoNice, but the output isn't the same. A flame graph can show multiple levels of subdirectories at the same time, proportionally sized to their total bytes, up to a maximum of the screen hight (say, 24). Looks like ncdu can only show 1 directory level at a time.
- vthriller 10y agoThe thing with flame graphs, pie charts and all that is that I rarely find it that useful to see multiple directory levels at the same time. Unless it is /var/log/omg-why-dont-you-rotate that occupies hundreds of gigabytes, I always end up navigating through the largest near-the-root folders with baobab just to make room for subdirectories on the screen, which is hardly different from the typical ncdu (or even 'du -hs *, then cd') workflow.
- rajachan 10y ago+1 to the ncdu recommendation. Of all the other tricks I've used to reclaim space, this is the most efficient. Found it when an admin of an HPC system I was using complained about my usage on the site-wide Lustre. Turns out, I had core dumps neatly tucked away in a project dir that I hadn't looked at in 3+ years.
- adambrenecki 10y agoThere's a bunch of other tools[1][2] that do this sort of thing already, although all the ones I've seen display file size as a sort of flame graph/pie chart hybrid (imagine a flame graph wrapped around a circle). This flat representation is probably better because it doesn't exaggerate the size of deeply nested files, but I find the example in the article a bit harder to read. [1]: https://daisydiskapp.com/ https://daisydiskapp.com/ [2]: http://www.jgoodies.com/freeware/jdiskreport/ http://www.jgoodies.com/freeware/jdiskreport/
- brendangregg 10y agoYou're right, the flame graphs (which is really an adjacency diagram with an inverted icicle layout) doesn't exaggerate subdirectories like a sunburst layout does. I wrote about problems with the sunburst layout before in ACMQ[1]: > The sunburst layout is equivalent to the icicle layout as used by flame graphs, but it uses polar coordinates.7 While this can generate interesting shapes, there are some difficulties: function names are harder to draw and read from sunburst slices than they are in the rectangular flame-graph boxes. Also, comparing two functions becomes a matter of comparing two angles rather than two line lengths, which has been evaluated as a more difficult perceptual task.10 (I should have mentioned that they visually exaggerate deeper slices, too). I think they are pretty, but, more difficult to read. The other app has a pie chart and trees. Both can't visually show everything at once, all subdirectories. [1] http://queue.acm.org/detail.cfm?id=2927301 http://queue.acm.org/detail.cfm?id=2927301
- masklinn 10y agoI agree with that criticism with respect to profiling, but for filesystems I find the increased surfaces of the outer rings makes the evaluation easier as the filesystem reaches smaller amounts of data, especially as FS tend to be relatively shallow in POI terms.
- dkersten 10y agoThe problem is that humans are not very good at comparing angles, which is one of the reasons why pie and donut charts aren't good (and sunburst is kinda like a donut, so the same criticism applies).
- jonny_eh 10y agoVery nice! On the Mac I recommend GrandPerspective: http://grandperspectiv.sourceforge.net/ http://grandperspectiv.sourceforge.net/
- wiiittttt 10y agoFor Windows: https://windirstat.net/ https://windirstat.net/
- hug 10y agoI prefer Wiztree (link below), especially on systems with much larger disks. Since Wiztree works by scanning the MFT it's about an order of magnitude faster than Windirstat or Treesize. http://antibody-software.com/web/software/software/wiztree-finds-the-files-and-folders-using-the-most-disk-space-on-your-hard-drive/ http://antibody-software.com/web/software/software/wiztree-f...
- kermire 10y agoHey, thanks for this. Was using WinDirStat. This is really fast! Plus I like the minimalistic interface.
- lsmod 10y agoThanks for the recommendation. faster than "space sniffer" as well.
- ptman 10y agoYES! I really wish there were others that scanned the MFT. I don't think Wiztree is as flexible with visualizations.
- IsmaOlvey 10y ago> Since Wiztree works by scanning the MFT it's > about an order of magnitude faster than [...] > Treesize. That's misleading - TreeSize also works by scanning the MFT.
- radiowave 10y ago
- ubercow 10y agoThis is beautiful. Great stuff. I've always had tools on desktops to do this, but never anything I could just run on a server.
- foxhop 10y agocd / && du -sh *
- thyselius 10y agoWhat does it do?
- foxhop 10y agoIt will list the size of every file and directory in human readable form. Output is related to your present working directory.
- deleted 10y ago[deleted]
- brendangregg 10y agoI'd try that first, and if it didn't find my space, then flame graphs. du's output does the first level of directories, but not subdirectories, and also requires reading of text rather than visually comparing line lengths (easier).
- foxhop 10y agoYou can quickly move toward the problem and continue to run 'du'.
- brendangregg 10y agoIf I'm at the command line already, that may well work. Or, it may not (the initial problem I had, I already knew the high level breakdowns, and was hunting for wasted 1%'s here and there -- the flame graph made it easy to spy everything at once). Another use of the flame graph approach is with automated build software. Imagine automatically generating one with every linux version, to keep track of where growth is.
- deleted 10y ago[deleted]
- foxhop 10y agocd / && du -sh
- chris_wot 10y agoA bit off topic, but Windows needs this baked in. And Windows needs hard links to update ALL links when the underlying file changes, because right now if you update a file from one hardlinked file the other hardlinked files don't change their metadata. Quite ridiculous, it's not like this hasn't been something Unix has been doing since almost the beginning! Right now we have a ridiculous situation where the winsxs folder gets out of sync with the c:\windows\system32 folder. Nothing treeview or any other utility can you daily do about it either. And until recently that winsxs store was holding gigabytes of old and useless updates, because Microsoft's updates never removes these components (recently - last year some time I think- they updated the disk cleanup GUI to delete this stuff from Windows 7 upwards. IMO they recognised a big stuff up caused by their product management team's decision, which in turn caused this unheralded enhancement).
- drinkjuice 10y agoAnother oldie but goldie for Windows: http://www.win.tue.nl/sequoiaview/ http://www.win.tue.nl/sequoiaview/
- throwawayish 10y agoKDE: Filelight, https://docs.kde.org/trunk5/en/kdeutils/filelight/introduction.html https://docs.kde.org/trunk5/en/kdeutils/filelight/introducti... Windows: WinDirStat
- janus24 10y agoMac: Disk Inventory X http://www.derlien.com/ http://www.derlien.com/
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- joe_developer 10y agoDownload link is broken. Alternative ways to download: a) Use brew: brew install Caskroom/cask/disk-inventory-x b) Another download link (same link used in brew/cask)[0]: http://www.derlien.com/diskinventoryx/downloads/dev/DIX1.0Universal.dmg http://www.derlien.com/diskinventoryx/downloads/dev/DIX1.0Un... 0: https://github.com/caskroom/homebrew-cask/blob/master/Casks/disk-inventory-x.rb https://github.com/caskroom/homebrew-cask/blob/master/Casks/...
- dageshi 10y agoWinDirStat helped me figure out what was using 25 gig on my c: last week, very useful tool.
- Nexxxeh 10y agoSeconded. My initial suspect was Dropbox. Windirstat helped me find it was some Trusted Installer-related bug filling the machine with corrupted cab files. http://www.infoworld.com/article/3112358/microsoft-windows/windows-7-log-file-compression-bug-can-fill-up-your-hard-drive.html http://www.infoworld.com/article/3112358/microsoft-windows/w...
- 10y ago
- dioltas 10y agoThis is really cool. I really like the cpu / profiling usage too for flame graphs. For disk usage I would normally use du and/or filelight which is also great. This is a nice way to visualise all the sub directories too in one go though.
- falsedan 10y agoWhen you have a hammer, you hammer out a reimplementation of `ncdu`. I'm a die-hard fan of `du -kx dir | sort -rn | less`.
- brendangregg 10y agodo they show multiple levels of subdirectories at the same time? no?
- falsedan 10y agoYou can try running it. yes
- brendangregg 10y agohttp://www.brendangregg.com/blog/images/2017/files_linux49.svg http://www.brendangregg.com/blog/images/2017/files_linux49.s... That's all the files and subdirectories, and multiple levels of subdirectories, shown _at the same time_. Without needing to navigate in and out of directories. If there's a way to do this using ncdu, then I've failed to find it. Screenshot please.
- brendangregg 10y agoBTW, here's my ncdu screenshot: --- /mnt/src/6/linux-4.9-rc5 --------------------- 401.7 MiB [##########] /drivers 135.6 MiB [### ] /arch 37.3 MiB [ ] /fs 35.3 MiB [ ] /include 34.4 MiB [ ] /Documentation 32.2 MiB [ ] /sound 27.6 MiB [ ] /net 13.4 MiB [ ] /tools 7.5 MiB [ ] /kernel 5.9 MiB [ ] /firmware 3.6 MiB [ ] /lib 3.4 MiB [ ] /scripts 3.3 MiB [ ] /mm 3.2 MiB [ ] /crypto 2.3 MiB [ ] /security 1.1 MiB [ ] /block 880.0 KiB [ ] /samples 424.0 KiB [ ] /virt [...] I don't see subdirectories. Linux source is here: https://www.kernel.org/ https://www.kernel.org/
- vacri 10y ago
- chronial 10y agoThis tool for windows is not known well enough: http://www.steffengerlach.de/freeware/ http://www.steffengerlach.de/freeware/ Screenshot: http://www.steffengerlach.de/freeware/scnshot.gif http://www.steffengerlach.de/freeware/scnshot.gif
- maxxxxx 10y agoOn Windows I like Treesize http://www.jam-software.de/treesize_free/?language=EN http://www.jam-software.de/treesize_free/?language=EN
- deleted 10y ago[deleted]
- m1el 10y agoI had this exact idea almost a year ago: https://github.com/m1el/du-flamegraph https://github.com/m1el/du-flamegraph find $1 -type f | xargs -n32 -d'\n' -- du -ab \ | awk -F'\t' '{print $2" "$1;}' | sed 's#/##;s#/#;#g' \ | flamegraph.pl --title "Disk usage" --countname "bytes" \ --nametype "file" --width 1900 --minwidth 0.05 ... and tested it using linux sources: http://m1el.github.io/du-flamegraph/linux-sources.svg http://m1el.github.io/du-flamegraph/linux-sources.svg http://m1el.github.io/du-flamegraph/illumos-gate.svg http://m1el.github.io/du-flamegraph/illumos-gate.svg http://m1el.github.io/du-flamegraph/dota2.svg http://m1el.github.io/du-flamegraph/dota2.svg
- brendangregg 10y agoI've done this before with awk, but with "find ... -ls". I don't know why you want to use du with xargs (is it really to get du's measure of bytes rather than the stat's count, for when they are different?) -- as that's calling over a thousand du's to walk the linux tree, when find can print bytes built-in with -ls (much more efficient).
- m1el 10y agoWell, this could be explained by my ignorance in *nix tools.
- surki 10y agoI do something like find $PWD -type f -ls | awk '{gsub("/", ";", $11); print $11" "$7}' | flamegraph.pl --title "Disk usage" --countname "bytes" --nametype "file" --width 1900 --minwidth 0.05
- jackyinger 10y agoFor Linux Baobab is a great utility like this. It does the circular equivalent of a flame though, which can be much easier to read as the circumference of deeper directories gets larger by nature.
- CarVac 10y agoAnd the name is amazing (a Little Prince reference).
- grzm 10y agoI'd love for this to be true. Do you have a reference for this? Baobab trees are spectacular in their own right. Might the usage in the Little Prince be just a coincidence?
- CarVac 10y agoSorry, that's just my assumption (though I'd be willing to bet on it). The circular graph reminds me of nothing more than the image of baobabs swallowing up a miniature planet.
- beefsack 10y agoFor a lot of people it might be installed already as part of their distro or Gnome software metapackages, it'll appear in your menus as "Disk Usage Analyzer."
- theoh 10y agoThere was a simple X application called "xdu" from the 90s that did this job adequately. At the other end of the practicality scale was the SGI tool fsn, recreated in open source by this: https://en.m.wikipedia.org/wiki/File_System_Visualizer https://en.m.wikipedia.org/wiki/File_System_Visualizer
- gjkood 10y agoI am partial to DaisyDisk on OSX. Nice UI. It comes in very handy on the lower capacity Flash Disks (64GB) on the early generation Macbook Airs. Needs constant housekeeping on those.
- jwatte 10y agoWinDirStat for the win!
- ajnin 10y agoA flame graph does not feel like the best visualization for this kind of data. You can't really make anything of the top spikes and there is a lot of blank space that is wasted. I prefer tree maps as they make more efficient use of the available screen space. My favorite tool for this is SpaceMonger v1.4.0 (http://www.aplusfreeware.com/categories/LFWV/SpaceMonger.html http://www.aplusfreeware.com/categories/LFWV/SpaceMonger.htm...), which has a very neat layout algorithm. It's a Windows app but it works OK using Wine with only a few minor graphic glitches.
- ryao 10y agoI use fsview for this on Linux. It works well, although it is slow when applied to the entire file system.
- jamesrom 10y agoOn Windows, WinDirStat is a great tool. https://windirstat.net/ https://windirstat.net/
- Dylan16807 10y agoI use it on linux too. I haven't seen anything that comes close in ability to visualize an entire drive.
- ghostly_s 10y agoI believe Windirstat is basically a direct port of K4dirstat, though I've found the outdated interface and KDE dependencies frustrating. I discovered Baobab has a similar box visualization available via an easy-to-miss icon at the bottom of the window, in addition to the default pretty-but-not-very-useful ring graph. Unfortunately both have the major shortcoming of not including individual large files.
- Dylan16807 10y agoIt is, but the rendering method is importantly different. Kdirstat decays into a low-contrast blob on a lot of files.
- partycoder 10y agoFor me a tree map is clearer and more screen-space efficient than a flamegraph. qcachegrind renders profiling output as tree maps, and for filesystems there are many alternatives, e.g: Baobab (aka Disk usage analyzer) does this, also KDirStat.
- Laforet 10y agoThere exists a 4 year old bug in Ubuntu LTS releases (not sure whether mainline Debian is affected too but it is very likely) in which kernel upgrades fail to remove old headers. Because the way headers are structured it is possible to run out of inodes long before free space is exhausted if you don't pay attention to inode use. https://bugs.launchpad.net/ubuntu/+source/update-manager/+bug/1089195 https://bugs.launchpad.net/ubuntu/+source/update-manager/+bu...
- jaimex2 10y agoI recommend Baobab, its been around for ages and is on pretty much every distro's repo already. Shows you a radial chart of your disk you can explore and zoom into and even better lets you delete directly from its ui.
- ghostly_s 10y agoConveniently, I no longer need to delve into any of these visualization tools, because the answer to the question in the title is always "Debian's overly-lax default logrotate.d"
- nobrains 10y agoIt's nice that you created something on your own. On Android I use DiskUsage: https://play.google.com/store/apps/details?id=com.google.android.diskusage https://play.google.com/store/apps/details?id=com.google.and... Screenshot: http://imgur.com/CrVcdEk http://imgur.com/CrVcdEk
- hrjet 10y agoInteresting that "drm" occupies almost all of the "drivers/gpu" space!
- Pengwin 10y agoWell in this case DRM means Direct Rendering Manager (https://en.wikipedia.org/wiki/Direct_Rendering_Manager https://en.wikipedia.org/wiki/Direct_Rendering_Manager) What I find even more interesting is AMD takes over half of that.
- brendangregg 10y agoAfter all the comments on treemaps and sunburst layouts, I think many readers might enjoy the following article if they haven't seen it already: A "Tour through the Visualization Zoo", by Jeffrey Heer, Michael Bostock, and Vadim Ogievetsky.[1] See the section on Hierarchies. In that article, the icicle graph (same as the flame graph) has long rectangles suitable for labeling -- although their example has the font rotated 90 degrees and not making the best use of that space. But as you can see, these are all related. Hierarchy visualizations. If I were serious about building a tool to do disk space visualizations, I'd let the user toggle between all three visualizations. I should also note that I think both treemaps and sunburst layout have their own cons, which I've mentioned here on HN. [1] http://queue.acm.org/detail.cfm?id=1805128 http://queue.acm.org/detail.cfm?id=1805128
- annnnd 10y agoNice - ideal use of flame graphs. When you are running out of disk space, you want to know which parts are using what proportion of disk space and IMHO this visualization is perfect for that. Kudos! On a side note, when I'm on a server with disk space problems, I usually debug it like this: # cd / # du -sm | sort -n | tail (lists the biggest space users) # cd <unusually big subdir> (goto step 2: du -sm...) Works like charm, but can be a bit slow when applied to large and slow disks.
- djaychela 10y agoOn Windows, I've always used HDGraph[1], but as no one has suggested it I'm wondering if I'm missing something important as to why not? I've never really liked the appearance of WinDirStat, but that's just a personal niggle, I know it works well. [1] http://www.hdgraph.com http://www.hdgraph.com
- beeswax 10y agoCan anyone recommend a tool for macOS that monitors disk usage changes over time? I usually use Disk Inventory X but I'd really like to correlate usage increases to specific dates / app installs, so it'd be nice to see stats over time, e.g: - Installed Android Studio on Feb 1st: Usage in /Applications increased by 850MB, usage in User folder increased by 10G (450MB for android-studio-2.x.dmg, 8.4GB in /Users/name/.android, largest leaf in /Users/name/.android/sdk etc) I tried to do this with the 'du' tools once but simply writing the current output to disk would take ages and diffs would need some heavy lifting to make sense of.
- Sukram21 10y agoFor visualizing your Dropbox' contents as a pie chart, a friend and I developed dgraph: https://github.com/joplapp/dgraph https://github.com/joplapp/dgraph
- 47_ 10y agowow that looks nice, I've always wondered why there isn't some tool like that
- akerro 10y agoKDE has this thing built-in, I think it's called kdirstat and you can get to it by clicking on directory size (bottom right corner) in Dolphin.
- pbhjpbhj 10y agoWow - I have an add-on to add "open with k4dirstat" to the right-click menu. Have used KDE5 since it came out and did not know about that. There is no affordance there at all; it doesn't even highlight on mouse-over. I get a menu with options for not-installed apps - filelight, kdiskfree and two partition tools. 'sudo apt install filelight' and it works, but TBH I'm not keen on filelight, would be nice to have k4dirstat as an option there (sounds like a bug report is due).
- emodendroket 10y agoI always liked WinDirStat.
- cogs 10y agoWe built Crab so we could run SQL queries over the filesystem. It's for Win and Mac, free for personal use, and lets you slice this anyway you want and plot the data how you like. I like the flexibility. http://etia.co.uk/win/about/ http://etia.co.uk/win/about/ e.g. SELECT sum(bytes) FROM files GROUP BY extension ORDER BY sum(bytes) DESC; You can GROUP BY any file attributes
- LeifCarrotson 10y agoThe height of the flame in this graph does not seem very informative. In the example, the 3.88% used by the tower of "linux-4.9-rc5/drivers/gpu/drm/amd/include" looks larger than "linux-4.9-rc5/net" over on the right just because the tree is deeper. Similarly, on my Windows machine at work, "C:\Users\LeifCarrotson\AppData\Local\Microsoft\Outlook\Leif@Carrotson.com.pst" (where my email is stored) would take up more screen space than "C:\hiberfil.sys", just because it's taller. I don't much care about the depth of the filesystem. The various tree views seem more useful.
- brendangregg 10y agoI posted a follow up: http://www.brendangregg.com/blog/2017-02-06/flamegraphs-vs-treemaps-vs-sunburst.html http://www.brendangregg.com/blog/2017-02-06/flamegraphs-vs-t...