3 ms·
I will tell about my most recent more significant hack. I was building a Yocto image and it failed, because it filled remaining space of my 128G disk. I run som
by hawski 4y ago
I will tell about my most recent more significant hack. I was building a Yocto image and it failed, because it filled remaining space of my 128G disk. I run some diagnostics. ncdu showed clearly that it is all Yocto builds. Then remembering how Yocto gets lots of the same files in the build process I was curious and run fdupes to find duplicates. I took a hacky python script that interprets fdupes output to calculate the size duplicates take, I wrote the script a few months earlier for unrelated issue. It showed around 36G of duplicates. From the same directory I took a script that replaces duplicates with hardlinks, small change to the mentioned script - I know that it is a common thing for duplicate software to do, but I wanted it to work on the fdupes output, so it would not have to recalculate anything for lots of time.
I run the build and eventually it failed. It failed, because Yocto keeps inode numbers in an SQLite file for an LD_PRELOAD library, that fakes root privileges or something like that. I did a simple SQLite utility in C before and I modified it so it would put correct inode numbers for all the files. After I made it to compile I was surprised that it just worked. I run a find with -exec of my tool for those database files.
Now the build mostly worked, but occasionally it failed. I don't remember exactly, but some of the hard linked files had to be replaced with a copy. I did not investigate further to make it automatic, but I considered checking the build log and doing the copying automatically or maybe to get to know how differentiate which files to leave alone. There was not a lot of files like that and after the build I did not have to do a big rebuild like that often enough to care.
It was not smooth, but I made it work. I wondered about creating a separate BTRFS partition (even in a file) that would do the dedup hidden or a fuse filesystem that would mask my own dedup.