3 ms·
HN: Help make Golang more (resource|env) friendly
- codesniperjoe 4y agoHi, just discovered that the (sometimes excessive large) golang mod disk cache is - in opposite to the compiler cache - not (yet) properly deduplicated. Try yourself: Made a little hack/tool to deduplicate data via hardlinks (unix/i-node-filesystems tested only, not windows yet). https://github.com/paepckehh/fsdd https://github.com/paepckehh/fsdd (less than 1kLOC, 100% golang stdlib, no external deps) go install paepcke.de/fsdd/cmd/fsdd@latest cd $GOMODCACHE && fsdd --hard-link . FSDD [start] [/usr/store/gomod][hash:MAPHASH] FSDD [_done] [time: 39.000221ms] FSDD [stats] [files:13329] [inode(s): 8680] [sym.valid: 0] [sym.invalid: 0] [data blocks: 277.1 Mbytes] FSDD [_info] [new deduplication savings][inode(s): 4649] [data blocks: 66.9 Mbytes] If you can reproduce and has it shows no downsides (remove/add should auto-dereference) it should be part of src/cmd/go/internal/mod* pack? If you can confirm/field test, please report back! I would love to write a quick stdlib/proposal. At golang scale, it could make a real resource / enviromental dent! It least for me, it cut my disk cache in half and reduced vfs fs mem-cache required as well, so should scale even on deduplicated fs like zfs/btrfs. Made my batch builds fast and snappy again! Michael (edit:first-post-ever-style-fixes)