3 ms·
I helped rewrite Reposurgeon. It was originally written in Python, and is now written in Go. In the process we shed all of the overhead from the Python interpre
by db48x 6y ago
I helped rewrite Reposurgeon. It was originally written in Python, and is now written in Go. In the process we shed all of the overhead from the Python interpreter, which sped the program up significantly. I made further improvements that cut the memory usage of the Go version in half (from >256GB to <128GB of ram for the task of converting the GCC repository from SVN to Git), and dropped the run time to about a quarter of what it was.
Now that I've learned Go, it wouldn't be my first choice. Around 70% of CPU time is currently spent in the heap scanner, looking for garbage to collect. Initial experiments with Rust show a marked decrease in memory usage and run time, and that's without any really tricky optimizations that Rust could make quite easy to do. Just the storage space used by all of the filenames in the GCC repository is a huge amount of ram, and in Rust a string interning system is quite easy to write. (In Go it would be impossible, though perhaps now that generics are on the horizon that will start to change.) All I need to finish the task is about 4–6 months of free time… or a grant, now that I think of it.
- PragmaticPulp 6y ago> Now that I've learned Go, it wouldn't be my first choice. Funny you mention that. I worked at a company that rewrote a project in Go when it was the hot language. Go helped us solve the problems from the old language, but it introduced a new set of problems. Later, someone suggested we re-rewrite the project again in Rust. Obviously, that didn't go anywhere. > All I need to finish the task is about 4–6 months of free time… 4-6 months of programmer time is not cheap. You can buy a lot of faster hardware for the price of tying up a single programmer for half a year.
- db48x 6y agoThat is frequently true. However, when it is your customers buying the hardware then it's a bit rude to be wasteful. On the other hand, the repository conversion that GCC did was an extreme case. Their SVN repository had 287k commits in it, and I've only heard tales of larger ones.