9 ms·

Sensenmann: Code Deletion at Scale

308 points by gslin 4y ago

DamonHD 4y ago
I worry about archival and enough history for diagnosing long-standing subtle issues that take a long time to surface as bugs. This is not theoretical: apparently a TeX bug picked up after many years had been there from the start.
- kragen 4y ago
  i think piper saves the full history of the whole monorepo; if that's correct it's not 'deletion' in that sense
  - gravypod 4y ago
    (opinions are my own) > Its goal is simple (at least, in principle): automatically identify dead code, and send code review requests ('changelists') to delete it. It sends CLs (pull requests) and shows up as a commit. You get a chance to approve or deny the deletion
    - kragen 4y ago
      yeah but even if you approve it the code is still there in the code history, right?
      - bradfitz 4y ago
        Yes.
        DamonHD 4y ago
        Good. But relatively hard to find and work with.
        tonfa 4y ago
        Still fairly easy to find IMO. You can search deleted code easily, and blame layers allows finding how code evolved fairly quickly.
        larsrc 4y ago
        I still would like a better way to follow a line back through history. But what we have is already pretty good.
        kevinoconnor7 4y ago
        Not really. If you're in the very rare situation where you need to diagnose a bug in long-since dead code, you can just view repository synced to where the version was cut.
        pradn 4y ago
        No you just go to the folder and select “show deleted”.
        dmoy 4y ago
        Don't even need to do that. Codesearch with a `from:0` qualifier does full regex search on the history iirc
        dekhn 4y ago
        I used to do archeaology on Google's monorepo (IE, looking far into the past of mapreduce, search engine, ads, and other products) and it wasn't really that hard. Heck, there was even a sythetic filesystem where you could just cd to a historical commit # and see a view of the repo at that timepoint (google's version control is based on an always-increasing globally shared commit numbers).
        eru 4y ago
        I wrote a similar fuse based file system to do the same thing for git. It was quite pleasant to write, because both the guy who designed git, Linus Torvalds, is also a file system guy.
        kragen 4y ago
        thank you
croes 4y ago
FYI: Sensenmann is the german word for Grim Reaper
- Zetobal 4y ago
  I still don't get the tech industries fascination with random german words at least here it's sort of fitting.
  - meibo 4y ago
    Looks like this was made by a team in Zürich, which is mostly German, so I imagine it came to them fairly naturally, and who doesn't want to pick cool names for hackathon projects.
    - evmar 4y ago
      It was kind of an internal joke at Google for German-speaking teams to make German-named projects. (It's maybe only a joke that makes sense to the infamous German sense of humor.)
      - unlikelymordant 4y ago
        The german sense of humour is no laughing matter!
      - munificent 4y ago
        Q: How many Germans does it take to change a lightbulb? A: One. They are very efficient and don't have much sense of humor.
  - aardvarkr 4y ago
    Citation needed ^ Maybe there are just some great German devs and you’re using a lot of their software?
    - mflendrich 4y ago
falcor84 4y ago
>For example, if an engineer is unsure how to use a library, they can find examples just by searching Isn't that the case with all libraries? How does the monorepo help here?
- er4hn 4y ago
  Discoverability. It's a lot easier to search one repo then it is to search a set of repos. For the latter you need to have all the repos listed somewhere, and have them be accessible.
  - knutzui 4y ago
    It's just as easy to index multiple repos as it is to index one, which means that the same goes for searching. Why would it be any different for one as opposed to many?
    - codetrotter 4y ago
      We use GitLab in the company I work for. There may be repositories created by others in the company, that depend on repos I work on, where I don’t have access to said other repos. So to me these are invisible. If everything was in one monorepo, it’d all be visible to me easily.
      - Kwpolska 4y ago
        That’s more of a culture thing. Your company chose to enforce more granular permissions. Perhaps there are good reasons behind it, e.g. code for different clients being under different contracts and NDAs.
      - jkrems 4y ago
        That's a question of policy, not of repo structure. A monorepo can still have certain parts of the repo protected by access control. One repo doesn't mean "all files share one read permission".
  - 4y ago
lifeisstillgood 4y ago
There is a meta-meta situation surrounding really good software management. You can knock up some code that say solves a specific business problem right now. (meta:0) But you need an environment that can take a new piece of code and deploy it and test it (meta:1) how is that code running - this is shadingnfrom production monitoringninto QA and performance (meta:2) Compare all the running code and its performance against the benefits of replacing code or going back to level 0 and just fixing a business problem (meta:3) Then this death eater - meta 4 I think. And to me this is why comments like "software needs to solve business problems" is naive - once you start using software you need more software to manage the software - it's going to grow till it consumes the business.
- groestl 4y ago
  > once you start using software you need more software to manage the software - it's going to grow till it consumes the business. You can replace "software" with "people" and you'll end up with a sentence that's equally true.
- valenterry 4y ago
  True. Which is why it's important to not measure the ROI of a feature by the time and resources you spend on it vs. how it increases your revenue. You also have to include that every further feature is now _more_ expensive than before.
jawns 4y ago
The most difficult part about code deletion is practicing the Chesterton's Fence principle: > In the matter of reforming things, as distinct from deforming them, there is one plain and simple principle; a principle which will probably be called a paradox. There exists in such a case a certain institution or law; let us say, for the sake of simplicity, a fence or gate erected across a road. The more modern type of reformer goes gaily up to it and says, “I don’t see the use of this; let us clear it away.” To which the more intelligent type of reformer will do well to answer: “If you don’t see the use of it, I certainly won’t let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it. https://wiki.lesswrong.com/wiki/Chesterton%27s_Fence https://wiki.lesswrong.com/wiki/Chesterton%27s_Fence While this tool certainly does the job of proposing code deletions, that's the easier part. The harder part is knowing why the code exists in the first place, which is necessary to know whether it's truly a good idea to remove it. Google, smartly, is leaving that part up to a human (for now).
- breck 4y ago
  As a counter to Chesterton's Fence: sometimes the fastest way to understand what something does is to remove it and see what complains. You might get only 1 complainer for every 10 fences you take down. Putting that one fence back up takes much longer than taking it down, but the time saved from removing the other 9 unnecessary ones makes it a net win. And this time you can add Documentation to the rebuilt fence.
  - IshKebab 4y ago
    A further counterpoint: if you follow the Fence proponents' logic to its conclusion you can never remove any code which is clearly an absurd situation. I think the real logical flaw is that Fencers (as I will now call them) put the blame on the person who removes an apparently useless fence. But they're wrong. The real blame lies with the person who built the apparently useless fence and didn't put a sign on it explaining why it shouldn't be removed.
    - hgsgm 4y ago
btilly 4y ago
I liked most of the blog, but it bothers me to see stuff like, "... just as with the introduction of unit testing 20 years ago...". No, unit testing was NOT introduced 20 years ago. As an example, Perl 1 was released about 35 years ago with a unit test suite that got run on every install. Every version of Perl has done so since, and since CPAN came along, most Perl modules have followed suit. This was the secret sauce behind Perl's reputation for being so portable. Nor was Perl a pioneer. In fact unit testing was used in the 1960s on the Apollo program, and was even called unit testing. I believe that the concept can be dated back to a 1950s textbook but I can't find the reference. So unit testing is over 60 years old.
- allanrbo 4y ago
  Maybe they just mean that Google started investing seriously in unit testing 20 years ago?
- UncleMeat 4y ago
  This refers to Google, I'm pretty sure. Early Google didn't believe in unit testing and it took a few particularly stubborn engineers to demonstrate the value of the practice and convert the culture to promoting unit testing.
mkoubaa 4y ago
Sounds like a useful system from which almost nothing is usable outside of Google.
- vamega 4y ago
  Sure; but working at another very large company, I can say I wish we had this. Old unused code is a huge problem for us. The coordination costs of trying to update company wide problems are made much more severe by old code. I wish we had something like this. We’re large enough we’d need our own system anyway. We don’t have a monorepo, and we don’t use tools so many others do.
- habitue 4y ago
  Sure, but this kinda tool only works because of their monorepo, build system, deployment run tracking etc. Let's say they open sourced it, you'd have to basically adopt the entire google stack to make use of it
jmyeet 4y ago
The most important part of this is that the build units are hermetic and all dependencies are explicit. This is why you need to use something like Bazel/Blaze vs older build systems like make where identifying what's used, particularly when you get into meta-rules, becomes all but impossible. As the article points out, you also have to look at what's actually run. This is the real advantage of Google infrastructure: the vertical integration so if a binary is run on Borg, or even on the command line, that can be tracked.
deleted 4y ago
[deleted]
joebiden2 4y ago
Sincere quesion: what is interesting or novel about this? Is it just the scale or did I miss some subtle aspect? This is more (or less?) the same as industry best practices, just scaled up. There is a challenge in scaling up, as there is more potential for someone to mess it up. But it's the same technique. So what am I missing?
- codemac 4y ago
  > the same as industry best practices, just scaled up. That's like saying S3 is the same as ext4, their the same, just scaled up! This is a poor argument, you'll note that S3 and ext4 are entirely different things, not "challenges", fundamentally different implementations. Google is the only company I've ever worked for that automatically deleted dead code, let alone across a company of 100k+ SWE.
  - joebiden2 4y ago
    Fair enough and thanks for the reply. Still, for anything bothering engineers more than a bit repeatedly, anyone will write tools to remove the manual burden. Our internal practice is to delete code if you suspect it's unused, run tests, and if it doesn't affect any tests, go for it. This could be automated, but it is not pressing enough, so we didn't automate it yet. We could though, and it may even be a good idea, but I still don't get the novelty. But I appreciate your point of view.
    - brunooliv 4y ago
      Not to criticize your POV and argument directly, but, in the end, a lot of things, especially like these, are always easily subjected to the "we could do it, but just didn't bother to yet" kind of argument, and, when it comes down to the real work, things are much harder than they superficially appear to be. So yeah this isn't new... But... You know eheh
      - joebiden2 4y ago
        Well, I'd politely agree to disagree. Google scale is defined by novel, radical approaches, like for example inventing map/reduce, writing papers on LLMs others then implement successfully, or creating something like Kubernetes. The specific topic here is not one of those google problems to me, as I can compare it to other problems we already solved. But yes, we could miss that critical point where a totally different problem domain emerges just from one order of magnitude more, so fair game.
charcircuit 4y ago
Does this only get rid of unused binaries? Or is the system smart enough to use the profiling infrastructure to identify dead code in general?
- Jolter 4y ago
  Just read the article. It’s not that long.
  - charcircuit 4y ago
    I was hoping to spark a conversation about this approach as no such thing was mentioned in the article even though it should be possible to do. The whole point of these comment sections is to have a discussion. If the point of this site was just to read articles there wouldn't be a comment section.
    - Jolter 4y ago
      I think your question was phrased in a way that suggested you had not understood the article. The described system detects unused libraries and binaries, but does not seem to look for dead code paths in source files. Dead code path detection is an interesting topic which I think the linked article doesn’t address at all.
- kccqzy 4y ago
  Profiling is inherently probabilistic and I don't think it should be used. Anything that inspects the runtime (dynamic) behavior of code isn't good enough for a code deletion tool. Only static analysis will do.
  - charcircuit 4y ago
    >Profiling is inherently probabilistic And with enough samples we can learn what code paths are being taken. You can have your dead code bot look at the number of samples that have been collected to know if it should be enabled for something. For example if you have 6 mounths worth of samples of a piece of software running on 10k machines you will get a good idea on what code isn't being used. >Anything that inspects the runtime (dynamic) behavior of code isn't good enough for a code deletion tool. I disagree since the code that is being run is by definitions not dead code.
sitkack 4y ago
I hope everyone involved gets L+1!
oneplane 4y ago
On a non-google level just being aware of code sitting around costing resources is pretty important. Often, tests and maintenance are just ignored or not calculated in as cost (be it time, money, effort or otherwise). It is almost in the same realm as "I don't know why it works" which is as dangerous as "I don't know why it doesn't work".
- clintonb 4y ago
  The costs at a non-Google level are relatively low, and probably worth ignoring.
quickthrower2 4y ago
Code deletion is much easier when you have a typed language with no meta programming. I guess Go and C code is easier to tree shake like that, than say Python or JS.
- blauditore 4y ago
  C is certainly not very strongly typed.
  - jaggederest 4y ago
    Oh come on, it's extremely strongly typed, everything is made up entirely of bits.
    - syngrog66 4y ago
      and pointers! and address offsets! C was my main/daily lang by age 18 or 19 and I got drunk on its power
      - jaggederest 4y ago
        Those are just suspicious bits. Bits in the shape of an integer, but only knowed and understood by the compiler. And the preprocessor... dark arts indeed, manipulating the bits of the program itself!
  - saghm 4y ago
    Not sure if the parent comment was edited or not, but at the time I'm reading it, it just says "typed", not "strongly typed". Strong versus weak typing isn't something I've seen a consistent definition of that everyone agrees on (although most definitions I've seen would consider C to be somewhat weakly typed because generally almost any type can be coerced to any other type without much ability to tell if the result is valid or not), but statically versus dynamically typed is pretty well defined, and C certainly is a statically typed language.
    - 4y ago
didip 4y ago
I am guessing it would be very hard to do this on a dynamic language.
deleted 4y ago
[deleted]
- deleted 4y ago
  [deleted]
kgeist 4y ago
The codebase of an internal system our team inherited is pretty old (14 years) and large, and there's a feeling a lot of the stuff is unused (dead code or unused ancient features) and can be deleted. Fortunately, it's a monolith and all that matters is HTTP endpoints defined in a few config files. So we've made a script which downloads all access logs from the live server for the last 3 months and tries to find endpoints with zero or very low usage (it did find quite a few unused or rarely used endpoints). I'm planning the second step to be a script which builds a dependency graph and finds everything which is only reachable from unused endpoints. I wanted it to be a one-time endeavor but this blog post made me think it should be run on a regular basis.
- jiggawatts 4y ago
  I wonder if APM performance traces could be used to determine unused code paths. A lot of these tools sample stack traces at intervals, collecting statistics of how often functions are used. If you had enough samples, then with very high confidence you could find code paths that are never used in production.
  - emptysea 4y ago
    Another option is tail based sampling which allows you to make the choice at the end of the request whether to keep the trace. With that you could configure the APM to keep the first $N traces for all endpoints and then start sampling after that
  - chrnola 4y ago
    That’s an interesting idea! Reminds me of how Pyroscope’s continuous profiling data can be linked to OpenTelemetry traces[1]. [1]: https://pyroscope.io/docs/java-tracing/ https://pyroscope.io/docs/java-tracing/
- kortilla 4y ago
  Be very careful using this methodology. This is how you end up deleting stuff only called in rare but critical cases: - during outage - at end of year - during audits - when the one special customer that paid a fortune for an obscure feature decides to use it It might be the case that none of this applies to your application, but it’s why “check what got hit recently” (3 months is an eye blink in the business and govt world) is dangerous.
pabs3 4y ago
They should also look at deleting third_party directories and replacing them with package manager dependencies. For example Qt WebEngine embeds Chromium, which embeds Blink, which embeds wpt, which embeds a bunch of Python libraries and one of those Python libraries (six) is embedded in three other locations within Chromium. https://sources.debian.org/src/qtwebengine-opensource-src/5.15.2+dfsg-3/src/3rdparty/chromium/third_party/blink/tools/blinkpy/third_party/wpt/wpt/tools/third_party/ https://sources.debian.org/src/qtwebengine-opensource-src/5.... https://codesearch.debian.net/search?q=package%3Aqtwebengine-opensource-src+Useful+for+very+coarse+version+differentiation https://codesearch.debian.net/search?q=package%3Aqtwebengine.... I wonder how many copies of the Python six module the Google monorepo contains.
- raverbashing 4y ago
  And while six is 'kinda' well behaved, trying to unify dependencies of dependencies is an exercise in frustration usually. Code repetition is cheap
  - pabs3 4y ago
    Code repetition is only cheap if you abandon and forget about the repeated code. Any time you need to have to care about the code to fix bugs, security issues or improve performance, it is expensive do each of those things once, repeating them multiple times can mean an order of magnitude difference in the worse case.
- habosa 4y ago
  I’d assume one. The “one version policy” is one of the strictest codebase rules at Google. But the Chrome codebase is generally the giant exception to everything due to the major open source components (Android too).
- secondcoming 4y ago
  This is a good plan but I've often found that with for example, Debian, the official packages are well behind the times compared to the direct third party code. Ubuntu is not too bad.
daxfohl 4y ago
"deleting code" should have been included in the "two hardest things in computer science".
valenterry 4y ago
Upvoted for the nice name :)
aktuel 4y ago
The Devel's in the Details: typo or pun?
terom 4y ago
It's about time someone gets a KPI for lines of code removed :)
meindnoch 4y ago
It's not that simple. A lot of dead code doesn't look dead to static analysis. E.g. gated by long-forgotten feature flags, or platform checks that are no longer relevant.
- GeneralMayhem 4y ago
  This is only the simplest possible reaper tool. You can (and Google does) also write tools to find flags that have only ever been set to one value, codepaths that haven't been exercised in months, databases that haven't been accessed in years, etc.
izwasm 4y ago
Looks like graph algorithms comes in handy at the end
whirlwin 4y ago
Vert interesting. At our company we have a spring cleaning event next week, and one of the goals is to delete old code. Some of the comments mention some good techniques. Anyone experienced with deep analysis, e.g. BPF on this field?
specialist 4y ago
Conducting an inventory (of services) would be cool too. If a chunk of code isn't actually deployed somewhere, mark it as a candidate for culling. Probably requires some kind of metadata provenance for deployed artifacts. For Java, I thought Maven had a stock manifest.mf entry for the source repo. Alas, a quick search only reveals that the archiver plugin has an entry for the project's "url". https://maven.apache.org/shared-archives/maven-archiver-2.5/examples/manifestEntries.html https://maven.apache.org/shared-archives/maven-archiver-2.5/... Which for in-house projects is probably sufficient.
amai 4y ago
[flagged]
elesbao 4y ago
I'm generally not fond of monorepos but reading this and right away being targeted by this tweet https://twitter.com/nikitabier/status/1652764613962760196 https://twitter.com/nikitabier/status/1652764613962760196 made me think on how much time goes on decisions that won't impact users directly and how hard it becomes to justify that on regular companies in an environment where CEOs are jumping the layoff bandwagon for no reason.

308 points by gslin 4y ago

I worry about archival and enough history for diagnosing long-standing subtle issues that take a long time to surface as bugs. This is not theoretical: apparently a TeX bug picked up after many years had been there from the start.

kragen 4y ago
i think piper saves the full history of the whole monorepo; if that's correct it's not 'deletion' in that sense
- gravypod 4y ago
  (opinions are my own) > Its goal is simple (at least, in principle): automatically identify dead code, and send code review requests ('changelists') to delete it. It sends CLs (pull requests) and shows up as a commit. You get a chance to approve or deny the deletion
  - kragen 4y ago
    yeah but even if you approve it the code is still there in the code history, right?
    - bradfitz 4y ago
      Yes.
      - DamonHD 4y ago
        Good. But relatively hard to find and work with.
        tonfa 4y ago
        Still fairly easy to find IMO. You can search deleted code easily, and blame layers allows finding how code evolved fairly quickly.
        larsrc 4y ago
        I still would like a better way to follow a line back through history. But what we have is already pretty good.
        kevinoconnor7 4y ago
        Not really. If you're in the very rare situation where you need to diagnose a bug in long-since dead code, you can just view repository synced to where the version was cut.
        pradn 4y ago
        No you just go to the folder and select “show deleted”.
        dmoy 4y ago
        Don't even need to do that. Codesearch with a `from:0` qualifier does full regex search on the history iirc
        dekhn 4y ago
        I used to do archeaology on Google's monorepo (IE, looking far into the past of mapreduce, search engine, ads, and other products) and it wasn't really that hard. Heck, there was even a sythetic filesystem where you could just cd to a historical commit # and see a view of the repo at that timepoint (google's version control is based on an always-increasing globally shared commit numbers).
        eru 4y ago
        I wrote a similar fuse based file system to do the same thing for git. It was quite pleasant to write, because both the guy who designed git, Linus Torvalds, is also a file system guy.
      - kragen 4y ago
        thank you

croes 4y ago

FYI: Sensenmann is the german word for Grim Reaper

Zetobal 4y ago
I still don't get the tech industries fascination with random german words at least here it's sort of fitting.
- meibo 4y ago
  Looks like this was made by a team in Zürich, which is mostly German, so I imagine it came to them fairly naturally, and who doesn't want to pick cool names for hackathon projects.
  - evmar 4y ago
    It was kind of an internal joke at Google for German-speaking teams to make German-named projects. (It's maybe only a joke that makes sense to the infamous German sense of humor.)
    - unlikelymordant 4y ago
      The german sense of humour is no laughing matter!
    - munificent 4y ago
      Q: How many Germans does it take to change a lightbulb? A: One. They are very efficient and don't have much sense of humor.
- aardvarkr 4y ago
  Citation needed ^ Maybe there are just some great German devs and you’re using a lot of their software?
  - mflendrich 4y ago

falcor84 4y ago

>For example, if an engineer is unsure how to use a library, they can find examples just by searching Isn't that the case with all libraries? How does the monorepo help here?

er4hn 4y ago
Discoverability. It's a lot easier to search one repo then it is to search a set of repos. For the latter you need to have all the repos listed somewhere, and have them be accessible.
- knutzui 4y ago
  It's just as easy to index multiple repos as it is to index one, which means that the same goes for searching. Why would it be any different for one as opposed to many?
  - codetrotter 4y ago
    We use GitLab in the company I work for. There may be repositories created by others in the company, that depend on repos I work on, where I don’t have access to said other repos. So to me these are invisible. If everything was in one monorepo, it’d all be visible to me easily.
    - Kwpolska 4y ago
      That’s more of a culture thing. Your company chose to enforce more granular permissions. Perhaps there are good reasons behind it, e.g. code for different clients being under different contracts and NDAs.
    - jkrems 4y ago
      That's a question of policy, not of repo structure. A monorepo can still have certain parts of the repo protected by access control. One repo doesn't mean "all files share one read permission".
- 4y ago

lifeisstillgood 4y ago

There is a meta-meta situation surrounding really good software management. You can knock up some code that say solves a specific business problem right now. (meta:0) But you need an environment that can take a new piece of code and deploy it and test it (meta:1) how is that code running - this is shadingnfrom production monitoringninto QA and performance (meta:2) Compare all the running code and its performance against the benefits of replacing code or going back to level 0 and just fixing a business problem (meta:3) Then this death eater - meta 4 I think. And to me this is why comments like "software needs to solve business problems" is naive - once you start using software you need more software to manage the software - it's going to grow till it consumes the business.

groestl 4y ago
> once you start using software you need more software to manage the software - it's going to grow till it consumes the business. You can replace "software" with "people" and you'll end up with a sentence that's equally true.
valenterry 4y ago
True. Which is why it's important to not measure the ROI of a feature by the time and resources you spend on it vs. how it increases your revenue. You also have to include that every further feature is now _more_ expensive than before.

jawns 4y ago

The most difficult part about code deletion is practicing the Chesterton's Fence principle: > In the matter of reforming things, as distinct from deforming them, there is one plain and simple principle; a principle which will probably be called a paradox. There exists in such a case a certain institution or law; let us say, for the sake of simplicity, a fence or gate erected across a road. The more modern type of reformer goes gaily up to it and says, “I don’t see the use of this; let us clear it away.” To which the more intelligent type of reformer will do well to answer: “If you don’t see the use of it, I certainly won’t let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it. https://wiki.lesswrong.com/wiki/Chesterton%27s_Fence https://wiki.lesswrong.com/wiki/Chesterton%27s_Fence While this tool certainly does the job of proposing code deletions, that's the easier part. The harder part is knowing why the code exists in the first place, which is necessary to know whether it's truly a good idea to remove it. Google, smartly, is leaving that part up to a human (for now).

breck 4y ago
As a counter to Chesterton's Fence: sometimes the fastest way to understand what something does is to remove it and see what complains. You might get only 1 complainer for every 10 fences you take down. Putting that one fence back up takes much longer than taking it down, but the time saved from removing the other 9 unnecessary ones makes it a net win. And this time you can add Documentation to the rebuilt fence.
- IshKebab 4y ago
  A further counterpoint: if you follow the Fence proponents' logic to its conclusion you can never remove any code which is clearly an absurd situation. I think the real logical flaw is that Fencers (as I will now call them) put the blame on the person who removes an apparently useless fence. But they're wrong. The real blame lies with the person who built the apparently useless fence and didn't put a sign on it explaining why it shouldn't be removed.
  - hgsgm 4y ago

btilly 4y ago

I liked most of the blog, but it bothers me to see stuff like, "... just as with the introduction of unit testing 20 years ago...". No, unit testing was NOT introduced 20 years ago. As an example, Perl 1 was released about 35 years ago with a unit test suite that got run on every install. Every version of Perl has done so since, and since CPAN came along, most Perl modules have followed suit. This was the secret sauce behind Perl's reputation for being so portable. Nor was Perl a pioneer. In fact unit testing was used in the 1960s on the Apollo program, and was even called unit testing. I believe that the concept can be dated back to a 1950s textbook but I can't find the reference. So unit testing is over 60 years old.

allanrbo 4y ago
Maybe they just mean that Google started investing seriously in unit testing 20 years ago?
UncleMeat 4y ago
This refers to Google, I'm pretty sure. Early Google didn't believe in unit testing and it took a few particularly stubborn engineers to demonstrate the value of the practice and convert the culture to promoting unit testing.

mkoubaa 4y ago

Sounds like a useful system from which almost nothing is usable outside of Google.

vamega 4y ago
Sure; but working at another very large company, I can say I wish we had this. Old unused code is a huge problem for us. The coordination costs of trying to update company wide problems are made much more severe by old code. I wish we had something like this. We’re large enough we’d need our own system anyway. We don’t have a monorepo, and we don’t use tools so many others do.
habitue 4y ago
Sure, but this kinda tool only works because of their monorepo, build system, deployment run tracking etc. Let's say they open sourced it, you'd have to basically adopt the entire google stack to make use of it

jmyeet 4y ago

The most important part of this is that the build units are hermetic and all dependencies are explicit. This is why you need to use something like Bazel/Blaze vs older build systems like make where identifying what's used, particularly when you get into meta-rules, becomes all but impossible. As the article points out, you also have to look at what's actually run. This is the real advantage of Google infrastructure: the vertical integration so if a binary is run on Borg, or even on the command line, that can be tracked.

deleted 4y ago

[deleted]

joebiden2 4y ago

Sincere quesion: what is interesting or novel about this? Is it just the scale or did I miss some subtle aspect? This is more (or less?) the same as industry best practices, just scaled up. There is a challenge in scaling up, as there is more potential for someone to mess it up. But it's the same technique. So what am I missing?

codemac 4y ago
> the same as industry best practices, just scaled up. That's like saying S3 is the same as ext4, their the same, just scaled up! This is a poor argument, you'll note that S3 and ext4 are entirely different things, not "challenges", fundamentally different implementations. Google is the only company I've ever worked for that automatically deleted dead code, let alone across a company of 100k+ SWE.
- joebiden2 4y ago
  Fair enough and thanks for the reply. Still, for anything bothering engineers more than a bit repeatedly, anyone will write tools to remove the manual burden. Our internal practice is to delete code if you suspect it's unused, run tests, and if it doesn't affect any tests, go for it. This could be automated, but it is not pressing enough, so we didn't automate it yet. We could though, and it may even be a good idea, but I still don't get the novelty. But I appreciate your point of view.
  - brunooliv 4y ago
    Not to criticize your POV and argument directly, but, in the end, a lot of things, especially like these, are always easily subjected to the "we could do it, but just didn't bother to yet" kind of argument, and, when it comes down to the real work, things are much harder than they superficially appear to be. So yeah this isn't new... But... You know eheh
    - joebiden2 4y ago
      Well, I'd politely agree to disagree. Google scale is defined by novel, radical approaches, like for example inventing map/reduce, writing papers on LLMs others then implement successfully, or creating something like Kubernetes. The specific topic here is not one of those google problems to me, as I can compare it to other problems we already solved. But yes, we could miss that critical point where a totally different problem domain emerges just from one order of magnitude more, so fair game.

charcircuit 4y ago

Does this only get rid of unused binaries? Or is the system smart enough to use the profiling infrastructure to identify dead code in general?

Jolter 4y ago
Just read the article. It’s not that long.
- charcircuit 4y ago
  I was hoping to spark a conversation about this approach as no such thing was mentioned in the article even though it should be possible to do. The whole point of these comment sections is to have a discussion. If the point of this site was just to read articles there wouldn't be a comment section.
  - Jolter 4y ago
    I think your question was phrased in a way that suggested you had not understood the article. The described system detects unused libraries and binaries, but does not seem to look for dead code paths in source files. Dead code path detection is an interesting topic which I think the linked article doesn’t address at all.
kccqzy 4y ago
Profiling is inherently probabilistic and I don't think it should be used. Anything that inspects the runtime (dynamic) behavior of code isn't good enough for a code deletion tool. Only static analysis will do.
- charcircuit 4y ago
  >Profiling is inherently probabilistic And with enough samples we can learn what code paths are being taken. You can have your dead code bot look at the number of samples that have been collected to know if it should be enabled for something. For example if you have 6 mounths worth of samples of a piece of software running on 10k machines you will get a good idea on what code isn't being used. >Anything that inspects the runtime (dynamic) behavior of code isn't good enough for a code deletion tool. I disagree since the code that is being run is by definitions not dead code.

sitkack 4y ago

I hope everyone involved gets L+1!

oneplane 4y ago

On a non-google level just being aware of code sitting around costing resources is pretty important. Often, tests and maintenance are just ignored or not calculated in as cost (be it time, money, effort or otherwise). It is almost in the same realm as "I don't know why it works" which is as dangerous as "I don't know why it doesn't work".

clintonb 4y ago
The costs at a non-Google level are relatively low, and probably worth ignoring.

quickthrower2 4y ago

Code deletion is much easier when you have a typed language with no meta programming. I guess Go and C code is easier to tree shake like that, than say Python or JS.

blauditore 4y ago
C is certainly not very strongly typed.
- jaggederest 4y ago
  Oh come on, it's extremely strongly typed, everything is made up entirely of bits.
  - syngrog66 4y ago
    and pointers! and address offsets! C was my main/daily lang by age 18 or 19 and I got drunk on its power
    - jaggederest 4y ago
      Those are just suspicious bits. Bits in the shape of an integer, but only knowed and understood by the compiler. And the preprocessor... dark arts indeed, manipulating the bits of the program itself!
- saghm 4y ago
  Not sure if the parent comment was edited or not, but at the time I'm reading it, it just says "typed", not "strongly typed". Strong versus weak typing isn't something I've seen a consistent definition of that everyone agrees on (although most definitions I've seen would consider C to be somewhat weakly typed because generally almost any type can be coerced to any other type without much ability to tell if the result is valid or not), but statically versus dynamically typed is pretty well defined, and C certainly is a statically typed language.
  - 4y ago

didip 4y ago

I am guessing it would be very hard to do this on a dynamic language.

deleted 4y ago

[deleted]

deleted 4y ago
[deleted]

kgeist 4y ago

The codebase of an internal system our team inherited is pretty old (14 years) and large, and there's a feeling a lot of the stuff is unused (dead code or unused ancient features) and can be deleted. Fortunately, it's a monolith and all that matters is HTTP endpoints defined in a few config files. So we've made a script which downloads all access logs from the live server for the last 3 months and tries to find endpoints with zero or very low usage (it did find quite a few unused or rarely used endpoints). I'm planning the second step to be a script which builds a dependency graph and finds everything which is only reachable from unused endpoints. I wanted it to be a one-time endeavor but this blog post made me think it should be run on a regular basis.

jiggawatts 4y ago
I wonder if APM performance traces could be used to determine unused code paths. A lot of these tools sample stack traces at intervals, collecting statistics of how often functions are used. If you had enough samples, then with very high confidence you could find code paths that are never used in production.
- emptysea 4y ago
  Another option is tail based sampling which allows you to make the choice at the end of the request whether to keep the trace. With that you could configure the APM to keep the first $N traces for all endpoints and then start sampling after that
- chrnola 4y ago
  That’s an interesting idea! Reminds me of how Pyroscope’s continuous profiling data can be linked to OpenTelemetry traces[1]. [1]: https://pyroscope.io/docs/java-tracing/ https://pyroscope.io/docs/java-tracing/
kortilla 4y ago
Be very careful using this methodology. This is how you end up deleting stuff only called in rare but critical cases: - during outage - at end of year - during audits - when the one special customer that paid a fortune for an obscure feature decides to use it It might be the case that none of this applies to your application, but it’s why “check what got hit recently” (3 months is an eye blink in the business and govt world) is dangerous.

pabs3 4y ago

They should also look at deleting third_party directories and replacing them with package manager dependencies. For example Qt WebEngine embeds Chromium, which embeds Blink, which embeds wpt, which embeds a bunch of Python libraries and one of those Python libraries (six) is embedded in three other locations within Chromium. https://sources.debian.org/src/qtwebengine-opensource-src/5.15.2+dfsg-3/src/3rdparty/chromium/third_party/blink/tools/blinkpy/third_party/wpt/wpt/tools/third_party/ https://sources.debian.org/src/qtwebengine-opensource-src/5.... https://codesearch.debian.net/search?q=package%3Aqtwebengine-opensource-src+Useful+for+very+coarse+version+differentiation https://codesearch.debian.net/search?q=package%3Aqtwebengine.... I wonder how many copies of the Python six module the Google monorepo contains.

raverbashing 4y ago
And while six is 'kinda' well behaved, trying to unify dependencies of dependencies is an exercise in frustration usually. Code repetition is cheap
- pabs3 4y ago
  Code repetition is only cheap if you abandon and forget about the repeated code. Any time you need to have to care about the code to fix bugs, security issues or improve performance, it is expensive do each of those things once, repeating them multiple times can mean an order of magnitude difference in the worse case.
habosa 4y ago
I’d assume one. The “one version policy” is one of the strictest codebase rules at Google. But the Chrome codebase is generally the giant exception to everything due to the major open source components (Android too).
secondcoming 4y ago
This is a good plan but I've often found that with for example, Debian, the official packages are well behind the times compared to the direct third party code. Ubuntu is not too bad.

daxfohl 4y ago

"deleting code" should have been included in the "two hardest things in computer science".

valenterry 4y ago

Upvoted for the nice name :)

aktuel 4y ago

The Devel's in the Details: typo or pun?

terom 4y ago

It's about time someone gets a KPI for lines of code removed :)

meindnoch 4y ago

It's not that simple. A lot of dead code doesn't look dead to static analysis. E.g. gated by long-forgotten feature flags, or platform checks that are no longer relevant.

GeneralMayhem 4y ago
This is only the simplest possible reaper tool. You can (and Google does) also write tools to find flags that have only ever been set to one value, codepaths that haven't been exercised in months, databases that haven't been accessed in years, etc.

izwasm 4y ago

Looks like graph algorithms comes in handy at the end

whirlwin 4y ago

Vert interesting. At our company we have a spring cleaning event next week, and one of the goals is to delete old code. Some of the comments mention some good techniques. Anyone experienced with deep analysis, e.g. BPF on this field?

specialist 4y ago

Conducting an inventory (of services) would be cool too. If a chunk of code isn't actually deployed somewhere, mark it as a candidate for culling. Probably requires some kind of metadata provenance for deployed artifacts. For Java, I thought Maven had a stock manifest.mf entry for the source repo. Alas, a quick search only reveals that the archiver plugin has an entry for the project's "url". https://maven.apache.org/shared-archives/maven-archiver-2.5/examples/manifestEntries.html https://maven.apache.org/shared-archives/maven-archiver-2.5/... Which for in-house projects is probably sufficient.

amai 4y ago

[flagged]

elesbao 4y ago

I'm generally not fond of monorepos but reading this and right away being targeted by this tweet https://twitter.com/nikitabier/status/1652764613962760196 https://twitter.com/nikitabier/status/1652764613962760196 made me think on how much time goes on decisions that won't impact users directly and how hard it becomes to justify that on regular companies in an environment where CEOs are jumping the layoff bandwagon for no reason.