9 ms·

I’m not sure how it’s done with the Linux Kernel (and git definately makes it hard), but at Google (using Perfoce model) there are tools for mass renaming that

I’m not sure how it’s done with the Linux Kernel (and git definately makes it hard), but at Google (using Perfoce model) there are tools for mass renaming that try to guarantee that there are no regressions. Of course in Linux the amount of C macros make it much harder to work on the syntax trees, but comparision after the preprocessor step should be possible.

klodolph 5y ago
IIRC Google works the same way. The automated refactoring tools (Rosie?) may make it go faster, but you generally don't fix up 15,000 call sites in a single change... you break it up into smaller batches spread across the tree.
- acidbaseextract 5y ago
  It's been a long time since I was there, but I thought there were plenty of 15,000 call site refactorings done in a single final CL. Not that the Linux kernel should do the same!
  - skybrian 5y ago
    When I was there (several years ago) it was rare to do that across many projects. You don't want to have to roll back everyone due to a problem that affects one project. Also, there's a risk that changes that are happening in the meantime might mean the patch doesn't apply.
    - Filligree 5y ago
      For a global change? These days that 'risk' is more of a certainty.
  - ithkuil 5y ago
    It may take a lot of time to get all the approvals from all the code owners. One of the big advantages of a tool like Rosie is precisely that it splits up the update into smaller CLs and has automation that nags all necessary reviewers and reruns the transformation once head moves.
deckard1 5y ago
What does git or Perforce have to do with refactoring? Why would whatever version control you're using matter? > tools for mass renaming that try to guarantee that there are no regressions beyond sed or whatever a fancy IDE already does? What kind of voodoo magic are we talking about here? Did Google solve the halting problem and I just haven't noticed yet...
- bri3d 5y ago
  Disclaimer: I haven't worked at Google, but you can easily read their papers like https://dl.acm.org/doi/pdf/10.1145/2854146 https://dl.acm.org/doi/pdf/10.1145/2854146 and understand what's going on. Refactoring tooling is integrated with the revision control system because in a giant codebase with thousands of contributors, you need to split a refactoring patch into smaller chunks and ideally use semantic rather than text-based diff tooling, or the changes will never get reviewed or merged before they conflict with more changes. As for the actual renaming, yes, definitely beyond what sed does but probably around what the fanciest IDE does. Imagine a big global symbol dependency graph produced by the entire build toolchain all at once and cached, I think? Also, this isn't the halting problem unless your codebase allows for fully dynamic invocation :)
  - worthless-trash 5y ago
    Aren't all function pointers dynamic invocation ? Or is "fully" one of those words that i'm not grasping in context.
    - DaiPlusPlus 5y ago
      > Aren't all function pointers dynamic invocation? Given that compilers can be smart-enough to detect "non-dynamic" function-pointer invocation (e.g. when an execution-trace proves that a function-pointer parameter always points to the same function address), it's not safe to say that "all" function pointer invocations are dynamic. Another case to consider is when one implements (Smalltalk-style) OOP with message-passing: in many cases it's possible to build that without needing to use function-pointers at all.
arka2147483647 5y ago
The point is that the change is not a simple rename: struct foo *iterator; list_for_each_entry(iterator, &foo_list, list) { do_something_with(iterator); } While the new version would be something like: list_for_each_entry_v2(iterator, &foo_list, list) { do_something_with(iterator); } So the ’iterator’ variable may be removed, but as this is c, might also be reused multiple times, or come from some struct member, union, or whatever. So i presume something like this would have to be fixed case by case.
- klodolph 5y ago
  You can do these kinds of changes with clang-tidy, if necessary. Clang-tidy is extensible and you can write your own rules for transforming source code. Mind you, there are not a ton of people around who have the time to learn how to write custom clang-tidy rules. It might be done for 15,000 changes that potentially fix kernel vulnerabilities, though.
  - pitaj 5y ago
    Unless something has changed in the last month, clang-tidy is not extensible. You can't write your own rules without forking the whole project.
    - klodolph 5y ago
      I’ve worked at a couple companies that had custom clang-tidy checks. This is well-documented enough at this point, but it is still a bit arcane. http://bbannier.github.io/blog/2015/05/02/Writing-a-basic-clang-static-analysis-check.html http://bbannier.github.io/blog/2015/05/02/Writing-a-basic-cl... Yes, it involves forking. Forking isn’t that big a deal. We even had checks that were specific to internal libraries.
      - andi999 5y ago
        At the start forking is like no deal at all. But how do you get new clang tidy features from the main branch later into your fork; I think this is the only instance which I would really call technical debt.
staticassertion 5y ago
Linux doesn't even have consistent testing let alone the kind of tooling or will that Google does.
- colin_mccabe 5y ago
  The Linux kernel developers have pretty good tooling for refactoring. For example, they have Coccinelle and Sparse. 1. https://en.wikipedia.org/wiki/Coccinelle_(software) https://en.wikipedia.org/wiki/Coccinelle_(software) 2. https://en.wikipedia.org/wiki/Sparse https://en.wikipedia.org/wiki/Sparse
- marcan_42 5y ago
  Linux definitely has testing. Some helpful person added our repo base to the kernel-test-robot list, and now I regularly get e-mails when something I'm working on in some random WIP branch broke the build on some other architecture (often in COMPILE_TEST mode), or some other random driver. Didn't even have to do anything myself. It's great for avoiding "this broke the build on $obscure_config" issues.
eru 5y ago
I don't see how Perforce makes this easier than Git?

klodolph 5y ago

IIRC Google works the same way. The automated refactoring tools (Rosie?) may make it go faster, but you generally don't fix up 15,000 call sites in a single change... you break it up into smaller batches spread across the tree.

acidbaseextract 5y ago
It's been a long time since I was there, but I thought there were plenty of 15,000 call site refactorings done in a single final CL. Not that the Linux kernel should do the same!
- skybrian 5y ago
  When I was there (several years ago) it was rare to do that across many projects. You don't want to have to roll back everyone due to a problem that affects one project. Also, there's a risk that changes that are happening in the meantime might mean the patch doesn't apply.
  - Filligree 5y ago
    For a global change? These days that 'risk' is more of a certainty.
- ithkuil 5y ago
  It may take a lot of time to get all the approvals from all the code owners. One of the big advantages of a tool like Rosie is precisely that it splits up the update into smaller CLs and has automation that nags all necessary reviewers and reruns the transformation once head moves.

deckard1 5y ago

What does git or Perforce have to do with refactoring? Why would whatever version control you're using matter? > tools for mass renaming that try to guarantee that there are no regressions beyond sed or whatever a fancy IDE already does? What kind of voodoo magic are we talking about here? Did Google solve the halting problem and I just haven't noticed yet...

bri3d 5y ago
Disclaimer: I haven't worked at Google, but you can easily read their papers like https://dl.acm.org/doi/pdf/10.1145/2854146 https://dl.acm.org/doi/pdf/10.1145/2854146 and understand what's going on. Refactoring tooling is integrated with the revision control system because in a giant codebase with thousands of contributors, you need to split a refactoring patch into smaller chunks and ideally use semantic rather than text-based diff tooling, or the changes will never get reviewed or merged before they conflict with more changes. As for the actual renaming, yes, definitely beyond what sed does but probably around what the fanciest IDE does. Imagine a big global symbol dependency graph produced by the entire build toolchain all at once and cached, I think? Also, this isn't the halting problem unless your codebase allows for fully dynamic invocation :)
- worthless-trash 5y ago
  Aren't all function pointers dynamic invocation ? Or is "fully" one of those words that i'm not grasping in context.
  - DaiPlusPlus 5y ago
    > Aren't all function pointers dynamic invocation? Given that compilers can be smart-enough to detect "non-dynamic" function-pointer invocation (e.g. when an execution-trace proves that a function-pointer parameter always points to the same function address), it's not safe to say that "all" function pointer invocations are dynamic. Another case to consider is when one implements (Smalltalk-style) OOP with message-passing: in many cases it's possible to build that without needing to use function-pointers at all.

arka2147483647 5y ago

The point is that the change is not a simple rename: struct foo *iterator; list_for_each_entry(iterator, &foo_list, list) { do_something_with(iterator); } While the new version would be something like: list_for_each_entry_v2(iterator, &foo_list, list) { do_something_with(iterator); } So the ’iterator’ variable may be removed, but as this is c, might also be reused multiple times, or come from some struct member, union, or whatever. So i presume something like this would have to be fixed case by case.

klodolph 5y ago
You can do these kinds of changes with clang-tidy, if necessary. Clang-tidy is extensible and you can write your own rules for transforming source code. Mind you, there are not a ton of people around who have the time to learn how to write custom clang-tidy rules. It might be done for 15,000 changes that potentially fix kernel vulnerabilities, though.
- pitaj 5y ago
  Unless something has changed in the last month, clang-tidy is not extensible. You can't write your own rules without forking the whole project.
  - klodolph 5y ago
    I’ve worked at a couple companies that had custom clang-tidy checks. This is well-documented enough at this point, but it is still a bit arcane. http://bbannier.github.io/blog/2015/05/02/Writing-a-basic-clang-static-analysis-check.html http://bbannier.github.io/blog/2015/05/02/Writing-a-basic-cl... Yes, it involves forking. Forking isn’t that big a deal. We even had checks that were specific to internal libraries.
    - andi999 5y ago
      At the start forking is like no deal at all. But how do you get new clang tidy features from the main branch later into your fork; I think this is the only instance which I would really call technical debt.

staticassertion 5y ago

Linux doesn't even have consistent testing let alone the kind of tooling or will that Google does.

colin_mccabe 5y ago
The Linux kernel developers have pretty good tooling for refactoring. For example, they have Coccinelle and Sparse. 1. https://en.wikipedia.org/wiki/Coccinelle_(software) https://en.wikipedia.org/wiki/Coccinelle_(software) 2. https://en.wikipedia.org/wiki/Sparse https://en.wikipedia.org/wiki/Sparse
marcan_42 5y ago
Linux definitely has testing. Some helpful person added our repo base to the kernel-test-robot list, and now I regularly get e-mails when something I'm working on in some random WIP branch broke the build on some other architecture (often in COMPILE_TEST mode), or some other random driver. Didn't even have to do anything myself. It's great for avoiding "this broke the build on $obscure_config" issues.

eru 5y ago

I don't see how Perforce makes this easier than Git?