10 ms·

Now, would it even make sense to merge into the mainline codebase of LLVM backend code targeting a platform that noone else can test or use? Is Google planning

Now, would it even make sense to merge into the mainline codebase of LLVM backend code targeting a platform that noone else can test or use? Is Google planning on eventually releasing this architecture publicly or maybe licensing it to third party manufacturers, a-la ARM?

lhecker 11y ago
Maybe they just don't want to backport their backend code again and again for every LLVM snapshot they do, because that can be a tedious thing to do compared to just having it in origin/master forever for free.
- iainmerrick 11y ago
  What makes that tedious? And can't it be fixed?
  - coldpie 11y ago
    Suppose the LLVM devs change a function signature used by Google's private backend. If the backend is private, Google will have to update their usage of that API when they do a merge. But if the backend is present upstream when the LLVM devs make that change, then the LLVM devs are responsible for making sure all supported backends update their usage of the API.
    - iainmerrick 11y ago
      But as people have noted, if you don't have the hardware, it's impossible to test that any changes actually work. If we're just talking about pure refactoring that doesn't change any output, you could test that the generated machine code is identical. But then you have to ask, why isn't there a stable API rather than all this refactoring churn? I guess this is just the way LLVM and Clang are designed -- all components really tightly coupled together. And it's a successful project so it must be working out for them. But...!
      - microcolonel 11y ago
        In this case you would just need to compile to know it's broken.
      - coldpie 11y ago
        I'm just explaining why Google might want it merged upstream. The work you're talking about has to be performed in either case.
      - mywittyname 11y ago
        So I'd venture to presume that there's an extensive battery to tests for each target processor. It seems unlikely to me that developers on the LLVM/Clang team all have physical access to every target CPU.
      - DannyBee 11y ago
        "If we're just talking about pure refactoring that doesn't change any output, you could test that the generated machine code is identical. But then you have to ask, why isn't there a stable API rather than all this refactoring churn? " LLVM deliberately does not want a stable API. It wants people to keep up with trunk. They do this because they saw what has happened with other compilers, where the stable API became literally impossible to change over time. This is one of the reasons GCC still has a crappy backend. You either have to build a new API and port everyone over, or you have to find an incremental way to change an API interface with hundreds of random interface points.
      - thrownaway2424 11y ago
        That is definitely not true about needing the hardware. There are loads of compilers for fictitious hardware and they work perfectly well. There was an entire x86-64 tool chain before anyone ever manufactured one of those.
- jacquesm 11y ago
  Without the ability to test though they will still have to do a bunch of work in order to be able to use a new release. Possibly the release breaks their backend but nobody will know until google tries to use it.
  - VikingCoder 11y ago
    If their contribution includes unit tests, it could be enough to guarantee most changes won't break it.
usrusr 11y ago
For Google, sure. Everybody working with the in-house hardware could do so without hunting for a version of the internal patch/fork. Using the fork would surely be easy enough when directly using LLVM, but the deeper it is hidden within a dependency tree, the more friction it adds to whatever the user at the root of that tree is doing. Everybody else pays a small "maintenance tax" when working on the codebase or relying on work being done on that codebase. Being by no means a compiler expert I do suspect however, that this "tax" is tiny and likely to be dwarfed by other contributions from Google, so letting them solve distribution of their private backend by piggybacking on the public release is most likely the right course of action.
tyingq 11y ago
It appears to be for the Myricom LanAI processor, which is available to others. It's an older 32 bit RISC processor, used as the offload engine for their NIC cards. Here's gcc for it: https://github.com/myri/lanai-gcc https://github.com/myri/lanai-gcc Sounds like it's not particularly interesting unless you want to write your own network offload code. Edit: Probably just the offload processor they are choosing for their in-house routers/switches/servers. And, probably they want their own firmware either for security/nsa reasons, or performance, or both.
- mikecb 11y ago
  I thought this too, but someone in the email thread explicitly mentioned the myricom, and the _response_ to that message was that it was purely internal hardware, not useful for others...so it may just be a coincidence.
  - geofft 11y ago
    But then they go on to say that their "previous binutils used" was https://github.com/myri/lanai-binutils https://github.com/myri/lanai-binutils. It certainly seems believable that this is a third-party processor architecture, but Google has a contract with them to build specific models of that processor that meet their needs (possibly alongside some other proprietary hardware, like a high-speed NIC), and those models aren't sold to the general public. That's pretty common outside x86, right? For instance, is there a way for me to buy a BCM2835 other than by buying a Raspberry Pi?
    - ShinyCyril 11y ago
      Chips like the BCM2835 are available for purchase, just you need to have a very large purchase order.
    - tyingq 11y ago
      If you search a little, you'll find that Google hired Myricom's CEO, Founder/CTO, and several engineers. I suppose it's possible they licensed the IP too.
Sir_Cmpwn 11y ago
No, I don't think it really makes sense for LLVM to mainline an architecture no one can use. I doubt they'll be on board with this.
- verbatim 11y ago
  It's not exactly unprecedented. GCC supports mmix, which hardware doesn't even exist for. (An educational platform from TAOCP.) https://gcc.gnu.org/backends.html https://gcc.gnu.org/backends.html
  - iainmerrick 11y ago
    MMIX is fully documented, though. Being an educational platform is a valid reason to be supported.
  - cmrx64 11y ago
    "Only has simulators" is very different from "nobody can use". MMIX has value.
  - FullyFunctional 11y ago
    The difference between hardware and software is getting fuzzy (Transmeta, NVIDIA Denver, ...), but there's at least one (admittedly terrible) FPGA implementation of a subset of MMIX large enough to execute small graphical demos: https://github.com/tommythorn/fpgammix https://github.com/tommythorn/fpgammix
- kenferry 11y ago
  Looks like there's consensus to accept it as an "experimental" backend. I think that means the code is in tree, but other people changing LLVM aren't obligated to keep it working. It may be accepted as a full backend in the future, but that discussion is deferred since people are happy with it being experimental for now.
- jakejake 11y ago
  Ordinarily I'd agree with you. Except the tradeoff of having Google's other contributions to the main branch is probably worth it for a little bit of code that isn't harming anyone else. It's perhaps not fair, but just a reality that a company like Google has a lot to offer so they can sometimes get special treatment.
maaku 11y ago
Having been in this position before, I can tell you that maintaining large patch sets increases friction against submitting anything upstream. Even if this was a Google-only CPU (it's not), that'd still be an argument for merging upstream.

lhecker 11y ago

Maybe they just don't want to backport their backend code again and again for every LLVM snapshot they do, because that can be a tedious thing to do compared to just having it in origin/master forever for free.

iainmerrick 11y ago
What makes that tedious? And can't it be fixed?
- coldpie 11y ago
  Suppose the LLVM devs change a function signature used by Google's private backend. If the backend is private, Google will have to update their usage of that API when they do a merge. But if the backend is present upstream when the LLVM devs make that change, then the LLVM devs are responsible for making sure all supported backends update their usage of the API.
  - iainmerrick 11y ago
    But as people have noted, if you don't have the hardware, it's impossible to test that any changes actually work. If we're just talking about pure refactoring that doesn't change any output, you could test that the generated machine code is identical. But then you have to ask, why isn't there a stable API rather than all this refactoring churn? I guess this is just the way LLVM and Clang are designed -- all components really tightly coupled together. And it's a successful project so it must be working out for them. But...!
    - microcolonel 11y ago
      In this case you would just need to compile to know it's broken.
    - coldpie 11y ago
      I'm just explaining why Google might want it merged upstream. The work you're talking about has to be performed in either case.
    - mywittyname 11y ago
      So I'd venture to presume that there's an extensive battery to tests for each target processor. It seems unlikely to me that developers on the LLVM/Clang team all have physical access to every target CPU.
    - DannyBee 11y ago
      "If we're just talking about pure refactoring that doesn't change any output, you could test that the generated machine code is identical. But then you have to ask, why isn't there a stable API rather than all this refactoring churn? " LLVM deliberately does not want a stable API. It wants people to keep up with trunk. They do this because they saw what has happened with other compilers, where the stable API became literally impossible to change over time. This is one of the reasons GCC still has a crappy backend. You either have to build a new API and port everyone over, or you have to find an incremental way to change an API interface with hundreds of random interface points.
    - thrownaway2424 11y ago
      That is definitely not true about needing the hardware. There are loads of compilers for fictitious hardware and they work perfectly well. There was an entire x86-64 tool chain before anyone ever manufactured one of those.
jacquesm 11y ago
Without the ability to test though they will still have to do a bunch of work in order to be able to use a new release. Possibly the release breaks their backend but nobody will know until google tries to use it.
- VikingCoder 11y ago
  If their contribution includes unit tests, it could be enough to guarantee most changes won't break it.

usrusr 11y ago

For Google, sure. Everybody working with the in-house hardware could do so without hunting for a version of the internal patch/fork. Using the fork would surely be easy enough when directly using LLVM, but the deeper it is hidden within a dependency tree, the more friction it adds to whatever the user at the root of that tree is doing. Everybody else pays a small "maintenance tax" when working on the codebase or relying on work being done on that codebase. Being by no means a compiler expert I do suspect however, that this "tax" is tiny and likely to be dwarfed by other contributions from Google, so letting them solve distribution of their private backend by piggybacking on the public release is most likely the right course of action.

tyingq 11y ago

It appears to be for the Myricom LanAI processor, which is available to others. It's an older 32 bit RISC processor, used as the offload engine for their NIC cards. Here's gcc for it: https://github.com/myri/lanai-gcc https://github.com/myri/lanai-gcc Sounds like it's not particularly interesting unless you want to write your own network offload code. Edit: Probably just the offload processor they are choosing for their in-house routers/switches/servers. And, probably they want their own firmware either for security/nsa reasons, or performance, or both.

mikecb 11y ago
I thought this too, but someone in the email thread explicitly mentioned the myricom, and the _response_ to that message was that it was purely internal hardware, not useful for others...so it may just be a coincidence.
- geofft 11y ago
  But then they go on to say that their "previous binutils used" was https://github.com/myri/lanai-binutils https://github.com/myri/lanai-binutils. It certainly seems believable that this is a third-party processor architecture, but Google has a contract with them to build specific models of that processor that meet their needs (possibly alongside some other proprietary hardware, like a high-speed NIC), and those models aren't sold to the general public. That's pretty common outside x86, right? For instance, is there a way for me to buy a BCM2835 other than by buying a Raspberry Pi?
  - ShinyCyril 11y ago
    Chips like the BCM2835 are available for purchase, just you need to have a very large purchase order.
  - tyingq 11y ago
    If you search a little, you'll find that Google hired Myricom's CEO, Founder/CTO, and several engineers. I suppose it's possible they licensed the IP too.

Sir_Cmpwn 11y ago

No, I don't think it really makes sense for LLVM to mainline an architecture no one can use. I doubt they'll be on board with this.

verbatim 11y ago
It's not exactly unprecedented. GCC supports mmix, which hardware doesn't even exist for. (An educational platform from TAOCP.) https://gcc.gnu.org/backends.html https://gcc.gnu.org/backends.html
- iainmerrick 11y ago
  MMIX is fully documented, though. Being an educational platform is a valid reason to be supported.
- cmrx64 11y ago
  "Only has simulators" is very different from "nobody can use". MMIX has value.
- FullyFunctional 11y ago
  The difference between hardware and software is getting fuzzy (Transmeta, NVIDIA Denver, ...), but there's at least one (admittedly terrible) FPGA implementation of a subset of MMIX large enough to execute small graphical demos: https://github.com/tommythorn/fpgammix https://github.com/tommythorn/fpgammix
kenferry 11y ago
Looks like there's consensus to accept it as an "experimental" backend. I think that means the code is in tree, but other people changing LLVM aren't obligated to keep it working. It may be accepted as a full backend in the future, but that discussion is deferred since people are happy with it being experimental for now.
jakejake 11y ago
Ordinarily I'd agree with you. Except the tradeoff of having Google's other contributions to the main branch is probably worth it for a little bit of code that isn't harming anyone else. It's perhaps not fair, but just a reality that a company like Google has a lot to offer so they can sometimes get special treatment.

maaku 11y ago

Having been in this position before, I can tell you that maintaining large patch sets increases friction against submitting anything upstream. Even if this was a Google-only CPU (it's not), that'd still be an argument for merging upstream.