13 ms·

Rich Felker of musl libc comments on Google's LLVM libc proposal

387 points by maximilianroos 7y ago

new_realist 7y ago
Why not start with musl?
- gpm 7y ago
  This is a common complaint about companies starting new open source projects, and it really annoys me. Anyone who works with software knows there are tons of very good reasons why you might want to start a new project from scratch instead of contributing to an existing one. You might find the existing architecture limiting. The trade offs taken to be non optimal (for your use case or in general). Consider the existing code to be low quality. The code contribution process to lax (introducing bugs) or strict (slowing work). Find the community toxic. Want to utilize existing internal code bases. Etc. There are tons of valid reasons to make a new project, and for most of them you don't want to say it out loud. Saying "your code sucks" or "we don't like you (i.e. toxic community)" is rude. Saying "we don't like the processes" is asking for a long political fight, and likely to be taken as rude by the people who invented them. Saying that they want to utilize internal code bases could be problematic internally. Etc. In this case I get the feeling that quite a bit of it is disagreeing with the tradeoff's musl took (see: Wanting to make libc more like a library), which has to be about the only reason you are allowed to publicly say. Maybe there are other reasons as well, maybe not. Sorry to jump down your throat, it's just a common type of comment that I wanted to give a general reply to. (Edit: To be clear I'm not involved with this project, or a google employee for that matter)
  - Pyxl101 7y ago
    When a new project is proposing to do something that's been done before, potentially multiple times, I think it is reasonable to want to understand how the new project will compare and contrast to work that's already been done. Asking "How will this project compare to <existing implementation>?" seems quite reasonable to me. In fact, I would expect the person making the proposal to address this question as part of the proposal. It is not desirable to reinvent the wheel without good reason. It's totally fine if the reason is something like: we want to use a different license, or a different development process, or a different coding style, or whatever. Whatever the actual motivations are, clarifying them up front will be good for everyone.
  - nine_k 7y ago
    This is reasonable. A simple answer like "we are not starting with musl because ..." would be quite welcome, though.
    - yc12340 7y ago
      I assume, that Google's answer would sound like this: "We are not starting with musl, because we want to own all our software the same way people own hamsters or dogs: care for them, feed them, train them, single handedly decide whether to castrate or kill them. We don't need the risk of owning something we don't fully control." As far, as I am aware, musl isn't owned by public company or foundation. Google can't exert desirable degree of control over musl project, because there is no board of directors to buy their way into, — it is all maintained by one guy (and chattel slavery is outlawed in US...)
      - AstralStorm 7y ago
        Bribing or hiring the dev can still work.
  - g82918 7y ago
    I agree this with as some one working at a company that avoids most open source like the plague. But in this particular case, the code quality in musl is pretty good. I haven't had an issue with it so far, and after trying to upstream some things in glibc I have to say I can't see why they wouldn't use musl.
    - rurban 7y ago
      Well, there are still huge holes in musl. towlower e.g uses a linear search. my binary search made it 2x faster, but we still have to wait for that to appear. Implemented Unicode standards are years old. You don't know which standard is implemented. Still many problems with dynamic linking. And: musl is POSIX. POSIX blocking IO should not be used in a modern OS or language. You can never guarantee concurrency safety with POSIX. This is the Fuchsia problem. They went for unsafety and POSIX compat. L4 ditto. Microsoft's Singularity not. A modern libc would need to get rid of blocking IO and implement strings as utf8 only. There's no need for anything else. The C standard made too many mistakes.
      - saagarjha 7y ago
        I don't see how you're planning on writing a libc that chooses to interpret the C standard as full of "mistakes" without writing an entirely different language.
        rurban 7y ago
        I'm already maintaining the libc security extensions, which are in the standard but all libc's refuse to implement. I also added the missing string API's for the wrong wide API, but thinking about adding the proper u8 api instead. Nobody really should use the wide (UTF-32/UTF-16) api. But the C standard committee is somewhere completely else. There are not even aware of the problems, even if I'm getting interviewed from time to time, so that they will change their attitudes towards insecurity and functionality. searching for strings should be a pretty common usage pattern, but it's still not implemented. coreutils or grep or sed would like to use it, I'm sure. But not 10x slower as the current multibyte patches. Which I'm also maintaining btw. for coreutils. Non-blocking IO into the standard? I only know about libuv which was fine before the drama. Despised it since then, but I'm still using it. The standard mostly needs to provide a namespace to block blocking IO if someone needs a concurrency-safe subset. Then the unhealthy need for POSIX compat can be circumvented somewhat. And gets() can be finally.
  - anth_anm 7y ago
    > There are tons of valid reasons to make a new project, and for most of them you don't want to say it out loud. There are also a few that are just scummy and self-serving. Since google is involved, I immediately assume the worst.
    - dang 7y ago
      If you can't stop posting flamebait and unsubstantive comments to HN, we're going to have to ban you. We've asked you repeatedly and you've continued to break the guidelines repeatedly. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
      - anth_anm 7y ago
        You can keep posting the guidelines all you want. I don't bother trying because I know that you will just pop up with the word "unsubstantive" for anything you don't happen to personally think is good.
  - hinkley 7y ago
    Yes, there are reasons to rewrite a library, but people expect you to tell them why you are rewriting it. So... why are they rewriting it?
- MichaelSalib 7y ago
  Which do you think looks better on your promotion packet? 1. Write a new libc and get it used all over Google, generating lots of users automatically; or 2. Shepard Google's adoption of code some other folks wrote.
  - chandlerc1024 7y ago
    Either one looks completely fantastic, I assure you. I have helped people on my team write promotion packets in both forms. Both worked perfectly well.
  - DannyBee 7y ago
    I don't think this level of cynicism is warranted However, as the person who actually runs the relevant promo committees for the org, both would be fine, i'd care a lot more about why the choice was made.
    - izacus 7y ago
      I love how your repose with 1st hand experience is downvoted as opposed to a pointless jab of the original poster :)
      - finnthehuman 7y ago
        Googlers have the experience but not the distance to meaningfully comment on google.
        DannyBee 7y ago
        Sorry, but this is just BS through and through. Painting 100k people with a brush of "can't be objective" based on no data is probably the most biased think i've actually seen in a long time.
        finnthehuman 7y ago
        I’ll paint 7 billion with the “proximity changes perception” brush.
- mda 7y ago
  Instead of resorting to theoretical rhetorics on meta issues on the given proposal,Rich Felker could ask this question as well. But he did not, I am actually quite disappointed with his response, it looks unprofessional.
kbumsik 7y ago
Doesn't google use newlib already?
- altmind 7y ago
  Wait, weren't they using Bionic on android?
  - murderfs 7y ago
    Google is too big for the left hand to know what the right hand is doing. Various parts of Google use bionic, newlib, musl, and multiple forks of glibc.
    - hinkley 7y ago
      So does that mean all that propaganda about them having a monorepo is bullshit?
      - murderfs 7y ago
        Android (bionic), Chrome (forks of glibc and newlib in NaCl), ChromeOS (glibc with patches), and Fuchsia (musl) are all in separate Gerrit repositories. The monorepo for Google proper probably contains all of these, plus the libc used for Google.
nemothekid 7y ago
What does “LLVM implementation of Libc” mean here? Does that just mean an implementation of libc maintained by the LLVM project?
- fourier_mode 7y ago
  Yeah, I am also confused. Isn't there Clang already? (Please correct me if I am wrong)
  - wyldfire 7y ago
    The C library is generally considered a part of the OS and the overall toolchain. Clang is "merely" a C compiler. The C library provides implementations for the platform-neutral language functions like "open/read/write/close/ioctl/dup/dup2" and more.
    - fourier_mode 7y ago
      So implementations of those subroutines are written in assembly lang?
      - wyldfire 7y ago
        No, it's very common for them to be written in primarily C. But I'm obligated to mention (as a member of the Rust Evangelism Strike Force) that you don't have to write them in C. ;) Many implementations do have performance-critical parts be target-dependent and implemented in assembly. IIRC glibc has more of this than musl/newlib/bionic, but it's been a while since I last looked.
        fourier_mode 7y ago
        > No, it's very common for them to be written in primarily C Then can't you just get the LLVM IR out of it using Clang?
        7y ago
m0zg 7y ago
This sounds like Android NDK development will get quite painful in a few years. It previously included a botched libc++ missing a bunch of features. This is libc, but I imagine subtle feature disparities and incompatibilities there will cause plenty of pain as well.
- uep 7y ago
  There are already some real pain points with bionic. I primarily write Linux code that sometimes has to run on Android (notably not any other UNIX). I've run into tons of problems porting to old versions of bionic, despite the kernel being the same.
- mrpippy 7y ago
  I don’t think the Android team is the interested party here, given the original poster’s interest only in x86_64
  - m0zg 7y ago
    Google doesn't do "only x86_64". Even when I was there (years ago) at a minimum all software had to run on PowerPC as well, and non-portable assembly/intrinsics required explicit approval from "Eng Elders" and a portable fallback. I wouldn't be surprised if ARM compat is also required now.
    - saagarjha 7y ago
      That might be true, but the email posted here explicitly lists x86_64 as the architecture they are targeting initially.
- jdswain 7y ago
  In my experience NDK development is quite painful now. Most of my NDK work is a little old now but I found the debugging process quite painful. This was for C++ code that was running on other platforms so almost all of the debugging was due to platform incompatibilities or bugs. But I did notice in the linked article that they are only targeting x86_64, so I don't think this libc is targeted at Android.
catern 7y ago
One of the replies is this email from an LLVM developer working at Google: http://lists.llvm.org/pipermail/llvm-dev/2019-June/133315.html http://lists.llvm.org/pipermail/llvm-dev/2019-June/133315.ht... I can't say it looks particularly good for someone at Google to be using their moderator powers on this unrelated-to-Google list to tell Felker not to include parts of his message which reflect badly on Google.
- scarface74 7y ago
  I see nothing wrong with this. The last thing you want in a technical discussion are petty personal attacks. The moderator sounds a lot like something you would see dang say on HN.
  - catern 7y ago
    I don't think they're petty personal attacks. They are important concerns which I share. Regardless, since this particular member of the project has clear incentives to be biased in favor of Google, they should have recused themselves, and asked another member of the project/moderator of the list to judge whether the message was appropriate.
    - shereadsthenews 7y ago
      Chandler is as related to the person responsible for putting musl in Fuchsia as are any two randomly selected software developers from anywhere in the industry. Having to disqualify someone from moderating a mailing list based on the union of all the conflicts of interest of tens of thousands of unrelated people is very silly.
      - catern 7y ago
        The conflict of interest is not related to putting musl in Fuschia. The conflict of interest is created by Chandler's employment by Google. Someone paid by a company should not be making moderation decisions about comments related to that company.
Sir_Cmpwn 7y ago
Copy-pasting a comment from "cup" on lobsters, because I think it's on-point: >1000 times yes. If Google wants their own libc that’s their business. But LLVM should not be part of their “manifest destiny”. The corporatization of OSS is a scary prospect, and should be called out loud and clear like this every time it’s attempted. https://lobste.rs/s/dgmoyi/response_libc_llvm#c_hg8dvg https://lobste.rs/s/dgmoyi/response_libc_llvm#c_hg8dvg
- favorited 7y ago
  There has been huge corporate involvement and investment in LLVM from its earliest days as anything beyond a research project. When Apple open-sourced Clang, or their C++ standard library implementation and contributed them to the LLVM project, was that something to be afraid of as well? It seems to have gone pretty well.
  - svnpenn 7y ago
    Fair point, but Google is not Apple. Google has a history of railroading standards to suit their needs, to the point where the standard becomes "what Google is doing". See the Chrome project.
    - kevin_thibedeau 7y ago
      Uh... Firewire.
    - flukus 7y ago
      Apple ignores standards in the first place and opts for a completely closed and proprietary ecosystem wherever possible (see vulkan support and many others). They're both out to screw you but at least google does a bit of romancing first.
    - Conan_Kudo 7y ago
      There are plenty of examples on Apple's side, too. To name a few: Floppy disks. CDROM. USB. Firewire. WebKit.
  - 7y ago
fourier_mode 7y ago
I think its primarily because Chris Lattner[0] is at Google now. I wonder even if they pull out this effort how much of that would still be maintained if Lattner changes jobs. [0]: https://en.wikipedia.org/wiki/Chris_Lattner https://en.wikipedia.org/wiki/Chris_Lattner
- favorited 7y ago
  Chris Lattner said that he has nothing to do with this project http://lists.llvm.org/pipermail/llvm-dev/2019-June/133274.html http://lists.llvm.org/pipermail/llvm-dev/2019-June/133274.ht...
- theresistor 7y ago
  No. Chris is working in the AI/ML space, and specifically commented in this thread that he is not involved with this project.
- CUViper 7y ago
  FWIW Roland McGrath, the original glibc author, also works at Google. But of course not every possible person will be involved here, no matter how relevant.
jasonhansel 7y ago
Filed under "This is why we need the GPL"
- flukus 7y ago
  This. I definitely sounds like they want a libc that they are free to fork and add proprietary incompatibilities to, which they can do with the GNU libc (under LGPL) as long as they don't want to distribute it, so we can assume they want to distribute it and that this is a further closing of their consumer tech like android. I would've though musl would be a better starting point for them.
- taneq 7y ago
  How does the GPL help with this? Google could build their own libc replacement, and if they push it hard enough to gain widespread community support, it could become the (or at least, a) new standard. At that point Google is free to do whatever they want with it as long as they make it useful enough that the community won't abandon it. The GPL just means that it could be forked later, but if Google is still putting significant resources into developing the main branch then no-one will switch to the forks.
- izacus 7y ago
  GPL is literally made to allow forks like these, what exactly did you mean with your post?!
ajxs 7y ago
> A corporate-led project does not have to answer to the community, and will leave whatever bugs they introduce in place for the sake of bug-compatibility with their own software rather than fixing them. This speaks volumes very clearly. This highlights an immense hazard. Enterprise scale companies contributing to open-source is a fantastic thing, but enterprise scale companies thrusting their own proprietary libraries onto the open-source world is not. I'm already actively avoiding becoming beholden to Google in my work as it is already, let alone in the world where important software uses a libc written by Google. If you're not concerned by this, refer to the immense power that Google already wields over the extremely ubiquitous web-standards through the market dominance that Chrome has.
- jchw 7y ago
  (Disclosure, Googler here, not working on projects related to any of the things being discussed, speaking in a purely personal capacity.) I hate to disagree with Rich Felker. I think most people have a lot of respect for musl and the amount of effort it took. But, I disagree with the basic conclusions. For one thing, I don't think it's fair to compare this to Chrome, an end user product. libc is a fundamental piece of architecture, like (edit: as in, fundamental in stacks that they are used in, to clarify) protobufs, gRPC, Kubernetes, etc. I heavily sympathize with folks uneasy about the Chrome situation - like I've mentioned maybe hundreds of times now I use and recommend Firefox personally, but I feel this particular discussion will become more productive when that stops being a focal point. Their other points are not really related, but I disagree with them also. I don't think there's a good motivation to keep bugs for backwards compatibility: Google runs non-Google software, too; keeping subtle incompatibilities could be a disaster. Also, Google has a pretty good corpus of code to test their libc on, so I think they have a unique opportunity to fast-track a pretty battle-tested libc (I did not know of this project before this Hacker News post, so this is purely a hot take, but for now I stand by it.) No idea what their plan is, but open governance would probably work wonders here. After all, LLVM has plenty of corporate contributors, and it seems to do fine.
  - stochastic_monk 7y ago
    I think trying to work with protobufs by analogy here fails because the scopes of power are so vastly different.
xvilka 7y ago
Quite relevant then: "Why I'm donating $150/month (10% of my income) to the musl libc project"[1]. [1] https://andrewkelley.me/post/why-donating-to-musl-libc-project.html https://andrewkelley.me/post/why-donating-to-musl-libc-proje...
gok 7y ago
This would have been a much better email without the needless stone throwing from one libc maintainer's glass house.
cpeterso 7y ago
I'm surprised the proposed initial scope would include x86-64 but not ARMv7 or AArch64. That suggests Google's motivation here is their server software, not mobile devices running Android or Fuchsia. Also, portability and consistency is an important attribute of a libc. Supporting multiple architectures from the beginning would help shape a portable design. For example, the original target processor for Windows NT development was the Intel i860 specially because it wasn't the standard i386 architecture. Fun fact: the Intel i860 was codenamed "N10" aka "N-Ten", which may have been the origin of the Windows "NT" moniker.
- deleted 7y ago
  [deleted]
- floatboth 7y ago
  > mobile devices running Android They already have Bionic! > or Fuchsia They already have a weird fork of musl that Rich described in the email! typical Google "Not Invented in this Team" approach..
  - pjmlp 7y ago
    Bionic was a very hard child birth. Check NDK roadmap to see how much years (yes years) has taken them between Android and NDK releases to provide a proper libc.
lone_haxx0r 7y ago
Google are going to make it with or without LLVM's cooperation. If it's done with LLVM, it's highly probable that LLVM won't let them extend,extinguish libc. Google's code will be carefully evaluated by a relatively neutral party: LLVM. If Google does it alone, they will write whatever code they want. They'll be free to extend, extinguish libc.
- saagarjha 7y ago
  Why do you think Google has the ability to EEE libc by themselves but somehow be voluntarily reigned in by LLVM?
rurban 7y ago
Rich is of course totally right. There's not only the botched musl in Fuchsia, also Android Bionic, which is a libc written in C++. It's utterly unusable. They had a better, older, safer libc implementation based on some HP code, but that is gone. Some of Rich' points are moot though. Google already have their own libc's, in Android and Fuchsia with enhanced security measures. Similar to Huawei or Intel or Cisco. Better than Apple's freebsd libc. Now reaching out to the community to outsource their development efforts under llvm is a good thing. There's obviously a need, musl, bsd and glibc cannot address. I'm sympathetic to that.
- wbl 7y ago
  And the need is? I read the thread and I still don't know.
_pmf_ 7y ago
Make vs buy decisions work differently for companies with virtually unlimited buying power. The absolute control that doing it yourself provides cannot be bought. And Google has to care only the platforms it wants to support (as some guy at BSDcon said, you have Windows, Linux Intel/AMD/ARM and rounding errors). Do you think the 100000 cookie cutter game developers for Android will stop developing for it because the platform libc has subtle issues? They rightfully don't give a shit.
eqvinox 7y ago
His 3rd argument is absolutely and utterly compelling and a total knockout against Google's approach to this IMHO: > Some folks have told me that I should press LLVM to make musl the "LLVM libc" instead of whatever Google wants to do, but that misses the point: there shouldn't be a "LLVM libc", or any one library implementation that's "first class" for use with LLVM while others are only "second class". If they want to create a great libc - or fuck up another libc - they're much welcome to do that, but they need to go do it outside of LLVM. Dumping a libc on LLVM is just (1.) appropriating credibility from LLVM, (2.) sticking others with the maintenance bill, and (3.) horseshit.
- qalmakka 7y ago
  This, and I also strongly argue that compiler implementations should be untied from standard libraries, and this also applies to libc++/libstdc++. A compiler shipping its runtime is a bad idea that totally screws up compatibility (just look at how painful it is to use MINGW + C++ on Windows because it can't link with VS's C++ Runtime - I understand that MS headers are crap to parse, but with a bit of effort clang now does it fine), and ties them together and prevents reuse. In an ideal world shipping a runtime should be the OS's role, not the compiler's, and the compiler should make use of whatever the platform ships, instead of having N compilers shipping N different, incompatible libraries. The interface between the library and the compiler would then become a true language standard, instead of "library for compiler version X.Y.Z"
  - nly 7y ago
    Raymond Chen wrote about this in his 2014 blog post "Windows is not a Microsoft Visual C/C++ Run-Time delivery channel" https://devblogs.microsoft.com/oldnewthing/20140411-00/?p=1273 https://devblogs.microsoft.com/oldnewthing/20140411-00/?p=12... Stable ABIs are good, but the ability to break it and iterate/improve is even more so. GNU platforms are currently stuck with a shitty std::string implementation (32 bytes but only 15 bytes of SSO capacity) because breaking ABI means recompiling your entire Linux distro.
    - Asooka 7y ago
mehdix 7y ago
These are the highlights IMO: > Avoiding monoculture preserves the motivation for consensus-based standards processes rather than single-party control > there shouldn't be a "LLVM libc", or any one library implementation that's "first class" for use with LLVM while others are only "second class".
rjvehn 7y ago
Well glibc kind of proved that individual maintainers can be just as bad for stewardship[0]. [0]: https://news.ycombinator.com/item?id=7914022 https://news.ycombinator.com/item?id=7914022

new_realist 7y ago

Why not start with musl?

gpm 7y ago
This is a common complaint about companies starting new open source projects, and it really annoys me. Anyone who works with software knows there are tons of very good reasons why you might want to start a new project from scratch instead of contributing to an existing one. You might find the existing architecture limiting. The trade offs taken to be non optimal (for your use case or in general). Consider the existing code to be low quality. The code contribution process to lax (introducing bugs) or strict (slowing work). Find the community toxic. Want to utilize existing internal code bases. Etc. There are tons of valid reasons to make a new project, and for most of them you don't want to say it out loud. Saying "your code sucks" or "we don't like you (i.e. toxic community)" is rude. Saying "we don't like the processes" is asking for a long political fight, and likely to be taken as rude by the people who invented them. Saying that they want to utilize internal code bases could be problematic internally. Etc. In this case I get the feeling that quite a bit of it is disagreeing with the tradeoff's musl took (see: Wanting to make libc more like a library), which has to be about the only reason you are allowed to publicly say. Maybe there are other reasons as well, maybe not. Sorry to jump down your throat, it's just a common type of comment that I wanted to give a general reply to. (Edit: To be clear I'm not involved with this project, or a google employee for that matter)
- Pyxl101 7y ago
  When a new project is proposing to do something that's been done before, potentially multiple times, I think it is reasonable to want to understand how the new project will compare and contrast to work that's already been done. Asking "How will this project compare to <existing implementation>?" seems quite reasonable to me. In fact, I would expect the person making the proposal to address this question as part of the proposal. It is not desirable to reinvent the wheel without good reason. It's totally fine if the reason is something like: we want to use a different license, or a different development process, or a different coding style, or whatever. Whatever the actual motivations are, clarifying them up front will be good for everyone.
- nine_k 7y ago
  This is reasonable. A simple answer like "we are not starting with musl because ..." would be quite welcome, though.
  - yc12340 7y ago
    I assume, that Google's answer would sound like this: "We are not starting with musl, because we want to own all our software the same way people own hamsters or dogs: care for them, feed them, train them, single handedly decide whether to castrate or kill them. We don't need the risk of owning something we don't fully control." As far, as I am aware, musl isn't owned by public company or foundation. Google can't exert desirable degree of control over musl project, because there is no board of directors to buy their way into, — it is all maintained by one guy (and chattel slavery is outlawed in US...)
    - AstralStorm 7y ago
      Bribing or hiring the dev can still work.
- g82918 7y ago
  I agree this with as some one working at a company that avoids most open source like the plague. But in this particular case, the code quality in musl is pretty good. I haven't had an issue with it so far, and after trying to upstream some things in glibc I have to say I can't see why they wouldn't use musl.
  - rurban 7y ago
    Well, there are still huge holes in musl. towlower e.g uses a linear search. my binary search made it 2x faster, but we still have to wait for that to appear. Implemented Unicode standards are years old. You don't know which standard is implemented. Still many problems with dynamic linking. And: musl is POSIX. POSIX blocking IO should not be used in a modern OS or language. You can never guarantee concurrency safety with POSIX. This is the Fuchsia problem. They went for unsafety and POSIX compat. L4 ditto. Microsoft's Singularity not. A modern libc would need to get rid of blocking IO and implement strings as utf8 only. There's no need for anything else. The C standard made too many mistakes.
    - saagarjha 7y ago
      I don't see how you're planning on writing a libc that chooses to interpret the C standard as full of "mistakes" without writing an entirely different language.
      - rurban 7y ago
        I'm already maintaining the libc security extensions, which are in the standard but all libc's refuse to implement. I also added the missing string API's for the wrong wide API, but thinking about adding the proper u8 api instead. Nobody really should use the wide (UTF-32/UTF-16) api. But the C standard committee is somewhere completely else. There are not even aware of the problems, even if I'm getting interviewed from time to time, so that they will change their attitudes towards insecurity and functionality. searching for strings should be a pretty common usage pattern, but it's still not implemented. coreutils or grep or sed would like to use it, I'm sure. But not 10x slower as the current multibyte patches. Which I'm also maintaining btw. for coreutils. Non-blocking IO into the standard? I only know about libuv which was fine before the drama. Despised it since then, but I'm still using it. The standard mostly needs to provide a namespace to block blocking IO if someone needs a concurrency-safe subset. Then the unhealthy need for POSIX compat can be circumvented somewhat. And gets() can be finally.
- anth_anm 7y ago
  > There are tons of valid reasons to make a new project, and for most of them you don't want to say it out loud. There are also a few that are just scummy and self-serving. Since google is involved, I immediately assume the worst.
  - dang 7y ago
    If you can't stop posting flamebait and unsubstantive comments to HN, we're going to have to ban you. We've asked you repeatedly and you've continued to break the guidelines repeatedly. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
    - anth_anm 7y ago
      You can keep posting the guidelines all you want. I don't bother trying because I know that you will just pop up with the word "unsubstantive" for anything you don't happen to personally think is good.
- hinkley 7y ago
  Yes, there are reasons to rewrite a library, but people expect you to tell them why you are rewriting it. So... why are they rewriting it?
MichaelSalib 7y ago
Which do you think looks better on your promotion packet? 1. Write a new libc and get it used all over Google, generating lots of users automatically; or 2. Shepard Google's adoption of code some other folks wrote.
- chandlerc1024 7y ago
  Either one looks completely fantastic, I assure you. I have helped people on my team write promotion packets in both forms. Both worked perfectly well.
- DannyBee 7y ago
  I don't think this level of cynicism is warranted However, as the person who actually runs the relevant promo committees for the org, both would be fine, i'd care a lot more about why the choice was made.
  - izacus 7y ago
    I love how your repose with 1st hand experience is downvoted as opposed to a pointless jab of the original poster :)
    - finnthehuman 7y ago
      Googlers have the experience but not the distance to meaningfully comment on google.
      - DannyBee 7y ago
        Sorry, but this is just BS through and through. Painting 100k people with a brush of "can't be objective" based on no data is probably the most biased think i've actually seen in a long time.
        finnthehuman 7y ago
        I’ll paint 7 billion with the “proximity changes perception” brush.
mda 7y ago
Instead of resorting to theoretical rhetorics on meta issues on the given proposal,Rich Felker could ask this question as well. But he did not, I am actually quite disappointed with his response, it looks unprofessional.

kbumsik 7y ago

Doesn't google use newlib already?

altmind 7y ago
Wait, weren't they using Bionic on android?
- murderfs 7y ago
  Google is too big for the left hand to know what the right hand is doing. Various parts of Google use bionic, newlib, musl, and multiple forks of glibc.
  - hinkley 7y ago
    So does that mean all that propaganda about them having a monorepo is bullshit?
    - murderfs 7y ago
      Android (bionic), Chrome (forks of glibc and newlib in NaCl), ChromeOS (glibc with patches), and Fuchsia (musl) are all in separate Gerrit repositories. The monorepo for Google proper probably contains all of these, plus the libc used for Google.

nemothekid 7y ago

What does “LLVM implementation of Libc” mean here? Does that just mean an implementation of libc maintained by the LLVM project?

fourier_mode 7y ago
Yeah, I am also confused. Isn't there Clang already? (Please correct me if I am wrong)
- wyldfire 7y ago
  The C library is generally considered a part of the OS and the overall toolchain. Clang is "merely" a C compiler. The C library provides implementations for the platform-neutral language functions like "open/read/write/close/ioctl/dup/dup2" and more.
  - fourier_mode 7y ago
    So implementations of those subroutines are written in assembly lang?
    - wyldfire 7y ago
      No, it's very common for them to be written in primarily C. But I'm obligated to mention (as a member of the Rust Evangelism Strike Force) that you don't have to write them in C. ;) Many implementations do have performance-critical parts be target-dependent and implemented in assembly. IIRC glibc has more of this than musl/newlib/bionic, but it's been a while since I last looked.
      - fourier_mode 7y ago
        > No, it's very common for them to be written in primarily C Then can't you just get the LLVM IR out of it using Clang?
        7y ago

m0zg 7y ago

This sounds like Android NDK development will get quite painful in a few years. It previously included a botched libc++ missing a bunch of features. This is libc, but I imagine subtle feature disparities and incompatibilities there will cause plenty of pain as well.

uep 7y ago
There are already some real pain points with bionic. I primarily write Linux code that sometimes has to run on Android (notably not any other UNIX). I've run into tons of problems porting to old versions of bionic, despite the kernel being the same.
mrpippy 7y ago
I don’t think the Android team is the interested party here, given the original poster’s interest only in x86_64
- m0zg 7y ago
  Google doesn't do "only x86_64". Even when I was there (years ago) at a minimum all software had to run on PowerPC as well, and non-portable assembly/intrinsics required explicit approval from "Eng Elders" and a portable fallback. I wouldn't be surprised if ARM compat is also required now.
  - saagarjha 7y ago
    That might be true, but the email posted here explicitly lists x86_64 as the architecture they are targeting initially.
jdswain 7y ago
In my experience NDK development is quite painful now. Most of my NDK work is a little old now but I found the debugging process quite painful. This was for C++ code that was running on other platforms so almost all of the debugging was due to platform incompatibilities or bugs. But I did notice in the linked article that they are only targeting x86_64, so I don't think this libc is targeted at Android.

catern 7y ago

One of the replies is this email from an LLVM developer working at Google: http://lists.llvm.org/pipermail/llvm-dev/2019-June/133315.html http://lists.llvm.org/pipermail/llvm-dev/2019-June/133315.ht... I can't say it looks particularly good for someone at Google to be using their moderator powers on this unrelated-to-Google list to tell Felker not to include parts of his message which reflect badly on Google.

scarface74 7y ago
I see nothing wrong with this. The last thing you want in a technical discussion are petty personal attacks. The moderator sounds a lot like something you would see dang say on HN.
- catern 7y ago
  I don't think they're petty personal attacks. They are important concerns which I share. Regardless, since this particular member of the project has clear incentives to be biased in favor of Google, they should have recused themselves, and asked another member of the project/moderator of the list to judge whether the message was appropriate.
  - shereadsthenews 7y ago
    Chandler is as related to the person responsible for putting musl in Fuchsia as are any two randomly selected software developers from anywhere in the industry. Having to disqualify someone from moderating a mailing list based on the union of all the conflicts of interest of tens of thousands of unrelated people is very silly.
    - catern 7y ago
      The conflict of interest is not related to putting musl in Fuschia. The conflict of interest is created by Chandler's employment by Google. Someone paid by a company should not be making moderation decisions about comments related to that company.

Sir_Cmpwn 7y ago

Copy-pasting a comment from "cup" on lobsters, because I think it's on-point: >1000 times yes. If Google wants their own libc that’s their business. But LLVM should not be part of their “manifest destiny”. The corporatization of OSS is a scary prospect, and should be called out loud and clear like this every time it’s attempted. https://lobste.rs/s/dgmoyi/response_libc_llvm#c_hg8dvg https://lobste.rs/s/dgmoyi/response_libc_llvm#c_hg8dvg

favorited 7y ago
There has been huge corporate involvement and investment in LLVM from its earliest days as anything beyond a research project. When Apple open-sourced Clang, or their C++ standard library implementation and contributed them to the LLVM project, was that something to be afraid of as well? It seems to have gone pretty well.
- svnpenn 7y ago
  Fair point, but Google is not Apple. Google has a history of railroading standards to suit their needs, to the point where the standard becomes "what Google is doing". See the Chrome project.
  - kevin_thibedeau 7y ago
    Uh... Firewire.
  - flukus 7y ago
    Apple ignores standards in the first place and opts for a completely closed and proprietary ecosystem wherever possible (see vulkan support and many others). They're both out to screw you but at least google does a bit of romancing first.
  - Conan_Kudo 7y ago
    There are plenty of examples on Apple's side, too. To name a few: Floppy disks. CDROM. USB. Firewire. WebKit.
- 7y ago

fourier_mode 7y ago

I think its primarily because Chris Lattner[0] is at Google now. I wonder even if they pull out this effort how much of that would still be maintained if Lattner changes jobs. [0]: https://en.wikipedia.org/wiki/Chris_Lattner https://en.wikipedia.org/wiki/Chris_Lattner

favorited 7y ago
Chris Lattner said that he has nothing to do with this project http://lists.llvm.org/pipermail/llvm-dev/2019-June/133274.html http://lists.llvm.org/pipermail/llvm-dev/2019-June/133274.ht...
theresistor 7y ago
No. Chris is working in the AI/ML space, and specifically commented in this thread that he is not involved with this project.
CUViper 7y ago
FWIW Roland McGrath, the original glibc author, also works at Google. But of course not every possible person will be involved here, no matter how relevant.

jasonhansel 7y ago

Filed under "This is why we need the GPL"

flukus 7y ago
This. I definitely sounds like they want a libc that they are free to fork and add proprietary incompatibilities to, which they can do with the GNU libc (under LGPL) as long as they don't want to distribute it, so we can assume they want to distribute it and that this is a further closing of their consumer tech like android. I would've though musl would be a better starting point for them.
taneq 7y ago
How does the GPL help with this? Google could build their own libc replacement, and if they push it hard enough to gain widespread community support, it could become the (or at least, a) new standard. At that point Google is free to do whatever they want with it as long as they make it useful enough that the community won't abandon it. The GPL just means that it could be forked later, but if Google is still putting significant resources into developing the main branch then no-one will switch to the forks.
izacus 7y ago
GPL is literally made to allow forks like these, what exactly did you mean with your post?!

ajxs 7y ago

> A corporate-led project does not have to answer to the community, and will leave whatever bugs they introduce in place for the sake of bug-compatibility with their own software rather than fixing them. This speaks volumes very clearly. This highlights an immense hazard. Enterprise scale companies contributing to open-source is a fantastic thing, but enterprise scale companies thrusting their own proprietary libraries onto the open-source world is not. I'm already actively avoiding becoming beholden to Google in my work as it is already, let alone in the world where important software uses a libc written by Google. If you're not concerned by this, refer to the immense power that Google already wields over the extremely ubiquitous web-standards through the market dominance that Chrome has.

jchw 7y ago
(Disclosure, Googler here, not working on projects related to any of the things being discussed, speaking in a purely personal capacity.) I hate to disagree with Rich Felker. I think most people have a lot of respect for musl and the amount of effort it took. But, I disagree with the basic conclusions. For one thing, I don't think it's fair to compare this to Chrome, an end user product. libc is a fundamental piece of architecture, like (edit: as in, fundamental in stacks that they are used in, to clarify) protobufs, gRPC, Kubernetes, etc. I heavily sympathize with folks uneasy about the Chrome situation - like I've mentioned maybe hundreds of times now I use and recommend Firefox personally, but I feel this particular discussion will become more productive when that stops being a focal point. Their other points are not really related, but I disagree with them also. I don't think there's a good motivation to keep bugs for backwards compatibility: Google runs non-Google software, too; keeping subtle incompatibilities could be a disaster. Also, Google has a pretty good corpus of code to test their libc on, so I think they have a unique opportunity to fast-track a pretty battle-tested libc (I did not know of this project before this Hacker News post, so this is purely a hot take, but for now I stand by it.) No idea what their plan is, but open governance would probably work wonders here. After all, LLVM has plenty of corporate contributors, and it seems to do fine.
- stochastic_monk 7y ago
  I think trying to work with protobufs by analogy here fails because the scopes of power are so vastly different.

xvilka 7y ago

Quite relevant then: "Why I'm donating $150/month (10% of my income) to the musl libc project"[1]. [1] https://andrewkelley.me/post/why-donating-to-musl-libc-project.html https://andrewkelley.me/post/why-donating-to-musl-libc-proje...

gok 7y ago

This would have been a much better email without the needless stone throwing from one libc maintainer's glass house.

cpeterso 7y ago

I'm surprised the proposed initial scope would include x86-64 but not ARMv7 or AArch64. That suggests Google's motivation here is their server software, not mobile devices running Android or Fuchsia. Also, portability and consistency is an important attribute of a libc. Supporting multiple architectures from the beginning would help shape a portable design. For example, the original target processor for Windows NT development was the Intel i860 specially because it wasn't the standard i386 architecture. Fun fact: the Intel i860 was codenamed "N10" aka "N-Ten", which may have been the origin of the Windows "NT" moniker.

deleted 7y ago
[deleted]
floatboth 7y ago
> mobile devices running Android They already have Bionic! > or Fuchsia They already have a weird fork of musl that Rich described in the email! typical Google "Not Invented in this Team" approach..
- pjmlp 7y ago
  Bionic was a very hard child birth. Check NDK roadmap to see how much years (yes years) has taken them between Android and NDK releases to provide a proper libc.

lone_haxx0r 7y ago

Google are going to make it with or without LLVM's cooperation. If it's done with LLVM, it's highly probable that LLVM won't let them extend,extinguish libc. Google's code will be carefully evaluated by a relatively neutral party: LLVM. If Google does it alone, they will write whatever code they want. They'll be free to extend, extinguish libc.

saagarjha 7y ago
Why do you think Google has the ability to EEE libc by themselves but somehow be voluntarily reigned in by LLVM?

rurban 7y ago

Rich is of course totally right. There's not only the botched musl in Fuchsia, also Android Bionic, which is a libc written in C++. It's utterly unusable. They had a better, older, safer libc implementation based on some HP code, but that is gone. Some of Rich' points are moot though. Google already have their own libc's, in Android and Fuchsia with enhanced security measures. Similar to Huawei or Intel or Cisco. Better than Apple's freebsd libc. Now reaching out to the community to outsource their development efforts under llvm is a good thing. There's obviously a need, musl, bsd and glibc cannot address. I'm sympathetic to that.

wbl 7y ago
And the need is? I read the thread and I still don't know.

_pmf_ 7y ago

Make vs buy decisions work differently for companies with virtually unlimited buying power. The absolute control that doing it yourself provides cannot be bought. And Google has to care only the platforms it wants to support (as some guy at BSDcon said, you have Windows, Linux Intel/AMD/ARM and rounding errors). Do you think the 100000 cookie cutter game developers for Android will stop developing for it because the platform libc has subtle issues? They rightfully don't give a shit.

eqvinox 7y ago

His 3rd argument is absolutely and utterly compelling and a total knockout against Google's approach to this IMHO: > Some folks have told me that I should press LLVM to make musl the "LLVM libc" instead of whatever Google wants to do, but that misses the point: there shouldn't be a "LLVM libc", or any one library implementation that's "first class" for use with LLVM while others are only "second class". If they want to create a great libc - or fuck up another libc - they're much welcome to do that, but they need to go do it outside of LLVM. Dumping a libc on LLVM is just (1.) appropriating credibility from LLVM, (2.) sticking others with the maintenance bill, and (3.) horseshit.

qalmakka 7y ago
This, and I also strongly argue that compiler implementations should be untied from standard libraries, and this also applies to libc++/libstdc++. A compiler shipping its runtime is a bad idea that totally screws up compatibility (just look at how painful it is to use MINGW + C++ on Windows because it can't link with VS's C++ Runtime - I understand that MS headers are crap to parse, but with a bit of effort clang now does it fine), and ties them together and prevents reuse. In an ideal world shipping a runtime should be the OS's role, not the compiler's, and the compiler should make use of whatever the platform ships, instead of having N compilers shipping N different, incompatible libraries. The interface between the library and the compiler would then become a true language standard, instead of "library for compiler version X.Y.Z"
- nly 7y ago
  Raymond Chen wrote about this in his 2014 blog post "Windows is not a Microsoft Visual C/C++ Run-Time delivery channel" https://devblogs.microsoft.com/oldnewthing/20140411-00/?p=1273 https://devblogs.microsoft.com/oldnewthing/20140411-00/?p=12... Stable ABIs are good, but the ability to break it and iterate/improve is even more so. GNU platforms are currently stuck with a shitty std::string implementation (32 bytes but only 15 bytes of SSO capacity) because breaking ABI means recompiling your entire Linux distro.
  - Asooka 7y ago

mehdix 7y ago

These are the highlights IMO: > Avoiding monoculture preserves the motivation for consensus-based standards processes rather than single-party control > there shouldn't be a "LLVM libc", or any one library implementation that's "first class" for use with LLVM while others are only "second class".

rjvehn 7y ago

Well glibc kind of proved that individual maintainers can be just as bad for stewardship[0]. [0]: https://news.ycombinator.com/item?id=7914022 https://news.ycombinator.com/item?id=7914022