7 ms·
Genuine question: since these are core utils, and probably used billions of times every day, is anyone actually going to switch to this version? I see that the
by austinjp 3y ago
Genuine question: since these are core utils, and probably used billions of times every day, is anyone actually going to switch to this version? I see that the intention is for this to be a drop-in replacement, but some options and behaviours are still different.
To clarify, I'm not intending this as a negative comment. It's an impressive project, and aiming for cross-platform sharing of scripts seems a worthy goal. The graph of progress against the GNU test suite is neat and encouraging. However, I can't imagine anyone in Unixy lands moving to this - although I may be wrong there - so it feels like a "compatibility" project for MacOS and Windows. I'm not familiar with the situation on MacOS but how bad is the compatibility issue? Similarly, on Windows what's the situation with WSL or VMs or even Cygwin? Is performance the issue?
Do people actually want this, is there a market for it? It's got 17.8k stars so seemingly so? Again, I'm not intending to cast aspersions, just trying to understand the audience. This is clearly different from a "for the fun of it" side project.
- noirscape 3y agoWell for one, uutils is in rust. That alone can be pretty desirable for some people because that infers assumptions about memory safety, especially compared to the coreutils which are in C. The license stuff is what people focus on, but to me that is much more interesting. C is a pretty decrepit language and while I don't care much for the "rewrite it in rust" cult, the coreutils are exactly the type of program rust excels at - a non-iterative target that doesn't change often and can be made with "best practices" in mind. Not having to deal with the utterly inane amount of dead architectures that GNU projects inherit probably helps them too on that end.
- lohnjemon 3y agoWhat kinds of memory safety bugs do you really care about in coreutils? Genuinely curious. Given how mature and well defined the GNU Coreutils are, how small their scope is, how they are used, I really don't see the supposed security upside here. There simply has to be a better reason to me, than "Rust good".
- VBprogrammer 3y agoWasn't it just a couple of years ago that we had shell shock? According to Wikipedia that bug was introduced in 1989. Of course, that wasn't a memory safety issue (and it's probably important to reiterate that memory safety doesn't mean free of bugs and / or exploits). But it does demonstrate that issues can exist for a long time in mature code without anyone noticing.
- ReleaseCandidat 3y ago> Wasn't it just a couple of years ago that we had shell shock? Yes, but Bash (or any other shell) isn't part of coreutils. https://en.wikipedia.org/wiki/List_of_GNU_Core_Utilities_commands https://en.wikipedia.org/wiki/List_of_GNU_Core_Utilities_com...
- KingLancelot 3y ago[dead]
- wongarsu 3y agoSome CVEs have happened in the past [1]. None of them memory issues, but a couple that seem unlikely in idiomatic rust or are much easier to prevent in rust. Specifically, integer overflow is much easier to correctly handle in rust, making bugs like CVE-2015-4042 less likely, and correct handling of multibyte strings is basically enforced by the standard library, making issues like CVE-2015-4041 very unlikely in a rust implementation 1: https://www.cvedetails.com/vulnerability-list/vendor_id-72/product_id-5075/GNU-Coreutils.html https://www.cvedetails.com/vulnerability-list/vendor_id-72/p...
- lohnjemon 3y agoSo this one integer overflow in sort, a command which is never ran as root is an issue somehow, because it can cause a denial of service(it crashes)? Am I missing something here. Can I use this to exploit someone's machine? I can search uutils/coreutils for "overflow" and get way more hits, I don't see how this is a rational thing to be afraid of within GNU Coreutils considering it's a collection of tools, that have been developed and maintained for decades and used by millions over that time period. https://github.com/uutils/coreutils/issues/1420 https://github.com/uutils/coreutils/issues/1420 https://github.com/uutils/coreutils/issues/886 https://github.com/uutils/coreutils/issues/886 https://github.com/uutils/coreutils/issues/5149 https://github.com/uutils/coreutils/issues/5149 To be clear, I don't see any problems personally with any of these issues, they don't seem very exploitable to me. However, I think that relying on Rust to be the bastion of safety merely because the name "Rust" is mentioned is nothing but a fallacy. To me, logic bugs are the far more egregious category in something like coreutils. Me making the assumption, that something works the way it's documented, but doesn't can lead to horrible things down the road. Much more so, than any integer overflow crash could ever dream to.
- smt88 3y ago> Differences with GNU are treated as bugs. That tells me that "some options and behaviours are still different" is also being treated like a bug. Even if this is only used by huge companies like Meta or for new projects, it will still justify its existence..
- YoshiRulz 3y agoI use NixOS so it's feasible for me to use these as true drop-in replacements when they're done. And the reason I'd want to do that is for hardening, as the sibling commenter suggests. The fact it's released under MIT instead of Apache (or GPL) does worry me though.
- Snow_Falls 3y agoYeah, I'm not a fan of the rust ecosystem trying to move everything from FSF-style free/libre licenses to permissive licenses.
- m4rtink 3y agoYeah - memory safety, why not (though its not a silver bullet) but why change te license to one that can be quite dangerous over time for something this important?
- silon42 3y agoSomeone could release it with a relicense to GPL (and maybe LGPL if dynamically linked).
- dartos 3y agoYou can’t just release someone else’s BSD code under GPL.
- dspillett 3y ago> so it feels like a "compatibility" project for MacOS and Windows I get a similar feeling. This is more attractive than ports like cygwin as a solution because if you have this in every place you can be more sure your environments match. With ports there will be more delay getting changes/fixes in different environments, at least in theory, than with a solution that is cross-platform as a core goal. The other main options, running Linux in a VM (directly or via WSL2) adds an extra layer of friction. A cross-platform solution is more likely to match bug-for-bug in different environments, which can be as important as matching feature-for-feature: if something is going to fail out there it will fail the same way in your dev/test environments. Also some will want to use it from a licensing PoV. Many¹ find GPL related licences bothersome and use the GNU coreutils because there isn't (until this matches their requirements) an alternative. Similarly, the language might be an attraction to some, either ideologically or because they might want to dig into the source, though this is more of a factor for projects developing new features rather than trying to be drop-in replacements. I expect the attraction to be relatively niche though, people won't start using it as a drop-in for the GNU tools en-mass until, for instance, popular distributions use it (or a niche distribution using it becomes more popular for some other reason). ---- [1] I am not one of that many, but they are many so this is a notable consideration.
- pletnes 3y agoWhere I’m working, WSL doesn’t work (well), VMs are banned, and personal experience with cygwin is awful (that was on win7, but still). Everything that helps me develop software on windows which runs on linux (of course!) is warmly welcome. I’m sure there are a thousand similar but different enterprise environments like this. At home / solo dev it makes no sense, you’d just install ubuntu / get a macbook and get working without any fuss. Docker works but it makes test runs 100x slower (yes, I measured, it’s not made up).
- VBprogrammer 3y agoIf your test runs are slower by two orders of magnitude then something is going wrong. For example, I've had problems on docker for Mac where accessing lots of (python) files in a volume mount was slow. I had to use some beta setting at the time which was far better (though I had to do a OS upgrade to use it, which also entailed some IT nonsense). The overheads of docker are there but if it was that bad people wouldn't use it.
- pletnes 3y agoFile IO seems to be the problem. I’m guessing antivirus + virtualization overhead. Don’t see what I can change about that (use linux, sure). Docker is still useful, it’s great for sharing a complicated setup across machines.
- prewett 3y agoIf I recall correctly, NTFS updates a file's access time (or some such) every time anything happens to it, so by default Windows' file I/O is slow. (Antivirus, of course, will not be helping that.) Once upon a time you could disable access time updating in the registry, although I assume if you had those kind of permissions you wouldn't be having this problem.
- ParetoOptimal 3y agoIt is that bad. I use a Linux VM with docker from OSX and coworkers use docker with OSX. Their compiles take 2m, mine take 30s.
- ranguna 3y agoI don't see why I wouldn't switch.
- cdogl 3y agoI mostly agree with your comment, but the source code of of many GNU coreutils is quite gnarly. It's ancient code (from my reference point as a 35yo) developed at a time when coding style was different and a much smaller community maintained it. I think it's important for free software that people coming into the community are enthusiastic to maintain it. It took the wind out of my sails a little when I realised the GNU code base, while it produces critical tools I use every day, is written in a way that I found extremely (unnecessarily) terse and "clever". Tracing how different combinations of flags are handled is not much fun. Documentation is helpful for the user, less so for the tinkerer who is trying to understand the stack. If this project manages to hit parity with GNU coreutils, and my distro(s) provide support, I'll switch to it purely on that basis.
- Throw839 3y agoThis! I am horrified to touch anything old from GNU!
- viraptor 3y agoThis is true. I've tried to read some gnu code in the past (tar) and patch another (df) and... ran away instead. It's doable, but so messy I'd rather write my own very specific command, than try improving one of the old-style gnu ones.
- dmd 3y agoI've interacted quite a bit with some of the authors, and when I've asked "why on earth did you do it this way" the answer is generally some form of "well, it saves nearly 6 bytes on disk in the source code! disk isn't free, son". It's a different mindset and one that is no longer useful.
- bayindirh 3y agoWhen I asked a graybeard why the variable names were so short, he said that "longer names affected compile duration very severely in the past, so this is why we used the shortest name possible". While user facing programs are not faster by any means, computers got faster in some aspects after all.
- 0x1ceb00da 3y agoCross compiling rust programs is much easier, so if you end up on a niche system that doesn't come with coreutils, you can install your own version.
- bayindirh 3y agoThis is “just” for changing the license of the coreutils so companies can use a compatible version and do not share the source of they get to modify them. In other words, it’s mostly for people who dislike GPL because of its virality. More extreme people may see it as a major milestone in “killing” GPL.
- wirrbel 3y agoIt would definitely not be my motivation. I would not modify the gnu coreutils in many cases because they lack tests and are sometimes written in pre-ansi c. At some point the GNU project focused their attention more on leading “gnu + Linux” debates and not so much on developing a stable and secure OS. So now it’s legacy cruft.
- bayindirh 3y agoThen, why not license uutils in GNU/GPLv2+ (or v3+) again as a spiritual successor to GNU Coreutils, but use a "permissive" license like BSD to allow "free for all" incl. but not limited to closing the source? What happens if the uutils team says that they're not releasing the source of the latest version but the $CURRENT-5 from now on, moving to a closed source, open baggage model? What prevents them from pulling an effective EEE?
- Sanzig 3y ago> What happens if the uutils team says that they're not releasing the source of the latest version but the $CURRENT-5 from now on, moving to a closed source, open baggage model? What prevents them from pulling an effective EEE? I don't follow this argument. This can happen under GPL as well. Nothing stops the copyright holder from relicensing future versions of the software under a different license. The existing versions already out there under GPL/LPGL/MIT/BSD, there's no take backs, but the copyright holder is free to do whatever they want with future versions. What would happen in your hypothetical scenario is that everyone would get really really angry with the uutils team and the latest open source version would get forked by the community. The proprietary one would wither and die, because who in their right mind wants a proprietary set of coreutils? Fundamentally, the choice between copyleft and permissive is simply if you care that someone takes your software and incorporates it into a proprietary package. Clearly, the uutils team doesn't care if eg. Apple makes a proprietary fork of uutils for OS X. And that's their prerogative.
- wvh 3y agoThe people who originally wrote those utils are slowly disappearing, and with them the knowledge and ancient C hackery skills. It might be a good idea to have a new generation that can take some of that knowledge and interest and carry it forward into the future.
- tahnyall 3y agoThat'd be great if it were migrating those utilities to modern C using improved modern coding practices but re-writing in some goofy language like rust is dumb.
- thecodedmessage 3y agoWriting in Rust when appropriate is improved modern coding practice.
- ParetoOptimal 3y agoIf it were GPL licensed, I might try switching because I'm more interested in learning Rust than learning more C.
- nonameiguess 3y agoProbably makes sense mostly on Mac. Right now, it's common for developers using Macs to install GNU coreutils from Homebrew as a first step to ensure scripts and build systems that were originally written for Linux still work. Macs give greater ease of corporate device management, so Macbooks are a fairly typical compromise for companies that need to give developers a Unix environment but still want to maintain the level of control over endpoints you'd get with Windows. I'm sure there's desire out there for a pure-Rust close-to-POSIX userspace you can put on top of Linux, but I don't exactly see a whole lot of progress toward that. At minimum, you'd need coreutils, findutils, diffutils, tar, grep, sed, compression libraries, crypto libraries comparable to OpenSSL or GnuTLS, an editor, a POSIX shell, an init system, a service manager, system-wide DNS and DHCP. Going beyond pure POSIX to a more practical server distro would probably include a PGP implementation, a package manager, sudo or equivalent, ssh, iproute2, probably many other things I'm forgetting. uutils seems to give you findutils and coreutils. ripgrep is not a drop-in replacement for grep. I'm sure there are other "rewrite it in Rust" projects out there I don't know about, but low-level system utilities aren't exactly the gloryland most developers are interested in rewriting and POSIX includes a lot of stuff. Frankly, I don't think this is a realistic terrain for any single organization. GNU itself took over a decade to get a practical distro and that was only because other developers provided a kernel, package managers, boot loaders, crypto, and what not. And the GNU project is not really a single organization. It encompasses many people from many different employers. Systems software in general wasn't as fragmented. If you were working in the space at all and cared about being cross-platform and collaborating across organizations in the 70s and 80s, you were using C. Everything else was proprietary. If we're ever going to get something comparable but in Rust, it's not going to be from a single Github org like this.
- asteba 3y agoIt makes a lot of sense in embedded devices, where the GPLv3 license cannot be used for commercial devices which are using signed images (secure-boot / verified-boot) for security reasons. I expect a lot of adoption there.
- giancarlostoro 3y ago> I'm not familiar with the situation on MacOS but how bad is the compatibility issue? Similarly, on Windows what's the situation with WSL or VMs or even Cygwin? Is performance the issue? If I remember correctly, Mac has any GNU utility from before they all switched to GPL v3. Which was decades ago now.
- duped 3y agoThe canonical GNU packages are pretty arcane and difficult to build/understand (*), so alternatives with modern tooling/languages should be welcome. But also, things like BusyBox and ToyBox are very popular alternatives which contain non-standard ports of only a subset of coreutils. A complete and mostly compatible set of coreutils would be more popular than either, but they also have some different constraints that Rust builds make difficult (eg: binary size). * nb4 "it's just configure/make/install what do you mean" consider if you want coreutils without glibc, cross compiled, or want to bootstrap your environment without any of them. The GNU ecosystem is "easy" to build/use within the GNU ecosystem, which is not unopinionated about what that looks like and how it works.
- deleted 3y ago[deleted]
- apatheticonion 3y agoI use them on my Windows machine from PowerShell to make using the Windows command line more tolerable. You can find the GNU coreutils for Windows on sourceforge but it always feels like spyware when I download something from that site, plus I don't know how old they are or if they are actively maintained. With this rewrite, I like that I can download them from the repo's releases page. There is an issues page for problems and I can see the project is actively worked on. Installing them as binaries is simple and, this goes for all Rust/Go projects, I actually know how to compile them (sorry, I know C/C++ is great but between gcc||msvc||clang, missing deps, ./configure ./install and make - I rarely have a pleasant experience compiling something in C/C++).