27 ms·
Cross-platform Rust rewrite of the GNU coreutils
- faaef 11y agoIt certainly beats using NodeJS. EDIT: Yea, this was meant as a reference to the coreutils-in-nodejs post on the frontpage.
- deleted 11y ago[deleted]
- plugnburn 11y agoNodeJS is for applications, Rust is for system programming. How does one beat another? It's akin to saying "ARMv7 assembly beats using Java for Android programming".
- grok2 11y agoHe is probably referring to this: https://news.ycombinator.com/item?id=11334597 https://news.ycombinator.com/item?id=11334597 -- discussion on CoreUtils in Javascript!
- plugnburn 11y agoAh, understood. I think neither of these beats actual coreutils and their support base (still under impression of Termux capabilities...), at least not yet.
- tibiapejagala 11y agoParent comment probably refers to [1] submitted earlier today. [1] https://news.ycombinator.com/item?id=11334597 https://news.ycombinator.com/item?id=11334597
- midgetjones 11y agoProbably a response to this: https://news.ycombinator.com/item?id=11334597 https://news.ycombinator.com/item?id=11334597
- pjmlp 11y agoRust is a general programming language, no need to constraint oneself to a single domain.
- plugnburn 11y agoSo is JS. But like I said, both implementations suck compared to actual coreutils.
- jamescun 11y agoCould be useful given the choice of MIT license rather than a GPL variant.
- mikerichards 11y agoI guess this would matter to busy-box/router type applications.
- davexunit 11y agoThey really should use the GPL and not a pushover license.
- quotemstr 11y agoWhat is it with rewrite-the-world-in-Rust people being stridently anti-copyleft?
- cyphar 11y agoI don't understand it either. It's as though they want to ignore the history of free software, which simply wouldn't have happened without copyleft.
- kps 11y agoTautologically, it wouldn't have happened as it did, but free software long predates the FSF.
- cyphar 11y agoSure, but the FSF and the GNU project were reactions to proprietary software becoming more common. Free software wasn't recognised as such until Stallman saw the issue and decided to act. Saying that "free software existed before the FSF" is ignoring the fact that free software would've died almost entirely without the FSF and GNU.
- 11y ago
- masklinn 11y agoRewriting coreutils is neat, but a project I'd really look forward would be a strict POSIX base expanded with warnings or errors on valid but risky constructs e.g. echo -n. Even more so if it included a shell (with static analysis of useless uses and dangerous patterns). That would make writing cross-shell scripts much easier.
- gjem97 11y agoOk, I'll bite. What's wrong with "echo -n"?
- cbr 11y agoThe FreeBSD manpages have: Note that the -n option as well as the effect of `\c' are implementation-defined in IEEE Std 1003.1-2001 (``POSIX.1'') as amended by Cor. 1-2002. -- http://www.freebsd.org/cgi/man.cgi?echo So there's probably some system (or shell) out there where -n doesn't work?
- hyperbovine 11y agoOn some Unix variants that will print "-n" followed by a newline. On others it will print the empty string. http://steve-parker.org/sh/echo.shtml http://steve-parker.org/sh/echo.shtml
- masklinn 11y agohttp://www.in-ulm.de/~mascheck/various/echo+printf/ http://www.in-ulm.de/~mascheck/various/echo+printf/ is the raw data version. POSIX defines echo as only taking string parameters and no options, but notes that behaviour facing `-n` is implementation-defined. BSD and GNU echo implement `echo -n` as not printing a trailing newline, but `echo` commonly calls to a shell builtin which may or may not follow that behaviour (and may switch behaviour depending on whether the shell is in "sh mode" or not), so `echo -n` could print `-n<newline>` or nothing whatsoever (empty string and suppressed newline) depending on the utils set, the shell, and the shell's runmode. GNU echo also supports -e, -E, --version and --help options, and much like -n shells and other utils set may or may not support these. For instance on my machine (OSX 10.11) * zsh (builtin) interprets -e, -E and -n as options (but not version or help) * bash (builtin) also does, except when invoked as sh in which case it does not and all parameters are literal (this may also apply to zsh) * dash (builtin) interprets -n, but none of the others. bash note may also apply to it. * BSD echo interprets -n but will print -e, -E, version and help literally * GNU echo interprets all of the above So if you use echo with any non-literal parameter, or with one of the parameters listed above, in a script you distribute to un-controlled third-parties as an sh script (rather than e.g. a bash or zsh script specifically) you will suffer from portability issues. And that's just for measly trivial echo (and incidentally why you should always use printf rather than echo in scripts you try to make portable).
- pjc50 11y agoThe rationale talks about easy portability to Windows, but I'm skeptical - the underlying API and filesystem is different enough that this is likely to cause problems. '\' vs '/' to start with.
- deleted 11y ago[deleted]
- Someone1234 11y agoWindows supports forward-slash fine. Has for at least 25 years. No need to change / to \ or visa versa. Now obviously any Windows port would have to call Win32 APIs instead of POSIX APIs (unless you're writing for a Cygwin-like layer).
- tacos 11y agoYou might want to review the Xamarin bug reports as the slash alone nearly put them out of business at least three times.
- 2Xheadpalm 11y agoSupported, just but it's a second class citizen, for example, no tab completion of directories on the cmd prompt when using forward slash. Some commands will not except it; e.g. type c:\home\foo.txt -> cats it out type c:/home/foo.txt -> 'The syntax of this command is incorrect.' whereas cd will except either forward or backslash cd c:\home/foo cd c:/home/foo work as expected.
- pjmlp 11y agoJust use Powershell, it will work as expected.
- Someone1234 11y agoFirstly I was talking about API support. Since that is the topic at hand. Secondly, Cmd is deprecated. Powershell supports both slashes and "type c:/home/foo.txt" works perfectly.
- jffry 11y agoSee also today's discussion of `cash` ("CoreUtils implemented in pure JavaScript"): https://news.ycombinator.com/item?id=11334597 https://news.ycombinator.com/item?id=11334597
- ignoramous 11y agoRust is a great choice for systems programming it'd seem. Esp considering that a well-tested, battle-hardened code-base like SQLite faces problems [1] solely due to the nature of the language its written in. [1] https://news.ycombinator.com/item?id=11312918 https://news.ycombinator.com/item?id=11312918
- thegeomaster 11y agoNot sure why you're being downvoted. I don't see why we shouldn't be moving to languages like Rust given the chance, as C makes it far more difficult to write safe and correct code.
- masklinn 11y agoI expect it's because because the first part is essentially preaching to the choir, and because Richard Hipp very much disagrees with the second part[0] > Rewriting SQLite in Rust, or some other trendy “safe” language, would not help. In fact it might hurt. (see link for expansion on that matter, which is a question of tooling and testing) [0] http://blog.regehr.org/archives/1292#comment-18452 http://blog.regehr.org/archives/1292#comment-18452
- kbenson 11y agoProf. Regehr did not find problems with SQLite. He found constructs in the SQLite source code which under a strict reading of the C standards have “undefined behaviour”, which means that the compiler can generate whatever machine code it wants without it being called a compiler bug. That’s an important finding. But as it happens, no modern compilers that we know of actually interpret any of the SQLite source code in an unexpected or harmful way. At some point some popular compiler is going to make a subtle but important change to some undefined behavior that's not going to be immediately obvious as to it's repercussions, and the fallout will be massive. It boggles my mind the mental contortions people will go though to justify what is essentially an argument of "it hasn't caused a problem yet" while ignoring that it's caused many problems already, just not that they've noticed or that have affected them.
- gbersac 11y agoIs there a real need for this project ? I mean, this project has been used for a long time, could it really be better ? Do you intend to replace GNU coreutils ? Not to be mean, just a genuine question (and yes I read the Why section of the readme).
- wyldfire 11y agoHow great would it be if coreutils could get replaced with this implementation? We've seen vulnerabilities come from here before.
- im3w1l 11y agoWhat is the primary danger of bugs in coreutils? Are there any setuid programs in there that could be used to elevate? Or is it the ability to exploit shell scripts using them? Or files that when you manually process them with coreutils install a trojan?
- oldmanjay 11y agoYour last question seems reasonable, but the first is a classic tactic taken by that nerd subculture who loves to shit on things. This seems particularly true in light of your edit indicating you already know the answer.
- amyjess 11y agoHonestly, I wouldn't mind a non-GNU alternative to their stuff that isn't either a) feature-bare like Busybox or b) tightly-coupled with other software like the BSD utilities. Why non-GNU? To completely and thoroughly dispel the notion of "GNU/Linux".
- yincrash 11y agoThe reason I contributed was solely to learn how to program in Rust.
- deleted 11y ago[deleted]
- 11y ago
- z3phyr 11y agoRust is a fine language, no doubt. What worries me is the exact rewrite of the C code using the unsafe construct. Also a little out of topic; Can more light be shed by fellow HNers on the debugging tools for rust, the debugging experiences from the security research point of view?
- steveklabnik 11y agoIt looks like there are 134 instances of 'unsafe', in 23,000 lines of Rust. And it looks like a lot of that unsafe is to FFI into libc. Rust works with GDB, so you end up debugging like anything else. IDE integration is being actively worked on, and sorta-kinda works in my understanding.
- jblow 11y agoWhat is the roadmap on getting rid of the need for libc? Given how terrible libc is, I personally would make that a high priority, though I guess in Linux you can't even start up a process without libc (maybe that is a misunderstanding?), which makes the situation less clean, but at least you could get to a point where you never call back into it after entry into main.
- steveklabnik 11y agoI don't know what this project's stance is towards libc.
- jblow 11y agoOh, but I mean in Rust generally. Ideally any language that is not C would not depend on libc.
- steveklabnik 11y agoAh. Well, right now, the standard library relies on a libc of some kind. If you use only core, then there's no reliance on a libc. There has been some interest in making it easy to have stdlib without a libc; and some refactorings to std that may make it easier to do so, but those haven't come to fruition just yet.
- gravypod 11y agoI'd really love to use rust, given the large growth of projects like this. I only need an IDE to get started, but it seems like there isn't one yet. Anyone have any pointers on where I can find a good IDE for Rust? Should I just start praying to lord-JetBrains for something that works?
- theonewolf 11y agoI don't understand why you need an IDE, I believe Rust is simple enough that you can work on things in Notepad++ or Vim or Emacs.
- orbitur 11y agoAutocomplete is a godsend, I don't type nearly as much as I hit my arrow keys+tab. Whenever I have to write anything outside of ObjC (in Xcode) or Java (in Android Studio), I get annoyed very quickly.
- cballard 11y agoI tried to use Rust briefly but found that discovery was very difficult without autocomplete - I'm used to Swift's SourceKit, which is quite good if it hasn't crashed.
- gravypod 11y agoHere are the features I like in an IDE that make be very productive: - Autocompletion - Mass rename - Source formatting - Integrated debugger interface with breakpoint insertion and overlying of state on source - Integrated VCS control - Automated deploy - Error display - Automatic importing of modules - Source cleanup (Automatic loop transformation) - Automatically building my project That's just a few things that I like an IDE to have. Some provide even more features that I like. I understand that there are other tools that do the tasks better, maybe even faster, but that's not what I want. I want to be able to learn one thing, and learn how to use all of it's features well.
- coldpie 11y ago
- mapleoin 11y agoWhat are the licensing implications for this kind of work? I assume the authors used GNU coreutils as more than just inspiration. They probably read all the original code and reused some of the solutions (obviously ported to Rust). Shouldn't the derivative work still be covered by the GPL?
- steveklabnik 11y agoWell, that's a very large assumption, and it is a big part of that answer. I have sent in a few PRs to this project, and I have never done more than maybe glance at the source of coreutils, and it was for unrelated reasons. can't speak to the regular contributors, though.
- pron 11y ago"Maybe glance" might well make it derivative.
- steveklabnik 11y agoIt might. I'm hedging because I can't even asset that I've _never_ looked at it, I've been doing C off and on for a very long time. But regardless, my two or three patches aren't the worry here. Or rather, every patch is, but if it came into question, they'd be easily removed.
- epistasis 11y agoThis is very very weak grounds for any sort of lawsuit. But if having glanced at GPL source code prevents implementing similar functionalities in an entirely different language, that's a pretty darn strong argument for me to never look at GPL code again.
- ktRolster 11y agoIf you're not trying to copy someone else's product, then you're probably ok
- 11y ago
- acqq 11y agoHow big are the executables when compiled, especially compared to the C versions? I hope it's not Rust version 2 MB, C version 6 KB.
- steveklabnik 11y agoWell, that would be a statically linked Rust version vs a dynamically linked C version. Either of those languages can produce binaries of either size. Statically link a MUSL libc and it'll be bigger than 6kb, dynamically link the rust code and use the system allocator instead of jemalloc and it'll be smaller than 2MB. (Or rather, it should be, I haven't literally tried it. The smallest known Rust executable is 151 bytes.)
- acqq 11y agoThat's what I wanted to hear, could really the mentioned in Rust implemented coreutils be compiled and linked equivalently to C or not and can we see the resulting sizes? Not "if I don't use any library I can get the 151 byte binary" but exactly these coreutils that are presented. It gives a good idea if the replacements can actually be comparable replacements. The second part is then to compare the number of features implemented and the behavior.
- jabl 11y agoBy default Rust statically links the Rust runtime library, so yeah, 2 MB is roughly it. You can get around it by linking dynamically, and uutils also supports busybox style single binary + links installation (that is, check argv[0] (or whatever the equivalent in Rust is) for which code to run).
- gkya 11y ago> Many GNU, Linux and other utils are pretty awesome, and obviously some effort has been spent in the past to port them to Windows. However, those projects are either old, abandoned, hosted on CVS, written in platform-specific C, etc. I have seen such a paragraph in another projects README, IIRC a Go rewrite of standard utilities. I do not understand why a project would be obsolete because it's on CVS or is old. CVS is simpler than Git, albeit less capable. I, for one prefer it over Git for this reason, and others may do so too. Why would the end user care? And why would we care about the age of a programme if it works? Now, that said, the authors need not justify anything, they are free to do whatever they want, and I guess it's fun to code this stuff. I tried this just to play with Golang when it was 1.1. And lastly, that Makefile is really a bunch of shell scripts, and some common environment variables. There is no real dependency tracking in it, and it is easier to maintain a bunch of shell scripts than a seriously ugly and complex Makefile like this. N.b. that when I say dependency tracking I mean dependencies among input and output files of processing commands, not tasks. I guess Cargo would know how to do that, and how to not build if the build artefact is already there. It's a useless use of make. And that makefile is very GNU-specific, not complying with a project whose purpose is to be cross-platform. Also, I guess, tho I'm really unfamiliar with Rust and Cargo, if that makefile was removed, maybe on Windows they'd be able to drop development dependencies on Cygwin or Msys.
- micampe 11y agoI could see preferring svn to git because of the simpler model, but cvs? No thanks, a vcs without atomic commits is not much better than snapshot archives, maybe worse actually.
- maaku 11y agoSometimes snapshot archives is exactly what's needed.
- gkya 11y agoFor my use case, it's way more preferrable: a) SVN seems daunting and complex, tho I didn't ever dive into it. CVS is so simple and easy, a half-arsed programmer like me can actually understand it. Things like git and mercurial are way more complex. b) RCS is real handy for single files, e.g. a free-standing text file or shell script. But when the thing grows up, it is very easy to integrate the fileset into a CVS repo preserving it's history: move the ,v files to $CVSROOT/$MODULE/. c) The repository model of CVS is as transparent as it gets. d) The keywords like $Id$ are really useful. e.g. I keep my system configuration in "~/Checkouts/system-config", and I have a script that cp's the files to appropriate locations using a map file. When I'm not sure if the active config is not up to date, I can verify very easily. And I can be sure that dirty files won't be active as long as I don't expressly copy them. I know that SVN has this too, but I find CVS easier to use in general. I guess for fast paced, very active development, yes CVS is sub-par, but for personal stuff, or for something that is patched say at most two-three times a month, it's O.K. It boils down to personal preference.
- kazinator 11y agoComplete rubbish: Many GNU, Linux and other utils are pretty awesome, and obviously some effort has been spent in the past to port them to Windows. However, those projects are either old, abandoned, hosted on CVS, written in platform-specific C, etc. Rust provides a good, platform-agnostic way of writing systems utils that are easy to compile anywhere, and this is as good a way as any to try and learn it. Cygwin has a complete, up-to-date port of the Coreutils for Windows. The POSIX + C language provides a a good, platform-agnostic way of writing systems utils that are easy to compile anywhere.
- dikaiosune 11y agoIs the cygwin port just some different build files, or is it genuinely different code? A Rust codebase probably wouldn't (in most cases) need to be modified to run on different supported platforms.
- frutiger 11y agoI'm not going to comment about cygwin (because its build process is not as transparent) but instead about msys2, the spiritual successor to cygwin. See for yourself: https://github.com/Alexpux/MSYS2-packages/blob/master/coreutils/PKGBUILD https://github.com/Alexpux/MSYS2-packages/blob/master/coreut... is the `pacman` style `PKGBUILD` used to build the `coreutils` package.
- dikaiosune 11y agoThat looks pretty clean, but they're still applying looks like 5 patches which would need to be maintained separately. It's certainly not as bad as a full port, but it's still not the same codebase. Rust's cross-platform efforts would (I think?) obviate that process.
- kazinator 11y agoBut, nevertheless, it's not exactly a port of Coreutils to Windows per se. For instance, let's take the "cp" utility. A Windows port "cp" should be able to copy Windows-specific file attributes. Does the Cygwin "cp" do that? How about NTFS forks? I don't believe that you can write some portable code in Rust, and have a "cp" utility which correctly handles every quirk of every OS's file structure.
- berkut 11y ago"easy to compile anywhere" Doesn't this imply there needs to be the Rust compiler / Cargo (if it needs that) installed on the system? And what versions - stable / unstable ?
- codedokode 11y agoThough I like ideas behind Rust but the code is awful. If you look at the code you'll see a lot of misterious symbols (as in Perl) and strange constructs like wrap, unwrap, Arc etc. They make the code less readable. Also Rust doesn't have exceptions so you have to wrap almost any function call with let/match/Ok/Err. Ugly. I looked at one random file which turned out to be a `du' command implementation: https://github.com/uutils/coreutils/blob/master/src/du/du.rs#L89 https://github.com/uutils/coreutils/blob/master/src/du/du.rs... Are they really starting a new OS level thread for every directory found? Looks like an easy way to exhaust system resources to me. Also I don't see the code that would collect error information if if the thread panics.
- 0xFFC 11y ago>Also Rust doesn't have exceptions so you have to wrap almost any function call with let/match/Ok/Err. Ugly. respectfully, This is pure nonsense. First of all rust does not have runtime and AFAIK for providing exception you should have runtime to manage stack. Second not every language should be like high-level languages, it is not the rule to be like C#,Java,Python,etc. I use a lot of them for my work when I need simple thing to do, but rust designed to do low-level stuff, and I cannot understand how having not having exception makes a language ugly (specially when you code in lowlevel).
- bigdubs 11y agoComment OP's stance is valid. "Magic" in this context are keywords or symbols that are not immediately clear to programmers that don't work in rust. One of the reasons golang is so successful is that there is very little magic in the syntax, and even when there is it's fairly easy to grok (an example would be the `go` keyword). FWIW I also share their opinion that rust is unapproachable.
- codedokode 11y agoGolang has some magic too - for example magic with capital letters in exported symbols. But generally golang is easier to read than line like this: > static NAME: &'static str = "du";
- spaceheeder 11y agoWritten in Rust, MIT-licensed, designed to be portable...I wonder if this is something that might eventually get merged into Redox?
- steveklabnik 11y agoRedox uses a different, BSD style coreutils, not GNU style.
- spaceheeder 11y agoPlease pardon my ignorance, but what are some of the key differences?
- steveklabnik 11y agoI am not an expert, but the general difference, in my understanding, is that the BSD coreutils is significantly "lighter", both in features and in code size.
- fsiefken 11y agoSome people worry about the file size (which should be similar), but I worry about the speed. How much faster can we grep or sed with a Rust compiled grep vs a C compiled grep?
- teamhappy 11y agoLikely not as fast as with a grep written in C, but FreeBSD's grep is (was?) pretty slow compared to the GNU version and nobody really cares (read: it's probably fast enough).
- steveklabnik 11y agoThere shouldn't be any theoretical reason that a Rust grep should be slower than a C grep. In practice, a rust grep would be a new code base, and the existing grep has seen a lot of development over years.
- joejev 11y agoIt is a bit ironic that these "gnu" core utils are not licensed under the GPL
- rhelmer 11y agoRedox has been working on smaller, more BSD-like versions of the basic Unix utilities, also written in Rust: https://github.com/redox-os/coreutils https://github.com/redox-os/coreutils I am excited to see this as I was working on a similar project last year (rewriting the BSD userland in Rust), but it's from pre-1.0 Rust so not really as idiomatic as what is coming out of the Redox project.
- spaceheeder 11y agoFrom the README: "These are based on BSD coreutils rather than GNU coreutils as these tools should be as minimal as possible." If all of these utilities are running in userspace, per Redox's microkernel architecture, then what is the advantage of intentionally not making them feature-rich?
- icebraining 11y agoIt has the advantages that come from following the UNIX style. To learn more, I recommend reading this short paper by Rob Pike and Brian Kernighan, published in 1984: http://harmful.cat-v.org/cat-v/unix_prog_design.pdf http://harmful.cat-v.org/cat-v/unix_prog_design.pdf
- spiralpolitik 11y ago"Those days are dead and gone and the eulogy was delivered by Perl" - Rob Pike (2004)
- jorgecastillo 11y agoI love UNIX too much, nothing can replace C in my heart! :)
- Tunabrain 11y ago(on Windows use MinGW/MSYS or Cygwin make and make sure you have rustc in PATH) This made me chuckle - if you have MSYS installed, which already comes with a windows port of coreutils, why would you want to use it to build a windows port of coreutils?
- nwah1 11y agoBecause you don't trust the memory safety of programs written in C. Or because you think the coreutils would progress faster with a language with modern semantics like Rust.
- yeukhon 11y agoThis is actually a surprisingly good place to learn utils and learn how to write Rust...
- Animats 11y agoYes! No more buffer overflow exploits in the core utils.
- randommodnar 11y agoMIT license? Is open-source dying?
- hnbroseph 11y agothe question seems non sequitur.
- _ZeD_ 11y agoif you prefer "is free software dying"?
- schoen 11y agoThe MIT license is free and open source. https://opensource.org/licenses/MIT https://opensource.org/licenses/MIT https://www.gnu.org/licenses/license-list.html#X11License https://www.gnu.org/licenses/license-list.html#X11License Maybe "copyleft" or "reciprocal" licensing?
- cyphar 11y agoWithout copyleft, we wouldn't have free software in the form it is today.
- mc808 11y agoI was going to say it only takes about 30 seconds to fork and change the license if you want to. Then I decided to check if someone had already done so. Yep. https://github.com/ivegotasthma/coreutils/commit/4c7dcbd912a78fda77900806c8bb3a59335b2d6d https://github.com/ivegotasthma/coreutils/commit/4c7dcbd912a... I didn't count, but that may have taken less than 30 seconds.
- biot 11y agoIs that valid? The original license had: > The above copyright notice and this permission notice shall be > included in all copies or substantial portions of the Software. As those have now been stripped away, it seems like a license violation.
- alixaxel 11y agoThe CoreUtils sort is amazingly efficient, it can sort files nearly as big as your hard drive without matching memory requirements. With all these node.js / Go / Rust CoreUtils implementations I'm still hoping for one of them to actually match the efficiency of the original implementation.
- yincrash 11y agoI wrote the initial sort about a year ago (w/o looking at GNU version's source). I just looked at the current version, and confirmed uutil's sort does not do any external sorting, meaning the entire input is stored in memory. Maybe someone can confirm that GNU's sort uses temporary file storage to do something like an external merge sort to reduce memory usage? edit: confirmed. http://stackoverflow.com/questions/930044/how-could-the-unix-sort-command-sort-a-very-large-file http://stackoverflow.com/questions/930044/how-could-the-unix...
- CiPHPerCoder 11y agoObservation: https://github.com/uutils/coreutils/blob/master/LICENSE https://github.com/uutils/coreutils/blob/master/LICENSE GNU coreutils is GPL, this is MIT.
- alphacome 11y agowhich company is using Rust?
- steveklabnik 11y agoThere's a lot of startups that you may or may not have heard of, but the most well-known company using Rust in production today is Dropbox.
- deleted 11y ago[deleted]
- randhnuser 11y agoOfftopic: I'm currently indulging myself in clojure but as I only know dynamic high-level languages I am also a bit curious about low-level stuff. For someone starting at zero, would you recommend C rather than Rust or the other way around?
- botw 11y agodoes GNU coreutils include sed/awk?