12 ms·
Unix core utilities implemented in Haskell
- anardil 2y agoWow, hello! This is my repository. I'm happy to answer any questions.
- aeonik 2y agoYou specify "fast", can you elaborate on the performance of the collection? How does it compare to the standard core utils? Great work, looks amazing.
- deleted 2y ago[deleted]
- anardil 2y agoPerformance (execution, memory) is generally in the same ballpark as the BSD versions, with some caveats specific to utils that do lots of in place data manipulation. cut comes to mind as an example, slicing and dicing lines into fields quickly without a ton of copies isn't easy. Using Streaming.ByteString generally makes a huge difference, but it's extremely difficult to use unless you get can your mind to meld with the types it wants. Picking it up again months later takes some serious effort.
- faragon 2y agoVery beautiful implementation of the awk interpreter in less than 600 lines! https://github.com/Gandalf-/coreutils/blob/master/Coreutils/Awk.hs https://github.com/Gandalf-/coreutils/blob/master/Coreutils/...
- cosmic_quanta 2y agoCould you speak to the advantages of Haskell's lazy IO? I only hear about its disadvantages usually
- habitue 2y agoI imagine for streaming tools like these it's pretty convenient. You don't have to manage buffers etc, just write code against a massive string and haskell takes care of streaming it for you and pulling in more data when needed. There are libraries that handle it, but they probably have weird types, you can just use functions in the prelude to write a lot of these basic utilities.
- jerf 2y agoUnfortunately, while that may be the dream, it doesn't work out that way if you want good performance. If you look at the source you'll see that it uses things like https://hackage.haskell.org/package/streaming-bytestring-0.3.3/docs/Streaming-ByteString-Char8.html https://hackage.haskell.org/package/streaming-bytestring-0.3... a lot. For one thing, a "string" in Haskell by default is a linked list of unicode characters, so right out of the gate you've got big performance problems if you want to use strings. The exact way laziness is done also has serious performance consequences as well; when dealing with things as small as individual characters all the overhead looms large as a percentage basis. One of the major purposes of any of the several variants of ByteString is to bundle the bytes together, but that means you're back to dealing with chunks. Haskell does end up with a nice API that can abstract over the chunks but it still means you sometimes have to deal with chunks as chunks; if you turn them back into a normal Haskell "string" you lose all the performance advantages. It can still come out fairly nice, but if you want performance it is definitely not just a matter of opening a file and pretending you've just got one big lazy string and you can just ignore all the details; some of the details still poke out.
- habitue 2y agoI mean, I'm aware of the downsides, the OP asked why someone might use it. Ease of use seems like a reasonable upside
- bts 2y agoHi! A few years ago I found myself wanting an equivalent of `column` that didn’t strip color codes. After I implemented it in Haskell, I found it was useful to use Nix to force statically linking against libraries like gmp to reduce startup time. Perhaps what I ended up doing might be helpful for you too: https://github.com/bts/columnate/blob/master/default.nix https://github.com/bts/columnate/blob/master/default.nix
- anardil 2y agoThank you for the suggestion, I'll give this a whirl! I've fussed around with `--ghc-options '-optl-static -fPIC'` and the like in years past without success.
- Vosporos 2y agoFantastic work, thank you so much!
- anacrolix 2y agoLOTR fan detected
- wizerno 2y agoThe author has a few blog posts covering the tee, split, which, and cat commands [1]. [1] https://www.anardil.net/tag/coreutils.html https://www.anardil.net/tag/coreutils.html
- chasil 2y ago"On Windows - Symlinking doesn't appear to change the name reported by System.Environment.getProgName, so you'll need to create copies of the binary with different names." I have not found this to be the case with "mklink /h".
- deleted 2y ago[deleted]
- zanie 2y agoThat'd be a hard link though
- eviks 2y ago> Symlinking doesn't appear to change the name reported by System.Environment.getProgName, so you'll need to create copies of the binary with different names Would hardlinks help (would avoid multiple copies)?
- anardil 2y agoI'll have to give this a try, thank you for the suggestion!
- Waterluvian 2y agoI’m curious if there are any that were especially uncomfortable to implement in Haskell. I’m a very very novice “functional programmer” but on occasion I find problems that feel just ridiculous to implement FP style.
- sfn42 2y agoI'm not much of a functional programmer either, but generally in the beginning you'll feel this way because you're not used to thinking about problems in a functional way. You're used to solving problems with imperative structures. So you try to implement that in a functional language and it turns out completely stupid because the language isn't made to solve problems that way. But it usually has good ways to solve things.
- galangalalgol 2y agoI got stuck trying to have an intuition about what would explode my memory usage due to lazy eval, I never got a grip on the transition from pseudofunctional at the io boundaries into real functional internally either.
- anonzzzies 2y agoI like fp and array programming and logic programming (and the overlap) to solve problems that are (considered or for real; I don't usually know going in) really a bad fit for those and then implement them shorter and faster than in their imperative implementation. For some things it does not work obviously, but it's a really nice exercise for when in the gym, walking, cooking, driving and such; the performant or LoC optimising parts fit in my head so I can do the rewrites there and find ways to make them fit. For instance; going from an imperative piece of code to array programming (k/j), you have to find a way to translate the problem to a vector/matrix manipulation problem and that you can do while doing something else entirely. When you find a good one fit, it can be many, many times faster than the original just like that.
- anardil 2y agoDefinitely. Depending on how long you've spent staring at the contents of /bin/ and /usr/bin/ you'll notice there are definitely some array or matrix oriented utils (or options) missing like column. cut comes to mind as a difficult one. In C, you can just hop around the char buffer[] and drop nulls in place for fields, etc before printing. You could go that way a Data.Array Char, but that's hard to justify as functional.
- amelius 2y agoIt would be cool if RFCs would be turned into Haskell, so we could use __that__ as the specification instead. Same for all the remaining web standards.
- fuzztester 2y agoexecutable specs?
- johnisgood 2y agoOpinion: I would rather prefer OCaml or just C (instead of say, Rust or Python).
- bionade24 2y agoC is way too ambiguous to serve for that purpose.
- johnisgood 2y agoAre we referring to reference implementations? I have always found it easier to understand the C code, as Rust and Python tend to obscure too many details behind high-level abstractions. C is simple enough to know what is going on, IMO. For example: numbers = [1, 2, 3, 4, 5] squared_numbers = [num * num for num in numbers] print(squared_numbers) vs. fn main() { let numbers = vec![1, 2, 3, 4, 5]; let squared_numbers: Vec<i32> = numbers.iter().map(|&num| num * num).collect(); println!("{:?}", squared_numbers); } vs. #include <stdio.h> int main() { int numbers[] = {1, 2, 3, 4, 5}; int squared_numbers[5]; // Squaring each number for (int i = 0; i < 5; i++) { squared_numbers[i] = numbers[i] * numbers[i]; } // Printing the squared numbers printf("["); for (int i = 0; i < 5; i++) { printf("%d", squared_numbers[i]); if (i < 4) printf(", "); } printf("]\n"); return 0; } Python and Rust here seem concise and elegant, but can be more difficult to follow to the unaccustomed due to obscurity. I am sure there are examples that involve more higher-level abstractions, and reference implementations typically seem to use a lot of said abstractions. In case of this Rust code, abstractions (like iterators and closures) can also make the code more challenging to follow for those who are not accustomed to functional programming paradigms. What I am trying to say is that the C implementation is more straightforward, IMO.
- tightbookkeeper 2y agoAwesome work! I noticed that they are about as long and complex as the C versions. In early C++/STL days part of the pitch was reimplementing core utils in a much more concise way. In wonder if the is a result of focusing on that application, or a coincidence of design decisions.
- mytec 2y agoI'm used to reading about a given C/C++ program being implemented in Rust, and was delighted to see such an effort in a functional programming language. I know little about functional programming languages but I've always been interested in how languages like Ada and now Rust can help programmers write safer code. I'm curious what advantages a rewrite of a C/C++ app in a FP language provides and also what advantages a FP language brings in comparison to a language like Rust.
- Maro 2y agoI looked at the wc implementation, there's no way it's compatible with the GNU tools, how it handles wide characters, which is heavily ties to the idiosyncratic C implementation. I know because I once wrote a wc implementation in modern C++ for my own education, and that's where most of my time and code complexity went. Also, the C versions are devilishly fast, with all flags, with optimized code branches. https://bytepawn.com/tag/wc.html https://bytepawn.com/tag/wc.html
- galangalalgol 2y agoPoking at the rust coreutils project it seems like they all faster than the c versions. The issue is the decades of features that have accumulated that someone somewhere uses but seem safe to disregard because so few do.
- Maro 2y agoThere are millions of lines of shell scripts relying on those flags.
- chongli 2y agoI would place the blame for that on the GNU coreutils project. At best, it’s feature creep. At worst, it’s embrace and extend on the POSIX standard.
- erik_seaberg 2y agoPOSIX doesn't forbid new flags or options. It's up to the author to read the spec and test his portability, or else willingly rely on certain distros. Some GNU tools have had strict modes as a courtesy (they used to jokingly call it POSIX_ME_HARDER).
- prmoustache 2y agoAnd those scripts should check they are indeed calling the gnu coreutils version before they proceed.
- parasti 2y agoIve seen several coreutils reimplementations just using the name "coreutils" for themselves. I doubt anyone in the original project cares, but seems in bad taste to me.
- bqmjjx0kac 2y agoI mean, it gets the point across and they're not claiming to be "GNU coreutils". Maybe some derivative name would be better for uniqueness.
- xolve 2y agoTIL that GitHub usernames can end with a `-`
- anardil 2y agoThe rules were different in 2014 when I made my account! It's actually quite annoying because lots of 3rd party GitHub integrations puke immediately saying I have an invalid username.
- hello_computer 2y agoThe original coreutils are very weak on wide-char formatting, item delimiting (i.e. -null / -print0), flag name consistency, discoverability, and overall input/output uniformity. Solving the edge cases rigorously would probably require minor breakages, but would be worthwhile. Most coreutil re-workings i’ve seen either double-down on JSON output, 24-bit color, sixels, & other lipsticks on the pig—without addressing any of the basic breakages we’ve become accustomed to—or they go off into an entirely different direction like nushell. There is definitely room for improvement, but it is a thankless job.
- Muromec 2y agoWhy not have bug to bug compatible vetsion of coreutils, but in rust and then have a nice thing too?
- hello_computer 2y agoMost are simple programs. If bug-per-bug compat + memory safety is the goal, carefully reviewing the C would have been a vastly better investment of time compared to making 13,498 commits. https://www.cvedetails.com/product/5075/GNU-Coreutils.html?vendor_id=72 https://www.cvedetails.com/product/5075/GNU-Coreutils.html?v... https://github.com/uutils/coreutils https://github.com/uutils/coreutils
- mhd 2y agoThis reminds me of a similar attempt with Perl[0], done a few… erm, twenty years ago. Might be interesting to do a three-way comparison for a few well-covered cases. [0]: https://metacpan.org/release/CWEST/ppt-0.14 https://metacpan.org/release/CWEST/ppt-0.14
- niobe 2y agoRead the readme, only one question, why?
- haskman 2y agoGreat work! As a nitpick, I noticed that there is some contrived indirection going on inside `Coreutils.Util`. Instead of the Utility class they could have just used an datatype. The current way could be confusing to newcomers looking to learn from this repo (if that was the goal).
- anardil 2y agoI'll admit it's mostly this way because I thought ExistentialQuantification sounded cool and wanted to give a try with classes - this could definitely be tidied up