4 ms·

Starting off in Rust: Trying to write a shell

88 points by madars 12y ago

lmm 12y ago
Does a shell really exercise the use of data structures? To my mind it seems there's very little state, and all of it in substantially simpler formats than even e.g. JSON.
- chubot 12y ago
  Yeah, probably not. In any program, if you don't care about performance, you essentially don't need anything more than hashes and arrays (what Python/Perl/Ruby/JS provide). In a shell you don't care about runtime performance because all the cost is in the invoked programs. Ironically you probably care about parsing performance more than execution performance, because parsing happens at startup and can cause latency. You will have a stack since I think shells have some form of scope, but that's pretty much an array.
  - sanxiyn 12y ago
    > In a shell you don't care about runtime performance because all the cost is in the invoked programs. This is not strictly true. In a reasonable shell many commands will be builtin, because invoking processes for, say, echo, is slow.
    - chubot 12y ago
      What you said is true, but it doesn't make what I said untrue. It could be that 99% of the statements are executed by shell builtins. But for essentially any shell script I've seen, speeding those up by 10x won't make any difference, because the real work is invoking compilers, linkers, invoking curl, tar, gzip, etc. As well as simple process creation overhead.
- hugofirth 12y ago
  Agreed - when I play with a language I normally like to building something like a simple Trie implementation to get a feel for the collections library. I guess a shell is a pretty good learning project though, as you can just expand its scope to touch upon most of a language's API surface area as you go: - Want to see how a language handles IO? cd, ls, mkdir and friends will give you a pretty good introduction. - Want to look at the http lib? A simple wget clone.
- JoshTriplett 12y ago
  Rust is supposed to be a language for systems programming, and a shell seems like a good exercise for that.
  - lmm 12y ago
    Is it? I understood it was a language for Mozilla (i.e. with the browser as a core use case) - so for very high-performance applications (those that need so much control of memory management that even Haskell is unacceptable) rather than systems per se. No?
    - burntsushi 12y ago
      Seems like pretty nebulous territory to me (which I think means it's probably not worth splitting hairs over it). In any case, rust-lang.org says in big giant letters: "Rust is a systems programming language ..."
    - JoshTriplett 12y ago
      Writing a browser is very much a systems programming task, including careful memory management, access to numerous platform features, implementation of a high-performance JIT and language runtime...
      - lmm 12y ago
        It's not really systems programming in the sense that I understand it (or in e.g. the wikipedia article). More to the point, I think a browser involves a lot more manipulation of complex data structures - even the basic DOM is more complex than anything you'd use in a shell - and this is an area in which I expect Rust to shine. So I don't think a shell provides the best opportunity to appreciate Rust's strengths.
justinwr 12y ago
Shame the blog is so horrible because I'd love to read the text and enjoy a hypertext document or two.
- kevin_thibedeau 12y ago
  I can't fathom why Google doesn't streamline that crap so that it doesn't depend on scripts from two or three separate domains and will properly fallback render to something decent without any scripts.
ajross 12y ago
> The POSIX shell grammar is ridiculously complicated, and would require the use of a parser-generator to reasonably implement. Wat? Admittedly Bourne shell syntax is ad-hoc, glitchy to implement and difficult to specify. But it's absolutely not something that requires a parser generator. It was always and remains a string replacement language without deep structure.
- kps 12y ago
  Bourne shell syntax is inspired by Algol 68¹, of which Bourne was a fan — the shell itself² was written in a highly idiosyncratic dialect of C, using the preprocessor to provide an Algol68ish syntax for control structures: WHILE i-- DO t=trapcom[i]; IF t==0 ORF *t THEN clrsig(i); FI trapflg[i]=0; OD The parser³ is straightforward recursive-descent in 400 lines including comments & blanks (the whole shell is about 4000 lines of code). ¹ http://en.wikipedia.org/wiki/ALGOL_68 http://en.wikipedia.org/wiki/ALGOL_68 ² http://minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/src/cmd/sh http://minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/src/cmd/... ³ http://minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/src/cmd/sh/cmd.c http://minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/src/cmd/...
majika 12y ago
To read this without JavaScript, you can get the post text from the (current) top of the RSS feed: http://blog.achernya.com/feeds/posts/default http://blog.achernya.com/feeds/posts/default The source code of the (unfinished) shell is at: https://github.com/achernya/xv6-shell.rs/blob/master/src/main.rs https://github.com/achernya/xv6-shell.rs/blob/master/src/mai...
- pasbesoin 12y ago
  For these Blogger templates, I just pull the relevant page from Google Cache. E.g. http://webcache.googleusercontent.com/search?q=cache:http%3A%2F%2Fblog.achernya.com%2F2015%2F03%2Fstarting-off-in-rust-trying-to-write.html http://webcache.googleusercontent.com/search?q=cache:http%3A...
shanemhansen 12y ago
Is it really true that rust doesn't support i/o redirection as the article states? It looks like the apis are there: https://doc.rust-lang.org/nightly/std/process/struct.Command.html#method.stdin https://doc.rust-lang.org/nightly/std/process/struct.Command... In addition to null() you can use piped().
Animats 12y ago
Good project, to write a shell in Rust. There are other efforts underway to rewrite the low-level UNIX utilities. An all-Rust BusyBox is probably not far off. The author's problems with language and library stability, though, comparable to my experience with Rust. The libraries aren't there yet. I've been trying to write a program that uses RSS feeds. There's no RSS feed reader library, so I have to write that. RSS requires an XML parsing library, an HTTP library, and an ISO date parser. There are several choices for an XML parsing library, and one of them worked and produced a good tree. It doesn't have any functions for finding elements in the tree, though, so I had to write those. There are at least three HTTP libraries. One is obsolete, one isn't done yet, and one, "hyper" is usable. Or it was, until last week, when the "io" library had a major redesign. Using "old_io"? That's so last week. Right now, the "hyper" HTTP library is broken, due to some upstream problems with the SSL and cookie libraries being broken by the change at the "io" level. The appropriate issue tickets have been opened and many people are working hard on fixing everything that does I/O to do it the new way. ISO date parsing wasn't implemented for the date/time module, so I wrote that, forked the date/time module, and submitted a pull request. The author of the date/time library rewrote the code and added it, and now the stock "chrono" module does the date/time formats needed for RSS and Atom feeds. Compiling my own code, I now get messages like rssread.rs:262:23: 262:25 warning: obsolete syntax: [] rssread.rs:262 p.feed_str(&string[]); note: write `[..]` instead The syntax for a slice that includes everything changed since last week. The documentation for all these changes runs behind the code, of course, so you have to read blogs, issues, and pull requests to figure out what the language does this week. Daily Rust development cycle: "sh updaterust.sh", then for each project "cargo clean; cargo update; cargo build; cargo test", then fix what's now broken. I can understand what the Rust developers are going through. Their public pronouncements [1][2], though, create the illusion of a higher degree of stability than is actually the case. If you have to do any real work, I'd hold off on Rust until the 1.0 release, and maybe a few months after that. Rust is going to be an important language, but it's not ready for prime time yet. [1] http://blog.rust-lang.org/2015/01/09/Rust-1.0-alpha.html http://blog.rust-lang.org/2015/01/09/Rust-1.0-alpha.html [2] http://blog.rust-lang.org/2015/02/20/Rust-1.0-alpha2.html http://blog.rust-lang.org/2015/02/20/Rust-1.0-alpha2.html
- burntsushi 12y ago

88 points by madars 12y ago

lmm 12y ago

Does a shell really exercise the use of data structures? To my mind it seems there's very little state, and all of it in substantially simpler formats than even e.g. JSON.

chubot 12y ago
Yeah, probably not. In any program, if you don't care about performance, you essentially don't need anything more than hashes and arrays (what Python/Perl/Ruby/JS provide). In a shell you don't care about runtime performance because all the cost is in the invoked programs. Ironically you probably care about parsing performance more than execution performance, because parsing happens at startup and can cause latency. You will have a stack since I think shells have some form of scope, but that's pretty much an array.
- sanxiyn 12y ago
  > In a shell you don't care about runtime performance because all the cost is in the invoked programs. This is not strictly true. In a reasonable shell many commands will be builtin, because invoking processes for, say, echo, is slow.
  - chubot 12y ago
    What you said is true, but it doesn't make what I said untrue. It could be that 99% of the statements are executed by shell builtins. But for essentially any shell script I've seen, speeding those up by 10x won't make any difference, because the real work is invoking compilers, linkers, invoking curl, tar, gzip, etc. As well as simple process creation overhead.
hugofirth 12y ago
Agreed - when I play with a language I normally like to building something like a simple Trie implementation to get a feel for the collections library. I guess a shell is a pretty good learning project though, as you can just expand its scope to touch upon most of a language's API surface area as you go: - Want to see how a language handles IO? cd, ls, mkdir and friends will give you a pretty good introduction. - Want to look at the http lib? A simple wget clone.
JoshTriplett 12y ago
Rust is supposed to be a language for systems programming, and a shell seems like a good exercise for that.
- lmm 12y ago
  Is it? I understood it was a language for Mozilla (i.e. with the browser as a core use case) - so for very high-performance applications (those that need so much control of memory management that even Haskell is unacceptable) rather than systems per se. No?
  - burntsushi 12y ago
    Seems like pretty nebulous territory to me (which I think means it's probably not worth splitting hairs over it). In any case, rust-lang.org says in big giant letters: "Rust is a systems programming language ..."
  - JoshTriplett 12y ago
    Writing a browser is very much a systems programming task, including careful memory management, access to numerous platform features, implementation of a high-performance JIT and language runtime...
    - lmm 12y ago
      It's not really systems programming in the sense that I understand it (or in e.g. the wikipedia article). More to the point, I think a browser involves a lot more manipulation of complex data structures - even the basic DOM is more complex than anything you'd use in a shell - and this is an area in which I expect Rust to shine. So I don't think a shell provides the best opportunity to appreciate Rust's strengths.

justinwr 12y ago

Shame the blog is so horrible because I'd love to read the text and enjoy a hypertext document or two.

kevin_thibedeau 12y ago
I can't fathom why Google doesn't streamline that crap so that it doesn't depend on scripts from two or three separate domains and will properly fallback render to something decent without any scripts.

ajross 12y ago

> The POSIX shell grammar is ridiculously complicated, and would require the use of a parser-generator to reasonably implement. Wat? Admittedly Bourne shell syntax is ad-hoc, glitchy to implement and difficult to specify. But it's absolutely not something that requires a parser generator. It was always and remains a string replacement language without deep structure.

kps 12y ago
Bourne shell syntax is inspired by Algol 68¹, of which Bourne was a fan — the shell itself² was written in a highly idiosyncratic dialect of C, using the preprocessor to provide an Algol68ish syntax for control structures: WHILE i-- DO t=trapcom[i]; IF t==0 ORF *t THEN clrsig(i); FI trapflg[i]=0; OD The parser³ is straightforward recursive-descent in 400 lines including comments & blanks (the whole shell is about 4000 lines of code). ¹ http://en.wikipedia.org/wiki/ALGOL_68 http://en.wikipedia.org/wiki/ALGOL_68 ² http://minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/src/cmd/sh http://minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/src/cmd/... ³ http://minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/src/cmd/sh/cmd.c http://minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/src/cmd/...

majika 12y ago

To read this without JavaScript, you can get the post text from the (current) top of the RSS feed: http://blog.achernya.com/feeds/posts/default http://blog.achernya.com/feeds/posts/default The source code of the (unfinished) shell is at: https://github.com/achernya/xv6-shell.rs/blob/master/src/main.rs https://github.com/achernya/xv6-shell.rs/blob/master/src/mai...

pasbesoin 12y ago
For these Blogger templates, I just pull the relevant page from Google Cache. E.g. http://webcache.googleusercontent.com/search?q=cache:http%3A%2F%2Fblog.achernya.com%2F2015%2F03%2Fstarting-off-in-rust-trying-to-write.html http://webcache.googleusercontent.com/search?q=cache:http%3A...

shanemhansen 12y ago

Is it really true that rust doesn't support i/o redirection as the article states? It looks like the apis are there: https://doc.rust-lang.org/nightly/std/process/struct.Command.html#method.stdin https://doc.rust-lang.org/nightly/std/process/struct.Command... In addition to null() you can use piped().

Animats 12y ago

Good project, to write a shell in Rust. There are other efforts underway to rewrite the low-level UNIX utilities. An all-Rust BusyBox is probably not far off. The author's problems with language and library stability, though, comparable to my experience with Rust. The libraries aren't there yet. I've been trying to write a program that uses RSS feeds. There's no RSS feed reader library, so I have to write that. RSS requires an XML parsing library, an HTTP library, and an ISO date parser. There are several choices for an XML parsing library, and one of them worked and produced a good tree. It doesn't have any functions for finding elements in the tree, though, so I had to write those. There are at least three HTTP libraries. One is obsolete, one isn't done yet, and one, "hyper" is usable. Or it was, until last week, when the "io" library had a major redesign. Using "old_io"? That's so last week. Right now, the "hyper" HTTP library is broken, due to some upstream problems with the SSL and cookie libraries being broken by the change at the "io" level. The appropriate issue tickets have been opened and many people are working hard on fixing everything that does I/O to do it the new way. ISO date parsing wasn't implemented for the date/time module, so I wrote that, forked the date/time module, and submitted a pull request. The author of the date/time library rewrote the code and added it, and now the stock "chrono" module does the date/time formats needed for RSS and Atom feeds. Compiling my own code, I now get messages like rssread.rs:262:23: 262:25 warning: obsolete syntax: [] rssread.rs:262 p.feed_str(&string[]); note: write `[..]` instead The syntax for a slice that includes everything changed since last week. The documentation for all these changes runs behind the code, of course, so you have to read blogs, issues, and pull requests to figure out what the language does this week. Daily Rust development cycle: "sh updaterust.sh", then for each project "cargo clean; cargo update; cargo build; cargo test", then fix what's now broken. I can understand what the Rust developers are going through. Their public pronouncements [1][2], though, create the illusion of a higher degree of stability than is actually the case. If you have to do any real work, I'd hold off on Rust until the 1.0 release, and maybe a few months after that. Rust is going to be an important language, but it's not ready for prime time yet. [1] http://blog.rust-lang.org/2015/01/09/Rust-1.0-alpha.html http://blog.rust-lang.org/2015/01/09/Rust-1.0-alpha.html [2] http://blog.rust-lang.org/2015/02/20/Rust-1.0-alpha2.html http://blog.rust-lang.org/2015/02/20/Rust-1.0-alpha2.html

burntsushi 12y ago