10 ms·
A statically typed scripting language that transpiles to Posix sh
- JustLurking2022 5y agoDefinitely a niche that needs a solution - cool to see one!
- throwaway894345 5y agoThis seems neat. I love that there are examples front-and-center on the README page. This is exactly what I want and the first thing I would think to put when demonstrating a new language, but sadly this is rare. I'm not sure how I'd use something like this, but I've often found myself wanting simple statically typed scripting languages for generating YAML/HCL/etc configs. Dhall seems close, but config is something that every engineer has to touch and the burden of making everyone learn its syntax and functional programming idioms probably negates the advantages of using static typing. I've also looked into CUE, but it seems like it's largely neglecting the problem that I'm trying to solve DRY configs in favor of some problem I didn't know I had (unifying a bunch of layers of mostly-static configs?). This seems even worse than Dhall with respect to burden:reward ratio. I haven't yet checked out Nickel (the statically typed Nix language)--would be curious to hear from those who have. EDIT: just skimmed Nickel. It seems like it's trying to hew closely to Nix syntax and functional programming conventions, which suggests it probably shares many of the same problems as Dhall. Too bad. It seems like no one understands the assignment.
- lakecresva 5y agoThe functional stuff in Nickel is really pretty vanilla, and the syntax seems much more inspired by ML than Nix. There's some info in the repo's `RATIONALE.md` that specifically answers "how is this different/better than Dhall/JSonnet/Cue/Starlark".
- kazinator 5y ago> I'm not sure how I'd use something like this. It seems that you could use this to program embedded systems that have a shell, yet which don't support any other usable language (due to flash space concerns, product management approval or whatever).
- sdevonoes 5y agoWhat I miss in the README page is the output of the transpilation.
- JyB 5y agoI guess the output is not really meant to be readable in any way
- em-bee 5y agoif it transpiles to sh then the output should be an sh script. that should be at least somewhat readable.
- xyzzy123 5y agoYes although it might be readable in the same way that autotools configure scripts are readable -- technically you can, but you probably don't want to, and if you need to then something has gone badly wrong.
- em-bee 5y agoyeah, that's kind of what i meant. especially if the script has external commands you should be able to at least recognize something.
- deleted 5y ago[deleted]
- xg15 5y agoI mean yes. Things do go wrong, especially with bleeding-edge projects such as this. You'll have to look at that output if you want to debug things, understand error messages. As such, I'd absolutely see it as a quality measure for a transpiler how readable the code it produces is.
- hnlmorg 5y ago
- em-bee 5y agoI love that there are examples front-and-center on the README page except that those examples are to simple. for something that transpiles to sh i would like to use it to write better shell scripts. so i'd like to see examples that resemble actual shell scripts in that they call lots of external commands, do file operations, terminal interaction and other things commonly done with sh.
- smoe 5y agoFelt the same. Most complete example seems to be the installer for the language which is written in it. This covers some of the things you mentioned https://github.com/cotowali/konryu/blob/main/konryu.li https://github.com/cotowali/konryu/blob/main/konryu.li and the generated shell script https://raw.githubusercontent.com/cotowali/konryu/dist/konryu.sh https://raw.githubusercontent.com/cotowali/konryu/dist/konry... There is also the website, heavily under construction, but has some more documentation on the language than the readme https://cotowali.org/ https://cotowali.org/
- remram 5y agoSo this 366-line script compiles into 588 lines of shell, 2,229 lines if we count the parts that seem to come from the imported modules. That includes a lot of auto-generated names like `_cotowali_tmp_166` or `first8b04d5e3775d298e78455efc5ca404d5` and replacements for built-in shell functions like `eprintln` or `rm_r`. In fact that script seems to use `eprint`, `eprintln`, `println`, `print`, AND `echo` which seems a little much. It would have been nice to see something more transparent, e.g. a type-checked extension to shell, like TypeScript is for JavaScript. Something to make it easier to write correct shell scripts, but would generate understandable, editable shell scripts.
- Willamin 5y ago> simple statically typed scripting languages for generating YAML/HCL/etc configs Do you actually want a statically typed _imperative_ language for generating configs? Or do you want a sane, statically typed, declarative language for configs?
- throwaway894345 5y agoI haven’t seen a sane declarative language. I’m pretty sure what I want is an imperative language, especially since that’s already well-understood by the target audience. I’m open to persuasion, but I gave Cue an honest shot but it seemed to regard reuse as an afterthought at best rather than the main problem to solve (I think reuse was possible, but it wasn’t obvious and it seemed like it was only supported by accident).
- kirbyfan64sos 5y agoYou might find Cue interesting: https://cuelang.org/ https://cuelang.org/ It's a bit overambitious IMO, but it tries taking a relatively different approach to the problem that I find to be quite interesting.
- sidpatil 5y agoParent already mentioned their investigation of CUE.
- mixedCase 5y agoAlright, here's the hot take: The assignment is for any developer without basic knowledge of FP beyond .map and .filter to complete their intermediate-level programming education. All problem domains that involve declarative programming, such as configuration (Terraform, Kubernetes, Nix, many build systems), dynamic HTML, relational databases, and pretty much anything that is manipulating data in a pipeline fits functional programming like a glove and is hard to get right in a strictly traditionally imperative fashion. As an industry we should stop pretending FP isn't a necessary part of a professional programmer's education and that we can solve the problem by coming up with the umpteenth "we promise it's not FP" abstraction with its own take on fooling developers into writing functional code through an imperative-looking layer full of foot guns that subtly break imperative developer expectations, like React.
- verall 5y agoI think it's a super hot take because the real world has demonstrated (over and over again) that a huge portion of professional software developers have no interest in learning functional paradigms just to write some config files. Just like a huge portion of professional software developers have no interest in learning RTL or digital logic just to program an FPGA. HLS is a massively complex system to solve this problem. People will also build massively complex systems to solve the problem of "No FP here, please".
- klekticist 5y agoI agree with your point here, though imo it's worth drawing a distinction between the FP concepts the language designers / implementors must understand and the FP concepts the language users must understand. Ideally languages prevent bug classes like null deref, use after free while still being familiar enough for most programmers to use. It's analogous to the spectrum of SQL knowledge folks have. For instance knowing what a group by does vs how a group by reshuffles data on a single node / across nodes. Ideally the language design hides the hard details. A very tall order admittedly.
- mixedCase 5y agoThat'd be fine if it was just config files, but that's not the problem now isn't it?. It's an enormous amount of problem domains that FP handles trivially and is a complete pain to manage imperatively. We've had entire groups of people coming up with ridiculous ideas like replacing relational databases with key-value stores for relational data just because they find SQL "icky" because it fits a mental model they never developed (syntax issues with SQL aside) and it isn't until they're hand rolling data aggregation that they realize the mess they got themselves into. Which is why I stand by what I said: this knowledge is not optional, and engineers not willing to learn them are going to be stuck with sub-optimal tools, dragging the whole industry down with them. While I can't be confident this is the definitive solution, I believe that the best way to move forward is to start treating basic FP (purity, data transformation pipelines, immutability, favoring data structures over code) as a baseline that everyone needs to know to work as a developer, and stop treating it as if it is on a superior level of difficulty compared to learning whatever bullshit combination of half-templated server side rendering, 4D DOMs and incantations needed to upgrade Webpack we need to know to be "up to speed" nowadays.
- soraminazuki 5y ago> making everyone learn its syntax and functional programming idioms probably negates the advantages of using static typing I think this is an over-exaggeration of the complexity involved because Dhall/Nickel/Nix are just JSON with variables and functions. If you're familiar with JSON and any general purpose programming language, say Python, you can learn the aforementioned languages in less than an hour. > Too bad. It seems like no one understands the assignment. You mean all those language designers for static configuration languages have no idea what they're doing? I think there's a better explanation. For complex configurations that warrant the kind of high level configuration languages we're talking about, declarative syntax seems to a better fit. Tracking mutation and state for large scale configuration would put the burden on everyone, beginners and experts alike.
- neoyagami 5y agonice. I do a lot of bash/sh things when need to automate some task, now they can be programmed in a propper lenguaje( sh can be painful sometimes) and just transpile it to sh
- kazinator 5y agoThere are no examples which show that the source language is good for shell scripting tasks. Making shell programming tasks nicer doesn't appear to be a goal.
- zakuro 5y agoAlthough not yet in the sample, Cotwali is also intended to use for shell programming tasks.
- deleted 5y ago[deleted]
- norswap 5y agoIt's pretty nice! I'd love to see the compiled output of the examples though!
- nerdponx 5y agoI'm surprised to see that the compiler is written in V (https://vlang.io/ https://vlang.io/). Last I heard, V was vaporware, and a lot of its claims about performance etc. ended up being "todo" items rather than facts about the language as it actually existed. But that was several years ago; has anything changed? Otherwise, fun project! I could see this actually being useful in some niche cases, e.g. generating scripts that need to work on some exotic architecture that has a POSIX Sh implementation. It might also be useful for writing interactive shell "plugins" that set up things like a fancy PS1 prompt.
- yisonPylkita 5y agoI would love to hear more about that. Does someone here have an experience with V lang and can tell us if its claims are at least close to truth?
- camgunz 5y agoThis was my impression too and it looks like they've made really impressive progress since I last saw them on HN. Their bug list [0] looks kind of scary, but that's maybe par for the course for a language? I remember the bad old days of Go's 32-bit GC not being wonderful for example. [0]: https://github.com/vlang/v/issues?q=is%3Aissue+is%3Aopen+label%3A%22Status%3A+Confirmed%22 https://github.com/vlang/v/issues?q=is%3Aissue+is%3Aopen+lab...
- KerrAvon 5y agoMaybe? Some of those sound like unit tests should have caught them.
- rurban 5y agolooks very much like Rust's issue tracker to me
- johnisgood 5y agoAgreed! Also... I was just going through smhasher. Thank you for it!
- hnlmorg 5y agoWow this is a great advert for V (the programming language it was written in). Having never written a line of V before I can still read and understand the code.
- IggleSniggle 5y agoI had exactly the same reaction. Extremely digestible syntax. Didn’t even notice until after looking it over that it had been written in a language I don’t know.
- Shadonototra 5y agoV is such a nice and simple language, it's Go for system programming
- reincarnate0x14 5y agoThis is amazingly great for how simple the idea is, going to have to play with the output a bit to poke around for strange edge cases (it is sh, after all) but given all the random devices out there with Busybox or similar this could end up being significantly useful for configuration management, etc. Also seeing quick, readable examples right on the github landing page is such a small but wonderful thing.
- ducktective 5y agoSo it's APE (actually portable executable) but for POSIX systems which instead of binary uses scripting. nice
- perlgeek 5y agoNice, a language written a pretty obscure language (V) and using a pretty obscure build tool (z). I have a fondness for people using unusual tools for their hobby projects. POSIX shell is a good compilation target if you want to write a non-trivial build system, but be able to run it on some old or otherwise challenging systems.
- smasher164 5y agoNeat! A really cool enhancement to this would be if the compiler could target a polyglot script that is both Shell and Batch. For example: https://stackoverflow.com/a/17623721 https://stackoverflow.com/a/17623721 Then you'd get a single script that would automatically be portable and executable on pretty much every conceivable platform.
- codeflo 5y agoWhat’s the main use case? I guess build scripts?
- zakuro 5y agoBuild scripts are one of the assumed use cases. Build scripts are one of the assumed use cases. The other use case is installation scripts. cotowali's installer is written in cotowali itself. https://github.com/cotowali/konryu https://github.com/cotowali/konryu
- adriangrigore 5y agoMight as well check out my POSIX sh static site generator https://mkws.sh/ https://mkws.sh/ if you're into this stuff.
- beepbooptheory 5y agolooks like the task runner part [1] has unfortunate name overlap with the family of programs that include z.lua[2] or zoxide [3] or z.sh [4]. Not really that big of a deal, I just like tracking command line apps naming because it is an interesting exercise of being short, intuitive, and unique. When you decide a single letter is your name, surely you are going to run into this. 1. https://github.com/zakuro9715/z https://github.com/zakuro9715/z 2. https://github.com/skywind3000/z.lua https://github.com/skywind3000/z.lua 3. https://github.com/ajeetdsouza/zoxide https://github.com/ajeetdsouza/zoxide 4. https://github.com/rupa/z https://github.com/rupa/z
- jokethrowaway 5y agoWhy not just build a binary? What's the benefit of transpiling to sh?
- zakuro 5y agoBinary files are different for each architecture, but shell script is not so. Most scripting languages require a interpreter to be installed. Output of Cotowali is shell script. so no need to install any interpreter.
- dogprez 5y agoWouldn't it just be better to make a new shell that has this syntax?
- eyelidlessness 5y agoBetter in that you can use it directly, not necessarily great for interop. I love Fish shell, but it can be a pain when trying to use tools that expect posix syntax to work.
- hnlmorg 5y agoThere's already a plethora of shells that fit that domain: + Nu Shell + NGS + Powershell + Oil Shell + Elvish + Murex (which is one I'd written) Not to mention all the LISP, Python, and other such high level language REPLs come shells. But they solve a different problem. This seems to be more like what Typescript is to Javascript: a alternative "safer" language where your end point is limited to supporting a different language. So you need to compile down to a known portable base language.
- tux1968 5y agoWould be nice if the language supported a #! bang line in order to make it a bit easier to execute, rather than manually typing "lic run X". Although of course on Linux you can always use https://en.wikipedia.org/wiki/Binfmt_misc https://en.wikipedia.org/wiki/Binfmt_misc without needing the bang line.
- nikeee 5y agoAs I understand shebangs, one could use something like this: #!/usr/bin/lic run
- metadat 5y agoAwesome idea, but "cotowali" is a truly terrible gibberish sounding name, maybe even worse than klubernetes.
- westurner 5y agoCould this be done with LLVM?
- Merad 5y agoNeat idea, but it seems to me like a critical part of making a new language of any kind is documenting it so that others can learn it and use it. On that note, can anyone explain the syntax that seems to use pipes in a function declaration (i.e. every function in the readme example other than fib)?
- mbarbar 5y agoIt appears to me those are the types the functions expect to consume and output when used in a pipeline.
- nixpulvis 5y agoUnless I'm misinterpreting the example, it seems rather unfortunate that you need to create a sentential value before a `read` call to update some `&n` location. The `twice` function really could be just `return read() * 2` no? I guess I should look into how this is actually compiled to sh. I agree the README would benefit from adding some examples of this.
- jpgvm 5y agoWould appear that V isn't as vaporware as I initially thought. Going to give this a fairly through once-over because I think Posix sh is an under-appreciated transpilation target.
- bhussai20 5y agoLet me preface by saying this is pretty cool. I feel that this seems like a good idea on paper, and a bad idea in the wild. Last time I looked into shell syntax and such, I found out that most shells don't. They vary so greatly that very large context-free grammars have to be used to cover most of their context-sensitive rules. I don't think being POSIX compliant changes that. I suspect that if this got popular usage, the bugs would be quite numerous.
- remus 5y agoI wonder whether there's a subset of shell syntax that's stable enough and powerful enough that it could be used as the compilation target whilst avoiding the incompatibilities you mention?
- goombacloud 5y agoI also wonder if targeting bash which has things like pipefail and arrays could make things easier in the implementation side?
- nomemory 5y agoBack in the day i had a similar idea, and i even started implementing it, but never had the resilience to finish. It waa too difficult for me at that time (probably even now). I am glad that finally someone had the courage to do it.
- Brian_K_White 5y agoI see 2 possible rationales to transpile to posix sh: 1: To remove bashisms/kshisms/zshisms to create output that works on ash/dash/busybox/xenix/sun/... In which case, the more useful source language is bash or ksh etc, not a new invented one. 2: To generate a portable output like above from a better language than any *sh (*). In which case the source language should be something already common like python or perl or ruby etc (even if just a subset). In either of those two use cases, I don't think it makes a great deal of sense to make a new language, and especially not to write the compiler for it in V of all things. That relegates the whole project to being just a curiosity or a purely academic excercise. (*) I use every trick bash and ksh offer, so I know it's possible to do all kinds of stuff in some *sh's, but a lot of the features in them are encoded into practically inscrutable special syntax and rules and tricks, where more purpose built programming languages make all the features more explicit and spelled out -> readable. bash is a powerful enough language, but not a readable one, except for rairly stupid scripts that either don't do much, or do everything the hard way.
- maxbond 5y agoWhat about, to gain saner semantics (static typing) while retaining the portability and versatility of shell scripts?
- chalst 5y agoTypical shell programming idioms are quite dynamic. You can give saner semantics, allowing a fair amount of static checking, without abandoning this dynamicity. What Andy Chu has been doing with Oil Shell is promising in this respect. It supports optional type annotations. Cf. https://www.oilshell.org/blog/2021/01/why-a-new-shell.html https://www.oilshell.org/blog/2021/01/why-a-new-shell.html
- Brian_K_White 5y ago"What about, to gain saner semantics (static typing) while retaining the portability and versatility of shell scripts?" I think I was insufficiently critical in a misguided intention to be nice. Let me correct that a little :) If the need is great enough for a DSL, then the need is great enough for python or c or any of 100 other fully fledged universally available languages, and either require it's interpreter or ship a binary. However annoying it may be to require a ruby interpreter or something, there is essentially no case where the most sensible answer is this "Vala-for-sh". It's the worst of all worlds. Taking the whole life cycle into account, it's not saner or portable or versatile, it reduces all of those.
- heavyset_go 5y agoDoes this provide real data structures like arrays, hash maps, etc? And can you pass them around?
- pabs3 5y agoI wonder how many complaints shellcheck would have about the resulting script :)
- hnarn 5y agoI don't see any other comments about this so I'll ask: what other options are there for similar use cases? I.e. "statically typed scripting languages that transpile to Posix sh".