15 ms·

Why Dark didn't choose Rust

130 points by pimterry 6y ago

brundolf 6y ago
Small note for the author, I believe this was supposed to be outside of the quote block: > It received 39 pretty enthusiast[ic] replies.
- lilyball 6y ago
  No, the quote block is correct. It was quoting a HN comment (which includes the typo).
- mumblemumble 6y ago
  No, that's correct. The sentence is part of the message that was being quoted (which is linked up above).
cordite 6y ago
A follow up to this is what they did choose: F# This is posted at https://blog.darklang.com/new-backend-fsharp/ https://blog.darklang.com/new-backend-fsharp/
lubesGordi 6y ago
Did I miss something? "I don't like x,y,z, languages so I picked F#." I didn't see anything about F# in this post.
- marcinzm 6y ago
  He links in the blog to his previous long post explaining why he chose F#: https://blog.darklang.com/new-backend-fsharp/ https://blog.darklang.com/new-backend-fsharp/
- mumblemumble 6y ago
  It's in a separate post, referenced by the 2nd link in the article. The link text is just the word "posts", so it's easy to miss. https://blog.darklang.com/new-backend-fsharp/ https://blog.darklang.com/new-backend-fsharp/
  - pbiggar 6y ago
    Thanks. Just updated the post to be a bit clearer on this.
- brmgb 6y ago
  No, they did three parts so they could hit the front page three times. The whole thing is clearly a Dark marketing campaign but people upvote it so the community must like it.
  - robocat 6y ago
    Your comment should be flagged, but perhaps review the guidelines: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html And factually I think your opinion is incorrect. The first post didn’t seem like it was tailored for HN, and he said he was surprised: https://news.ycombinator.com/item?id=24981505 https://news.ycombinator.com/item?id=24981505 https://news.ycombinator.com/item?id=24980661 https://news.ycombinator.com/item?id=24980661 Meanwhile it looks like this discussion has been modded down to remove it from front page due to the low quality of comments (yes: this comment doesn’t meet standards either but maybe it helps you?...)
    - Dylan16807 6y ago
      Which part of the guidelines are you referring to? I don't think this counts as an accusation of shilling/astroturfing or similar. And they didn't say it shouldn't have been submitted.
qppo 6y ago
If you're writing async code you don't need to worry about pinning unless you're manually writing futures or designing an executor. But if you are curious, this chapter explains it pretty good: https://rust-lang.github.io/async-book/04_pinning/01_chapter.html https://rust-lang.github.io/async-book/04_pinning/01_chapter... > Apparently, recursion adds a new level of complexity to async. It really doesn't - it's just some users may be surprised by the lack of magic when it comes to Rust. Async is implemented with futures, futures are types, and you can't recursively define a type without boxing. > Apparently, this boxing and pinning is what you get when you don't have a GC, and that when you do have a GC, you simply don't need to deal with it. So that was the final straw for me. It's kind of weird to me that language developers would pass on Rust because they don't understand memory management. How is memory going to be managed in Dark? If it's GC'd, how is that implemented?
- throwaway894345 6y ago
  I’m guessing Dark is interpreted and that its objects are managed by the host language’s GC?
  - pbiggar 6y ago
    Yes, that's right.
- pbiggar 6y ago
  > If it's GC'd, how is that implemented? The language in which the Dark interpreter is built has a GC, and we just use that. A GC is a huge undertaking, and we already have to build an editor, a language, a stdlib, and infrastructure.
  - qppo 6y ago
    Why do you need to build an editor too? Side note, I've written a GC in rust (stop the world, compacting). It's not a cutting edge, concurrent generational GC but it's not that big of an undertaking. And it performs very well for small problem sets since memory is all slab allocated. Only a few hundred lines in Rust. I do definitely see how writing a tree walking interpreter is non ideal in Rust if you don't have a ton of experience writing Rust. I have a tiny lisp in Rust compared to doing it in dynamic languages it is quite verbose and hard to grok.
jbirer 6y ago
As a C, Go and Javascript programmer (nearly 10 years of experience with both of them), I really feel what the author is saying when they say there are too many ways to do something and fighting the compiler. There comes a point when the languages quirks start to eat into your productivity. That being said, I have a feeling that the author lacks programming expertise to develop a language (I am not saying this in an offensive way). They are dodging things that are needed to build a language (memory management, choices of libraries etc.).
- luminati 6y ago
  IIRC the author has a PhD in CS, and his thesis work was on PLT.
  - ook 6y ago
    http://www.tara.tcd.ie/handle/2262/77562 http://www.tara.tcd.ie/handle/2262/77562 - "Design and implementation of an ahead-of-time compiler for PHP" is the PhD thesis in question. A google tech talk related to this work is on youtube https://www.youtube.com/watch?v=kKySEUrP7LA https://www.youtube.com/watch?v=kKySEUrP7LA You can find multiple other papers on google scholar where Paul was an author all related to programming languages, compilers etc.
  - mlindner 6y ago
    Some people are much better with theory than the application of that theory. I've met a lot of theory people that just care about implementing something that does something to get the result they want rather than the nitty gritty of making things efficient.
- busterarm 6y ago
  > That being said, I have a feeling that the author lacks programming expertise to develop a language (I am not saying this in an offensive way). They are dodging things that are needed to build a language (memory management, choices of libraries etc.). The author is Paul Biggar, a former Sr. Compiler Engineer @ Mozilla and founder of CircleCI.
- 6y ago
mmastrac 6y ago
Yeah, the problem with jumping into Rust headfirst is that you assume that the most complex parts of the language are what you need to use to solve your problem. It turns out that the vast majority of Rust code is just fine using Arc and letting the reference counts handle your memory management. The lifetimes are a really cool feature for library/data structure authors, but if you're writing "web server" style code just use Arc, call clone, and be done with it, unless you've got a specific perf issue.
- brundolf 6y ago
  > It turns out that the vast majority of Rust code is just fine using Arc and letting the reference counts handle your memory management. There are disadvantages to Rc/Arc compared with GC, though: - They tend to be less performant - Reference cycles can leak - In languages where they exist at a library level instead of a language level, they're much less ergonomic than GC So if your core problem domain involves creating and disposing of lots of heap-allocated stuff all the time, you're probably better off just using a language with a proper GC. The Rust version may not even perform better if you're just using Arc, everywhere, by default. Use Rust when your core domain/hot paths can mostly stick to the single-ownership model (and ideally the stack). Rc/Arc/RefCell are a band-aid for when that fails.
  - mc10 6y ago
    As someone who's relatively new to Rust, I'm curious: what is an example of a situation where someone might lean on Rc/Arc as a "band-aid", and what would be a more "idiomatic", non-Rc/Arc solution?
    - brundolf 6y ago
      Rust's key feature - the borrow-checker - relies on the idea that each value has a single "owner" at any given time. This owner can be a function, another value (a parent struct), etc. You can put these values on the heap, but if you use Box (the go-to for heap allocation), that pointer still has to have a single logical "owner". Under idiomatic Rust, each value effectively lives in one single "place". This allows the compiler to determine with 100% confidence at what point it's no longer being used and can therefore be de-allocated. Now, these values can be lent out ("borrowing") to sub-functions and such via references (mutable or immutable). Multiple immutable references can be handed out at once, but a mutable reference to a value has to be the only reference to that value of any kind, at a given time. The problem is, some domains really don't lend themselves to this restricted model. No two objects or functions can point, mutably, to the same object at the same time. You simply can't create a graph of inter-referenced objects where a single value may have multiple "parents". And sometimes even with a perfectly tree-like ownership structure moving values around can get complicated, because Rust has to know for sure that the ownership model is adhered to. This is where explicit lifetimes and such can come into play. Even writing a linked-list in Rust without using unsafe { } (or Rc's) is hard (https://rust-unofficial.github.io/too-many-lists/ https://rust-unofficial.github.io/too-many-lists/). In Rust, Rc's are kind of an admission of defeat. You're telling Rust not to perform its normal "compile-time" automatic deallocation, instead having it track references at runtime (which comes with overhead) to know when to de-allocate. What this buys you is basically an out from the ownership system: instead of handing off a plain reference to multiple places, which Rust may not let you do, you just clone the Rc and hand off that "new" value which can go anywhere it wants. That Rc is then what gets tracked by the ownership system and de-allocated, and when de-allocated it decrements the count (again, at runtime), and eventually that runtime mechanism (hopefully) decides the real value can be de-allocated. Basically any part of your code that uses Rc/Arc is giving up one of the biggest features of Rust. Which is totally fine, if you're reaping those advantages elsewhere and you just need to bridge a gap where ownership is too limiting. But if heap-juggling is going to be primary thing your program is doing, you'll probably have a better overall time with a GCed language.
brundolf 6y ago
Highly subjective and project-specific, but I still think it provides a nice counter-anecdote to the constant Rust praise we tend to see here. And I say that as a Rust fan myself. > ...but remember that garbage collectors are great. By having a GC, we don't have to do any of the stuff that causes all these problems in Rust. Maybe that costs performance, but I need the ability to quickly write code a lot more than I need the extra performance. This is very valid and sensible. Worth noting is that C++ would be a ridiculous choice for this type of application, and Rust manages to be a not ridiculous choice which is already a huge accomplishment. But that doesn't mean it's always a good choice.
hnarn 6y ago
The article ends with why Rust wasn’t chosen, but not before randomly disqualifying some other arguably more obscure languages; the author ends with saying that low level languages “suck”, and that garbage collectors are “great”, and yet Go isn’t mentioned once in the entire post? I’m not even arguing that Go would be the best or even a good choice here, but it seems strange to not even mention it as it also qualifies as great judging by the arbitrary statements of what does and doesn’t “suck” at the very end.
- ffijo32j 6y ago
  Go doesn't even have try-catch. In high level languages you can just write huge amount of code in a try-catch block that runs in a single transaction and if something goes bad, the entire transaction will be rolled back (without writing a single line of code for this to happen). In Go you have to handle return codes after each operation and that is 100x more code. I don't have time for such micro-optimizations.
  - throwaway894345 6y ago
    It’s not a micro-optimization, and you really shouldn’t naively optimize for lines of code. The explicit error handling is to encourage programmers to think about errors and to make it easier for readers to trace the error path through the call stack. Making it harder to punt on error handling is the whole point, just like static typing is tedious for the programmer who only cares if their code appears to be correct.
    - ffijo32j 6y ago
      I see absolutely no point in doing this. For example a network connection may occur any time I execute database query. Why should I handle it? Transaction will be automatically reverted to the previous state without me doing anything. Saves a lot of time (because I have to write much less code) and I never had any issues with this in 15 years.
      - throwaway894345 6y ago
blargmaster42_8 6y ago
Glorified code gen developer using his 15min of fame.
st1x7 6y ago
I'm suprised by: > Good parts: library ecosystem is great > Bad parts: having to fight the compiler My limited impression of Rust was the opposite on both points - that the compiler is super helpful but the crate ecosystem sucks.
- snazz 6y ago
  Depends on your point of reference and what you're using Rust for. The crate ecosystem is a huge breath of fresh air compared to package management in C and C++, but you can't expect it to have a very good solution for things like game engines and GUIs.
  - markrawls 6y ago
    Rust does actually have an excellent game engine crate, called tetra. It's heavily MonoGame/XNA inspired and is at a good stable point (in my opinion). I typically work more in the area of systems administration and automation, so I don't have much experience in the way GUI development, but there are Rust bindings for dear-imgui; and I would be very surprised if there weren't QT/GTK bindings.
unnouinceput 6y ago
TL;DR - I don't like Rust.
djhaskin987 6y ago
Ultimately it was a no-brainer to choose F#. Their current code base is in OCaml. Porting code from OCaml to F# sounds about as painful as moving from Python 2 to 3[1], which while painful, is waaay better than a total re-write. F# has much better libraries and ecosystem (because it is built on .NET). Their frustration was with the libraries and ecosystem of OCaml, not its syntax[2]. So, if they switch to F#, much of their core logic remains syntactically the same, and simply needs reworking (rather than rewriting). They also have to retool/call .NET libraries instead of C/OCaml ones, of course, but this is really the point of the move anyway. So: better libraries for what they're doing, not a total rewrite (just a re-work), preserve the overall codebase's structure, less disruption. It just makes good business sense to me. [1]: https://stackoverflow.com/questions/4239121/code-compatibility-between-ocaml-and-f https://stackoverflow.com/questions/4239121/code-compatibili... [2]: https://blog.darklang.com/leaving-ocaml/ https://blog.darklang.com/leaving-ocaml/
steveklabnik 6y ago
Thanks for writing this Paul; Rust is not going to be for everyone, and it's always nice to see reasoning laid out. For whatever it's worth, I wish I had more time to contribute some code back when you were asking for examples. It kinda felt to me like a lot of your code was trying to write OCaml in Rust, which causes a lot of this friction. But it was only a feeling; I didn't have the capacity to really dig in. It is also very true that taking time to get up to "what is the Rust-y way to do this" is a challenge for folks learning Rust, and a valid reason to use a language that works closer to how your brain works. I'm excited to see how F# works out for you; it's something I've always wanted to spend some time with, but never found the chance to actually do so.
- FullyFunctional 6y ago
  FWIW, I had the exact same experience as the OP; I was slowly getting comfortable with Rust until I wanted (high performance) cooperative multitasking, which in Rust meant getting into Async and all the complexities of Rust multiplied to a point where I'm reconsidering some life choices. And I have yet to match the performance of the C++ equivalent solution :( I suspect I would have liked the original green threads more, despite the drawbacks.
Scarbutt 6y ago
What I got from all these posts was that F# and Scala are the only statically typed functional programming languages with an advance type system that have access to a big ecosystem.
simias 6y ago
I think for people used to garbage collection, going to a non-gc ecosystem seems to be a big jump, akin to switching from an imperative to a functional language or from a dynamic to a statically typed language. Articles like this one make me think that maybe the Rust community should develop more resources to help people make this jump because the language is more likely to attract people with this background than other non-GC languages that are typically seen as lower level and relegated to high performance applications. I'm very anti-GC myself: I think the pros don't outweigh the cons in the overwhelming majority of the cases, they hide a complexity that you'll have to deal with eventually if you ever need to optimize your memory consumption and they encourage sloppy coding by not forcing you to think about ownership relations in your program's data. Having this debate over the years (often here on HN) I notice that the GC-fans tend to fall in two groups: those who really understand the tradeoffs and, unlike me, really think that GCs are worth it and those who simply have never really used languages with manual memory management - or not for very long - and seem to base their dislike of them on a lack of comprehension, cargo culting and associating it with C-style memory management where tracking memory allocations and frees adds a true cognitive overhead (unlike languages like C++ or Rust where usually RAII will take care of it 99% of the time). I realize that I'm building a strawman here, but when I see the way the author talks about memory management (and how much they hate it) I can't help thinking that they're Doing It Wrong: >I was actually surprised at how little the actual memory management bothered me. I'm a big believer in garbage collection, and not having to think about memory, so I expected to hate this part of Rust, but it turns out it's kinda OK. You put everything in a Box::new (regular heap memory) or RC::new (reference counted memory) or Arc::new (reference counted memory suitable to be used concurrently in different threads), and then when they go out of scope they'll be cleaned up. See, here's the problem. You don't like to think about memory management and ownership, so you think "oh, I'm just going to put everything in a Box/Rc". Spoiler alert: (A)Rc is garbage collection, it's just a very simplistic one. You haven't switched from GC to manual memory management, you've reimplemented a lousy GC in Rust and, unsurprisingly, it fails to compete with the good GCs of the languages you're used to. It's like when I started using Python and I naively tried to do RAII with destructors, only to realize that it's a very bad idea because you never know when the destructor is going to run in such a language. I'm currently implementing a rather asynchronous multicast Rust applications where I have 3 threads processing messages and passing them around to deal with retransmits, packet reconstruction etc... I have one(1) Arc, zero Rc and one(1) Box in the entire codebase (and I intend to get rid of this Box soon).
daxfohl 6y ago
The most surprising thing to me is that nobody has created a JVM port of F#.
- Multicomp 6y ago
  There is scala which is a functional proglang for jvm (but not F# ported to jvm)
  - daxfohl 6y ago
    Sure but the author explicitly rejected scala due to its too-many features, even though he'd have preferred jvm. So it sounds like there is still some opportunity.
- sideeffffect 6y ago
  F# has the expectation of reified generics (Don Syme, the language author created them for .NET before creating F#, after all), which are not a thing (yet?) on JVM. Conversely, Scala has the expectation of (partial) type erasure as is done on JVM, and that's probably why the .NET port didn't get very far.
  - daxfohl 6y ago
    But does F# depend on reified generics? My understanding is that everything is still expressible without them, except some reflection operations (that could just not carry over), and dealing with int vectors etc would be slower. Seems like not a big blocker unless I'm mistaken. Is there a deeper problem that I'm missing?
mamcx 6y ago
> I'm implementing a language that's basically F#/OCaml. So it makes sense that it's easier to implement in F#/OCaml. This is the key point. When your target and host language match in semantics is easier to be done in the host. You se how much gymnastics is necessary to implement near all langs on C. --- I done a lot of F# before move to Rust, and frankly, F# is great and Rust take a time to "click". I'm doing a lang on it (https://tablam.org https://tablam.org) and the truth is that Rust make things harder. Some of the points of the author are not that "bad" and with some time it will see that are non the real issues, but the fact is that anyone, specially (IMHO) if have done a lot of other langs (not named C or pascal) will have a miserable time the first try. Eventually, you Jump the wall of complexity and a lot of stuff click.. but then hit the REAL showstoppers (that are different depending according to what you do), and lets by real: Are truly hard. (maybe: But doing a lang, you hit this stuff more easily than "regular" coding. Doing a erp backend Rust is super productive to me) --- 2 things that make harder this stuff: Rc/Arc make complicated how mutate things. You need to wrap again with RefCell. I wish exist a super-charged RcRefCell that is "blessed" by the rust team as is with Rc, so the correct idioms to deal with this become widespread. Also, the trait system and the restrictions of object safety somehow make harder to mimic a OO system, that in part could make a lot easier to deal with lang implementation. Plus, I wish I can take alias everywhere, so i can cut the noise in syntax...
- zozbot234 6y ago
  > I wish exist a super-charged RcRefCell that is "blessed" by the rust team as is with Rc The syntax for that is Rc<RefCell<…>> and yes, this is clearly intended. The idea is that Rc is limited to managing multiple owners, while the RefCell part deals with shared mutability.
  - steveklabnik 6y ago
    Long ago, we did have those as one type: https://doc.rust-lang.org/0.5/std/arc.html#struct-mutexarc https://doc.rust-lang.org/0.5/std/arc.html#struct-mutexarc But yes, as you say, we decided to split them up, because it is easy to combine them, and you may want to combine other pieces in different ways. Rc<T> on its own is good in many cases, you may want Arc<RwLock<T>> instead of Arc<Mutex<T>>, etc.
cztomsik 6y ago
I wonder how one can not agree with Rich on Maybe/Option (and I am definitely not blindly in his camp, for example I think persistent data structures and STM are total overkill for my kinds of apps) Kotlin and TypeScript got null right. Rust got it wrong - it's not such a big deal because compiler will tell you but it's still wrong, it's still breaking change and it's also a kind of information leak.
oconnor663 6y ago
> Let's see if I can explain this. When you're writing an async, multi-threaded server in using the tokio runtime, async processes can be moved between threads. This means the memory can be copied, and so you need to ... pin things? OK, that's as much as I remember. Look in the HN comments after I publish this and I'm sure someone will explain better. Ok, here's my attempt to explain Pin. First a smidgen of background. In garbage collected languages, we can more or less say "everything is a pointer". And given how garbage collection works, we can extend that a little bit to say "anything can keep anything else alive, by holding onto a pointer to it." Now in C/C++/Rust, this is very much not the case. The most common example is if you try to return a pointer to one of your local variables. Your pointer cannot keep that variable alive, it becomes a "dangling pointer", and your code definitely does not work (but might appear to for a while). Nothing new so far. Now Rust comes along and decides to try to catch most of these bugs at compile time. So it makes you keep careful track of who points to what, and how long everything is planning on staying alive. As an important special case of this, one of the things Rust will essentially never let you do, is create object that holds a pointer to itself. It's basically the same problem we just mentioned: you might try to do something like returning that object, and the original memory location you took a pointer to would be destroyed, and the pointer in the returned copy would now be dangling. It's almost never safe, so Rust almost never allows it. That was pretty much the whole story for a while. But then things got trickier. Folks started working on async, and designing the new "await" keyword. Like in most other languages, the goal there was that you could write something that looked a lot like a regular function, but secretly the compiler would generate some sort of struct for you, and the variables in your function would actually be the fields of that struct. (Why anyone wants to do this is a long topic, but the reason is the same as in many other languages, basically because having lots of threads is slow.) And this runs is into a very tricky problem: Having one variable point to another variable is perfectly legal. We do that all the time. You can't write normal code without doing that. But now that variables are actually struct fields...that means we've got an object that points to itself. And we just said that was forbidden. The solution the Rust team landed on for this problem is called Pin. When you "pin" an object, you're essentially swearing that it will always stay in the same spot in memory, so that any pointers it has to itself will never become dangling. The exact way this is implemented involves a lot of Rust specifics (Pin is a struct, but Unpin is a trait), and the fact that it sits right at the boundary of safe code and unsafe code makes things trickier. But that's the general idea of pinning: a promise that I will never move this object or copy its bits somewhere else in memory.
fulafel 6y ago
Is there a place in the blog series where the async requirement is expanded on? Is it just the desire to transparently have Dark code use async IO so it's cheaper / can be said to be very scalable? edit: in https://blog.darklang.com/adventures-in-async/ https://blog.darklang.com/adventures-in-async/ there's a short clue saying "The Dark web server is currently synchronous, and so long or slow requests--at sufficient volume--can cause operational issues for us" but it doesn't go into details.
- pbiggar 6y ago
  Right, it's to make it more scalable. Since our users can write arbitrary code that runs on our servers, and can call 3rdparty servers, it's really not that hard to DOS us accidentally. Switching to async means that we're not just sitting there wasting resources when users do common stuff.

130 points by pimterry 6y ago

Small note for the author, I believe this was supposed to be outside of the quote block: > It received 39 pretty enthusiast[ic] replies.

lilyball 6y ago
No, the quote block is correct. It was quoting a HN comment (which includes the typo).
mumblemumble 6y ago
No, that's correct. The sentence is part of the message that was being quoted (which is linked up above).

cordite 6y ago

A follow up to this is what they did choose: F# This is posted at https://blog.darklang.com/new-backend-fsharp/ https://blog.darklang.com/new-backend-fsharp/

lubesGordi 6y ago

Did I miss something? "I don't like x,y,z, languages so I picked F#." I didn't see anything about F# in this post.

marcinzm 6y ago
He links in the blog to his previous long post explaining why he chose F#: https://blog.darklang.com/new-backend-fsharp/ https://blog.darklang.com/new-backend-fsharp/
mumblemumble 6y ago
It's in a separate post, referenced by the 2nd link in the article. The link text is just the word "posts", so it's easy to miss. https://blog.darklang.com/new-backend-fsharp/ https://blog.darklang.com/new-backend-fsharp/
- pbiggar 6y ago
  Thanks. Just updated the post to be a bit clearer on this.
brmgb 6y ago
No, they did three parts so they could hit the front page three times. The whole thing is clearly a Dark marketing campaign but people upvote it so the community must like it.
- robocat 6y ago
  Your comment should be flagged, but perhaps review the guidelines: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html And factually I think your opinion is incorrect. The first post didn’t seem like it was tailored for HN, and he said he was surprised: https://news.ycombinator.com/item?id=24981505 https://news.ycombinator.com/item?id=24981505 https://news.ycombinator.com/item?id=24980661 https://news.ycombinator.com/item?id=24980661 Meanwhile it looks like this discussion has been modded down to remove it from front page due to the low quality of comments (yes: this comment doesn’t meet standards either but maybe it helps you?...)
  - Dylan16807 6y ago
    Which part of the guidelines are you referring to? I don't think this counts as an accusation of shilling/astroturfing or similar. And they didn't say it shouldn't have been submitted.

qppo 6y ago

If you're writing async code you don't need to worry about pinning unless you're manually writing futures or designing an executor. But if you are curious, this chapter explains it pretty good: https://rust-lang.github.io/async-book/04_pinning/01_chapter.html https://rust-lang.github.io/async-book/04_pinning/01_chapter... > Apparently, recursion adds a new level of complexity to async. It really doesn't - it's just some users may be surprised by the lack of magic when it comes to Rust. Async is implemented with futures, futures are types, and you can't recursively define a type without boxing. > Apparently, this boxing and pinning is what you get when you don't have a GC, and that when you do have a GC, you simply don't need to deal with it. So that was the final straw for me. It's kind of weird to me that language developers would pass on Rust because they don't understand memory management. How is memory going to be managed in Dark? If it's GC'd, how is that implemented?

throwaway894345 6y ago
I’m guessing Dark is interpreted and that its objects are managed by the host language’s GC?
- pbiggar 6y ago
  Yes, that's right.
pbiggar 6y ago
> If it's GC'd, how is that implemented? The language in which the Dark interpreter is built has a GC, and we just use that. A GC is a huge undertaking, and we already have to build an editor, a language, a stdlib, and infrastructure.
- qppo 6y ago
  Why do you need to build an editor too? Side note, I've written a GC in rust (stop the world, compacting). It's not a cutting edge, concurrent generational GC but it's not that big of an undertaking. And it performs very well for small problem sets since memory is all slab allocated. Only a few hundred lines in Rust. I do definitely see how writing a tree walking interpreter is non ideal in Rust if you don't have a ton of experience writing Rust. I have a tiny lisp in Rust compared to doing it in dynamic languages it is quite verbose and hard to grok.

jbirer 6y ago

As a C, Go and Javascript programmer (nearly 10 years of experience with both of them), I really feel what the author is saying when they say there are too many ways to do something and fighting the compiler. There comes a point when the languages quirks start to eat into your productivity. That being said, I have a feeling that the author lacks programming expertise to develop a language (I am not saying this in an offensive way). They are dodging things that are needed to build a language (memory management, choices of libraries etc.).

luminati 6y ago
IIRC the author has a PhD in CS, and his thesis work was on PLT.
- ook 6y ago
  http://www.tara.tcd.ie/handle/2262/77562 http://www.tara.tcd.ie/handle/2262/77562 - "Design and implementation of an ahead-of-time compiler for PHP" is the PhD thesis in question. A google tech talk related to this work is on youtube https://www.youtube.com/watch?v=kKySEUrP7LA https://www.youtube.com/watch?v=kKySEUrP7LA You can find multiple other papers on google scholar where Paul was an author all related to programming languages, compilers etc.
- mlindner 6y ago
  Some people are much better with theory than the application of that theory. I've met a lot of theory people that just care about implementing something that does something to get the result they want rather than the nitty gritty of making things efficient.
busterarm 6y ago
> That being said, I have a feeling that the author lacks programming expertise to develop a language (I am not saying this in an offensive way). They are dodging things that are needed to build a language (memory management, choices of libraries etc.). The author is Paul Biggar, a former Sr. Compiler Engineer @ Mozilla and founder of CircleCI.
6y ago

mmastrac 6y ago

Yeah, the problem with jumping into Rust headfirst is that you assume that the most complex parts of the language are what you need to use to solve your problem. It turns out that the vast majority of Rust code is just fine using Arc and letting the reference counts handle your memory management. The lifetimes are a really cool feature for library/data structure authors, but if you're writing "web server" style code just use Arc, call clone, and be done with it, unless you've got a specific perf issue.

brundolf 6y ago
> It turns out that the vast majority of Rust code is just fine using Arc and letting the reference counts handle your memory management. There are disadvantages to Rc/Arc compared with GC, though: - They tend to be less performant - Reference cycles can leak - In languages where they exist at a library level instead of a language level, they're much less ergonomic than GC So if your core problem domain involves creating and disposing of lots of heap-allocated stuff all the time, you're probably better off just using a language with a proper GC. The Rust version may not even perform better if you're just using Arc, everywhere, by default. Use Rust when your core domain/hot paths can mostly stick to the single-ownership model (and ideally the stack). Rc/Arc/RefCell are a band-aid for when that fails.
- mc10 6y ago
  As someone who's relatively new to Rust, I'm curious: what is an example of a situation where someone might lean on Rc/Arc as a "band-aid", and what would be a more "idiomatic", non-Rc/Arc solution?
  - brundolf 6y ago
    Rust's key feature - the borrow-checker - relies on the idea that each value has a single "owner" at any given time. This owner can be a function, another value (a parent struct), etc. You can put these values on the heap, but if you use Box (the go-to for heap allocation), that pointer still has to have a single logical "owner". Under idiomatic Rust, each value effectively lives in one single "place". This allows the compiler to determine with 100% confidence at what point it's no longer being used and can therefore be de-allocated. Now, these values can be lent out ("borrowing") to sub-functions and such via references (mutable or immutable). Multiple immutable references can be handed out at once, but a mutable reference to a value has to be the only reference to that value of any kind, at a given time. The problem is, some domains really don't lend themselves to this restricted model. No two objects or functions can point, mutably, to the same object at the same time. You simply can't create a graph of inter-referenced objects where a single value may have multiple "parents". And sometimes even with a perfectly tree-like ownership structure moving values around can get complicated, because Rust has to know for sure that the ownership model is adhered to. This is where explicit lifetimes and such can come into play. Even writing a linked-list in Rust without using unsafe { } (or Rc's) is hard (https://rust-unofficial.github.io/too-many-lists/ https://rust-unofficial.github.io/too-many-lists/). In Rust, Rc's are kind of an admission of defeat. You're telling Rust not to perform its normal "compile-time" automatic deallocation, instead having it track references at runtime (which comes with overhead) to know when to de-allocate. What this buys you is basically an out from the ownership system: instead of handing off a plain reference to multiple places, which Rust may not let you do, you just clone the Rc and hand off that "new" value which can go anywhere it wants. That Rc is then what gets tracked by the ownership system and de-allocated, and when de-allocated it decrements the count (again, at runtime), and eventually that runtime mechanism (hopefully) decides the real value can be de-allocated. Basically any part of your code that uses Rc/Arc is giving up one of the biggest features of Rust. Which is totally fine, if you're reaping those advantages elsewhere and you just need to bridge a gap where ownership is too limiting. But if heap-juggling is going to be primary thing your program is doing, you'll probably have a better overall time with a GCed language.

brundolf 6y ago

Highly subjective and project-specific, but I still think it provides a nice counter-anecdote to the constant Rust praise we tend to see here. And I say that as a Rust fan myself. > ...but remember that garbage collectors are great. By having a GC, we don't have to do any of the stuff that causes all these problems in Rust. Maybe that costs performance, but I need the ability to quickly write code a lot more than I need the extra performance. This is very valid and sensible. Worth noting is that C++ would be a ridiculous choice for this type of application, and Rust manages to be a not ridiculous choice which is already a huge accomplishment. But that doesn't mean it's always a good choice.

hnarn 6y ago

The article ends with why Rust wasn’t chosen, but not before randomly disqualifying some other arguably more obscure languages; the author ends with saying that low level languages “suck”, and that garbage collectors are “great”, and yet Go isn’t mentioned once in the entire post? I’m not even arguing that Go would be the best or even a good choice here, but it seems strange to not even mention it as it also qualifies as great judging by the arbitrary statements of what does and doesn’t “suck” at the very end.

ffijo32j 6y ago
Go doesn't even have try-catch. In high level languages you can just write huge amount of code in a try-catch block that runs in a single transaction and if something goes bad, the entire transaction will be rolled back (without writing a single line of code for this to happen). In Go you have to handle return codes after each operation and that is 100x more code. I don't have time for such micro-optimizations.
- throwaway894345 6y ago
  It’s not a micro-optimization, and you really shouldn’t naively optimize for lines of code. The explicit error handling is to encourage programmers to think about errors and to make it easier for readers to trace the error path through the call stack. Making it harder to punt on error handling is the whole point, just like static typing is tedious for the programmer who only cares if their code appears to be correct.
  - ffijo32j 6y ago
    I see absolutely no point in doing this. For example a network connection may occur any time I execute database query. Why should I handle it? Transaction will be automatically reverted to the previous state without me doing anything. Saves a lot of time (because I have to write much less code) and I never had any issues with this in 15 years.
    - throwaway894345 6y ago

blargmaster42_8 6y ago

Glorified code gen developer using his 15min of fame.

st1x7 6y ago

I'm suprised by: > Good parts: library ecosystem is great > Bad parts: having to fight the compiler My limited impression of Rust was the opposite on both points - that the compiler is super helpful but the crate ecosystem sucks.

snazz 6y ago
Depends on your point of reference and what you're using Rust for. The crate ecosystem is a huge breath of fresh air compared to package management in C and C++, but you can't expect it to have a very good solution for things like game engines and GUIs.
- markrawls 6y ago
  Rust does actually have an excellent game engine crate, called tetra. It's heavily MonoGame/XNA inspired and is at a good stable point (in my opinion). I typically work more in the area of systems administration and automation, so I don't have much experience in the way GUI development, but there are Rust bindings for dear-imgui; and I would be very surprised if there weren't QT/GTK bindings.

unnouinceput 6y ago

TL;DR - I don't like Rust.

djhaskin987 6y ago

Ultimately it was a no-brainer to choose F#. Their current code base is in OCaml. Porting code from OCaml to F# sounds about as painful as moving from Python 2 to 3[1], which while painful, is waaay better than a total re-write. F# has much better libraries and ecosystem (because it is built on .NET). Their frustration was with the libraries and ecosystem of OCaml, not its syntax[2]. So, if they switch to F#, much of their core logic remains syntactically the same, and simply needs reworking (rather than rewriting). They also have to retool/call .NET libraries instead of C/OCaml ones, of course, but this is really the point of the move anyway. So: better libraries for what they're doing, not a total rewrite (just a re-work), preserve the overall codebase's structure, less disruption. It just makes good business sense to me. [1]: https://stackoverflow.com/questions/4239121/code-compatibility-between-ocaml-and-f https://stackoverflow.com/questions/4239121/code-compatibili... [2]: https://blog.darklang.com/leaving-ocaml/ https://blog.darklang.com/leaving-ocaml/

steveklabnik 6y ago

Thanks for writing this Paul; Rust is not going to be for everyone, and it's always nice to see reasoning laid out. For whatever it's worth, I wish I had more time to contribute some code back when you were asking for examples. It kinda felt to me like a lot of your code was trying to write OCaml in Rust, which causes a lot of this friction. But it was only a feeling; I didn't have the capacity to really dig in. It is also very true that taking time to get up to "what is the Rust-y way to do this" is a challenge for folks learning Rust, and a valid reason to use a language that works closer to how your brain works. I'm excited to see how F# works out for you; it's something I've always wanted to spend some time with, but never found the chance to actually do so.

FullyFunctional 6y ago
FWIW, I had the exact same experience as the OP; I was slowly getting comfortable with Rust until I wanted (high performance) cooperative multitasking, which in Rust meant getting into Async and all the complexities of Rust multiplied to a point where I'm reconsidering some life choices. And I have yet to match the performance of the C++ equivalent solution :( I suspect I would have liked the original green threads more, despite the drawbacks.

Scarbutt 6y ago

What I got from all these posts was that F# and Scala are the only statically typed functional programming languages with an advance type system that have access to a big ecosystem.

simias 6y ago

I think for people used to garbage collection, going to a non-gc ecosystem seems to be a big jump, akin to switching from an imperative to a functional language or from a dynamic to a statically typed language. Articles like this one make me think that maybe the Rust community should develop more resources to help people make this jump because the language is more likely to attract people with this background than other non-GC languages that are typically seen as lower level and relegated to high performance applications. I'm very anti-GC myself: I think the pros don't outweigh the cons in the overwhelming majority of the cases, they hide a complexity that you'll have to deal with eventually if you ever need to optimize your memory consumption and they encourage sloppy coding by not forcing you to think about ownership relations in your program's data. Having this debate over the years (often here on HN) I notice that the GC-fans tend to fall in two groups: those who really understand the tradeoffs and, unlike me, really think that GCs are worth it and those who simply have never really used languages with manual memory management - or not for very long - and seem to base their dislike of them on a lack of comprehension, cargo culting and associating it with C-style memory management where tracking memory allocations and frees adds a true cognitive overhead (unlike languages like C++ or Rust where usually RAII will take care of it 99% of the time). I realize that I'm building a strawman here, but when I see the way the author talks about memory management (and how much they hate it) I can't help thinking that they're Doing It Wrong: >I was actually surprised at how little the actual memory management bothered me. I'm a big believer in garbage collection, and not having to think about memory, so I expected to hate this part of Rust, but it turns out it's kinda OK. You put everything in a Box::new (regular heap memory) or RC::new (reference counted memory) or Arc::new (reference counted memory suitable to be used concurrently in different threads), and then when they go out of scope they'll be cleaned up. See, here's the problem. You don't like to think about memory management and ownership, so you think "oh, I'm just going to put everything in a Box/Rc". Spoiler alert: (A)Rc is garbage collection, it's just a very simplistic one. You haven't switched from GC to manual memory management, you've reimplemented a lousy GC in Rust and, unsurprisingly, it fails to compete with the good GCs of the languages you're used to. It's like when I started using Python and I naively tried to do RAII with destructors, only to realize that it's a very bad idea because you never know when the destructor is going to run in such a language. I'm currently implementing a rather asynchronous multicast Rust applications where I have 3 threads processing messages and passing them around to deal with retransmits, packet reconstruction etc... I have one(1) Arc, zero Rc and one(1) Box in the entire codebase (and I intend to get rid of this Box soon).

daxfohl 6y ago

The most surprising thing to me is that nobody has created a JVM port of F#.

Multicomp 6y ago
There is scala which is a functional proglang for jvm (but not F# ported to jvm)
- daxfohl 6y ago
  Sure but the author explicitly rejected scala due to its too-many features, even though he'd have preferred jvm. So it sounds like there is still some opportunity.
sideeffffect 6y ago
F# has the expectation of reified generics (Don Syme, the language author created them for .NET before creating F#, after all), which are not a thing (yet?) on JVM. Conversely, Scala has the expectation of (partial) type erasure as is done on JVM, and that's probably why the .NET port didn't get very far.
- daxfohl 6y ago
  But does F# depend on reified generics? My understanding is that everything is still expressible without them, except some reflection operations (that could just not carry over), and dealing with int vectors etc would be slower. Seems like not a big blocker unless I'm mistaken. Is there a deeper problem that I'm missing?

mamcx 6y ago

> I'm implementing a language that's basically F#/OCaml. So it makes sense that it's easier to implement in F#/OCaml. This is the key point. When your target and host language match in semantics is easier to be done in the host. You se how much gymnastics is necessary to implement near all langs on C. --- I done a lot of F# before move to Rust, and frankly, F# is great and Rust take a time to "click". I'm doing a lang on it (https://tablam.org https://tablam.org) and the truth is that Rust make things harder. Some of the points of the author are not that "bad" and with some time it will see that are non the real issues, but the fact is that anyone, specially (IMHO) if have done a lot of other langs (not named C or pascal) will have a miserable time the first try. Eventually, you Jump the wall of complexity and a lot of stuff click.. but then hit the REAL showstoppers (that are different depending according to what you do), and lets by real: Are truly hard. (maybe: But doing a lang, you hit this stuff more easily than "regular" coding. Doing a erp backend Rust is super productive to me) --- 2 things that make harder this stuff: Rc/Arc make complicated how mutate things. You need to wrap again with RefCell. I wish exist a super-charged RcRefCell that is "blessed" by the rust team as is with Rc, so the correct idioms to deal with this become widespread. Also, the trait system and the restrictions of object safety somehow make harder to mimic a OO system, that in part could make a lot easier to deal with lang implementation. Plus, I wish I can take alias everywhere, so i can cut the noise in syntax...

zozbot234 6y ago
> I wish exist a super-charged RcRefCell that is "blessed" by the rust team as is with Rc The syntax for that is Rc<RefCell<…>> and yes, this is clearly intended. The idea is that Rc is limited to managing multiple owners, while the RefCell part deals with shared mutability.
- steveklabnik 6y ago
  Long ago, we did have those as one type: https://doc.rust-lang.org/0.5/std/arc.html#struct-mutexarc https://doc.rust-lang.org/0.5/std/arc.html#struct-mutexarc But yes, as you say, we decided to split them up, because it is easy to combine them, and you may want to combine other pieces in different ways. Rc<T> on its own is good in many cases, you may want Arc<RwLock<T>> instead of Arc<Mutex<T>>, etc.

cztomsik 6y ago

I wonder how one can not agree with Rich on Maybe/Option (and I am definitely not blindly in his camp, for example I think persistent data structures and STM are total overkill for my kinds of apps) Kotlin and TypeScript got null right. Rust got it wrong - it's not such a big deal because compiler will tell you but it's still wrong, it's still breaking change and it's also a kind of information leak.

oconnor663 6y ago

> Let's see if I can explain this. When you're writing an async, multi-threaded server in using the tokio runtime, async processes can be moved between threads. This means the memory can be copied, and so you need to ... pin things? OK, that's as much as I remember. Look in the HN comments after I publish this and I'm sure someone will explain better. Ok, here's my attempt to explain Pin. First a smidgen of background. In garbage collected languages, we can more or less say "everything is a pointer". And given how garbage collection works, we can extend that a little bit to say "anything can keep anything else alive, by holding onto a pointer to it." Now in C/C++/Rust, this is very much not the case. The most common example is if you try to return a pointer to one of your local variables. Your pointer cannot keep that variable alive, it becomes a "dangling pointer", and your code definitely does not work (but might appear to for a while). Nothing new so far. Now Rust comes along and decides to try to catch most of these bugs at compile time. So it makes you keep careful track of who points to what, and how long everything is planning on staying alive. As an important special case of this, one of the things Rust will essentially never let you do, is create object that holds a pointer to itself. It's basically the same problem we just mentioned: you might try to do something like returning that object, and the original memory location you took a pointer to would be destroyed, and the pointer in the returned copy would now be dangling. It's almost never safe, so Rust almost never allows it. That was pretty much the whole story for a while. But then things got trickier. Folks started working on async, and designing the new "await" keyword. Like in most other languages, the goal there was that you could write something that looked a lot like a regular function, but secretly the compiler would generate some sort of struct for you, and the variables in your function would actually be the fields of that struct. (Why anyone wants to do this is a long topic, but the reason is the same as in many other languages, basically because having lots of threads is slow.) And this runs is into a very tricky problem: Having one variable point to another variable is perfectly legal. We do that all the time. You can't write normal code without doing that. But now that variables are actually struct fields...that means we've got an object that points to itself. And we just said that was forbidden. The solution the Rust team landed on for this problem is called Pin. When you "pin" an object, you're essentially swearing that it will always stay in the same spot in memory, so that any pointers it has to itself will never become dangling. The exact way this is implemented involves a lot of Rust specifics (Pin is a struct, but Unpin is a trait), and the fact that it sits right at the boundary of safe code and unsafe code makes things trickier. But that's the general idea of pinning: a promise that I will never move this object or copy its bits somewhere else in memory.

fulafel 6y ago

Is there a place in the blog series where the async requirement is expanded on? Is it just the desire to transparently have Dark code use async IO so it's cheaper / can be said to be very scalable? edit: in https://blog.darklang.com/adventures-in-async/ https://blog.darklang.com/adventures-in-async/ there's a short clue saying "The Dark web server is currently synchronous, and so long or slow requests--at sufficient volume--can cause operational issues for us" but it doesn't go into details.

pbiggar 6y ago
Right, it's to make it more scalable. Since our users can write arbitrary code that runs on our servers, and can call 3rdparty servers, it's really not that hard to DOS us accidentally. Switching to async means that we're not just sitting there wasting resources when users do common stuff.