12 ms·

About 2 months ago I would have said the same as the author, but I kept running against the hard edges of Rust: the borrow checker. I realised that while I real

About 2 months ago I would have said the same as the author, but I kept running against the hard edges of Rust: the borrow checker. I realised that while I really liked using algebraic data types (e.g. Enums) and pattern matching, the borrow checker and the low level memory concerns meant I spent a lot of time fighting the borrow checker instead of fighting the PL issues at the heart of my project. So while tokenising/parsing was nice, interpreting and typechecking became the bane of my existence With that realisation I started looking for another more suitable language - I knew the FP aspects of Rust are what I was looking for so at first I considered something like F# but I didn't like that it's tied to microsoft/.NET. Looking a bit further I could have gone with something like Zig/C but then I lose the FP niceness I'm looking for. I also spent a fair amount of time looking at Go, but eventually decided that 1. I wanted a fair amount of syntax sugar, and 2. golang is a server side language, a lot of its features and library are geared towards this use case. Finally I found OCaml, what really convinced me was seeing the syntax was like a friendly version of Haskell, or like Rust without lifetimes. In fact the first Rust compiler was written in OCaml, and OCaml is well known in the programming language space. I'm still learning OCaml so I'm not sure I can give a fair review yet, but so far it's exactly what I was looking for.

ralegh 2y ago
I wouldn't call Go a 'server side' language. The Go compiler is written in Go, for example! Cross compilation and (relatively) small binaries make it super easy for distribution. Syntax sugar is a fair point though, it doesn't lend itself to functional-y pattern matching.
- miningape 2y ago
  Well now I've got to go check out the go compiler! That sounds really interesting. I was mainly referring to go having a lot more developed concurrency features, which while they're great I didn't really want to use them for my toy language, it seemed like I was throwing away a lot of what makes golang great just because of the nature of my project. The rest of the golang ecosystem I found really nice actually, and imo it had a really great set of tools for reading/writing to files - and also I like that everything is apart of the go binary, it certainly is easier than juggling between opam and dune (used for OCaml for example).
  - ralegh 2y ago
    That's fair, the concurrency features are very handy though optional of course. The ecosystem and tooling are great, probably the best I've worked with. But the main reason I reach for Go is that it's got tiny mental overhead. There's a handful of language features so it becomes obvious what to use, so you can focus on the actual goal of the project. There are some warts of course. Heavy IO code can be riddled with err checks (actually, why I find it a bit awkward for servers). Similarly the stdlib is quite verbose when doing file system manipulation, I may try https://github.com/chigopher/pathlib https://github.com/chigopher/pathlib because Python's pathlib is by far my favourite interface.
- eminent101 2y ago
  > The Go compiler is written in Go, for example! Do you know how they avoid the GC in the Go implementation of the Go compiler? If I understand correctly they need to implement the Go garbage collector in their Go implementation of the Go compiler. But Go already has a garbage collector. So how do they avoid invoking Go's garbage collector so that they can implement the garbage collector of the Go language they are implementing? Not sure if I'm making sense but I'd like to know more about this from those who understand this more than I do.
  - afandian 2y ago
    I don't understand the question as it's written. But the shape of the question feels like you're asking about whether an interpreter (which the compiler is not) uses the GC of the host language?
    - ben0x539 2y ago
      I think they're asking how the code in the Go runtime (not the compiler, that being an interesting but also maybe non-obvious distinction!) that implements the garbage collector, a core feature of the language, avoids needing the garbage collector to already exist to be able to run, being written in the language that it's a core feature of. I suspect the answer is just something like "by very carefully not using language features that might tempt the compiler to emit something that requires an allocation". I think it's a fair question as it's not really obvious that that's possible--do you just avoid calling make() and new() and forming pointers to local variables that might escape? Do you need to run on a magical goroutine that won't try to grow its stack with gc-allocated segments? Can you still use slices (probably yes, just not append() or the literal syntax), closures (probably only trivial ones without local captures?), maps (probably no)...? I think the relevant code is https://github.com/golang/go/blob/master/src/runtime/mgc.go https://github.com/golang/go/blob/master/src/runtime/mgc.go and adjacent files. I see some annotations like //go:systemstack, //go:nosplit, //go:nowritebarrier that are probably relevant but I wouldn't know if there's any other specific requirements for that code.
      - 2y ago
ThePhysicist 2y ago
Lots of folks use Golang on the client side, even on mobile (for which Go has really great support with go-mobile). Of course it adds around 10-20 MB to your binary and memory footprint but in todays world that's almost nothing. I think Tailscale e.g. uses Golang as a cross-platform Wireguard layer in their mobile and desktop apps, seems to work really well. You wouldn't build a native UI using Golang of course but for low-level stuff it's fantastic. Tinygo even allows you to write Golang for microcontrollers or the web via Webassembly, lots of things aren't supported there but a large part of the standard library is.
- ducktapeofficer 2y ago
  Saying a the language adds 10-20 mb and go on to say it's almost nothing is avoiding the issue raised. The footprint always matters and we should use the right tool for the right job.
  - deleted 2y ago
    [deleted]
  - packetlost 2y ago
    It's not ignoring it, it's saying that 20Mb of data isn't really a lot these days which is objectively true for most contexts.
    - guappa 2y ago
      It's not just data. It's data that needs to be memory mapped and faulted into RAM. Go programs do have kinda slower startup times compared to regular dynamically linked C/C++ programs.
pimeys 2y ago
OCaml has been pretty common tool to write parsers for many years. Not a bad choice. I've written parsers professionally with Rust for two companies now. I have to say the issues you had with the borrow checker are just in the beginning. After working with Rust a bit you realize it works miracles for parsers. Especially if you need to do runtime parsing in a network service serving large traffic. There are some good strategies we've found out to keep the borrow checker happy and at the same time writing the fastest possible code to serve our customers. I highly recommend taking a look how flat vectors for the AST and using typed vector indices work. E.g. you have vector for types as `Vec<Type>` and fields in types as `Vec<(TypeId, Field)>`. Keep these sorted, so you can implement lookups with a binary search, which works quite well with CPU caches and is definitely faster than a hashmap lookup. The other cool thing with writing parsers with Rust is how there are great high level libraries for things like lexing: https://crates.io/crates/logos https://crates.io/crates/logos The cool thing with Logos is it keeps the source data as a string under the surface, and just refers to a specific locations in it. Now use these tokens as a basis for your AST tree, which is all flat data structures and IDs. Simplify the usage with a type: #[Clone, Copy] struct Walker<'a, Id> { pub id: Id, pub ast: &'a Ast, } impl<'a, Id> Walker<'a, Id> { pub fn walk<T>(self, other_id: T) -> Walker<'a, T> { Walker { id: other_id, ast: self.ast } } } Now you can specialize these with type aliases: type TypeWalker<'a> = Walker<'a, TypeId>; And implement methods: impl<'a> TypeWalker<'a> { fn as_ref(&self) -> &'a Type { &self.ast[self.id] } fn name(&self) -> &'a str { &self.as_ref().name } } From here you can introduce string interning if needed, it's easy to extend. What I like about this design is how all the IDs and Walkers are Copy, so you can pass them around as you like. There's also no reference counting needed anywhere, so you don't need to play the dance with Arc/Weak. I understand Rust feels hard especially in the beginning. You need to program more like you write C++, but with Rust you are enforced to play safe. I would say an amazing strategy is to first write a prototype with Ocaml, it's really good for that. Then, if you need to be faster, do a rewrite in Rust.
- miningape 2y ago
  Thanks for your comment, you've given me a lot to chew on and I think I'll need to bookmark this page. > I've written parsers professionally with Rust for two companies now If you don't mind me asking, which companies? Or how do you get into this industry within an industry? I'd really love to work on some programming language implementations professionally (although maybe that's just because I've built them non-professionally until now), > Especially if you need to do runtime parsing in a network service serving large traffic. I almost expected something like this, it just makes sense with how the language is positioned. I'm not sure if you've been following cloudflare's pingora blogs but I've found them very interesting because of how they are able to really optimise parts of their networking without looking like a fast-inverse-sqrt. > There's also no reference counting needed anywhere, so you don't need to play the dance with Arc/Weak. I really like the sound of this, it wasn't necessarily confusing to work with Rc and Weak but more I had to put in a lot of extra thought up front (which is also valuable don't get me wrong). > I would say an amazing strategy is to first write a prototype with Ocaml, it's really good for that. Thanks! Maybe then the Rust code I have so far won't be thrown in the bin just yet.
radicalbyte 2y ago
I've been writing a lot of Golang in the last year and I wouldn't use it for writing a parser. It's just a modernised C, the model it provides is very simple (coming from C# the simplicity actually made it harder to learn!) and is very well suited to small, focused applications where a low conceptual load are beneficial and the trade off of verbosity are acceptable. F# or even the latest version of C# are what I would recommend. Yes Microsoft are involved but if you're going to live in a world where you won't touch anything created by evil corporations then you're going to have a hard time. Java, Golang, Python, TypeScript/Javascript and Swift all suffer from this. That leaves you with very little choice. I'd be interested in hearing your thoughts over OCaml after a year or so of using it. The Haskell-likes are very interesting but Haskell itself has a poor learning curve / benefit ratio for me (Rust is similar there actually; I mastered C# and made heavy use of the type system but that involved going very very deep into some holes and I don't have the time to do that with Rust).
- guappa 2y ago
  I wouldn't use it to write an hello world :D
- galangalalgol 2y ago
  F# and Ocaml are still functionally identical to the point many programs would compile in either right? F# Ocaml and Rust seem a lot more similar to me than any of them are to haskell, or go for that matter. I like Haskell, but my brain hasn't made thinking that way native yet.
- MrMcCall 2y ago
  Python suffers by having been created by an evil corporation? Have I missed something GvR or his team did?
krick 2y ago
Bringing up goland always annoys me for some reason. Like, it's really practical, it is fast, but not actually low-level, it compiles fast, and most importantly it is very popular and has all the libraries. It seems like I should use it. But I just almost irrationally hate the language itself. Everything about it is just so ugly. It's a language invented in 2009 by some C-people who are apparently oblivious to everything that excited PL design folks for the last 20 years (as of 2009). PHP in 2009 was already a more modern and better designed language than goddamn golang. And golang didn't really improve since. I just cannot let it go somehow.
- pimeys 2y ago
  I know, I'm on the same boat. What I realized is I just need to avoid the companies using Go and I don't really need to be vocal about my dislike. It's not my loss if others find the language useful, but for me it either solves problems I'm not interested in solving, or the language and tooling just does not make it for me. But, I can always just write Rust and be happy where I am. Or, to be honest, would not be very unhappy with F#, Haskell or Ocaml either.
  - meowface 2y ago
    >What I realized is I just need to avoid the companies using Go What do you mean, exactly?
    - leftyspook 2y ago
      I'd imagine not seeking employment at Google is a big part.
      - tempest_ 2y ago
        Go is definitely used a lot more outside of google as of late. Anecdotally I would say where a lot of companies would have used Java in the past they are now turning to go for their server-side/backend service implementations.
smolder 2y ago
I think you missed something if you felt the borrow checker made things too hard. You can just copy and move on. Most languages do less efficient things anyway.
- miningape 2y ago
  Oh no, you're right - especially looking at my last few commits this is very much what some parts of the project became. And when I was looking at it I felt like I was throwing away so much of the goodness Rust provides and it really irritated me. Looking at Primeys' comment he actually gave some really interesting suggestions on how to manage this without needing Rc / weak pointers or copying loads of dynamic memory all over the place. Instead you have a flat structure of copy-able elements, giving you better cache locality and a really easy way to work with them.
rtpg 2y ago
I think one core thing that you have to do with ASTs in Rust is to absolutely not store strings of the like in your AST. Instead you want to use something like static string libraries (so you get cheap clones and interning) and for things like positions in text you want to use just indices. Absolutely avoid storing references if you can avoid it! The more your stuff is held in things that are cheap/free to clone, the less you have to fight the borrow checker… since you can get away with clones! And for actual interpretation there’s these libraries that can help a lot for memory management with arenas and the like. It’s super specialized stuff but helps to give you perf + usability. Projects like Ruffle use this heavily and it’s nice when you figure out the patterns Having said that OCaml and Haskell are both languages that will do all of this “for free” with their built in reference counting and GC… I just like the idea of going very fast in Rust
FrustratedMonky 2y ago
FSharp is OCaml to great extent. So if you don't have the need to stay away from MS/.NET, it is more 'open source' than the rest of MS products. MS did release Fsharp with an Open Source License. But, it does still run on .NET. At this point, isn't every major language controlled by one main corporate entity? Except Python? But Python doesn't have algebraic types, or very complete pattern matching.
- adsharma 2y ago
  I still believe that a variant of python that has algebraic types and pattern matching beats Rust for writing parsers quickly. My effort has been in adding these features to a front end language that transpiles to an underlying FP language, including but not limited to Rust.
- mikeschinkel 2y ago
  PHP is not controlled by one main corporate entity. {Runs, ducks, hides. :-D}
neonsunset 2y ago
> I considered something like F# but I didn't like that it's tied to microsoft/.NET. Could you explain your thought process when deciding to not use F# because it runs on top of .NET? (both of which are open-source, and .NET is what makes F# fast and usable in almost every domain)
- balencpp 2y ago
  I am genuinely curious too. .NET is a very mature, very performant runtime, and I think of F#, a beautiful, productive language, running on it a big pro. Perhaps things used to be different about/regarding Microsoft?
  - neonsunset 2y ago
    Yeah. I'm having so much fun with F# that I absolutely did not anticipate. Sure, it's something everyone using .NET knows about but I genuinely underestimated it and wish more people gave it a try. Such a good language. As for the hate - my pet theory is that developers need something like a sacrificial lamb to blame their misfortunes on, and a banner to rally under which often happens to be "against that other group" or "against that competing language", and because .NET is platform that happens to be made by microsoft and is a host for two very powerful multi-paradigm languages causes it to be a point of contention for many. From what I've seen, other languages do not receive so much undeserved hate and here on HN some like Go, Ruby or BEAM family receive copious amount of unjustified praise not rooted in technical merits.
almostdeadguy 2y ago
With some patience and practice, I think reasoning about borrows becomes second nature. And what it buys you with lexing/parsing is the ability to do zero-copy parsing.
omginternets 2y ago
I had a similar journey of enlightenment that likewise led me to OCaml. Unless you're doing low-level systems programming, OCaml will give you the "if it compiles, it's probably right" vibe with much less awkward stuff to type.
packetlost 2y ago
Go is not a great language to write parsers in IMO, it just doesn't have anything that makes such a task nice. That being said, people seem to really dislike Go here, which is fine, but somewhat surprising. Go is extremely simple. If you take a look at it's creators pedigree, that should make a ton of sense: they want you to make small, focused utilities and systems that use message passing (typically via a network) as the primary means of scaling complexity. What I find odd about this is that it was originally intended as a C++ replacement.
- marcosdumay 2y ago
  Message passing is an horrible means of scaling complexity, unless you are so big and have so many developers working with you that you can't use anything else.
- dinosaurdynasty 2y ago
  Go is simple at the cost of increasing the complexity of stuff written in it.
kemaru 2y ago
You wouldn't be losing FP niceness with Zig, and the pattern matching and enum situation is also similar to Rust. Even better, in a few areas, for example arbitrary-width integers and enum tagging in unions/structs. Writing parsers and low level device drivers is actually quite comfortable in Zig.
MrMcCall 2y ago
(OCaml Question Ahead) I agree on F#. It changed my C && OO perspective in fantastic ways, but I too can't support anything Microsoft anymore. But, seeing as OCaml was the basis for F#, I have a question, though: Does OCaml allow the use of specifically sized integer types? I seem to remember in my various explorations that OCaml just has a kind of "number" type. If I want a floating point variable, I want a specific 32- or 64- or 128-bit version; same with my ints. I did very much like F# having the ability to specify the size and signedness of my int vars. Thanks in advance, OCaml folks.
- neonsunset 2y ago
  F# is a far better option from a practical standpoint when compared to alternatives. By simple virtue of using .NET and having access to very wide selection of libraries that make it a non-issue when deciding to solve a particular business case. It also has an alternate compiler Fable which can target JS allowing the use of F# in front-end. Other options have worse support and weaker tooling, and often not even more open development process (e.g. you can see and contribute to ongoing F# work on Github). This tired opinion ".net bad because microsoft bad" has zero practical relevance to actually using C# itself and even more so F# and it honestly needs to die out because it borders on mental illness. You can hate microsoft products, I do so too, and still judge a particular piece of techology and the people that work on it on their merits.
  - MrMcCall 2y ago
    I was asking about whether or not OCaml has the ability to target specific integer size and signedness, as F# does. I would like to construct precise software that targets the specific kinds of ints that compilers have historically facilitated microprocessors to process, but using a functional language instead of C or C++. The F# folks (including Don Syme) did a fantastic job on the early versions of the language (I used it up to ver 2 or early 3), but I am tired of the corporate engine that funds that ecosystem. I now construct my software within an operating system of a different pedigree. Such considerations are important to me, but thanks for sharing your preference. As for me, I hate nothing or no one, but I am as picky as a poor man can be about whom I choose to rely on for my tools. As for your opinion on the borderlands of mental illness, I'll contact you at your outlook email address should I seek your opinion about such differently-technical topics. But I was only asking if the OCaml compiler can target specific varieties of ints, as the .NET compiler does.
PittleyDunkin 2y ago
The borrow checker is definitely a pain, but it stops being such a pain once you design your types around ownership and passing around non-owned pointers or references or indexes.
- lsllc 2y ago
  This. I've found the same, being effective in Rust really requires that you change your way of thinking about your data structures (and code layout). Once I realized that, I was no longer fighting the borrow checker and I've been about to build complex code that more or less worked immediately. As I look back on it, I think what a pain it would have been to write and debug in C, although doing it in C would appear to be "easier".
wyager 2y ago
Having written probably several hundred kloc of both Haskell and OCaml, I strongly prefer Haskell. A very simple core concept wrapped in an extremely powerful shell. Haskell is a lot better for parsing tasks because (among other considerations) its more powerful type system can better express constraints on grammars.
materielle 2y ago
I love Go for writing servers. And in fact, I do it professionally. But I totally agree that for parsers, it’s not the right tool for the job. First off, the only way to express union types is with runtime reflection. You might as well be coding in Python (but without the convenient syntax sugar). Second off, “if err != nil” is really terrible in parsers. I’m actually somewhat of a defender of Go’s error handling approach in servers. Sure, it could have used a more convenient syntax. But in servers, I almost never return an error without handling it or adding additional context. The same isn’t true in parser’s though. Almost half of my parser code was error checks that simply wouldn’t exist in other languages. For Rust, I think the value proposition is if you are also writing a virtual machine or an interpreter, your compiler front end can be written in the same language as your backend. Your other alternatives are C and C++, but then you don’t have sum types. You could write the front end in Ocaml, but then you would have to write the backend and runtime in some other language anyways.
balencpp 2y ago
Did you discover Scala 3 and give it a thought? I think of it as Rust with an _overall_ stronger type-system, but where you don't have to worry about memory management. It has an amazing standard library, particularly around collections. You get access to the amazing JVM ecosystem. And more. Martin Odersky in fact sees Scala's future lying in being a simpler Rust. Also, regarding F#. It runs on .NET, and indeed, since the ecosystem and community are very small, you need to rely on .NET (basically C#) libraries. But it's really not "tied" to Microsoft and is open source.

ralegh 2y ago

I wouldn't call Go a 'server side' language. The Go compiler is written in Go, for example! Cross compilation and (relatively) small binaries make it super easy for distribution. Syntax sugar is a fair point though, it doesn't lend itself to functional-y pattern matching.

miningape 2y ago
Well now I've got to go check out the go compiler! That sounds really interesting. I was mainly referring to go having a lot more developed concurrency features, which while they're great I didn't really want to use them for my toy language, it seemed like I was throwing away a lot of what makes golang great just because of the nature of my project. The rest of the golang ecosystem I found really nice actually, and imo it had a really great set of tools for reading/writing to files - and also I like that everything is apart of the go binary, it certainly is easier than juggling between opam and dune (used for OCaml for example).
- ralegh 2y ago
  That's fair, the concurrency features are very handy though optional of course. The ecosystem and tooling are great, probably the best I've worked with. But the main reason I reach for Go is that it's got tiny mental overhead. There's a handful of language features so it becomes obvious what to use, so you can focus on the actual goal of the project. There are some warts of course. Heavy IO code can be riddled with err checks (actually, why I find it a bit awkward for servers). Similarly the stdlib is quite verbose when doing file system manipulation, I may try https://github.com/chigopher/pathlib https://github.com/chigopher/pathlib because Python's pathlib is by far my favourite interface.
eminent101 2y ago
> The Go compiler is written in Go, for example! Do you know how they avoid the GC in the Go implementation of the Go compiler? If I understand correctly they need to implement the Go garbage collector in their Go implementation of the Go compiler. But Go already has a garbage collector. So how do they avoid invoking Go's garbage collector so that they can implement the garbage collector of the Go language they are implementing? Not sure if I'm making sense but I'd like to know more about this from those who understand this more than I do.
- afandian 2y ago
  I don't understand the question as it's written. But the shape of the question feels like you're asking about whether an interpreter (which the compiler is not) uses the GC of the host language?
  - ben0x539 2y ago
    I think they're asking how the code in the Go runtime (not the compiler, that being an interesting but also maybe non-obvious distinction!) that implements the garbage collector, a core feature of the language, avoids needing the garbage collector to already exist to be able to run, being written in the language that it's a core feature of. I suspect the answer is just something like "by very carefully not using language features that might tempt the compiler to emit something that requires an allocation". I think it's a fair question as it's not really obvious that that's possible--do you just avoid calling make() and new() and forming pointers to local variables that might escape? Do you need to run on a magical goroutine that won't try to grow its stack with gc-allocated segments? Can you still use slices (probably yes, just not append() or the literal syntax), closures (probably only trivial ones without local captures?), maps (probably no)...? I think the relevant code is https://github.com/golang/go/blob/master/src/runtime/mgc.go https://github.com/golang/go/blob/master/src/runtime/mgc.go and adjacent files. I see some annotations like //go:systemstack, //go:nosplit, //go:nowritebarrier that are probably relevant but I wouldn't know if there's any other specific requirements for that code.
    - 2y ago

ThePhysicist 2y ago

Lots of folks use Golang on the client side, even on mobile (for which Go has really great support with go-mobile). Of course it adds around 10-20 MB to your binary and memory footprint but in todays world that's almost nothing. I think Tailscale e.g. uses Golang as a cross-platform Wireguard layer in their mobile and desktop apps, seems to work really well. You wouldn't build a native UI using Golang of course but for low-level stuff it's fantastic. Tinygo even allows you to write Golang for microcontrollers or the web via Webassembly, lots of things aren't supported there but a large part of the standard library is.

ducktapeofficer 2y ago
Saying a the language adds 10-20 mb and go on to say it's almost nothing is avoiding the issue raised. The footprint always matters and we should use the right tool for the right job.
- deleted 2y ago
  [deleted]
- packetlost 2y ago
  It's not ignoring it, it's saying that 20Mb of data isn't really a lot these days which is objectively true for most contexts.
  - guappa 2y ago
    It's not just data. It's data that needs to be memory mapped and faulted into RAM. Go programs do have kinda slower startup times compared to regular dynamically linked C/C++ programs.

pimeys 2y ago

OCaml has been pretty common tool to write parsers for many years. Not a bad choice. I've written parsers professionally with Rust for two companies now. I have to say the issues you had with the borrow checker are just in the beginning. After working with Rust a bit you realize it works miracles for parsers. Especially if you need to do runtime parsing in a network service serving large traffic. There are some good strategies we've found out to keep the borrow checker happy and at the same time writing the fastest possible code to serve our customers. I highly recommend taking a look how flat vectors for the AST and using typed vector indices work. E.g. you have vector for types as `Vec<Type>` and fields in types as `Vec<(TypeId, Field)>`. Keep these sorted, so you can implement lookups with a binary search, which works quite well with CPU caches and is definitely faster than a hashmap lookup. The other cool thing with writing parsers with Rust is how there are great high level libraries for things like lexing: https://crates.io/crates/logos https://crates.io/crates/logos The cool thing with Logos is it keeps the source data as a string under the surface, and just refers to a specific locations in it. Now use these tokens as a basis for your AST tree, which is all flat data structures and IDs. Simplify the usage with a type: #[Clone, Copy] struct Walker<'a, Id> { pub id: Id, pub ast: &'a Ast, } impl<'a, Id> Walker<'a, Id> { pub fn walk<T>(self, other_id: T) -> Walker<'a, T> { Walker { id: other_id, ast: self.ast } } } Now you can specialize these with type aliases: type TypeWalker<'a> = Walker<'a, TypeId>; And implement methods: impl<'a> TypeWalker<'a> { fn as_ref(&self) -> &'a Type { &self.ast[self.id] } fn name(&self) -> &'a str { &self.as_ref().name } } From here you can introduce string interning if needed, it's easy to extend. What I like about this design is how all the IDs and Walkers are Copy, so you can pass them around as you like. There's also no reference counting needed anywhere, so you don't need to play the dance with Arc/Weak. I understand Rust feels hard especially in the beginning. You need to program more like you write C++, but with Rust you are enforced to play safe. I would say an amazing strategy is to first write a prototype with Ocaml, it's really good for that. Then, if you need to be faster, do a rewrite in Rust.

miningape 2y ago
Thanks for your comment, you've given me a lot to chew on and I think I'll need to bookmark this page. > I've written parsers professionally with Rust for two companies now If you don't mind me asking, which companies? Or how do you get into this industry within an industry? I'd really love to work on some programming language implementations professionally (although maybe that's just because I've built them non-professionally until now), > Especially if you need to do runtime parsing in a network service serving large traffic. I almost expected something like this, it just makes sense with how the language is positioned. I'm not sure if you've been following cloudflare's pingora blogs but I've found them very interesting because of how they are able to really optimise parts of their networking without looking like a fast-inverse-sqrt. > There's also no reference counting needed anywhere, so you don't need to play the dance with Arc/Weak. I really like the sound of this, it wasn't necessarily confusing to work with Rc and Weak but more I had to put in a lot of extra thought up front (which is also valuable don't get me wrong). > I would say an amazing strategy is to first write a prototype with Ocaml, it's really good for that. Thanks! Maybe then the Rust code I have so far won't be thrown in the bin just yet.

radicalbyte 2y ago

I've been writing a lot of Golang in the last year and I wouldn't use it for writing a parser. It's just a modernised C, the model it provides is very simple (coming from C# the simplicity actually made it harder to learn!) and is very well suited to small, focused applications where a low conceptual load are beneficial and the trade off of verbosity are acceptable. F# or even the latest version of C# are what I would recommend. Yes Microsoft are involved but if you're going to live in a world where you won't touch anything created by evil corporations then you're going to have a hard time. Java, Golang, Python, TypeScript/Javascript and Swift all suffer from this. That leaves you with very little choice. I'd be interested in hearing your thoughts over OCaml after a year or so of using it. The Haskell-likes are very interesting but Haskell itself has a poor learning curve / benefit ratio for me (Rust is similar there actually; I mastered C# and made heavy use of the type system but that involved going very very deep into some holes and I don't have the time to do that with Rust).

guappa 2y ago
I wouldn't use it to write an hello world :D
galangalalgol 2y ago
F# and Ocaml are still functionally identical to the point many programs would compile in either right? F# Ocaml and Rust seem a lot more similar to me than any of them are to haskell, or go for that matter. I like Haskell, but my brain hasn't made thinking that way native yet.
MrMcCall 2y ago
Python suffers by having been created by an evil corporation? Have I missed something GvR or his team did?

krick 2y ago

Bringing up goland always annoys me for some reason. Like, it's really practical, it is fast, but not actually low-level, it compiles fast, and most importantly it is very popular and has all the libraries. It seems like I should use it. But I just almost irrationally hate the language itself. Everything about it is just so ugly. It's a language invented in 2009 by some C-people who are apparently oblivious to everything that excited PL design folks for the last 20 years (as of 2009). PHP in 2009 was already a more modern and better designed language than goddamn golang. And golang didn't really improve since. I just cannot let it go somehow.

pimeys 2y ago
I know, I'm on the same boat. What I realized is I just need to avoid the companies using Go and I don't really need to be vocal about my dislike. It's not my loss if others find the language useful, but for me it either solves problems I'm not interested in solving, or the language and tooling just does not make it for me. But, I can always just write Rust and be happy where I am. Or, to be honest, would not be very unhappy with F#, Haskell or Ocaml either.
- meowface 2y ago
  >What I realized is I just need to avoid the companies using Go What do you mean, exactly?
  - leftyspook 2y ago
    I'd imagine not seeking employment at Google is a big part.
    - tempest_ 2y ago
      Go is definitely used a lot more outside of google as of late. Anecdotally I would say where a lot of companies would have used Java in the past they are now turning to go for their server-side/backend service implementations.

smolder 2y ago

I think you missed something if you felt the borrow checker made things too hard. You can just copy and move on. Most languages do less efficient things anyway.

miningape 2y ago
Oh no, you're right - especially looking at my last few commits this is very much what some parts of the project became. And when I was looking at it I felt like I was throwing away so much of the goodness Rust provides and it really irritated me. Looking at Primeys' comment he actually gave some really interesting suggestions on how to manage this without needing Rc / weak pointers or copying loads of dynamic memory all over the place. Instead you have a flat structure of copy-able elements, giving you better cache locality and a really easy way to work with them.

rtpg 2y ago

I think one core thing that you have to do with ASTs in Rust is to absolutely not store strings of the like in your AST. Instead you want to use something like static string libraries (so you get cheap clones and interning) and for things like positions in text you want to use just indices. Absolutely avoid storing references if you can avoid it! The more your stuff is held in things that are cheap/free to clone, the less you have to fight the borrow checker… since you can get away with clones! And for actual interpretation there’s these libraries that can help a lot for memory management with arenas and the like. It’s super specialized stuff but helps to give you perf + usability. Projects like Ruffle use this heavily and it’s nice when you figure out the patterns Having said that OCaml and Haskell are both languages that will do all of this “for free” with their built in reference counting and GC… I just like the idea of going very fast in Rust

FrustratedMonky 2y ago

FSharp is OCaml to great extent. So if you don't have the need to stay away from MS/.NET, it is more 'open source' than the rest of MS products. MS did release Fsharp with an Open Source License. But, it does still run on .NET. At this point, isn't every major language controlled by one main corporate entity? Except Python? But Python doesn't have algebraic types, or very complete pattern matching.

adsharma 2y ago
I still believe that a variant of python that has algebraic types and pattern matching beats Rust for writing parsers quickly. My effort has been in adding these features to a front end language that transpiles to an underlying FP language, including but not limited to Rust.
mikeschinkel 2y ago
PHP is not controlled by one main corporate entity. {Runs, ducks, hides. :-D}

neonsunset 2y ago

> I considered something like F# but I didn't like that it's tied to microsoft/.NET. Could you explain your thought process when deciding to not use F# because it runs on top of .NET? (both of which are open-source, and .NET is what makes F# fast and usable in almost every domain)

balencpp 2y ago
I am genuinely curious too. .NET is a very mature, very performant runtime, and I think of F#, a beautiful, productive language, running on it a big pro. Perhaps things used to be different about/regarding Microsoft?
- neonsunset 2y ago
  Yeah. I'm having so much fun with F# that I absolutely did not anticipate. Sure, it's something everyone using .NET knows about but I genuinely underestimated it and wish more people gave it a try. Such a good language. As for the hate - my pet theory is that developers need something like a sacrificial lamb to blame their misfortunes on, and a banner to rally under which often happens to be "against that other group" or "against that competing language", and because .NET is platform that happens to be made by microsoft and is a host for two very powerful multi-paradigm languages causes it to be a point of contention for many. From what I've seen, other languages do not receive so much undeserved hate and here on HN some like Go, Ruby or BEAM family receive copious amount of unjustified praise not rooted in technical merits.

almostdeadguy 2y ago

With some patience and practice, I think reasoning about borrows becomes second nature. And what it buys you with lexing/parsing is the ability to do zero-copy parsing.

omginternets 2y ago

I had a similar journey of enlightenment that likewise led me to OCaml. Unless you're doing low-level systems programming, OCaml will give you the "if it compiles, it's probably right" vibe with much less awkward stuff to type.

packetlost 2y ago

Go is not a great language to write parsers in IMO, it just doesn't have anything that makes such a task nice. That being said, people seem to really dislike Go here, which is fine, but somewhat surprising. Go is extremely simple. If you take a look at it's creators pedigree, that should make a ton of sense: they want you to make small, focused utilities and systems that use message passing (typically via a network) as the primary means of scaling complexity. What I find odd about this is that it was originally intended as a C++ replacement.

marcosdumay 2y ago
Message passing is an horrible means of scaling complexity, unless you are so big and have so many developers working with you that you can't use anything else.
dinosaurdynasty 2y ago
Go is simple at the cost of increasing the complexity of stuff written in it.

kemaru 2y ago

You wouldn't be losing FP niceness with Zig, and the pattern matching and enum situation is also similar to Rust. Even better, in a few areas, for example arbitrary-width integers and enum tagging in unions/structs. Writing parsers and low level device drivers is actually quite comfortable in Zig.

MrMcCall 2y ago

(OCaml Question Ahead) I agree on F#. It changed my C && OO perspective in fantastic ways, but I too can't support anything Microsoft anymore. But, seeing as OCaml was the basis for F#, I have a question, though: Does OCaml allow the use of specifically sized integer types? I seem to remember in my various explorations that OCaml just has a kind of "number" type. If I want a floating point variable, I want a specific 32- or 64- or 128-bit version; same with my ints. I did very much like F# having the ability to specify the size and signedness of my int vars. Thanks in advance, OCaml folks.

neonsunset 2y ago
F# is a far better option from a practical standpoint when compared to alternatives. By simple virtue of using .NET and having access to very wide selection of libraries that make it a non-issue when deciding to solve a particular business case. It also has an alternate compiler Fable which can target JS allowing the use of F# in front-end. Other options have worse support and weaker tooling, and often not even more open development process (e.g. you can see and contribute to ongoing F# work on Github). This tired opinion ".net bad because microsoft bad" has zero practical relevance to actually using C# itself and even more so F# and it honestly needs to die out because it borders on mental illness. You can hate microsoft products, I do so too, and still judge a particular piece of techology and the people that work on it on their merits.
- MrMcCall 2y ago
  I was asking about whether or not OCaml has the ability to target specific integer size and signedness, as F# does. I would like to construct precise software that targets the specific kinds of ints that compilers have historically facilitated microprocessors to process, but using a functional language instead of C or C++. The F# folks (including Don Syme) did a fantastic job on the early versions of the language (I used it up to ver 2 or early 3), but I am tired of the corporate engine that funds that ecosystem. I now construct my software within an operating system of a different pedigree. Such considerations are important to me, but thanks for sharing your preference. As for me, I hate nothing or no one, but I am as picky as a poor man can be about whom I choose to rely on for my tools. As for your opinion on the borderlands of mental illness, I'll contact you at your outlook email address should I seek your opinion about such differently-technical topics. But I was only asking if the OCaml compiler can target specific varieties of ints, as the .NET compiler does.

PittleyDunkin 2y ago

The borrow checker is definitely a pain, but it stops being such a pain once you design your types around ownership and passing around non-owned pointers or references or indexes.

lsllc 2y ago
This. I've found the same, being effective in Rust really requires that you change your way of thinking about your data structures (and code layout). Once I realized that, I was no longer fighting the borrow checker and I've been about to build complex code that more or less worked immediately. As I look back on it, I think what a pain it would have been to write and debug in C, although doing it in C would appear to be "easier".

wyager 2y ago

Having written probably several hundred kloc of both Haskell and OCaml, I strongly prefer Haskell. A very simple core concept wrapped in an extremely powerful shell. Haskell is a lot better for parsing tasks because (among other considerations) its more powerful type system can better express constraints on grammars.

materielle 2y ago

I love Go for writing servers. And in fact, I do it professionally. But I totally agree that for parsers, it’s not the right tool for the job. First off, the only way to express union types is with runtime reflection. You might as well be coding in Python (but without the convenient syntax sugar). Second off, “if err != nil” is really terrible in parsers. I’m actually somewhat of a defender of Go’s error handling approach in servers. Sure, it could have used a more convenient syntax. But in servers, I almost never return an error without handling it or adding additional context. The same isn’t true in parser’s though. Almost half of my parser code was error checks that simply wouldn’t exist in other languages. For Rust, I think the value proposition is if you are also writing a virtual machine or an interpreter, your compiler front end can be written in the same language as your backend. Your other alternatives are C and C++, but then you don’t have sum types. You could write the front end in Ocaml, but then you would have to write the backend and runtime in some other language anyways.

balencpp 2y ago

Did you discover Scala 3 and give it a thought? I think of it as Rust with an _overall_ stronger type-system, but where you don't have to worry about memory management. It has an amazing standard library, particularly around collections. You get access to the amazing JVM ecosystem. And more. Martin Odersky in fact sees Scala's future lying in being a simpler Rust. Also, regarding F#. It runs on .NET, and indeed, since the ecosystem and community are very small, you need to rely on .NET (basically C#) libraries. But it's really not "tied" to Microsoft and is open source.