5 ms·

I just spent the weekend learning Go and writing a single writer multi-reader hashtable for threadsafe access. I picked it deliberately because it's against the

by eloff 12y ago

I just spent the weekend learning Go and writing a single writer multi-reader hashtable for threadsafe access. I picked it deliberately because it's against the philosophy of the language, which is to share by communicating instead of sharing data structures directly. It was painful to write: // Do NOT reorder this, otherwise parallel lookup could find key but see empty value atomic.StoreInt32((*int32)(unsafe.Pointer(uintptr(uint64(slot)+4))), val) atomic.StoreInt32((*int32)(unsafe.Pointer(slot)), key) However, the non volatile, non unsafe parts of the code were an absolute joy. Testing was a joy, compiling was a joy, and benchmarking was a joy. I was impressed that it allowed me to bypass the type system completely and do nasty, nasty things in the pursuit of performance. I want a language that lets me do nasty things where I must, but that makes the other 95% of the program, and the job of compiling, testing, and maintaining that program easy. Go excels here. Rust, C++, Haskell, Scala will never be good at that because they're too damn complicated (although each of them make the nasty parts a little less painful!) The end result of my weekend's hacking? On an i7-4770K @ 3.5ghz BenchmarkGoMapInsert-4 20000000 110 ns/op BenchmarkHashIndexInsert-4 100000000 25.6 ns/op BenchmarkGoMapLookup-4 50000000 78.5 ns/op BenchmarkHashIndexLookup-4 100000000 17.7 ns/op About 4x faster than Go's builtin map, for int32 key/value on both insert and lookup. And it allows any number of reader threads concurrent access with a writer without any synchronization or blocking, unlike Go maps. It doesn't allow zero keys, unlike go maps, and it doesn't allow deletes. Hardly apples to apples, but the performance of pure Go code is impressive nonetheless. 200 LOC, not counting tests.

stcredzero 12y ago
it allows any number of reader threads concurrent access without any synchronization or blocking, unlike Go maps. You mean readers concurrent with a writer? AFAIK, any number of reader threads can get values out of a Go map when it isn't being written to.
- eloff 12y ago
  Yes, concurrent with a writer. If there is more than one writer, the writers must use a mutex, but the single writer and readers don't block each other. I updated the text to make it clear.
  - deleted 12y ago
    [deleted]
pornel 12y ago
Rust has `unsafe` blocks in which you're allowed to do all nasty hacks you want. Rust actually isn't that complicated. Don't get discouraged by comparisons to Haskell — it's still a C-family language where you can play with pointers and mutable state. To me Rust still feels like a "small" language (similar size as Go or ObjC, not even scratching complexity of C++). It's mostly just functions + structs + enums, but they're more flexible and can be combined to be more powerful than in C.
- thinkpad20 12y ago
  I think comparisons to Haskell are not too far off the mark. Haskell is not that complicated of a language either. You can do all sorts of complicated things with it, but the language itself is relatively simple. It just has a lot of stuff that you're likely to have never seen before (extensive use of higher-order functions, higher-kinded types, etc), and its type system produces error messages that seem obscure from the outside. Similarly Rust can have some rather obscure error messages that you're probably not going to have seen before during compilation - lifetime specifiers, errors about using things in the wrong contexts, heck, even "Bare str is not a type (what?)" I'm much more familiar with Haskell than Rust, but having played around with Rust I think they're on a par with each other in terms of difficulty, depending on your background.
  - deleted 12y ago
    [deleted]
    - thinkpad20 12y ago
      > Haskell is a research language; Rust is designed to be a practical language. It makes a lot of concessions to practicality, C-like syntax and imperative control flow being high among them. Perhaps I wasn't clear in the post you're responding to: my point was that both Rust and Haskell are fairly "simple" programming languages which seem more complicated, because they introduce a lot of features which are likely to be new to those who are using it for the first time. I wasn't really comparing them as languages per se; that's a separate discussion. > Common in any language, including Go. Higher-order functions are common in most languages, but not in the way that Haskell does. Most languages use first-class functions as lists of instructions (do some stuff, and perform the steps in this argument). Haskell makes them truly first-class, such that they're positively ubiquitous: an example is currying, which is everywhere in Haskell and rare in most other languages; another example is monads, which are obviously a core part of Haskell and which require first-class functions (e.g. in >>=) to do anything useful. There are other examples. > Rust does not have these. I know; I was speaking about Haskell. > This is because we want memory safety without performance tradeoffs (global concurrent garbage collection). Right; it's a perfectly understandable thing to have, but it's not something that (to my knowledge) exists in any other mainstream language. It's an example of something in Rust which is obscure to newcomers. > Could you elaborate? I'd have to write some code and run the compiler to get the actual error message, but I recall getting errors about using some reference outside of a context or something. In my fuzzy recollection, it would be something like where I had written `match foo { a => b; c => d}` and I would get some error message which would be fixed by writing `let foo1 = foo; match foo1 {a => b; c => d}`. Unfortunately I don't remember the specifics, but long story short: compiling Rust code produces a lot of very strange error messages to someone unfamiliar with the language. :) In this way it's not dissimilar from Haskell. > This is because dynamically-sized types are not yet implemented, but they will be for 1.0. Great to know! If only there were a more helpful error message than "Bare str is not a type." :)
dbaupp 12y ago
That map only works with keys and values both being int32, right?
- eloff 12y ago
  Correct. I have to duplicate much of the code for in64 keys and values, the lack of generics hurts there.
  - Peaker 12y ago
    You seem to be brushing this off as a minor nuisance, when in many cases it is a showstopper. Need it for floats, duplicate it again. This is a solved problem -- the fact Go doesn't have the solution reflects very poorly on Go.
    - coldtea 12y ago
      There are some batshit crazy "solutions" offered from Go fans, like "no biggie, just use a templating engine to generate the code for the types you want and compile it".
      - Peaker 12y ago
        I actually use that solution in c (we have some reasons to avoid c++). Of course it sucks, but at least c has the excuse of being from the 70's.
      - pjmlp 12y ago
        They suffer from a strange distortion field effect.
      - orbifold 12y ago
        Ironically that is the solution c++ essentially uses.
        Peaker 12y ago
pcwalton 12y ago
> Rust, C++, Haskell, Scala will never be good at that because they're too damn complicated (although each of them make the nasty parts a little less painful!) Why is Rust too complicated to allow you to do low-level hacking?
- eloff 12y ago
  I mean too complicated for writing, reading, and maintaining and generally just working with from day to day. It costs too much time to do the same thing.
  - pcwalton 12y ago
    Again, what specifically? I ask because this has not been my experience writing several hundreds of thousands of lines of Rust code, nor has this been the experience of anyone I have helped get up to speed. Moreover, I believe there is no feature in Rust that is not necessary to achieve safety without sacrificing performance.
    - cantankerous 12y ago
      Hundreds of thousands of lines of Rust? Out of curiosity, what kind of software are you writing in Rust? ... And can I get in on some of that? ;-)
      - Ygg2 12y ago
        > Hundreds of thousands of lines of Rust Just a very concurrent browser engine ;) One that does everything in parallel. Basically every task that can be parallelized becomes parallel (rendering, JS execution, CSS matcher, etc.). Also you can, there is Github for Servo (said parallel browser engine project by Mozilla). https://github.com/mozilla/servo/wiki/Design https://github.com/mozilla/servo/wiki/Design
      - dbaupp 12y ago
        The compiler and Mozilla's experimental rending engine Servo: - https://github.com/rust-lang/rust/ https://github.com/rust-lang/rust/ - https://github.com/mozilla/servo/ https://github.com/mozilla/servo/ (The 'contributors' graphs suggest that pcwalton has added/removed more than 1 million lines of code in those two repos combined.)
dbaupp 12y ago
> However, the non volatile, non unsafe parts of the code were an absolute joy. Testing was a joy, compiling was a joy, and benchmarking was a joy. I was impressed that it allowed me to bypass the type system completely and do nasty, nasty things in the pursuit of performance. I want a language that lets me do nasty things where I must, but that makes the other 95% of the program, and the job of compiling, testing, and maintaining that program easy. Go excels here. Rust, C++, Haskell, Scala will never be good at that FWIW, I actually think that Rust positively excels at this sort of isolated low-level work due to explicit `unsafe` blocks. Furthermore, the type system is more expressive meaning the need for this is rarer[1]. In my experience, the rest of the language (i.e. non-`unsafe` things) works very well for maintenance and testing, also in part due to the more expressive typesystem, and things like algebraic data types with exhaustive matches by default (I've done some huge bug-free refactorings to the standard library and compiler, mostly due to the compiler automatically catching all the places that need updating). On the other hand this comes with the cost of making the "job of compiling" more difficult: the compiler complains about more things. Re testing: there's unit testing and microbenchmarking built-in: http://doc.rust-lang.org/master/guide-testing.html http://doc.rust-lang.org/master/guide-testing.html [1]: in this case, the type system positively designed with making this sort of concurrency safer.

by eloff 12y ago

stcredzero 12y ago

it allows any number of reader threads concurrent access without any synchronization or blocking, unlike Go maps. You mean readers concurrent with a writer? AFAIK, any number of reader threads can get values out of a Go map when it isn't being written to.

eloff 12y ago
Yes, concurrent with a writer. If there is more than one writer, the writers must use a mutex, but the single writer and readers don't block each other. I updated the text to make it clear.
- deleted 12y ago
  [deleted]

pornel 12y ago

Rust has `unsafe` blocks in which you're allowed to do all nasty hacks you want. Rust actually isn't that complicated. Don't get discouraged by comparisons to Haskell — it's still a C-family language where you can play with pointers and mutable state. To me Rust still feels like a "small" language (similar size as Go or ObjC, not even scratching complexity of C++). It's mostly just functions + structs + enums, but they're more flexible and can be combined to be more powerful than in C.

thinkpad20 12y ago
I think comparisons to Haskell are not too far off the mark. Haskell is not that complicated of a language either. You can do all sorts of complicated things with it, but the language itself is relatively simple. It just has a lot of stuff that you're likely to have never seen before (extensive use of higher-order functions, higher-kinded types, etc), and its type system produces error messages that seem obscure from the outside. Similarly Rust can have some rather obscure error messages that you're probably not going to have seen before during compilation - lifetime specifiers, errors about using things in the wrong contexts, heck, even "Bare str is not a type (what?)" I'm much more familiar with Haskell than Rust, but having played around with Rust I think they're on a par with each other in terms of difficulty, depending on your background.
- deleted 12y ago
  [deleted]
  - thinkpad20 12y ago
    > Haskell is a research language; Rust is designed to be a practical language. It makes a lot of concessions to practicality, C-like syntax and imperative control flow being high among them. Perhaps I wasn't clear in the post you're responding to: my point was that both Rust and Haskell are fairly "simple" programming languages which seem more complicated, because they introduce a lot of features which are likely to be new to those who are using it for the first time. I wasn't really comparing them as languages per se; that's a separate discussion. > Common in any language, including Go. Higher-order functions are common in most languages, but not in the way that Haskell does. Most languages use first-class functions as lists of instructions (do some stuff, and perform the steps in this argument). Haskell makes them truly first-class, such that they're positively ubiquitous: an example is currying, which is everywhere in Haskell and rare in most other languages; another example is monads, which are obviously a core part of Haskell and which require first-class functions (e.g. in >>=) to do anything useful. There are other examples. > Rust does not have these. I know; I was speaking about Haskell. > This is because we want memory safety without performance tradeoffs (global concurrent garbage collection). Right; it's a perfectly understandable thing to have, but it's not something that (to my knowledge) exists in any other mainstream language. It's an example of something in Rust which is obscure to newcomers. > Could you elaborate? I'd have to write some code and run the compiler to get the actual error message, but I recall getting errors about using some reference outside of a context or something. In my fuzzy recollection, it would be something like where I had written `match foo { a => b; c => d}` and I would get some error message which would be fixed by writing `let foo1 = foo; match foo1 {a => b; c => d}`. Unfortunately I don't remember the specifics, but long story short: compiling Rust code produces a lot of very strange error messages to someone unfamiliar with the language. :) In this way it's not dissimilar from Haskell. > This is because dynamically-sized types are not yet implemented, but they will be for 1.0. Great to know! If only there were a more helpful error message than "Bare str is not a type." :)

dbaupp 12y ago

That map only works with keys and values both being int32, right?

eloff 12y ago
Correct. I have to duplicate much of the code for in64 keys and values, the lack of generics hurts there.
- Peaker 12y ago
  You seem to be brushing this off as a minor nuisance, when in many cases it is a showstopper. Need it for floats, duplicate it again. This is a solved problem -- the fact Go doesn't have the solution reflects very poorly on Go.
  - coldtea 12y ago
    There are some batshit crazy "solutions" offered from Go fans, like "no biggie, just use a templating engine to generate the code for the types you want and compile it".
    - Peaker 12y ago
      I actually use that solution in c (we have some reasons to avoid c++). Of course it sucks, but at least c has the excuse of being from the 70's.
    - pjmlp 12y ago
      They suffer from a strange distortion field effect.
    - orbifold 12y ago
      Ironically that is the solution c++ essentially uses.
      - Peaker 12y ago

pcwalton 12y ago

> Rust, C++, Haskell, Scala will never be good at that because they're too damn complicated (although each of them make the nasty parts a little less painful!) Why is Rust too complicated to allow you to do low-level hacking?

eloff 12y ago
I mean too complicated for writing, reading, and maintaining and generally just working with from day to day. It costs too much time to do the same thing.
- pcwalton 12y ago
  Again, what specifically? I ask because this has not been my experience writing several hundreds of thousands of lines of Rust code, nor has this been the experience of anyone I have helped get up to speed. Moreover, I believe there is no feature in Rust that is not necessary to achieve safety without sacrificing performance.
  - cantankerous 12y ago
    Hundreds of thousands of lines of Rust? Out of curiosity, what kind of software are you writing in Rust? ... And can I get in on some of that? ;-)
    - Ygg2 12y ago
      > Hundreds of thousands of lines of Rust Just a very concurrent browser engine ;) One that does everything in parallel. Basically every task that can be parallelized becomes parallel (rendering, JS execution, CSS matcher, etc.). Also you can, there is Github for Servo (said parallel browser engine project by Mozilla). https://github.com/mozilla/servo/wiki/Design https://github.com/mozilla/servo/wiki/Design
    - dbaupp 12y ago
      The compiler and Mozilla's experimental rending engine Servo: - https://github.com/rust-lang/rust/ https://github.com/rust-lang/rust/ - https://github.com/mozilla/servo/ https://github.com/mozilla/servo/ (The 'contributors' graphs suggest that pcwalton has added/removed more than 1 million lines of code in those two repos combined.)

dbaupp 12y ago

> However, the non volatile, non unsafe parts of the code were an absolute joy. Testing was a joy, compiling was a joy, and benchmarking was a joy. I was impressed that it allowed me to bypass the type system completely and do nasty, nasty things in the pursuit of performance. I want a language that lets me do nasty things where I must, but that makes the other 95% of the program, and the job of compiling, testing, and maintaining that program easy. Go excels here. Rust, C++, Haskell, Scala will never be good at that FWIW, I actually think that Rust positively excels at this sort of isolated low-level work due to explicit `unsafe` blocks. Furthermore, the type system is more expressive meaning the need for this is rarer[1]. In my experience, the rest of the language (i.e. non-`unsafe` things) works very well for maintenance and testing, also in part due to the more expressive typesystem, and things like algebraic data types with exhaustive matches by default (I've done some huge bug-free refactorings to the standard library and compiler, mostly due to the compiler automatically catching all the places that need updating). On the other hand this comes with the cost of making the "job of compiling" more difficult: the compiler complains about more things. Re testing: there's unit testing and microbenchmarking built-in: http://doc.rust-lang.org/master/guide-testing.html http://doc.rust-lang.org/master/guide-testing.html [1]: in this case, the type system positively designed with making this sort of concurrency safer.