9 ms·

Go channels, goroutines and GC available in Nim

147 points by pwernersbach 11y ago

kodablah 11y ago
I always wanted to see a Go/Nim interop project in a little different vein. Since Nim is a superset of Go (and Go is fairly simple), why not a Go implementation in Nim via cross-compilation? Then you can even build a Nim macro that does a "Go get" and Nim developers can use Golang libraries right in their code as imports. Not to mention we get a full, alternative Go impl. It should be easy to bootstrap by first using Go's parser to dump some AST that Nim code can then translate, and then once that version is done, just reference the Go parser using this new impl. The only real struggle with the project from what I can see is deciding which standard library items to use the Go transpiled implementation (probably most of them) vs which need to have Nim backends (e.g. runtime package). Meh, just rambling thoughts in my head I was considering playing with given the time...
- stefantalpalaru 11y ago
  > why not a Go implementation in Nim via cross-compilation? Nim's macros have one limitation that prevents an accurate implementation of another syntax: they can't fully modify the existing syntax. See how I had to use "scase" inside "select" blocks because the existing "case" keyword insists on having "of" after it. So Nim's semantics put some limits on the amount of hijacking one can inflict on it through macros.
  - stcredzero 11y ago
    If you front compilation with Go's parser, you could generate bog-standard Nim code from the AST.
    - stefantalpalaru 11y ago
      You can, but you'd be limiting yourself to pure Go. I'd rather have Go with Nim's generics, transpilation to C, macros, compile time computation, etc.
  - kodablah 11y ago
    I don't believe they should accept the full Golang syntax, just help with the import. E.g. macro goImport(path: string): stmt // TODO discard goImport("golang.org/x/crypto/nacl/box")
- aikah 11y ago
  > Since Nim is a superset of Go what ?
  - kodablah 11y ago
    I was under the impression this was true feature-set wise. I may be wrong though and am happy to be corrected.
    - stefantalpalaru 11y ago
      Almost. Go had the advantage of M:N threading with its goroutines and elegant CSP implementation.
    - worklogin 11y ago
      Calling something a superset usually means it's actually an expansion of something... kind of like C++11 might be a superset of C++03 (not sure if that's accurate).
    - MichaelGG 11y ago
      That's a really strange metric to use. It'll rarely be true, except in a sort of Turing completeness way.
- duaneb 11y ago
  > Since Nim is a superset of Go I seriously doubt that Nim is out of the box binary compatible with another runtime's fundamental types.
  - kodablah 11y ago
    I am not looking for binary compatibility. I am looking for feature parity. Go interfaces can be handled with concepts [1] or just macros. I am talking about an alternative Go implementation (i.e. just sitting on top of Nim), not ABI compat between the two runtimes. 1 - http://nim-lang.org/docs/manual.html#generics-concepts http://nim-lang.org/docs/manual.html#generics-concepts
    - duaneb 11y ago
      Along that way is C++ and "we should attempt to be a superset of all programming paradigms". Fewer features is better, and more restrictions allow better tooling and iteration if they don't functionally restrict the programmer. Basically, I'm not sure of the benefit of building go on top of nim. They appear to fill the same space in orthogonal ways.
bmurphy1976 11y ago
I'm very curious to hear the pros and cons of this from somebody who has intimate knowledge of Nim internals. I really really like Go's CSP model and would love to see it properly supported in another lightweight non-jvm language (yes I know about Erlang, it doesn't fit the bill for me).
- vbit 11y ago
  Well you can already use Nim threads (http://nim-lang.org/docs/threads.html http://nim-lang.org/docs/threads.html) and channels (http://nim-lang.org/docs/channels.html http://nim-lang.org/docs/channels.html) - the model is similar although the implementation uses system threads rather than coroutines. Nim also has lightweight coroutines using `async` and `await` (http://nim-lang.org/docs/asyncdispatch.html http://nim-lang.org/docs/asyncdispatch.html) - you can run a bunch of these within one thread. Also, have a look at gevent for Python.
  - dom96 11y ago
    In addition to the heavyweight threads (http://nim-lang.org/docs/threads.html http://nim-lang.org/docs/threads.html) there is also `spawn` (http://nim-lang.org/docs/manual.html#parallel-spawn-spawn-statement http://nim-lang.org/docs/manual.html#parallel-spawn-spawn-st..., http://nim-lang.org/docs/threadpool.html http://nim-lang.org/docs/threadpool.html)
  - rspeer 11y ago
    Don't look at gevent for Python, look at a responsible async library: asyncio if you like Py3, Twisted if you like Py2 and want something that's very featureful but old-school, Trollius if you wish you could be using asyncio but you're stuck on Py2. gevent is too much magic. It aims to give you async without changing any of your code. To accomplish this, it monkey-patches the entire Python standard library, in a way that is 99% compatible with Python, but the 1% will constantly surprise and infuriate you. Its compatibility shows no signs of increasing given how much its development has slowed down. You can use gevent as a quick hack, but you will hate yourself if you have to maintain gevent code.
Manishearth 11y ago
This is awesome! Go's channels and green threads were one of the features I really liked about Go. One more reason for me to go past basic prime number programs in Nim :) I'm hoping to see Rust get green threads/tasks/goroutines too. I'm working on a GC myself, and hopefully someone is trying out a green thread scheduler.
stcredzero 11y ago
This document describes how the GC works and how to tune it for (soft) realtime systems. The basic algorithm is Deferred Reference Counting with cycle detection. I'm sitting here feeling very impressed. Deferred reference counting has very good semantics for games. Even better, you can control the cycle detection part separately and run that part at an advantageous time. (Though it's probably better to just let GC do its thing, unless you really know what you're doing.) I am currently writing a multiplayer game server in golang, by making sure almost everything is allocated on the stack, and heap sizes are small. This gives me an efficient, nearly pauseless server. However, something like Nim could give me even more flexibility.
- pcwalton 11y ago
  > I'm sitting here feeling very impressed. Deferred reference counting has very good semantics for games. Not if you need to be thread-safe. > Even better, you can control the cycle detection part separately and run that part at an advantageous time. How would that work with multithreading? (Assuming you had a thread-safe GC, which Nim's isn't.)
  - Pharohbot 11y ago
    A more general approach to shared memory via lockable heaps is a feature Nim seems will implement soon.
    - pcwalton 11y ago
      Sounds interesting. Do you have any links to documentation? I'm interested in reading more about it, because Nim is doing a lot of experimental stuff and it's always interesting to look at its designs. (Edited to remove speculation about how well it will perform before reading about it.)
      - rbehrends 11y ago
        Well, this is currently highly speculative. What I proposed to Andreas was essentially a model based on Eiffel's SCOOP (with some additional influence from Erlang). Whether it's a practical design remains to be seen. Note that shared, lockable heaps need not be heavyweight structures. It is entirely possible to imagine a shared hash table with one heap per bucket and fine-grained locking, for example. Collections for such small heaps can be fast because the number of roots is limited, and (depending on what invariants you guarantee), you can even forgo stack scanning for most collections or limit the number of stack frames that need to be traversed.
jeremyjh 11y ago
I thought the Go runtime runs foreign code on M:M threads; e.g. when Go time calls foreign code, it dedicates a thread to it. This is so foreign libraries (which are unaware of yielding to the Go scheduler when they do I/O) don't block a thread with multiple goroutines scheduled on it. I do not think Nim code can run "in a goroutine".
- stefantalpalaru 11y ago
  > I do not think Nim code can run "in a goroutine". It does. Don't forget that this is gccgo so it is possible to use plain C functions as goroutines. Nim is translated to C and with the help of a macro I convert Nim functions with an arbitrary number of arguments into ones with a single void* arg that gccgo wants for its 'go' keyword implementation: extern void* __go_go(void (*f)(void *), void *);
  - jeremyjh 11y ago
    Yes, but when the Go runtime calls that foreign function pointer, it is going to schedule an entire OS thread for its duration isn't it? If it doesn't, then nothing prevents foreign code from blocking other goroutines. How does the Nim code yield the thread for other goroutines, does it have to register a callback?
    - stefantalpalaru 11y ago
      > when the Go runtime calls that foreign function pointer, it is going to schedule an entire OS thread for its duration isn't it? No. > How does the Nim code yield the thread for other goroutines, does it have to register a callback? There are no callbacks. Yielding happens automatically when launching another goroutine, when sending, receiving or selecting on a channel. You can also yield explicitly with go_yield() - the better named equivalent of Go's runtime.Gosched(). It's easier to understand if you realize that all those operations with goroutines and channels end up being done in the Go runtime.
      - nulltype 11y ago

147 points by pwernersbach 11y ago

I always wanted to see a Go/Nim interop project in a little different vein. Since Nim is a superset of Go (and Go is fairly simple), why not a Go implementation in Nim via cross-compilation? Then you can even build a Nim macro that does a "Go get" and Nim developers can use Golang libraries right in their code as imports. Not to mention we get a full, alternative Go impl. It should be easy to bootstrap by first using Go's parser to dump some AST that Nim code can then translate, and then once that version is done, just reference the Go parser using this new impl. The only real struggle with the project from what I can see is deciding which standard library items to use the Go transpiled implementation (probably most of them) vs which need to have Nim backends (e.g. runtime package). Meh, just rambling thoughts in my head I was considering playing with given the time...

stefantalpalaru 11y ago
> why not a Go implementation in Nim via cross-compilation? Nim's macros have one limitation that prevents an accurate implementation of another syntax: they can't fully modify the existing syntax. See how I had to use "scase" inside "select" blocks because the existing "case" keyword insists on having "of" after it. So Nim's semantics put some limits on the amount of hijacking one can inflict on it through macros.
- stcredzero 11y ago
  If you front compilation with Go's parser, you could generate bog-standard Nim code from the AST.
  - stefantalpalaru 11y ago
    You can, but you'd be limiting yourself to pure Go. I'd rather have Go with Nim's generics, transpilation to C, macros, compile time computation, etc.
- kodablah 11y ago
  I don't believe they should accept the full Golang syntax, just help with the import. E.g. macro goImport(path: string): stmt // TODO discard goImport("golang.org/x/crypto/nacl/box")
aikah 11y ago
> Since Nim is a superset of Go what ?
- kodablah 11y ago
  I was under the impression this was true feature-set wise. I may be wrong though and am happy to be corrected.
  - stefantalpalaru 11y ago
    Almost. Go had the advantage of M:N threading with its goroutines and elegant CSP implementation.
  - worklogin 11y ago
    Calling something a superset usually means it's actually an expansion of something... kind of like C++11 might be a superset of C++03 (not sure if that's accurate).
  - MichaelGG 11y ago
    That's a really strange metric to use. It'll rarely be true, except in a sort of Turing completeness way.
duaneb 11y ago
> Since Nim is a superset of Go I seriously doubt that Nim is out of the box binary compatible with another runtime's fundamental types.
- kodablah 11y ago
  I am not looking for binary compatibility. I am looking for feature parity. Go interfaces can be handled with concepts [1] or just macros. I am talking about an alternative Go implementation (i.e. just sitting on top of Nim), not ABI compat between the two runtimes. 1 - http://nim-lang.org/docs/manual.html#generics-concepts http://nim-lang.org/docs/manual.html#generics-concepts
  - duaneb 11y ago
    Along that way is C++ and "we should attempt to be a superset of all programming paradigms". Fewer features is better, and more restrictions allow better tooling and iteration if they don't functionally restrict the programmer. Basically, I'm not sure of the benefit of building go on top of nim. They appear to fill the same space in orthogonal ways.

bmurphy1976 11y ago

I'm very curious to hear the pros and cons of this from somebody who has intimate knowledge of Nim internals. I really really like Go's CSP model and would love to see it properly supported in another lightweight non-jvm language (yes I know about Erlang, it doesn't fit the bill for me).

vbit 11y ago
Well you can already use Nim threads (http://nim-lang.org/docs/threads.html http://nim-lang.org/docs/threads.html) and channels (http://nim-lang.org/docs/channels.html http://nim-lang.org/docs/channels.html) - the model is similar although the implementation uses system threads rather than coroutines. Nim also has lightweight coroutines using `async` and `await` (http://nim-lang.org/docs/asyncdispatch.html http://nim-lang.org/docs/asyncdispatch.html) - you can run a bunch of these within one thread. Also, have a look at gevent for Python.
- dom96 11y ago
  In addition to the heavyweight threads (http://nim-lang.org/docs/threads.html http://nim-lang.org/docs/threads.html) there is also `spawn` (http://nim-lang.org/docs/manual.html#parallel-spawn-spawn-statement http://nim-lang.org/docs/manual.html#parallel-spawn-spawn-st..., http://nim-lang.org/docs/threadpool.html http://nim-lang.org/docs/threadpool.html)
- rspeer 11y ago
  Don't look at gevent for Python, look at a responsible async library: asyncio if you like Py3, Twisted if you like Py2 and want something that's very featureful but old-school, Trollius if you wish you could be using asyncio but you're stuck on Py2. gevent is too much magic. It aims to give you async without changing any of your code. To accomplish this, it monkey-patches the entire Python standard library, in a way that is 99% compatible with Python, but the 1% will constantly surprise and infuriate you. Its compatibility shows no signs of increasing given how much its development has slowed down. You can use gevent as a quick hack, but you will hate yourself if you have to maintain gevent code.

Manishearth 11y ago

This is awesome! Go's channels and green threads were one of the features I really liked about Go. One more reason for me to go past basic prime number programs in Nim :) I'm hoping to see Rust get green threads/tasks/goroutines too. I'm working on a GC myself, and hopefully someone is trying out a green thread scheduler.

stcredzero 11y ago

This document describes how the GC works and how to tune it for (soft) realtime systems. The basic algorithm is Deferred Reference Counting with cycle detection. I'm sitting here feeling very impressed. Deferred reference counting has very good semantics for games. Even better, you can control the cycle detection part separately and run that part at an advantageous time. (Though it's probably better to just let GC do its thing, unless you really know what you're doing.) I am currently writing a multiplayer game server in golang, by making sure almost everything is allocated on the stack, and heap sizes are small. This gives me an efficient, nearly pauseless server. However, something like Nim could give me even more flexibility.

pcwalton 11y ago
> I'm sitting here feeling very impressed. Deferred reference counting has very good semantics for games. Not if you need to be thread-safe. > Even better, you can control the cycle detection part separately and run that part at an advantageous time. How would that work with multithreading? (Assuming you had a thread-safe GC, which Nim's isn't.)
- Pharohbot 11y ago
  A more general approach to shared memory via lockable heaps is a feature Nim seems will implement soon.
  - pcwalton 11y ago
    Sounds interesting. Do you have any links to documentation? I'm interested in reading more about it, because Nim is doing a lot of experimental stuff and it's always interesting to look at its designs. (Edited to remove speculation about how well it will perform before reading about it.)
    - rbehrends 11y ago
      Well, this is currently highly speculative. What I proposed to Andreas was essentially a model based on Eiffel's SCOOP (with some additional influence from Erlang). Whether it's a practical design remains to be seen. Note that shared, lockable heaps need not be heavyweight structures. It is entirely possible to imagine a shared hash table with one heap per bucket and fine-grained locking, for example. Collections for such small heaps can be fast because the number of roots is limited, and (depending on what invariants you guarantee), you can even forgo stack scanning for most collections or limit the number of stack frames that need to be traversed.

jeremyjh 11y ago

I thought the Go runtime runs foreign code on M:M threads; e.g. when Go time calls foreign code, it dedicates a thread to it. This is so foreign libraries (which are unaware of yielding to the Go scheduler when they do I/O) don't block a thread with multiple goroutines scheduled on it. I do not think Nim code can run "in a goroutine".

stefantalpalaru 11y ago
> I do not think Nim code can run "in a goroutine". It does. Don't forget that this is gccgo so it is possible to use plain C functions as goroutines. Nim is translated to C and with the help of a macro I convert Nim functions with an arbitrary number of arguments into ones with a single void* arg that gccgo wants for its 'go' keyword implementation: extern void* __go_go(void (*f)(void *), void *);
- jeremyjh 11y ago
  Yes, but when the Go runtime calls that foreign function pointer, it is going to schedule an entire OS thread for its duration isn't it? If it doesn't, then nothing prevents foreign code from blocking other goroutines. How does the Nim code yield the thread for other goroutines, does it have to register a callback?
  - stefantalpalaru 11y ago
    > when the Go runtime calls that foreign function pointer, it is going to schedule an entire OS thread for its duration isn't it? No. > How does the Nim code yield the thread for other goroutines, does it have to register a callback? There are no callbacks. Yielding happens automatically when launching another goroutine, when sending, receiving or selecting on a channel. You can also yield explicitly with go_yield() - the better named equivalent of Go's runtime.Gosched(). It's easier to understand if you realize that all those operations with goroutines and channels end up being done in the Go runtime.
    - nulltype 11y ago