6 ms·

Fibers in Guile Scheme

74 points by srean 10y ago

dmix 10y ago
Anyone using Guile for their projects? Any feedback on their experience with it? I only know Scheme from reading SICP and enjoyed Clojure but hated the java/JVM part of it. I currently use Erlang for when I need concurrency/performant backends. But I'm not totally satisfied with it (for ex: the weak type system and records). Edit: oh looks two programs I use all the time are written in Guile: GNU Make and WeeChat https://en.wikipedia.org/wiki/GNU_Guile#Programs_using_Guile https://en.wikipedia.org/wiki/GNU_Guile#Programs_using_Guile
- aban 10y ago
  I don't have much experience with Guile, but AFAIK both programs you mentioned are written in C, but support Guile in one way or another. GNU Make supports Guile as an embedded extension language [0] and WeeChat supports Guile as a scripting language for writing plugins [1]. There are a few other apps mentioned on the Guile website [2] and more on their Libraries page [3]. [0]: http://git.savannah.gnu.org/cgit/make.git/commit/?id=c992c4d80f51540699f33fed067caf6b7c38df79 http://git.savannah.gnu.org/cgit/make.git/commit/?id=c992c4d... [1]: http://dev.weechat.org/post/2011/10/27/New-plugin-Guile http://dev.weechat.org/post/2011/10/27/New-plugin-Guile [2]: https://www.gnu.org/software/guile/#apps-using-guile https://www.gnu.org/software/guile/#apps-using-guile [3]: https://www.gnu.org/software/guile/libraries/ https://www.gnu.org/software/guile/libraries/
- ZephyrP 10y ago
  Alright, total stream of thought here: Guile is cool, if possibly immature. To this day, the largest project I've done in Guile was the entirety of the exercises short of Chapter 5 in SICP, wrote unit tests for the interpreter and the 'Prolog' implementation, the whole lot. All told I generated approximately 10KLOC of Guile code, some 20% of it using features like GOOPs, threading, dynamic-unwind and so on. It was largely a positive experience and I am really glad I read ahead and implemented a few things as GOOPs classes rather than used the OOP system described in the Book (particularly for the Constraint Solver!) As part of this, I did run into some faulty code generation (inappropriate optimization) on the master branch while completing an early Chapter 4 exercise: "This code could do multiple things if we apply different models of an idealized computer language (each of which was advocated by 'Susan' and 'Bob' in typical SICP fashion)", I wrote an implementation inside my guile interpreter that got an unusual answer and asked the channel about it and everyone got to the bottom of it fairly quickly. Speaking of which, Guile has a community of absolutely rabid fans all over the world and so the support in #guile is absolutely phenomenal, people like ijp and mark_weaver are both true 31337s and always can be relied on to offer sage advice. You've brought up it's utility as a GNU extension language, and although I feel it's public documentation is not given a fair shake in the important arenas like GDB scripting, it is tremendously useful in this capacity. Of course, as an extension language itself, it excels. Excellent exposure and interaction with the bare-metal, good parallel & concurrent primitives (although they are more 'traditional', taking after the proud line of Pthreads than `link' or send/recieve in Erlang). It is quite large to statically link into your binary in comparison to other languages however, so be careful there. Weechat itself is written in C, but it does have Guile extensions (and unfortunately I may be the only person in the world to take advantage of this while writing an extension however https://github.com/zv/weechat-gateway-replacer https://github.com/zv/weechat-gateway-replacer ).
- mkeeter 10y ago
  I'm using Guile for Ao, a Scheme-based tool for solid modeling: https://mattkeeter.com/projects/ao https://mattkeeter.com/projects/ao (previous on HN at https://news.ycombinator.com/item?id=12319406 https://news.ycombinator.com/item?id=12319406) It's been a generally good experience. Scheme is fun, and Guile is a fine implementation. I'm not experienced enough to be pushing its limits – the only unusual corner I've explored is the C FFI, which is quite nice. My only complaint is the consistency of Guile's documentation. It's generally of high quality, but there are corners that are completely undocumented – I've had a few cases where I had to dig into the implementation to figure out how to do something.
- rubber_duck 10y ago
  I'm not too familiar with Guile and have been away from Clojure world for a while but from what I remember people compared CLJ to Racket which seemed to get a lot of stuff before clojure (IIRC, a lot of the stuff was ported over like type annotations). Might be worth looking in to. https://racket-lang.org/ https://racket-lang.org/
- IsmaOlvey 10y ago
  Neither Make nor WeeChat are written in Guile. They're both written in C. They do, however, use Guile as an extension language.
- davexunit 10y ago
  The biggest piece of software using Guile I know of is GNU Guix[0], a functional package manager and declarative GNU/Linux distribution. It has dozens of contributors actively working on it.[1] I contribute to Guix, but I also have my own projects[2] written in Guile. The only software that other people actually use, however, is Haunt, the static site generator that renders my blog (everyone needs to roll their own, right?) [0] https://gnu.org/s/guix https://gnu.org/s/guix [1] https://www.openhub.net/p/gnuguix https://www.openhub.net/p/gnuguix [2] https://dthompson.us/projects.html https://dthompson.us/projects.html
- Johnny_Brahms 10y ago
  I started using guile at work, since I like scheme, and guile was the scheme that was available without installing anything new. I prototype loads of small tools in guile, and some of the prototypes "stick" since they exceed expectations. The biggest one is a macro system for pascal that expands recursive macros and unrolls loops, generating _lots and lots_ of code. My scheme implementation of choice is chicken, since they have a package manager with quite a lot of packages, but I really enjoy using guile. Really looking forward to the new 2.2 branch. We are using that in production, and we probably hit a sweet spot where the magic optimization dust gathered the most. We got almost 2x improvement just by switching.
- rekado 10y ago
  I'm using Guile for a simple web interface to a scientific pipeline. Users upload a file, which is queued up in Redis, processed by a worker which spawns the pipeline. It's nice to be able to use a flexible and simple language like Scheme for tools like this. The HTML documents are written in SXML, so it's all S-expressions. (I'm not a fan of Python, the obvious alternative for things like this.)
flukus 10y ago
What's the state of other language interpreters for guile? It's one of this things I always wanted to have a play with until I remember how much I hate lisp.
- chrisseaton 10y ago
  What do you mean? Other languages running on top of the Guile interpreter? Or implementations of Guile in other languages. Why would you want to run Guile if you don't like Lisp?
  - flukus 10y ago
    > Other languages running on top of the Guile interpreter? This one. I've read about various efforts over the years like this one (https://wingolog.org/archives/2009/02/22/ecmascript-for-guile https://wingolog.org/archives/2009/02/22/ecmascript-for-guil...) for javascript and others for python etc. >Why would you want to run Guile if you don't like Lisp? It's the nicest way to embed a high level language in a program I've come across this tutorial is really cool (https://www.gnu.org/software/guile/docs/guile-tut/tutorial.html https://www.gnu.org/software/guile/docs/guile-tut/tutorial.h...). Other there any others that are so simple?
    - chrisseaton 10y ago
      > Other there any others that are so simple? I think the Lua language itself is designed for the implementation to be easy to embed. http://queue.acm.org/detail.cfm?id=1983083 http://queue.acm.org/detail.cfm?id=1983083
  - orangeshark 10y ago
    One of the features of Guile is different compiler front-ends. I recall one of the original intentions of Guile was to primarily support two languages, a lisp and a more imperative like language. I don't believe this feature is used that much though. You can find the documentation here https://www.gnu.org/software/guile/manual/html_node/Other-Languages.html#Other-Languages https://www.gnu.org/software/guile/manual/html_node/Other-La...
dkarapetyan 10y ago
What is the point of exposing a scheduler? Seems like that's another implementation detail that should not be exposed. The user should just instantiate some fibers and go.
- srean 10y ago
  Scheduling algorithms are dime a dozen and the default one provided by a runtime might not be suitable for a particular application. A user space scheduler would let you have finer control when you need it. You may have codepaths that are latency sensitive and those that are throughput sensitive.
  - dkarapetyan 10y ago
    That's not what this scheduler does. It simply bundles up some fibers. You don't have control over the actual scheduling. Relevant section of documentation When a fiber would block, it suspends to the scheduler from the current thread. The scheduler will arrange to re-start the fiber when the port or channel becomes readable or writable, as appropriate. For ports, the scheduler adds the file descriptor associated with the port to an epoll set. In either case, the scheduler remembers which fibers are waiting and for what, so that the user can inspect the state of their system. If no scheduler has been installed in the current thread, the fiber will... well, we don’t know yet! Either it blocks its thread, or it aborts. We don’t know. In fact it is pretty clear there is zero control user has over any scheduling activity other than suspending and resuming. It is all driven by polling events internal to the implementation.
    - srean 10y ago
      I was responding to your larger question. User space schedulers would be quite cool, don't know of a language that offers anything over and above a priority queue or timers.
JohnStrange 10y ago
It's not hard to come up with a working green thread system, I've done that in a VM I'm developing for fun, but the crucial question is: Does it distribute the fibers over native worker threads? That's my problem with most scheme implementations, they are great for single-threaded concurrency but require nasty hacks like Racket's 'places' for using all available CPU cores. The only fast multicore scheme I'm aware of is Chez, which has other problems, though. Does Guile do better than Racket with that respect?
mpweiher 10y ago
Yes! "But if your typey spidey senses are tingling, it’s for good reason: with promises, your whole program has to be transformed to return promises-for-values instead of values anywhere it would block." (Or lifted into a monad). To me, that's the big issue with a lot of the concurrency/react abstractions en-vogue now: you are programming indirectly, or more precisely you are programming in a specific (async) architectural style, but trying to express this in a call/return architectural style (that includes methods and FP functions). Depending on where you are, that often means your actual domain code is hidden inside a lot of maps() or flatMaps(). The first problem is that your domain code really should be primary, and your architectural support code as hidden as possible. The second (but probably not last) problem is that you can easily have multiple of these "lifts", for example another for error handling with Maybe. At some point, it just piles up and the actual domain code is completely swamped. async/await is sort of the high (or low depending on your POV) point of this approach of mapping other architectural styles to call/return: you just write your code as normal call/return, with normal control flow, and just two little annotations lift everything to a completely different architectural style. On the one hand, it is truly brilliant, hiding almost all the mechanics of the more asynchronous architectural style so that you can just keep writing procedural code as is. On the other hand, maybe it would be better to make it easier to express those alternate architectural styles directly in code. This seems like another great move in the direction of the latter.

74 points by srean 10y ago

dmix 10y ago

Anyone using Guile for their projects? Any feedback on their experience with it? I only know Scheme from reading SICP and enjoyed Clojure but hated the java/JVM part of it. I currently use Erlang for when I need concurrency/performant backends. But I'm not totally satisfied with it (for ex: the weak type system and records). Edit: oh looks two programs I use all the time are written in Guile: GNU Make and WeeChat https://en.wikipedia.org/wiki/GNU_Guile#Programs_using_Guile https://en.wikipedia.org/wiki/GNU_Guile#Programs_using_Guile

aban 10y ago
I don't have much experience with Guile, but AFAIK both programs you mentioned are written in C, but support Guile in one way or another. GNU Make supports Guile as an embedded extension language [0] and WeeChat supports Guile as a scripting language for writing plugins [1]. There are a few other apps mentioned on the Guile website [2] and more on their Libraries page [3]. [0]: http://git.savannah.gnu.org/cgit/make.git/commit/?id=c992c4d80f51540699f33fed067caf6b7c38df79 http://git.savannah.gnu.org/cgit/make.git/commit/?id=c992c4d... [1]: http://dev.weechat.org/post/2011/10/27/New-plugin-Guile http://dev.weechat.org/post/2011/10/27/New-plugin-Guile [2]: https://www.gnu.org/software/guile/#apps-using-guile https://www.gnu.org/software/guile/#apps-using-guile [3]: https://www.gnu.org/software/guile/libraries/ https://www.gnu.org/software/guile/libraries/
ZephyrP 10y ago
Alright, total stream of thought here: Guile is cool, if possibly immature. To this day, the largest project I've done in Guile was the entirety of the exercises short of Chapter 5 in SICP, wrote unit tests for the interpreter and the 'Prolog' implementation, the whole lot. All told I generated approximately 10KLOC of Guile code, some 20% of it using features like GOOPs, threading, dynamic-unwind and so on. It was largely a positive experience and I am really glad I read ahead and implemented a few things as GOOPs classes rather than used the OOP system described in the Book (particularly for the Constraint Solver!) As part of this, I did run into some faulty code generation (inappropriate optimization) on the master branch while completing an early Chapter 4 exercise: "This code could do multiple things if we apply different models of an idealized computer language (each of which was advocated by 'Susan' and 'Bob' in typical SICP fashion)", I wrote an implementation inside my guile interpreter that got an unusual answer and asked the channel about it and everyone got to the bottom of it fairly quickly. Speaking of which, Guile has a community of absolutely rabid fans all over the world and so the support in #guile is absolutely phenomenal, people like ijp and mark_weaver are both true 31337s and always can be relied on to offer sage advice. You've brought up it's utility as a GNU extension language, and although I feel it's public documentation is not given a fair shake in the important arenas like GDB scripting, it is tremendously useful in this capacity. Of course, as an extension language itself, it excels. Excellent exposure and interaction with the bare-metal, good parallel & concurrent primitives (although they are more 'traditional', taking after the proud line of Pthreads than `link' or send/recieve in Erlang). It is quite large to statically link into your binary in comparison to other languages however, so be careful there. Weechat itself is written in C, but it does have Guile extensions (and unfortunately I may be the only person in the world to take advantage of this while writing an extension however https://github.com/zv/weechat-gateway-replacer https://github.com/zv/weechat-gateway-replacer ).
mkeeter 10y ago
I'm using Guile for Ao, a Scheme-based tool for solid modeling: https://mattkeeter.com/projects/ao https://mattkeeter.com/projects/ao (previous on HN at https://news.ycombinator.com/item?id=12319406 https://news.ycombinator.com/item?id=12319406) It's been a generally good experience. Scheme is fun, and Guile is a fine implementation. I'm not experienced enough to be pushing its limits – the only unusual corner I've explored is the C FFI, which is quite nice. My only complaint is the consistency of Guile's documentation. It's generally of high quality, but there are corners that are completely undocumented – I've had a few cases where I had to dig into the implementation to figure out how to do something.
rubber_duck 10y ago
I'm not too familiar with Guile and have been away from Clojure world for a while but from what I remember people compared CLJ to Racket which seemed to get a lot of stuff before clojure (IIRC, a lot of the stuff was ported over like type annotations). Might be worth looking in to. https://racket-lang.org/ https://racket-lang.org/
IsmaOlvey 10y ago
Neither Make nor WeeChat are written in Guile. They're both written in C. They do, however, use Guile as an extension language.
davexunit 10y ago
The biggest piece of software using Guile I know of is GNU Guix[0], a functional package manager and declarative GNU/Linux distribution. It has dozens of contributors actively working on it.[1] I contribute to Guix, but I also have my own projects[2] written in Guile. The only software that other people actually use, however, is Haunt, the static site generator that renders my blog (everyone needs to roll their own, right?) [0] https://gnu.org/s/guix https://gnu.org/s/guix [1] https://www.openhub.net/p/gnuguix https://www.openhub.net/p/gnuguix [2] https://dthompson.us/projects.html https://dthompson.us/projects.html
Johnny_Brahms 10y ago
I started using guile at work, since I like scheme, and guile was the scheme that was available without installing anything new. I prototype loads of small tools in guile, and some of the prototypes "stick" since they exceed expectations. The biggest one is a macro system for pascal that expands recursive macros and unrolls loops, generating _lots and lots_ of code. My scheme implementation of choice is chicken, since they have a package manager with quite a lot of packages, but I really enjoy using guile. Really looking forward to the new 2.2 branch. We are using that in production, and we probably hit a sweet spot where the magic optimization dust gathered the most. We got almost 2x improvement just by switching.
rekado 10y ago
I'm using Guile for a simple web interface to a scientific pipeline. Users upload a file, which is queued up in Redis, processed by a worker which spawns the pipeline. It's nice to be able to use a flexible and simple language like Scheme for tools like this. The HTML documents are written in SXML, so it's all S-expressions. (I'm not a fan of Python, the obvious alternative for things like this.)

flukus 10y ago

What's the state of other language interpreters for guile? It's one of this things I always wanted to have a play with until I remember how much I hate lisp.

chrisseaton 10y ago
What do you mean? Other languages running on top of the Guile interpreter? Or implementations of Guile in other languages. Why would you want to run Guile if you don't like Lisp?
- flukus 10y ago
  > Other languages running on top of the Guile interpreter? This one. I've read about various efforts over the years like this one (https://wingolog.org/archives/2009/02/22/ecmascript-for-guile https://wingolog.org/archives/2009/02/22/ecmascript-for-guil...) for javascript and others for python etc. >Why would you want to run Guile if you don't like Lisp? It's the nicest way to embed a high level language in a program I've come across this tutorial is really cool (https://www.gnu.org/software/guile/docs/guile-tut/tutorial.html https://www.gnu.org/software/guile/docs/guile-tut/tutorial.h...). Other there any others that are so simple?
  - chrisseaton 10y ago
    > Other there any others that are so simple? I think the Lua language itself is designed for the implementation to be easy to embed. http://queue.acm.org/detail.cfm?id=1983083 http://queue.acm.org/detail.cfm?id=1983083
- orangeshark 10y ago
  One of the features of Guile is different compiler front-ends. I recall one of the original intentions of Guile was to primarily support two languages, a lisp and a more imperative like language. I don't believe this feature is used that much though. You can find the documentation here https://www.gnu.org/software/guile/manual/html_node/Other-Languages.html#Other-Languages https://www.gnu.org/software/guile/manual/html_node/Other-La...

dkarapetyan 10y ago

What is the point of exposing a scheduler? Seems like that's another implementation detail that should not be exposed. The user should just instantiate some fibers and go.

srean 10y ago
Scheduling algorithms are dime a dozen and the default one provided by a runtime might not be suitable for a particular application. A user space scheduler would let you have finer control when you need it. You may have codepaths that are latency sensitive and those that are throughput sensitive.
- dkarapetyan 10y ago
  That's not what this scheduler does. It simply bundles up some fibers. You don't have control over the actual scheduling. Relevant section of documentation When a fiber would block, it suspends to the scheduler from the current thread. The scheduler will arrange to re-start the fiber when the port or channel becomes readable or writable, as appropriate. For ports, the scheduler adds the file descriptor associated with the port to an epoll set. In either case, the scheduler remembers which fibers are waiting and for what, so that the user can inspect the state of their system. If no scheduler has been installed in the current thread, the fiber will... well, we don’t know yet! Either it blocks its thread, or it aborts. We don’t know. In fact it is pretty clear there is zero control user has over any scheduling activity other than suspending and resuming. It is all driven by polling events internal to the implementation.
  - srean 10y ago
    I was responding to your larger question. User space schedulers would be quite cool, don't know of a language that offers anything over and above a priority queue or timers.

JohnStrange 10y ago

It's not hard to come up with a working green thread system, I've done that in a VM I'm developing for fun, but the crucial question is: Does it distribute the fibers over native worker threads? That's my problem with most scheme implementations, they are great for single-threaded concurrency but require nasty hacks like Racket's 'places' for using all available CPU cores. The only fast multicore scheme I'm aware of is Chez, which has other problems, though. Does Guile do better than Racket with that respect?

mpweiher 10y ago

Yes! "But if your typey spidey senses are tingling, it’s for good reason: with promises, your whole program has to be transformed to return promises-for-values instead of values anywhere it would block." (Or lifted into a monad). To me, that's the big issue with a lot of the concurrency/react abstractions en-vogue now: you are programming indirectly, or more precisely you are programming in a specific (async) architectural style, but trying to express this in a call/return architectural style (that includes methods and FP functions). Depending on where you are, that often means your actual domain code is hidden inside a lot of maps() or flatMaps(). The first problem is that your domain code really should be primary, and your architectural support code as hidden as possible. The second (but probably not last) problem is that you can easily have multiple of these "lifts", for example another for error handling with Maybe. At some point, it just piles up and the actual domain code is completely swamped. async/await is sort of the high (or low depending on your POV) point of this approach of mapping other architectural styles to call/return: you just write your code as normal call/return, with normal control flow, and just two little annotations lift everything to a completely different architectural style. On the one hand, it is truly brilliant, hiding almost all the mechanics of the more asynchronous architectural style so that you can just keep writing procedural code as is. On the other hand, maybe it would be better to make it easier to express those alternate architectural styles directly in code. This seems like another great move in the direction of the latter.