4 ms·

Agreed. Colored functions can be solved, though. See Unison[1] or Zig[2]. [1] https://jaredforsyth.com/posts/whats-cool-about-unison/ https://jaredforsyth.com/

Agreed. Colored functions can be solved, though. See Unison[1] or Zig[2]. [1] https://jaredforsyth.com/posts/whats-cool-about-unison/ https://jaredforsyth.com/posts/whats-cool-about-unison/ [2] https://kristoff.it/blog/zig-colorblind-async-await/ https://kristoff.it/blog/zig-colorblind-async-await/

littlestymaar 6y ago
Zig has a really paradoxical reaction to “function coloring”: on the one hand, they create a specific system to avoid async/sync split, but on the other hand, because allocating functions need an allocator passed as argument, they are recreating the exact same problem in another place. (Allocating functions are red, zero-alloc are blue and you can't call a red function from a blue one). Same problem with Go's `Context` or error as a return value. Same problem with pure/impure functions, or functions returning optional, or platform-specific functions, etc. Sync/async is not special, it is just one instance of a problem that comes up pretty much everywhere (Algebraic effects may be a solution, but I'm not sure).
- dnautics 6y ago
  > Allocating functions are red, zero-alloc are blue and you can't call a red function from a blue one This is not entirely correct analogy. You can embed your allocator inside one of the parameters, too, as a strategy (perhaps a certain type of "object" should carry it's allocator with it). Then uncolored generic functions could operate on allocator-bound and allocator-unbound "objects" alike, even if say the generic function calls a "object method" that performs an allocation in one case but not the other.... You just have to be careful about how to handle errors. There is no equivalent for async. If you were going to be serious about function coloring in zig, the place where there is function coloring is functions with error return values but that's actually a reasonable place to have function coloring, and still, it's easy to build seams in both directions, so a red function can call a blue function, and a blue function can call a red function pretty easily.
  - littlestymaar 6y ago
    > There is no equivalent for async. Sure there is, a symmetrical one: you can embed the Future/Promise inside one of the return value of you sync function. Both work, but both are equally hackish. > If you were going to be serious about function coloring in zig, the place where there is function coloring is functions with error return values I actually address it a bit below.
    - dnautics 6y ago
      No, your understanding is not correct. It's not "hackish" to embed your allocator in your object, sometimes it is the correct, domain driven choice to make. I know it's done somewhere in zig stdlib (parser, iirc).
- snicker7 6y ago
  The painful aspect of async over the other "colors" is that they require the programmer to manually re-annotate all "recolored" functions. E.g. let f call g which calls h. And suppose you replace h with an async function. Then you'd have to manually annotate g and f as async functions too. You don't have to annotate a function as being allocating or not, so the allocation "color" isn't nearly as bad. Languages like Go or Julia or Zig that have "colorless" coroutines are much more robust.
  - littlestymaar 6y ago
    let f call g which calls h. And suppose you replace h with an allocating function, then in Zig you have to manually add a allocator as a parameter in g and f and update their call site accordingly, because now they are also allocating. And suppose you replace h with a function that returns an error, then in Go you have to manually add an error as a return value in g and f. h become impure? Then g and f are not pure anymore either. h becomes Windows specific, then g and f are Windows specific[1]. The situation is highly similar. [1]: or you need to write a non-windows version of the work done by the h function, which fundamentally equivalent to calling block_on on an async function to make it a blocking one.
    - deleted 6y ago
      [deleted]
- sobeston 6y ago
  This is silly. If you make code that requires allocation and does not have an allocator passed into it, there is literally nothing stopping you from allocating anyway. Zig does not stop you from, for example, heap allocating without the usage of an allocator (Zig does not know what heap allocation even is). You can directly use the std page_allocator (and others) wherever you want, too. There are very few good reasons to do this (e.g. spawning a thread requires particular allocation; using an allocator might not be sound depending on the OS). All you have to do is document that it allocates memory and that's fine; there is no split.
  - lhorie 6y ago
    I tend to agree that allocator param isn't that strong of an argument (e.g. one can easily just store a pointer to the allocator in a struct to hide it from method signatures). And it is quite valid to just create a local allocator and `defer a.free(thing)` to clean up at the end of a function (for example, if you just need a trivial std.mem.join and you know that you can fit the result in stack). One does not need to use the top-level GPA/arena for everything. But nonetheless I think there is a split (at least for now). Zig async function frames are different than regular functions, so even if syntactically, `foo()` could magically be switched between sync or async based on io_mode, in practice if you have a recursive call tree doing fs operations, it'll work in one mode (up until a stack overflow, that is) and throw a compile error upfront in the other. Presumably this would not be an issue anymore once [0] is implemented. [0] https://github.com/ziglang/zig/issues/1006 https://github.com/ziglang/zig/issues/1006
- earthboundkid 6y ago
  This is a really good insight. There are some architecture decisions you really need to anticipate because if you don't, it will really be a pain in the ass to refactor. I'm most familiar with Go, and there I've gotten very used to the idea that my functions will probably need an error return, so e.g. I write my main func to call into a run(...) error func first thing. I've had to go back and replumb things with context, and it's no fun. Similar strategies apply in JavaScript, where you just know that any kind of button press will probably need to be a Promise because it might have to wait on a network call or whatever. It's a pervasive problem.
  - littlestymaar 6y ago
    In a perfect word, this kind of refactor should be handled by the tooling (IDE/Language Server) though.
- AndyKelley 6y ago
  Rust has std and async_std as separate codebases because of function coloring. Zig has the same codebase that supports both evented I/O and blocking I/O because of the non-existence of function coloring. Zig has the same codebase with regards to allocation as well; there is no "allocator_std" or some nonsense like that. Where is the paradox, exactly?
  - steveklabnik 6y ago
    > Rust has std and async_std as separate codebases because of function coloring. It depends on what you mean by "separate codebases." asyc_std is a third party package someone made, so it's a separate codebase because of that. We could also include all of it (or something like it) and it would be a "single codebase," but I think that's more about your wording than your point.
  - littlestymaar 6y ago
    Hi Andrew, I'm a big fan of your work btw. As I said in the original comment, sync/async and no-alloc/alloc or no-error/error are in a parallel situation. And in a language design you can either expose “colors” to your users, or hide it through some magic and a global state. I personally prefer the former, and I liked Zig's idea to expose the allocation explicitly, instead of what Rust did initially (there are slowly rolling out the option to use a specific allocator for collections, which is good even if it took almost 6 years after 1.0). With error Handling, Zig also took the same route with error sets instead of exceptions. That's why I don't think the colorblind async mechanism is a good fit in the language (and it adds some cognitive burden, because newcomers need to learn that looking at code full of async/await doesn't necessarily mean they are looking at non-blocking code). (Also, async-std is a third party library with little in common with std and whose name has been chosen for “marketing” reason, so I' don't really think the comparison is relevant.)

littlestymaar 6y ago

Zig has a really paradoxical reaction to “function coloring”: on the one hand, they create a specific system to avoid async/sync split, but on the other hand, because allocating functions need an allocator passed as argument, they are recreating the exact same problem in another place. (Allocating functions are red, zero-alloc are blue and you can't call a red function from a blue one). Same problem with Go's `Context` or error as a return value. Same problem with pure/impure functions, or functions returning optional, or platform-specific functions, etc. Sync/async is not special, it is just one instance of a problem that comes up pretty much everywhere (Algebraic effects may be a solution, but I'm not sure).

dnautics 6y ago
> Allocating functions are red, zero-alloc are blue and you can't call a red function from a blue one This is not entirely correct analogy. You can embed your allocator inside one of the parameters, too, as a strategy (perhaps a certain type of "object" should carry it's allocator with it). Then uncolored generic functions could operate on allocator-bound and allocator-unbound "objects" alike, even if say the generic function calls a "object method" that performs an allocation in one case but not the other.... You just have to be careful about how to handle errors. There is no equivalent for async. If you were going to be serious about function coloring in zig, the place where there is function coloring is functions with error return values but that's actually a reasonable place to have function coloring, and still, it's easy to build seams in both directions, so a red function can call a blue function, and a blue function can call a red function pretty easily.
- littlestymaar 6y ago
  > There is no equivalent for async. Sure there is, a symmetrical one: you can embed the Future/Promise inside one of the return value of you sync function. Both work, but both are equally hackish. > If you were going to be serious about function coloring in zig, the place where there is function coloring is functions with error return values I actually address it a bit below.
  - dnautics 6y ago
    No, your understanding is not correct. It's not "hackish" to embed your allocator in your object, sometimes it is the correct, domain driven choice to make. I know it's done somewhere in zig stdlib (parser, iirc).
snicker7 6y ago
The painful aspect of async over the other "colors" is that they require the programmer to manually re-annotate all "recolored" functions. E.g. let f call g which calls h. And suppose you replace h with an async function. Then you'd have to manually annotate g and f as async functions too. You don't have to annotate a function as being allocating or not, so the allocation "color" isn't nearly as bad. Languages like Go or Julia or Zig that have "colorless" coroutines are much more robust.
- littlestymaar 6y ago
  let f call g which calls h. And suppose you replace h with an allocating function, then in Zig you have to manually add a allocator as a parameter in g and f and update their call site accordingly, because now they are also allocating. And suppose you replace h with a function that returns an error, then in Go you have to manually add an error as a return value in g and f. h become impure? Then g and f are not pure anymore either. h becomes Windows specific, then g and f are Windows specific[1]. The situation is highly similar. [1]: or you need to write a non-windows version of the work done by the h function, which fundamentally equivalent to calling block_on on an async function to make it a blocking one.
  - deleted 6y ago
    [deleted]
sobeston 6y ago
This is silly. If you make code that requires allocation and does not have an allocator passed into it, there is literally nothing stopping you from allocating anyway. Zig does not stop you from, for example, heap allocating without the usage of an allocator (Zig does not know what heap allocation even is). You can directly use the std page_allocator (and others) wherever you want, too. There are very few good reasons to do this (e.g. spawning a thread requires particular allocation; using an allocator might not be sound depending on the OS). All you have to do is document that it allocates memory and that's fine; there is no split.
- lhorie 6y ago
  I tend to agree that allocator param isn't that strong of an argument (e.g. one can easily just store a pointer to the allocator in a struct to hide it from method signatures). And it is quite valid to just create a local allocator and `defer a.free(thing)` to clean up at the end of a function (for example, if you just need a trivial std.mem.join and you know that you can fit the result in stack). One does not need to use the top-level GPA/arena for everything. But nonetheless I think there is a split (at least for now). Zig async function frames are different than regular functions, so even if syntactically, `foo()` could magically be switched between sync or async based on io_mode, in practice if you have a recursive call tree doing fs operations, it'll work in one mode (up until a stack overflow, that is) and throw a compile error upfront in the other. Presumably this would not be an issue anymore once [0] is implemented. [0] https://github.com/ziglang/zig/issues/1006 https://github.com/ziglang/zig/issues/1006
earthboundkid 6y ago
This is a really good insight. There are some architecture decisions you really need to anticipate because if you don't, it will really be a pain in the ass to refactor. I'm most familiar with Go, and there I've gotten very used to the idea that my functions will probably need an error return, so e.g. I write my main func to call into a run(...) error func first thing. I've had to go back and replumb things with context, and it's no fun. Similar strategies apply in JavaScript, where you just know that any kind of button press will probably need to be a Promise because it might have to wait on a network call or whatever. It's a pervasive problem.
- littlestymaar 6y ago
  In a perfect word, this kind of refactor should be handled by the tooling (IDE/Language Server) though.
AndyKelley 6y ago
Rust has std and async_std as separate codebases because of function coloring. Zig has the same codebase that supports both evented I/O and blocking I/O because of the non-existence of function coloring. Zig has the same codebase with regards to allocation as well; there is no "allocator_std" or some nonsense like that. Where is the paradox, exactly?
- steveklabnik 6y ago
  > Rust has std and async_std as separate codebases because of function coloring. It depends on what you mean by "separate codebases." asyc_std is a third party package someone made, so it's a separate codebase because of that. We could also include all of it (or something like it) and it would be a "single codebase," but I think that's more about your wording than your point.
- littlestymaar 6y ago
  Hi Andrew, I'm a big fan of your work btw. As I said in the original comment, sync/async and no-alloc/alloc or no-error/error are in a parallel situation. And in a language design you can either expose “colors” to your users, or hide it through some magic and a global state. I personally prefer the former, and I liked Zig's idea to expose the allocation explicitly, instead of what Rust did initially (there are slowly rolling out the option to use a specific allocator for collections, which is good even if it took almost 6 years after 1.0). With error Handling, Zig also took the same route with error sets instead of exceptions. That's why I don't think the colorblind async mechanism is a good fit in the language (and it adds some cognitive burden, because newcomers need to learn that looking at code full of async/await doesn't necessarily mean they are looking at non-blocking code). (Also, async-std is a third party library with little in common with std and whose name has been chosen for “marketing” reason, so I' don't really think the comparison is relevant.)