12 ms·

I think this design is very reasonable. However, I find Zig's explanation of it pretty confusing: they've taken pains to emphasize that it solves the function c

I think this design is very reasonable. However, I find Zig's explanation of it pretty confusing: they've taken pains to emphasize that it solves the function coloring problem, which it doesn't: it pushes I/O into an effect type, which essentially behaves as a token that callers need to retain. This is a form of coloring, albeit one that's much more ergonomic. (To my understanding this is pretty similar to how Go solves asynchronicity, expect that in Go's case the "token" is managed by the runtime.)

rowanG077 11mo ago
Having used zig a bit as a hobby. Why is it more ergonomic? Using await vs passing a token have similar ergonomics to me. The one thing you could say is that using some kind of token makes it dead simple to have different tokens. But that's really not something I run into often at all when using async.
- messe 11mo ago
  > The one thing you could say is that using some kind of token makes it dead simple to have different tokens. But that's really not something I run into often at all when using async. It's valuable to library authors who can now write code that's agnostic of the users' choice of runtime, while still being able to express that asynchronicity is possible for certain code paths.
  - rowanG077 11mo ago
    But that can already be done using async await. If you write an async function in Rust for example you are free to call it with any async runtime you want.
    - messe 11mo ago
      But you can't call it from synchronous rust. Zig is moving toward all sync code also using the Io interface.
      - tcfhgj 11mo ago
        yes, you can: runtime.block_on(async { }) https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=060fc0424005ecf2d892e55e39988a03 https://play.rust-lang.org/?version=stable&mode=debug&editio...
        whytevuhuni 11mo ago
        Here's a problem with that: Cannot start a runtime from within a runtime. This happens because a function (like `block_on`) attempted to block the current thread while the thread is being used to drive asynchronous tasks. https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=8abf9e00ef6039d9cd5f8fd7adc6bab0 https://play.rust-lang.org/?version=stable&mode=debug&editio...
        tcfhgj 11mo ago
        just pass around handles like you do in zig, alright? also: spawn_blocking for blocking code
        whytevuhuni 11mo ago
        But that's the thing, idiomatic Rust sync code almost never passes around handles, even when they need to do I/O. You might be different, and you might start doing that in your code, but almost none of either std or 3rd party libraries will cooperate with you. The difference with Zig is not in its capabilities, but rather in how the ecosystem around its stdlib is built. The equivalent in Rust would be if almost all I/O functions in std would be async; granted that would be far too expensive and disruptive given how async works.
        tcfhgj 11mo ago
        > But that's the thing, idiomatic Rust sync code almost never passes around handles, even when they need to do I/O. Because they don't use async inside. Zig code is passing around handles in code without io?
- hansvm 11mo ago
  Making it dead simple to have different tokens is exactly the goal. A smattering of examples recently on my mind: As a background, you might ask why you need different runtimes ever. Why not just make everything async and be done with it, especially if the language is able to hide that complexity? 1. In the context of a systems language that's not an option. You might be writing an OS, embedded code, a game with atypical performance demands requiring more care with the IO, some kernel-bypass shenanigan, etc. Even just selecting between a few builtin choices (like single-threaded async vs multi-threaded async vs single-threaded sync) doesn't provide enough flexibility for the range of programs you're trying to allow a user to write. 2. Similarly, even initializing a truly arbitrary IO effect once at compile-time doesn't always suffice. Maybe you normally want a multi-threaded solution but need more care with respect to concurrency in some critical section and need to swap in a different IO. Maybe you normally get to interact with the normal internet but have a mode/section/interface/etc where you need to send messages through stranger networking conditions (20s ping, 99% packet loss, 0.1kbps upload on the far side, custom hardware, etc). Maybe some part of your application needs bounded latency and is fine dropping packets but some other part needs high throughput and no dropped packets at any latency cost. Maybe your disk hardware is such that it makes sense for networking to be async and disk to be sync. And so on. You can potentially work around that in a world with a single IO implementation if you can hack around it with different compilation units or something, but it gets complicated. Part of the answer then is that you need (or really want) something equivalent to different IO runtimes, hot-swappable for each function call. I gave some high-level ideas as to why that might be the case, but high-level observations often don't resonate, so let's look at a concrete case where `await` is less ergonomic: 1. Take something like TLS as an example (stdlib or 3rd-party, doesn't really matter). The handshake code is complicated, so a normal implementation calls into an IO abstraction layer and physically does reads and writes (as opposed to, e.g., a pure state-machine implementation which returns some metadata about which action to perform next -- I hacked together a terrible version of that at one point [0] if you want to see what I mean). What if you want to run it on an embedded device? If it were written with async it would likely have enough other baggage that it wouldn't fit or otherwise wouldn't work. What if you want to hide your transmission in other data to sneak it past prying eyes (steganography, nowadays that's relatively easy to do via LLMs interestingly enough, and you can embed arbitrary data in messages which are human-readable and purport to discuss completely other things without exposing hi/lo-bit patterns or other such things that normally break steganography)? Then the kernel socket abstraction doesn't work at all, and "just using await" doesn't fix the problem. Basically, any place you want to use that library (and, arguably, that's the sort of code where you should absolutely use a library rather than rolling it yourself), if the implementer had a "just use await" mentality then you're SOL if you need to use it in literally any other context. I was going to write more concrete cases, but this comment is getting to be too long. The general observation is that "just use await" hinders code re-use. If you're writing code for your own consumption and also never need those other uses then it's a non-issue, but with a clever choice of abstraction it _might_ be possible (old Zig had a solution that didn't quite hit the mark IMO, and time will tell if this one is good enough, but I'm optimistic) to enable the IO code people naturally write to be appropriately generic by default and thus empower future developers via a more composable set of primitives. They really nailed that with the allocator interface, and if this works then my only real concern is a generic "what next" -- it's pushing toward an effect system, but integrating those with a systems language is mostly an unsolved problem, and adding a 3rd, 4th, etc explicit parameter to nearly every function is going to get unwieldy in a hurry (back-of-the-envelope idea I've had stewing if I ever write a whole "major" language is to basically do what Zig currently does and pack all those "effects" into a single effect parameter that you pass into each function, still allowing you to customize each function call, still allowing you to inspect which functions require allocators or whatever, but making the experience more pleasant if you have a little syntactic sugar around sub-effects and if the parent type class is comptime-known). [0] https://github.com/hmusgrave/rayloop/blob/d5e797967c42b9c891f1fb0603cc2498a896d75d/src/main.zig#L526 https://github.com/hmusgrave/rayloop/blob/d5e797967c42b9c891...
flohofwoe 11mo ago
If calling the same function with a different argument would be considered 'function coloring', every function in a program is 'colored' and the word loses its meaning ;) Zig actually also had solved the coloring problem in the old and abandondend async-await solution because the compiler simply stamped out a sync- or async-version of the same function based on the calling context (this works because everything is a single compilation unit).
- deleted 11mo ago
  [deleted]
- woodruffw 11mo ago
  > If calling the same function with a different argument would be considered 'function coloring', than every function in a program is 'colored' and the word loses its meaning ;) Well, yes, but in this case the colors (= effects) are actually important. The implications of passing an effect through a system are nontrivial, which is why some languages choose to promote that effect to syntax (Rust) and others choose to make it a latent invariant (Java, with runtime exceptions). Zig chooses another path not unlike Haskell's IO.
- rowanG077 11mo ago
  If your functions suddenly requires (currently)unconstructable instance "Magic" which you now have to pass in from somewhere top level, that indeed suffers from the same issue as async/await. Aka function coloring. But most functions don't. They require some POD or float, string or whatever that can be easily and cheaply constructed in place.
  - mk12 11mo ago
    Colors for 2 ways of doing IO vs colors for doing IO or not are so different that it’s confusing to call both of them “function coloring problem”. Only the former leads to having to duplicate everything (sync version and async version). If only the latter was a thing, no one would have coined the term and written the blog post.
dundarious 11mo ago
There is a token you must pass around, sure, but because you use the same token for both async and sync code, I think analogizing with the typical async function color problem is incorrect.
jayd16 11mo ago
Actually it seems like they just colored everything async and you pick whether you have worker threads or not. I do wonder if there's more magic to it than that because it's not like that isn't trivially possible in other languages. The issue is it's actually a huge foot gun when you mix things like this. For example your code can run fine synchronously but will deadlock asynchronously because you don't account for methods running in parallel. Or said another way, some code is thread safe and some code isn't. Coloring actually helps with that.
- flohofwoe 11mo ago
  > Actually it seems like they just colored everything async and you pick whether you have worker threads or not. There is no 'async' anywhere yet in the new Zig IO system (in the sense of the compiler doing the 'state machine code transform' on async functions). AFAIK the current IO runtimes simply use traditional threads or coroutines with stack switching. Bringing code-transform-async-await back is still on the todo-list. The basic idea is that the code which calls into IO interface doesn't need to know how the IO runtime implements concurrency. I guess though that the function that's called through the `.async()` wrapper is expected to work properly both in multi- and single-threaded contexts.
  - jayd16 11mo ago
    > There is no 'async' I meant this more as simply an analogy to the devX of other languages. >Bringing code-transform-async-await back is still on the todo-list. The article makes it seem like "the plan is set" so I do wonder what that Todo looks like. Is this simply the plan for async IO? > is expected to work properly both in multi- and single-threaded contexts. Yeah... about that.... I'm also interested in how that will be solved. RTFM? I suppose a convention could be that your public API must be thread safe and if you have a thread-unsafe pattern it must be private? Maybe something else is planned?
    - messe 11mo ago
      > The article makes it seem like "the plan is set" so I do wonder what that Todo looks like. Is this simply the plan for async IO? There's currently a proposal for stackless coroutines as a language primitive: https://github.com/ziglang/zig/issues/23446 https://github.com/ziglang/zig/issues/23446
doyougnu 11mo ago
Agreed. the Haskeller in me screams "You've just implemented the IO monad without language support".
- AndyKelley 11mo ago
  It's not a monad because it doesn't return a description of how to carry out I/O that is performed by a separate system; it does the I/O inside the function before returning. That's a regular old interface, not a monad.
  - endgame 11mo ago
    So it's the reader monad, then? ;-)
    - deleted 11mo ago
      [deleted]
    - tylerhou 11mo ago
      Yes.
      - AndyKelley 11mo ago
        Can you explain for those of us less familiar with Haskell (and monads in general)?
        dan-robertson 11mo ago
        Reader monad is a fancy way of saying ‘have the ability to read some constant value throughout the computation’. So here they mean the io value that is passed between functions.
        AndyKelley 11mo ago
        Well I don't think that fits at all. In Zig, an Io instance is an interface, passed as a parameter. You can draw some connections between what Zig is doing and what Haskell is doing but it's not a monad. It's plain old interfaces and parameters, just like Allocator.
SkiFire13 11mo ago
The function coloring problem actually comes up when you implement the async part using stackless coroutines (e.g. in Rust) or callbacks (e.g. in Javascript). Zig's new I/O does neither of those for now, so hence why it doesn't suffer from it, but at the same time it didn't "solve" the problem, it just sidestepped it by providing an implementation that has similar features but not exactly the same tradeoffs.
- bloppe 11mo ago
  How are the tradeoffs meaningfully different? Imagine that, instead of passing an `Io` object around, you just had to add an `async` keyword to the function, and that was simply syntactic sugar for an implied `Io` argument, and you could use an `await` keyword as syntactic sugar to pass whatever `Io` object the caller has to the callee. I don't see how that's not the exact same situation.
  - VMG 11mo ago
    Maybe I have this wrong, but I believe the difference is that you can create an Io instance in a function that has none
    - bloppe 11mo ago
      In Rust, you can always create a new tokio runtime and use that to call an async function from a sync function. Ditto with Python: just create a new asyncio event loop and call `run`. That's actually exactly what an Io object in Zig is, but with a new name. Looking back at the original function coloring post [1], it says: > It is better. I will take async-await over bare callbacks or futures any day of the week. But we’re lying to ourselves if we think all of our troubles are gone. As soon as you start trying to write higher-order functions, or reuse code, you’re right back to realizing color is still there, bleeding all over your codebase. So if this is isomorphic to async/await, it does not "solve" the coloring problem as originally stated, but I'm starting to think it's not much of a problem at all. Some functions just have different signatures from other functions. It was only a huge problem for JavaScript because the ecosystem at large decided to change the type signatures of some giant portion of all functions at once, migrating from callbacks to async. [1]: https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ https://journal.stuffwithstuff.com/2015/02/01/what-color-is-...
eikenberry 11mo ago
Function coloring is specifically about requiring syntax for a function, eg. the async keyword. So if you want an async and non-async function you need to write both in code. If you pass the "coloring" as an argument you avoid the need for extra syntax and multiple function definitions and therefor the function has no color. You can solve this in various ways with various tradeoffs but as long as there is a single function (syntactically) is all that matters for coloring.
- IshKebab 11mo ago
  > Function coloring is specifically about requiring syntax for a function, eg. the async keyword. It isn't really. It's about having two classes of functions (async and sync), and not being able to await async functions from sync ones. It was originally about Javascript, where it is the case due to how the runtime works. In a sync function you can technically call an async one, but it returns a promise. There's no way to get the actual result before you return from your sync function. That isn't the case for all languages though. E.g. in Rust: https://docs.rs/futures/latest/futures/executor/fn.block_on.html https://docs.rs/futures/latest/futures/executor/fn.block_on.... I think maybe Python can do something similar but don't quote me on that. There's a closely related problem about making functions generic over synchronicity, which people try and solve with effects, monads, etc. Maybe people call that "function colouring" now, but that wasn't exactly the original meaning.
- woodruffw 11mo ago
  > Function coloring is specifically about requiring syntax for a function, eg. the async keyword. Someone should tell the inventor of the phrase, because they don't mention the async keyword at all[1]. As-written, function coloring is about callbacks (since that's semantic mechanism that JavaScript happens to pick for their asynchronous model). Function coloring is just an informal way to describe encoding a function's effect. You can encode that in syntax if you want (an `async` keyword), or in the type system (returning `() -> T` instead of `T`), or in the runtime itself (by controlling all I/O and treating it the same). But you can't avoid it. [1]: https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ https://journal.stuffwithstuff.com/2015/02/01/what-color-is-...
throwawaymaths 11mo ago
1) zig's io is not a viral effect type, you can in principle declare a global io variable and use it everywhere that any library calls for it. Not best practice for a library writer, but if you're building an app, do what you want. 2) There are two things here, there is function coloring and the function coloring problem. The function coloring problem is five things: https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ https://journal.stuffwithstuff.com/2015/02/01/what-color-is-... 1. Every function has a color. 2. The way you call a function depends on its color. 3. You can only call a red function from within another red function. 4. Red functions are more painful to call. 5. Some core library functions are red. You'll have some convincing to do that zig's plan satisfies 4. It's almost certain that it won't satisfy 5. It's open to debate if zig's plan will work at all, of course.
- woodruffw 11mo ago
  > 1) zig's io is not an effect type, you can in principle declare a global io variable and use it everywhere that any library calls for it. That's an effect, akin to globally intermediated I/O in a managed runtime. To make it intuitive: if you have a global token for I/O, does your concurrent program need to synchronize on it in order to operate soundly? Do programs that fail to obtain the token behave correctly?
  - throwawaymaths 11mo ago
    how do you "fail to obtain the token"?
    - woodruffw 11mo ago
      The token guards a fallible resource (I/O). You can (temporarily or permanently) fail to obtain it for any reason that would affect the underlying I/O.
      - throwawaymaths 11mo ago
        the io isnt a single resource? it's a module grouping together a pile of code. and you can swap out implementations. the io modules should be responsible for handing out many failable resources, and synchronization is going to be up to the io module code, and thats whether or not it's globalized or passed.
dan-robertson 11mo ago
This solves a problem for library authors which is that blocking and event-based io implementations of functionality look the same but are not actually the same so users end up complaining when you do one but not the other. It adds a problem of needing to pass the global kind of io through a program. I think this mostly isn’t a huge problem because typical good program design has io on the periphery and so you don’t tend to need to pass this io object that ‘deep’. This is not too different from the type-system effect of IO in Haskell (except that one only does evented IO IIRC). It isn’t as bad because it only affects input types (data which can be closed over, I assume) rather than output types. Eg in Haskell you need various special functions to change from [ IO a ] to IO [ a ] but in the zig model you iterate over your list in the normal way using an io value from an outer scope. The one case where Io-colouring was annoying to me in Haskell was adding printf debugging (there is a function to cheat the type system for this). Zig may have other solutions to that, eg a global io value for blocking io in debug builds or some global logging system.
- themk 11mo ago
  There is nothing special about the [IO a] -> IO [a] in Haskell. You can iterate over it using the "normal" methods of iterating just fine. forM ios $ \io -> io But there are better ways to do it (e.g. sequence), but those are also not "special" to IO in any way. They are common abstractions usable by any Monad.
  - dan-robertson 11mo ago
    Haskell is a bit tricky to talk about here because it has other big differences on laziness and suchlike, and this means there are pervasive monads. If you instead consider a language more like JavaScript where async functions return values wrapped in promises and therefore require special versions of lots of things like Array.prototype.forEach for async-returning functions (ok, language feature of generators helps here). The point I’m trying to get at is that putting io-ness into an argument works better than putting it into the return type because it is easier to pass an extra argument when using other language features an harder to do an extra thing with returned values.
esjeon 11mo ago
AFACT, the only practically critical issue of colored function is the duplication of code b/w sync and async code paths. Zig avoids this with dependency injection, and that’s enough for practical usages (which basically means “ergonomic”). Other points raised by the original article (like calling async function is more difficult) are pretty much unavoidable for the sake of precise control.

rowanG077 11mo ago

Having used zig a bit as a hobby. Why is it more ergonomic? Using await vs passing a token have similar ergonomics to me. The one thing you could say is that using some kind of token makes it dead simple to have different tokens. But that's really not something I run into often at all when using async.

messe 11mo ago
> The one thing you could say is that using some kind of token makes it dead simple to have different tokens. But that's really not something I run into often at all when using async. It's valuable to library authors who can now write code that's agnostic of the users' choice of runtime, while still being able to express that asynchronicity is possible for certain code paths.
- rowanG077 11mo ago
  But that can already be done using async await. If you write an async function in Rust for example you are free to call it with any async runtime you want.
  - messe 11mo ago
    But you can't call it from synchronous rust. Zig is moving toward all sync code also using the Io interface.
    - tcfhgj 11mo ago
      yes, you can: runtime.block_on(async { }) https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=060fc0424005ecf2d892e55e39988a03 https://play.rust-lang.org/?version=stable&mode=debug&editio...
      - whytevuhuni 11mo ago
        Here's a problem with that: Cannot start a runtime from within a runtime. This happens because a function (like `block_on`) attempted to block the current thread while the thread is being used to drive asynchronous tasks. https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=8abf9e00ef6039d9cd5f8fd7adc6bab0 https://play.rust-lang.org/?version=stable&mode=debug&editio...
        tcfhgj 11mo ago
        just pass around handles like you do in zig, alright? also: spawn_blocking for blocking code
        whytevuhuni 11mo ago
        But that's the thing, idiomatic Rust sync code almost never passes around handles, even when they need to do I/O. You might be different, and you might start doing that in your code, but almost none of either std or 3rd party libraries will cooperate with you. The difference with Zig is not in its capabilities, but rather in how the ecosystem around its stdlib is built. The equivalent in Rust would be if almost all I/O functions in std would be async; granted that would be far too expensive and disruptive given how async works.
        tcfhgj 11mo ago
        > But that's the thing, idiomatic Rust sync code almost never passes around handles, even when they need to do I/O. Because they don't use async inside. Zig code is passing around handles in code without io?
hansvm 11mo ago
Making it dead simple to have different tokens is exactly the goal. A smattering of examples recently on my mind: As a background, you might ask why you need different runtimes ever. Why not just make everything async and be done with it, especially if the language is able to hide that complexity? 1. In the context of a systems language that's not an option. You might be writing an OS, embedded code, a game with atypical performance demands requiring more care with the IO, some kernel-bypass shenanigan, etc. Even just selecting between a few builtin choices (like single-threaded async vs multi-threaded async vs single-threaded sync) doesn't provide enough flexibility for the range of programs you're trying to allow a user to write. 2. Similarly, even initializing a truly arbitrary IO effect once at compile-time doesn't always suffice. Maybe you normally want a multi-threaded solution but need more care with respect to concurrency in some critical section and need to swap in a different IO. Maybe you normally get to interact with the normal internet but have a mode/section/interface/etc where you need to send messages through stranger networking conditions (20s ping, 99% packet loss, 0.1kbps upload on the far side, custom hardware, etc). Maybe some part of your application needs bounded latency and is fine dropping packets but some other part needs high throughput and no dropped packets at any latency cost. Maybe your disk hardware is such that it makes sense for networking to be async and disk to be sync. And so on. You can potentially work around that in a world with a single IO implementation if you can hack around it with different compilation units or something, but it gets complicated. Part of the answer then is that you need (or really want) something equivalent to different IO runtimes, hot-swappable for each function call. I gave some high-level ideas as to why that might be the case, but high-level observations often don't resonate, so let's look at a concrete case where `await` is less ergonomic: 1. Take something like TLS as an example (stdlib or 3rd-party, doesn't really matter). The handshake code is complicated, so a normal implementation calls into an IO abstraction layer and physically does reads and writes (as opposed to, e.g., a pure state-machine implementation which returns some metadata about which action to perform next -- I hacked together a terrible version of that at one point [0] if you want to see what I mean). What if you want to run it on an embedded device? If it were written with async it would likely have enough other baggage that it wouldn't fit or otherwise wouldn't work. What if you want to hide your transmission in other data to sneak it past prying eyes (steganography, nowadays that's relatively easy to do via LLMs interestingly enough, and you can embed arbitrary data in messages which are human-readable and purport to discuss completely other things without exposing hi/lo-bit patterns or other such things that normally break steganography)? Then the kernel socket abstraction doesn't work at all, and "just using await" doesn't fix the problem. Basically, any place you want to use that library (and, arguably, that's the sort of code where you should absolutely use a library rather than rolling it yourself), if the implementer had a "just use await" mentality then you're SOL if you need to use it in literally any other context. I was going to write more concrete cases, but this comment is getting to be too long. The general observation is that "just use await" hinders code re-use. If you're writing code for your own consumption and also never need those other uses then it's a non-issue, but with a clever choice of abstraction it _might_ be possible (old Zig had a solution that didn't quite hit the mark IMO, and time will tell if this one is good enough, but I'm optimistic) to enable the IO code people naturally write to be appropriately generic by default and thus empower future developers via a more composable set of primitives. They really nailed that with the allocator interface, and if this works then my only real concern is a generic "what next" -- it's pushing toward an effect system, but integrating those with a systems language is mostly an unsolved problem, and adding a 3rd, 4th, etc explicit parameter to nearly every function is going to get unwieldy in a hurry (back-of-the-envelope idea I've had stewing if I ever write a whole "major" language is to basically do what Zig currently does and pack all those "effects" into a single effect parameter that you pass into each function, still allowing you to customize each function call, still allowing you to inspect which functions require allocators or whatever, but making the experience more pleasant if you have a little syntactic sugar around sub-effects and if the parent type class is comptime-known). [0] https://github.com/hmusgrave/rayloop/blob/d5e797967c42b9c891f1fb0603cc2498a896d75d/src/main.zig#L526 https://github.com/hmusgrave/rayloop/blob/d5e797967c42b9c891...

flohofwoe 11mo ago

If calling the same function with a different argument would be considered 'function coloring', every function in a program is 'colored' and the word loses its meaning ;) Zig actually also had solved the coloring problem in the old and abandondend async-await solution because the compiler simply stamped out a sync- or async-version of the same function based on the calling context (this works because everything is a single compilation unit).

deleted 11mo ago
[deleted]
woodruffw 11mo ago
> If calling the same function with a different argument would be considered 'function coloring', than every function in a program is 'colored' and the word loses its meaning ;) Well, yes, but in this case the colors (= effects) are actually important. The implications of passing an effect through a system are nontrivial, which is why some languages choose to promote that effect to syntax (Rust) and others choose to make it a latent invariant (Java, with runtime exceptions). Zig chooses another path not unlike Haskell's IO.
rowanG077 11mo ago
If your functions suddenly requires (currently)unconstructable instance "Magic" which you now have to pass in from somewhere top level, that indeed suffers from the same issue as async/await. Aka function coloring. But most functions don't. They require some POD or float, string or whatever that can be easily and cheaply constructed in place.
- mk12 11mo ago
  Colors for 2 ways of doing IO vs colors for doing IO or not are so different that it’s confusing to call both of them “function coloring problem”. Only the former leads to having to duplicate everything (sync version and async version). If only the latter was a thing, no one would have coined the term and written the blog post.

dundarious 11mo ago

There is a token you must pass around, sure, but because you use the same token for both async and sync code, I think analogizing with the typical async function color problem is incorrect.

jayd16 11mo ago

Actually it seems like they just colored everything async and you pick whether you have worker threads or not. I do wonder if there's more magic to it than that because it's not like that isn't trivially possible in other languages. The issue is it's actually a huge foot gun when you mix things like this. For example your code can run fine synchronously but will deadlock asynchronously because you don't account for methods running in parallel. Or said another way, some code is thread safe and some code isn't. Coloring actually helps with that.

flohofwoe 11mo ago
> Actually it seems like they just colored everything async and you pick whether you have worker threads or not. There is no 'async' anywhere yet in the new Zig IO system (in the sense of the compiler doing the 'state machine code transform' on async functions). AFAIK the current IO runtimes simply use traditional threads or coroutines with stack switching. Bringing code-transform-async-await back is still on the todo-list. The basic idea is that the code which calls into IO interface doesn't need to know how the IO runtime implements concurrency. I guess though that the function that's called through the `.async()` wrapper is expected to work properly both in multi- and single-threaded contexts.
- jayd16 11mo ago
  > There is no 'async' I meant this more as simply an analogy to the devX of other languages. >Bringing code-transform-async-await back is still on the todo-list. The article makes it seem like "the plan is set" so I do wonder what that Todo looks like. Is this simply the plan for async IO? > is expected to work properly both in multi- and single-threaded contexts. Yeah... about that.... I'm also interested in how that will be solved. RTFM? I suppose a convention could be that your public API must be thread safe and if you have a thread-unsafe pattern it must be private? Maybe something else is planned?
  - messe 11mo ago
    > The article makes it seem like "the plan is set" so I do wonder what that Todo looks like. Is this simply the plan for async IO? There's currently a proposal for stackless coroutines as a language primitive: https://github.com/ziglang/zig/issues/23446 https://github.com/ziglang/zig/issues/23446

doyougnu 11mo ago

Agreed. the Haskeller in me screams "You've just implemented the IO monad without language support".

AndyKelley 11mo ago
It's not a monad because it doesn't return a description of how to carry out I/O that is performed by a separate system; it does the I/O inside the function before returning. That's a regular old interface, not a monad.
- endgame 11mo ago
  So it's the reader monad, then? ;-)
  - deleted 11mo ago
    [deleted]
  - tylerhou 11mo ago
    Yes.
    - AndyKelley 11mo ago
      Can you explain for those of us less familiar with Haskell (and monads in general)?
      - dan-robertson 11mo ago
        Reader monad is a fancy way of saying ‘have the ability to read some constant value throughout the computation’. So here they mean the io value that is passed between functions.
        AndyKelley 11mo ago
        Well I don't think that fits at all. In Zig, an Io instance is an interface, passed as a parameter. You can draw some connections between what Zig is doing and what Haskell is doing but it's not a monad. It's plain old interfaces and parameters, just like Allocator.

SkiFire13 11mo ago

The function coloring problem actually comes up when you implement the async part using stackless coroutines (e.g. in Rust) or callbacks (e.g. in Javascript). Zig's new I/O does neither of those for now, so hence why it doesn't suffer from it, but at the same time it didn't "solve" the problem, it just sidestepped it by providing an implementation that has similar features but not exactly the same tradeoffs.

bloppe 11mo ago
How are the tradeoffs meaningfully different? Imagine that, instead of passing an `Io` object around, you just had to add an `async` keyword to the function, and that was simply syntactic sugar for an implied `Io` argument, and you could use an `await` keyword as syntactic sugar to pass whatever `Io` object the caller has to the callee. I don't see how that's not the exact same situation.
- VMG 11mo ago
  Maybe I have this wrong, but I believe the difference is that you can create an Io instance in a function that has none
  - bloppe 11mo ago
    In Rust, you can always create a new tokio runtime and use that to call an async function from a sync function. Ditto with Python: just create a new asyncio event loop and call `run`. That's actually exactly what an Io object in Zig is, but with a new name. Looking back at the original function coloring post [1], it says: > It is better. I will take async-await over bare callbacks or futures any day of the week. But we’re lying to ourselves if we think all of our troubles are gone. As soon as you start trying to write higher-order functions, or reuse code, you’re right back to realizing color is still there, bleeding all over your codebase. So if this is isomorphic to async/await, it does not "solve" the coloring problem as originally stated, but I'm starting to think it's not much of a problem at all. Some functions just have different signatures from other functions. It was only a huge problem for JavaScript because the ecosystem at large decided to change the type signatures of some giant portion of all functions at once, migrating from callbacks to async. [1]: https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ https://journal.stuffwithstuff.com/2015/02/01/what-color-is-...

eikenberry 11mo ago

Function coloring is specifically about requiring syntax for a function, eg. the async keyword. So if you want an async and non-async function you need to write both in code. If you pass the "coloring" as an argument you avoid the need for extra syntax and multiple function definitions and therefor the function has no color. You can solve this in various ways with various tradeoffs but as long as there is a single function (syntactically) is all that matters for coloring.

IshKebab 11mo ago
> Function coloring is specifically about requiring syntax for a function, eg. the async keyword. It isn't really. It's about having two classes of functions (async and sync), and not being able to await async functions from sync ones. It was originally about Javascript, where it is the case due to how the runtime works. In a sync function you can technically call an async one, but it returns a promise. There's no way to get the actual result before you return from your sync function. That isn't the case for all languages though. E.g. in Rust: https://docs.rs/futures/latest/futures/executor/fn.block_on.html https://docs.rs/futures/latest/futures/executor/fn.block_on.... I think maybe Python can do something similar but don't quote me on that. There's a closely related problem about making functions generic over synchronicity, which people try and solve with effects, monads, etc. Maybe people call that "function colouring" now, but that wasn't exactly the original meaning.
woodruffw 11mo ago
> Function coloring is specifically about requiring syntax for a function, eg. the async keyword. Someone should tell the inventor of the phrase, because they don't mention the async keyword at all[1]. As-written, function coloring is about callbacks (since that's semantic mechanism that JavaScript happens to pick for their asynchronous model). Function coloring is just an informal way to describe encoding a function's effect. You can encode that in syntax if you want (an `async` keyword), or in the type system (returning `() -> T` instead of `T`), or in the runtime itself (by controlling all I/O and treating it the same). But you can't avoid it. [1]: https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ https://journal.stuffwithstuff.com/2015/02/01/what-color-is-...

throwawaymaths 11mo ago

1) zig's io is not a viral effect type, you can in principle declare a global io variable and use it everywhere that any library calls for it. Not best practice for a library writer, but if you're building an app, do what you want. 2) There are two things here, there is function coloring and the function coloring problem. The function coloring problem is five things: https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ https://journal.stuffwithstuff.com/2015/02/01/what-color-is-... 1. Every function has a color. 2. The way you call a function depends on its color. 3. You can only call a red function from within another red function. 4. Red functions are more painful to call. 5. Some core library functions are red. You'll have some convincing to do that zig's plan satisfies 4. It's almost certain that it won't satisfy 5. It's open to debate if zig's plan will work at all, of course.

woodruffw 11mo ago
> 1) zig's io is not an effect type, you can in principle declare a global io variable and use it everywhere that any library calls for it. That's an effect, akin to globally intermediated I/O in a managed runtime. To make it intuitive: if you have a global token for I/O, does your concurrent program need to synchronize on it in order to operate soundly? Do programs that fail to obtain the token behave correctly?
- throwawaymaths 11mo ago
  how do you "fail to obtain the token"?
  - woodruffw 11mo ago
    The token guards a fallible resource (I/O). You can (temporarily or permanently) fail to obtain it for any reason that would affect the underlying I/O.
    - throwawaymaths 11mo ago
      the io isnt a single resource? it's a module grouping together a pile of code. and you can swap out implementations. the io modules should be responsible for handing out many failable resources, and synchronization is going to be up to the io module code, and thats whether or not it's globalized or passed.

dan-robertson 11mo ago

This solves a problem for library authors which is that blocking and event-based io implementations of functionality look the same but are not actually the same so users end up complaining when you do one but not the other. It adds a problem of needing to pass the global kind of io through a program. I think this mostly isn’t a huge problem because typical good program design has io on the periphery and so you don’t tend to need to pass this io object that ‘deep’. This is not too different from the type-system effect of IO in Haskell (except that one only does evented IO IIRC). It isn’t as bad because it only affects input types (data which can be closed over, I assume) rather than output types. Eg in Haskell you need various special functions to change from [ IO a ] to IO [ a ] but in the zig model you iterate over your list in the normal way using an io value from an outer scope. The one case where Io-colouring was annoying to me in Haskell was adding printf debugging (there is a function to cheat the type system for this). Zig may have other solutions to that, eg a global io value for blocking io in debug builds or some global logging system.

themk 11mo ago
There is nothing special about the [IO a] -> IO [a] in Haskell. You can iterate over it using the "normal" methods of iterating just fine. forM ios $ \io -> io But there are better ways to do it (e.g. sequence), but those are also not "special" to IO in any way. They are common abstractions usable by any Monad.
- dan-robertson 11mo ago
  Haskell is a bit tricky to talk about here because it has other big differences on laziness and suchlike, and this means there are pervasive monads. If you instead consider a language more like JavaScript where async functions return values wrapped in promises and therefore require special versions of lots of things like Array.prototype.forEach for async-returning functions (ok, language feature of generators helps here). The point I’m trying to get at is that putting io-ness into an argument works better than putting it into the return type because it is easier to pass an extra argument when using other language features an harder to do an extra thing with returned values.

esjeon 11mo ago

AFACT, the only practically critical issue of colored function is the duplication of code b/w sync and async code paths. Zig avoids this with dependency injection, and that’s enough for practical usages (which basically means “ergonomic”). Other points raised by the original article (like calling async function is more difficult) are pretty much unavoidable for the sake of precise control.