43 ms·

Red and blue functions are a good thing

• rurban 6y ago
  All this color nonsense is written by people who have no idea about their problems with blocking IO. It's about blocking vs non-blocking, and many of those write about their problems when they cannot leave the blocking hell. Or even their presumed non-blocking hell, as seen with node/javascript folks. Well, you won't get secure multitasking if you allow blocking. You won't get fast multitasking neither if you allow blocking. The whole POSIX API is doomed. "Red" and "blue" are of course a bad thing. 90% of the time the kernel is waiting for some blocking IO, and keeps switching around, with all its context switches. With pure event driven non-blocking IO you don't wait in the kernel, you wait at the user code, who can decide to do something else useful in that time. And the user decides that he will need better tools to do more useful things in parallel. There are tons of success stories of pure non-blocking IO. Once the happy manager puts blocking in, on peer pressure, posix layers and red-blue advocates the system goes down, and you are in deadlock hell, slowness and insecure concurrency.
• otikik 6y ago
  > Inconvenient knowledge is better than convenient ignorance For perfect ideal beings, yes. Humans are a bit less straightforward, even for the particular case of red-blue functions.
• linkdd 6y ago
  In Erlang/Elixir, every function from another module is async by design, and the await is implicit. This is by design to allow for hot code reloading. This is a very sound design choice because true concurrency is achieved through the creation of other processes (agents, genservers, tasks, ...). You have nice things like Dynamic Supervisors to monitor those processes. And it works very well. Try to do the same with uncancellable promises in Javascript/Typescript, or with channels in Go, while it's still possible, you'll find your code growing in complexity way quicker. The actor model embedded in the language, like Erlang/Elixir, brings so much to the table, it's unfortunate that the author dismiss it without a second thought.
  • whizzter 6y ago
    Actually the entire Erlang runtime is "async"(in the terms of the other mentioned languages) almost all the time since it's based on green threads/processes(stuff like a NIF might block of course), the module call is more of a lookup-current-latest-definition call to enable a well defined point for the hot-loading whereas local functions would more likely to be in the middle of a computation (and thus be more likely to be in a complicated-to-upgrade state). async/await being implemented and popularized as it is today everywhere might just be our generations null-mistake since they're so error prone, they're great to show off parallel IO performance increases for classically synchronous languages but the programmer-burden of explicit await's (luckily IDE's/compilers can catch them) with "weird" side-effects was a big mistake. The author of the article seems to be firmly in a specific category of developers so the dismissal seems quite in-character.
    • linkdd 6y ago
      > Actually the entire Erlang runtime is "async" I wasn't sure so I did not want to claim it, so thanks :) > async/await being implemented and popularized as it is today everywhere might just be our generations null-mistake Agreed, especially since each language implements it differently. For example, Javascript's async/await is just syntactic sugar over promises and aren't truly "colored" functions.
      • singularity2001 6y ago
        The verbosity, hassle and error-proneness makes async/await definitely worse than yesteryears 'goto' failure
    • kannanvijayan 6y ago
      If you squint closely enough, you can recognize features of the "async/await" discussion in other longstanding issues such as the "effectless/effectful" distinction and whether that should be brought up in syntax, or the "continuations/finite-exits" control transfer distinction (finite-exits being traditional languages which provide a fixed set of control trasnfer such as return/throw/break). The "all things are (posslby) async" is a language design choice similar to "all things are (possibly) effectful" for a procedural language. If you keep your language semantics in the less generalized domain (e.g. functional vs. effectful), you have to invent syntax to talk about the other semantics easily. So you come up with monad systems in functional languages to talk about effects, and make some nice syntax for that. And you have your effectful functions and effectless functions segregated by type signatures that flow all the way up to the top. And you come up with async/await systems in traditional sync-semantics languages, and provide syntax to help with that - whether Javascript or Rust. In both Rust and JS, async functions are distinguished purely in the type-system, with some helpful syntax around it. In both, they desugar to `func(...): Promise<T>` or `func(...) -> impl Future<T>`. Those types too, flow "all the way to the top", imposing a colouring on code. Erlang solved this problem by adopting the general semantics for erlang to achieve erlang's goals, and it seems that the "everything is async" semantics works there (I agree that it's a model that deserves deeper exploration). Languages like Rust can't solve this problem the way erlang solves it, because it takes away from other fundamental aspects of the language. It would necessitate the inclusion of a very specific and very heavy (relative to Rust's expectations) runtime, along with lots of changes to the language that would diminish it along other dimensions.
• tome 6y ago
  Interesting. This author took a different interpretation of the original red/blue functions article than I did. I understood the original to mean "the distinction between red and blue functions is a bad thing when not made visible in the type system". Perhaps that's my Haskell bias showing and I missed the original point.
  • inglor 6y ago
    Haskell has one of the _clearest_ distinctions between red/blue functions that I know. Functions that perform I/O (sans unsafePerformIO and other strange things) are very explicit about it (by returning an `IO` monad). Most other languages (static or not) are equally explicit (returning a `Promise<A>` and not an `IO a` but it's very similar. Most differences are because call-by-need vs. call-by-value language semantics (Haskell being non-strict and the I/O not coming out of "thin air" but coming from somewhere - but that's kind of a different point than function color type)
• skywhopper 6y ago
  Reasonable point of view about the topic, but it’s unfortunate that the author needs to go out of their way to make broad statements about how every programming language should be statically typed and explicit about async state. It reveals such a narrow view of what programming can and should be used for. Yes, these are great features for Rust to have. But other languages make other choices because not everyone has the same problem space. If you’re going to make assertions about how programming languages should work, you need to bound that assertion with the specific use cases and work parameters you are talking about. Otherwise your statement is too general to be meaningful much less true.
• bsder 6y ago
  I remain unconvinced that red/blue functions are a good thing in a statically compiled language. If anything, the amount that async/await spirals outward to other areas in Rust demonstrates that it doesn't seem to be a well contained feature. In addition, async/await executors in Rust quite often fail on corner cases. If I want to only wait on a socket, stuff works great because that's what everybody wants. If, however, I want to do something like wait on sockets and wait on a character device file descriptor ... that tends to break in fairly mysterious ways.
  • willtim 6y ago
    The problem is more that Rust cannot (currently) abstract over these different function types; and that they are more limited (e.g. no async methods in traits). But the idea of effects tracking is a good thing and we are bound to see more of it in modern statically-typed languages.
    • kevincox 6y ago
      This is partly true. Unlike JS (where the colouring argument came from) you can always just block, so while you mean be "stealing" an event dispatcher thread nothing bad will happen, you just have to deal with the fact that the functions you are calling may task some time to return (as the article argues). There is no true function colouring in Rust. (Much like you can unwrap an Option or Result to handle that type of colouring.) In Rust an `async` function is just a fancy way to write a function that returns a `std::future::Future. You can certainly write a trait with functions that return a `std::future::Future<Output=T>`. Although you will probably have to box them at the moment. https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=0f2a4f6734293899112c9e4f7062904e https://play.rust-lang.org/?version=stable&mode=debug&editio... I think this restriction is because they are trying to figure out how to handle dynamic vs static generics. Presumably they will add support for this in the future. The compiler also recommends https://docs.rs/async-trait https://docs.rs/async-trait which has a pretty clean solution.
• faitswulff 6y ago
  > Feel free to create and use languages with invisible program effects or asynchrony, and I'll feel free to not pay any attention when you do. As a new Rust user myself, this kind of snobbery is still off-putting.
  • lelanthran 6y ago
    When the author of a blog post introduces the post with: > A mature and serious language such as Rust About a language as new as Rust you can be sure that they're getting their grinding stone and axe ready for some serious use.
  • justin_oaks 6y ago
    Although the author talks about Rust, I interpreted that point as generic. Rust happens to have characteristics that the author likes, such as static typing and declared "exceptions" in the form of Results. I agree that languages with invisible program effects are harmful. The author said it another way as "Inconvenient knowledge is better than convenient ignorance". I've been bitten many times when using languages which have weak types and perform implicit type conversion. I much prefer to know exactly what type of variable I'm using and perform explicit type conversion instead of having the language "take care of it" for me.
• douglaswlance 6y ago
  Using purple for link decorations is a non-optimal choice.
  • coldtea 6y ago
    Huh? It has been the standard in the web for time immemorial (blue for links, a shade akin to purple for visited links). What would be the "optimal" choice?
    • hu3 6y ago
      The point is that the blog uses purple for both new and visited links: a, a:visited { color: #692fac; }
  • declnz 6y ago
    ...but it's coloured both red and blue! :)
• willtim 6y ago
  And why restrict oneself to just two colours? Haskell monads also allow one to abstract over the "colour", such that one can write polymorphic code that works for any colour, which I think was the main objection from the original red/blue post. Microsoft's Koka is an example of a language that further empraces effect types and makes them easier to use: https://koka-lang.github.io/koka/doc/index.html https://koka-lang.github.io/koka/doc/index.html
  • the_duke 6y ago
    The Unison language [1] also has a very interesting effect system. [1] https://github.com/unisonweb/unison https://github.com/unisonweb/unison
  • inglor 6y ago
    Basically, in JS you could have multiple colors (with generators that imitate `do` notation pretty well) and you can obviously implement any monadic structure with bind you'd want. The thing is: having less generic abstractions (and fewer colors) makes certain things (like debugging tooling and assumptions when reading code) much much nicer.
    • fbn79 6y ago
      Javascript must solve the fundamental problem that generators are (by implementation) not clonable. So generators can not be used generally as do notation.
  • benrbray 6y ago
    Fun Fact: The author of Koka (Daan Leijen) wrote a LaTeX-flavored Markdown editor called Madoko [1] using Koka! It is used to render the "Programming Z3" book [2], for instance. [1] http://madoko.org/reference.html http://madoko.org/reference.html [2] https://theory.stanford.edu/~nikolaj/programmingz3.html https://theory.stanford.edu/~nikolaj/programmingz3.html
  • zozbot234 6y ago
    Monads are not the same thing as effect types - the latter are most commonly modeled as Lawvere theories. They are at a different level of composability. Regardless, this blogpost is about Rust which has yet to gain either feature, for well-known reasons (for one thing, monadic bind notation interacts badly w/ the multiple closure types in Rust). Talking about Haskell monads just does not seem all that relevant in this context.
• dustinmoris 6y ago
  > Inconvenient knowledge is better than convenient ignorance Can't disagree with this statement more. I've worked with a coloured language for a very long time (C#) and also recently worked a lot with Go and both have huge advantages and disadvantages. Go is not conveniently ignorant. The green thread model which Go uses is actually a very elegant solution where one can write simple code without leaking abstractions down the stack as it happens to be with red and blue functions. It's not ignorance. It's focus! Only the function which deals with async code has to know and think about it. Calling code doesn't need to adapt itself to the inner workings of another function which it has no control of. I quite like it and don't see any particular problem or traps with it. Quite the opposite, as a very experienced C# developer I see a lot of less experienced developers do very fatal mistakes with async/await as it's so easy to write code which can end up in a deadlock or exhaust resources and run into weird issues which only a person with a lot of experience would be able to spot and fix. I think Go's model is much more tolerant and forgiving to beginner mistakes which means that writing fatal code tends to be a lot harder and almost impossible if not done intentionally.
  • willtim 6y ago
    > I quite like it and don't see any particular problem or traps with it. While Go's model is certainly a valid point in the design space and keeps things nice and simple, there are of course going to be many trade-offs. For example, low-level control is very much lost. There are no opportunities for custom schedulers, optimising how non-blocking calls are made, or easily dealing with blocking syscalls (or foreign libraries that call them). It would not be the right approach for Rust.
• quelltext 6y ago
  > Most in the tech world are now familiar with the post describing red/blue functions and their downsides, especially in the context of asynchrony. We are?
  • ghengeveld 6y ago
    Despite being in the industry for a decade and authoring a library for async operations, I had never heard of this before.
    • toopok4k3 6y ago
      What, you don't breathe this magical glitter dust in and out every day? I feel old reading these kind of blog posts that assume that we all know what the latest unicorn has been spewing out of its butt. It's like I'm missing out on the greatest debate. On dung. I'm sure it's valuable debate for those in the know. But this is just how it translates to some of us.
      • coldtea 6y ago
        This is not about some unicorn company, or some far-fetched concept. Async/await has been in the discussion for most popular languages (C# and JS first for the masses, Python, Rust, Java, and others) for a decade now. And the colored functions analogy was a very popular post (and was subsequently referenced and discussed in lots of others). Besides, HN is not some website about just knowing the core concepts used in the trade. There's Hacker Noon and others for that. HN has always been all about these kind of intra-group discussions.
        • saurik 6y ago
          Right: there is a decade of discussion about this one narrow instantiation of a feature that is itself a mere part the past fifty years of discussion of the general class of thing it is from--coroutines--and an infinite amount of academic material on both that concept and monads--which is the correct mental abstraction to have here--and so the idea that I must have read this one random recent blog post (one which I honestly had heard about it in passing, but without any belief that it was important to anyone or that there was even one specific one that mattered) that was written by someone who probably, ironically, hasn't read all of those actually-seminal papers is more than a bit frustrating. (edit) Really, everyone should probably start by reading the 1978 papers "on the duality of operating system structures" and "communicating sequential processes", which, when combined with the appropriate background in the continuation passing style compiler transform (which I am pretty sure was also developed around the same time) could have hopefully prevented Ryan Dahl from accidentally causing the callback-hell "dark age" of software engineering from ever happening in the first place ;P. https://cs.oberlin.edu/~ctaylor/classes/341F2012/lauer78.pdf https://cs.oberlin.edu/~ctaylor/classes/341F2012/lauer78.pdf https://www.cs.cmu.edu/~crary/819-f09/Hoare78.pdf https://www.cs.cmu.edu/~crary/819-f09/Hoare78.pdf
• hnedeotes 6y ago
  For someone complaining about sloppiness... Don't know about go, but in elixir "We have to use Task.await", is like, you can use it if you want exactly those semantics, just because it has await and task in its name doesn't mean you have to - although simple in usage it's quite a higher level abstraction inside the BEAM - the base unit is the "process", and you can make any sort of asynchronous or synchronous flow on top of that. But even with Tasks, they can be called inside other processes and then instead of having to wait for all tasks to finish they'll be delivered one by one to that process mail box and you can decide at all points of execution what to do. Of course if you're running your code like a sequential block... You'll have to wait until they sequentially finish?
• bluetomcat 6y ago
  You never have such issues in well-designed programs with a main event loop where all possible events are waited on. Your program is either idle, or waken up by a set of events (any kind of IO, a timeout, or a signal from another thread). You handle all events by dispatching them in a timely manner to the responsible code, and wait again. Of course you need non-blocking APIs or dedicated threads for the IO and compute-intensive stuff. The whole async/await thing is just a convenience feature that looks great but distracts you from thinking about the true concurrency of your program.
  • zozbot234 6y ago
    Async await notation is for write "dispatch" based program you describe, as straight line code. Is convenience feature, but like syntactic sugar only.
  • chriswarbo 6y ago
    > well-designed programs with a main event loop This is a massive over-generalisation. In particular, you're making a huge assumption that looping is possible, leaping from there to having loops be desirable, then pole-vaulting to claim that anything not architected entirely around looping is "not well-designed". Based on this generalisation, I'm going to guess your experience is limited to languages which (a) have some sort of looping construct built-in (e.g. a `for` or `while` keyword), (b) do not eliminate tail-calls and (c) either don't support, or discourage (perhaps until recently) higher-order programming. This would include languages like C/C++/Go, Python/PHP/Perl, Java/C#, etc. In contrast, I consider looping to be a problematic, error-prone construct in general (for reasons outlined at http://lambda-the-ultimate.org/node/4683#comment-74302 http://lambda-the-ultimate.org/node/4683#comment-74302 ). Recursion is much safer, in languages which eliminate tail-calls. I also consider "main loops" to be an anti-pattern (for reasons outlined at http://disq.us/p/kk1da6 http://disq.us/p/kk1da6 ). They break all sorts of modularity/encapsulation; force bizzare control flow patterns (e.g. lots of highly non-local 'returns', AKA unlabelled GOTOs); force us to make arbitrary (yet highly impactful) decisions between 'carrying on' or 'trampolining off main' (these decisions are often hard to change after-the-fact, regardless of their suitability); require lots of boilerplate to marshal and manage intermediate values which, by their nature, must conflate all sorts temporary values from the entire codebase into amalgamated blobs; and so on. Rather than shoehorning important logic and decisions into a global, manually-kludged tail-call trampoline (AKA a 'main loop'); I find it far easier to write, reason-about and debug simple code which performs a 'step' then tail-calls to the next. As far as concurrency goes, I would claim that "well-designed programs" with control following data flow, it's usually pretty trivial to spin off parts as Futures/etc. as needed. A type system will tell us where to place the necessary calls to "map", etc. as needed.
• dukoid 6y ago
  I wish we had this for purity, too. But I guess it gets hard to define when handing in structured arguments...?
• slver 6y ago
  This article was a whole lot of "this pain you feel, cherish it, it's good". It embraces a defect introduced for BC reasons as if it's sound engineering. It really isn't. If you start over, there's no reason any blocking API should really BLOCK your program. Your functions can look completely synchronous, but yield to the event loop when they invoke a blocking API. Basically like Erlang works. Eventually it will be the case, but these kind of "love the flaw" attitudes are slowing us down.
  • rcxdude 6y ago
    This can work, but it's a fundamentally quite different async model and you give up a lot of the benefits because you have stackful coroutines, which are a lot more heavyweight than the state machines which rust's async functions compile down to. There's nothing stopping you implementing this concept in rust today (a la gevent in python), it's just a seperate thing (and you still need to be careful about accidentlly calling something which does block).
  • zozbot234 6y ago
    > Your functions can look completely synchronous, but yield to the event loop when they invoke a blocking API. This require either runtime-based or operating system support. If need be, you can always rely on the OS scheduler to be your "async blocking API".
    • jhgb 6y ago
      > If need be, you can always rely on the OS scheduler to be your "async blocking API". Well, the original red-blue function article did say "just use threads", didn't it?
      • zozbot234 6y ago
        IIRC the original article was talking about JavaScript and there's no threading in JS.
        • 6y ago
• coldtea 6y ago
  >Feel free to create and use languages with invisible program effects or asynchrony, and I'll feel free to not pay any attention when you do. As if you matter? The conclusion could be stripped, and the article would still make its point.
  • q3k 6y ago
    The snobbishness around Rust repeatedly makes me reconsider if I really should be investing even more time to continue learning the language.
    • whizzter 6y ago
      The core innovations were brilliant and it does seem to attract a certain kind of developers yes, that said there seems to be quite a few awesome people in the community as well so I guess the strategy is to use it where it's the best tool and not be too invested in the rest.
    • deleted 6y ago
      [deleted]
    • justin_oaks 6y ago
      I'm wondering if people are looking for Rust snobbery and finding it where it doesn't exist. This is not a dig at who I'm responding to, although the comment did trigger the thought. I interpreted the points in the article as fairly generic. The idea of "Inconvenient knowledge is better than convenient ignorance" goes well beyond Rust. The author of the article does imply that Rust does a lot of things right, but I'm curious how one could say "Hey, I like this language it does a lot of things well and fixes problems I see in other languages" without sounding like a snob.
      • q3k 6y ago
        > I'm curious how one could say "Hey, I like this language it does a lot of things well and fixes problems I see in other languages" without sounding like a snob. Just not writing “Feel free to create and use languages with invisible program effects or asynchrony, and I'll feel free to not pay any attention when you do.” would do the trick in this case. Or, generally, show your reasoning, show your arguments, acknowledge any tradeoffs, contextualize against other languages as needed. And perhaps do not jump into every programming language discussion with “this wouldn't be an issue with Rust” unprompted, as that's generally not constructive, and does not acknowledge that there is more to software engineering than just what programming language you pick.
• ztcfegzgf 6y ago
  you can make sync IO calls in Rust if i'm not mistaken. be it reading a file or doing a remote HTTP request. and with this being true, the article's argument does not work, there is no types-mark-io happening there. the argument applies better to Haskell, where you cannot do any IO without the types saying so. interestingly, it also applies somewhat to Javascript, where because of backward-compat and history, it's not possible to do sync IO (mostly :) welcome to javascript), so if you do not see an "async" annotation, you can be mostly sure that this is pure cpu-work.
• Skinney 6y ago
  While I agree with his points in principle, Rust isn't actually a great example to use. Rust allows to simply spawn threads, right? So a function not being marked 'async' doesn't necessarily give any guarantees about potential side effects. Spending most of my time writing Elm, having side-effects present in the type system is great. But colored functions in Rust (as well as most other async/await langs) only creates a devide between functions that provide a specific kind of asynchrony, and those which do not. So async/await improves the syntax of writing a specific kind of async code, at the cost of composability (the red/blue divide). It does not, however, make any guarantees about the asynchronous nature of your code.
• kaba0 6y ago
  Rust pretty much had to add function coloring, because of its low-level nature. Other languages can get away with it due to having a runtime, notably Go, and with the coming Project Loom, Java. For higher level languages, I do believe that coloring is not a useful concept, similarly to how you use a GC, instead of manual memory management. The runtime can “hijack” calls to IO or other blocking functions and can park a virtual thread, and resume execution afterwards automatically. There is no useful information in coloring in this case.
  • pas 6y ago
    Python has a runtime too, but it still did async in a half-assed way :/
    • kaba0 6y ago
      Well, Python is very dependent on C FFI, which may make it harder.
      • _old_dude_ 6y ago
        It's only an issue when you have a C callback that calls a Python code, because you have C code in the middle of your stack thus you can not move the stack frames back and forth (because the C code can reference stack addresses). I may be wrong but most of the C calls in Python do not have callbacks and you can pin the coroutine to the kernel thread if there is a callback.
  • coder543 6y ago
    Zig is just as low level as Rust, AFAICT, and it appears to have avoided this: https://kristoff.it/blog/zig-colorblind-async-await/ https://kristoff.it/blog/zig-colorblind-async-await/
• kbd 6y ago
  I really can't agree with this premise after seeing how Zig implements color-less async/await: https://kristoff.it/blog/zig-colorblind-async-await/ https://kristoff.it/blog/zig-colorblind-async-await/ I highly recommend watching Andrew Kelley's video (linked in that article) on this topic: https://youtu.be/zeLToGnjIUM https://youtu.be/zeLToGnjIUM
  • whizzter 6y ago
    Zig seems to be doing a lot of things right (just like Rust did). Sadly I think that their disregard for safety (the manual states that dangling pointers is the developers problem) kinda makes it a no-starter for many people. I personally consider possible information breach far worse than a crash or runtime stall.
    • kbd 6y ago
      > disregard for safety Your opinion seems to be that any future systems language that doesn't implement a heavy Rust-style borrow checker and explicit safe/unsafe modes "disregards safety"? Zig does have a lot of checks, such as bounds checking[1]. There are also test modes that help catch errors. I don't know what you're referring to about "information breach". > The manual states that dangling pointers is the developers problem... In a systems language where you can cast an int to a pointer: const ptr = @intToPtr(*i32, 0xdeadbee0); or where you have to manually manage memory, what else do you expect? Zig made the design choice to not split the world into safe vs unsafe. It seems a bit unwarranted to say that because they didn't make the same design choices Rust did that they have a "disregard for safety". [1] https://ziglang.org/documentation/master/#Index-out-of-Bounds https://ziglang.org/documentation/master/#Index-out-of-Bound...
      • whizzter 6y ago
        Out of bounds check is definitely a responsible start. As for the exact methods I'm not really partial to anything (Be it borrow-checker,gc,compiler-time-analysis,etc) but we've seen time and time out of bounds(handled by Zig atleast), UAF and other issues be exploited so having "safe" defaults for the majority of code isn't something I think we should skip on, especially if it's a "systems language" since the exploitation surface will end up everywhere. The Rust borrow checker can probably feel cludgy at times, and in many ways it's a result of creating a scheme that is verifiasble. Since Zig already has comptime I guess having expand compile time capabilties to analyze for common UAF conditions,etc shouldn't be unfeasible?
• avodonosov 6y ago
  If the author likes to see explicit `async` label on functions, he misses the pleasure of having `async` on every function. a = x + z; b = y + x; That's concurrency laying on the table - refactor your code to run those two lines in parallel.
  • TheCoelacanth 6y ago
    That's an entirely different kind of concurrency. Those statements are just going to run on the CPU. It's only faster to do them in parallel if you have an extra CPU core lying around and if the overhead of splitting it across CPUs doesn't outweigh the time saving of doing both at once (it almost never does for such a small piece of work). With async functions, the CPU isn't actually doing anything. It's just waiting for something to happen, like a network response from another machine. It's almost always faster to do them in parallel if possible.
    • avodonosov 6y ago
      > for such a small piece of work Don't take the example too literally, some function calls can be here. Running computations in parallel is often valuable. Or run computations in parallel with waiting for external resource - why does not the code in the article compute something while waiting for a, b and c? Anyways, if async functions are so good, why not have all functions async? The article says this a kind of "documentation" that tells you what functions can wait for some external data and what functions are "pure computation". If it was so, it would be OK. Such a documentation could be computed automatically based on the called function implementations and developer is hinted: "these two functions you call are both async, consider waiting for both in parallel". In reality, the async / await implementations prevent the non-async functions from becoming async without the calning code change. This restriction is just a limitation of how async / await is implemented, not something useful. As other commenter says, the article "embraces a defect introduced for BC reasons as if it's sound engineering. It really isn't." When my code is called by a 3rd party library, I can not change my code to async. That's the most unpleasant property of today's async / await. What yesterday was quick computation tomorrow can become a network call. For example, I may want the bodies of rarely used functions to only load when called first time (https://github.com/avodonosov/pocl https://github.com/avodonosov/pocl). The article suggests we have to decide upfront, at the top-level of the application / call stack, which parts can be implemented with as waiting blocks and which should never wait for anything external. This is not practical. > It's almost always faster to do them in parallel if possible. Yes, if possible. And it is determined, in particular, by dependencies between the operations. If the first call is writing to DB and the second call is a query that reads the same DB cell, they can't be parallelized. The same is with in memory computations - if next steps do not depend on previous steps they can be parallelized. Hopefully, compilers of the future will be able to automatically parallelize computations.
• kristoff_it 6y ago
  In Zig allocators are always passed around explicitly and yet we have a completely opposite stance on function coloring [1]. In my opinion the argument doesn't really apply to async await when you get down to the details, but I need to think about it more to be able to articulate why I think so with precision. [1] https://kristoff.it/blog/zig-colorblind-async-await/ https://kristoff.it/blog/zig-colorblind-async-await/
  • losvedir 6y ago
    It's really interesting that Zig seems to have managed to avoid the "color" distinction for async. I'm curious to see how it works out in the end (and am a happy sponsor, to help that out!). However, and maybe this is what you're alluding to with your mention of allocators, it seems that Zig has "colored" functions in its own way, with regard to allocation! If a certain function allocates, then to use it the caller will need to pass in an allocator, and that caller will need to have been given an allocator, etc. In other words, explicit allocation "infects" the codebase, too, right? I can't help but see the parallels there. I think Zig's stance on explicit allocation is one of its strengths, but it does make me wonder if some compiler trickery can make it colorblind as well.
    • AndyKelley 6y ago
      I can see why you would think that but if you try out some Zig coding you will quickly realize there is no parallel here. It just doesn't make sense to try to compare them. I'm at a loss as to how even to explain it.
      • dnautics 6y ago
        Red/blue means that in one direction a seam is impossible to create. That's not generally the case for allocators in zig.
  • dnautics 6y ago
    It's not true that in zig allocators are always passed around explicitly. You can: - declare a global allocator for your program and call it implicitly everywhere. This is very accepted by the community as bad practice for libraries but if you are writing end product who cares. In the end in most cases you will have to bootstrap your allocator against a global allocator anyways, and usually that in some form or another std.heap.page_allocator. - bind an allocator into your struct when you build it. This makes the allocator "implicit" instead of explicit. The important criteria about red/blue functions that a lot of people forget about is that one of the seams between the two colors is missing. As far as I can tell, for all of "red"/"blue" ish things in zig, (async, allocators, errors) there are clear and documented ways of annealing those seams in your code.
• diragon 6y ago
  First it was "Rust's async isn't f#@king colored!" (https://www.hobofan.com/blog/2021-03-10-rust-async-colored/ https://www.hobofan.com/blog/2021-03-10-rust-async-colored/) Then "Rust async is colored, and that’s not a big deal" https://morestina.net/blog/1686/rust-async-is-colored https://morestina.net/blog/1686/rust-async-is-colored And now it's a good thing. Honestly, this sounds like somebody struck Achilles' heel of Rust and it hurts. They're neatly covering 3 stages of grief over these 3 articles, and I wouldn't be surprised if we could uncover the other 2 in the blogosphere somewhere.
  • skohan 6y ago
    I actually have found all the angst and consternation around the "coloring" of Rust's async approach to be rather funny, because there are heaps of concepts in Rust which introduce something analogous to function coloring. A good example would be lifetimes. Once you introduce lifetime parameters into a data type, it starts to introduce ballooning complexity into everything that touches it.
    • orra 6y ago
      Is that problematic in practice? Write functions that are pure or take references (not ownership), and for the rest, use a judicious .copy()?
      • skohan 6y ago
        Yeah sure it's problematic. For instance, if you have a deeply nested struct, and you have to add a lifetime somewhere in the bottom, now you have to propagate it all the way up to the top level. Also if you run into a case where you need an explicit lifetime in a function somewhere, this ends up propagating outward. I've run into cases with Rust where I refactored a significant portion of a program to avoid nested references and explicit lifetimes in a way I haven't had to in other languages
        • fauigerzigerk 6y ago
          Yes, but you only have three choices basically: a) Accept the performance hit that comes with GC or defensive copying. b) Convince yourself that developers can keep all those potentially dangling references in their heads at all times. c) Use something like Rust lifetimes. My preference would be (a) whenever possible, (c) when absolutely necessary, and never (b)
• Sebb767 6y ago
  One point I massively miss from this discussion is retrofitting. Imagine the following situation: I have a webserver which calls a parser module which calls an external library which calls a closure in my code during parsing. Now this call needs to write to a log, which is async due to a file write or so. To implement this I need to touch the webserver core, implement a breaking change in the parser library and get a new external library, just to get my log line in this function. This is a massive refactoring effort just to get a line in my log. It's purer and better style, yes. But now I need a few days for what should take five minutes. If the codebase is designed from the ground-up with async in mind (which probably means to make basically every non-leaf async, to avoid the situation above), this is fine. If I'm working with an existing codebase, this does not work. I see why people don't want a fallback - it leads to bad code in cover. But allowing this and simply marking it via linter would make step-wise upgrading so much easier and alleviate 90% of the problems. EDIT: My concerns are specific about JavaScript/TypeScript. I think Rust got this right.
  • BenoitP 6y ago
    One of the best retrofitting effort there is is Java's Loom IMHO. Instead of trying to solve the IO bottleneck on a few threads with async, it just reduces the Thread footprint and allows you to have millions of them. Good'ol Tomcat is going to massively improve its IO concurrency overnight with just a simple `Thread.builder().virtual().factory()`.
  • zozbot234 6y ago
    > To implement this I need to touch the webserver core, implement a breaking change in the parser library and get a new external library, just to get my log line in this function. This is a massive refactoring effort just to get a line in my log. Not quite. It's very easy in Rust to purposefully block on an async call in sync-only code, or spawn blocking code on a side thread in async code and let it "join" whenever the async code block gets executed. Of course you'll get less-than-optimal performance, but no program-wide refactoring is required.
    • Sebb767 6y ago
• wrlkjh 6y ago
  This post's use of the term 'colored' is insensitive to the thousands of "colored" folks that were beaten, raped and tortured in the United States. Please change your terms and if you don't we will out you as a racist and you will lose your job.
• musingsole 6y ago
  So...are we going to just ignore the massive elephant in the room with people of a certain skin pigmentation pointing at anything and declaring it "colored"?
  • saurik 6y ago
    Yeah... I find this entire conversation pretty unpleasant--to the point where I haven't really wanted to ever interact with it directly, despite having tons of opinions on async/await due to my usage of either that specific feature or more generic related constructs (coroutines, monads) in many languages now (just not Rust)--as it isn't even like "colored" is being used in some happy way: things are always apparently somehow bad because they are "colored" :/, and I pretty much never hear the word "colored" used to talk about anything other than people (and, occasionally, functions). I realize a bunch of people are going to jump on me and say "I don't mean that", and I am not claiming you are: it just causes this continual looming mental state, as the word itself is so horribly weighted now.
    • q3k 6y ago
      Perhaps this discussion could be a opportunity to use the wording in a neutral fashion, letting yourself be okay with consciously and honestly using it, fighting against that looming feeling of unease? That is, if your issue with this wording is only that it's subjectively, personally unpleasant to you for inner reasons, and not that you see it as actively harmful.
      • saurik 6y ago
        The issue I noted is that most of the discussion surrounds an assumption--one that I happen to hold neither for people nor functions--that "colored" things are inherently bad, and that we have to defend the lack of badness of colored things... if you can't see why having to sit inside of that terminology cluster would make someone who separately has to spend a lot of time in arguments about race relations uneasy, I fear for your empathy :(.
        • deleted 6y ago
• one-more-minute 6y ago
  Being explicit about side effects in the type system isn't a bad idea, but don't conflate that with blocking/non-blocking operations. Some side effects aren't async (eg console.log in JS, mutating shared state) and some pure functions are (eg GPU linear algebra operations). Effect systems (as in Koka) are mentioned elsewhere; arguably their major innovation is being explicit about effects without sacrificing composability. Functions are only "coloured" by the type system, rather than syntactically, and can be coloured differently depending on context. This is quite different from the async/await situation.
• hn_throwaway_99 6y ago
  In this "async/await vs. threads" debate (which is primarily what this debate is about), I'm very interested to see what happens with Project Loom in Java. The primary downside with threads is really the thread-per-connection model, and how that means long-lived connections can prevent a server from scaling. If you can solve that problem, the appeal of single threaded languages with an event loop like Node really takes a hit.
• iovrthoughtthis 6y ago
  text based languages have two options for providing meta data to declared constructs. 1. syntax (explicit) 2. context (implicit) i cant wait until we have some other dimensions to play with. bring me the weird text interface to a rdf graph / syntax tree database plz.
• clktmr 6y ago
  The example screams "Premature Optimization". What if fetch_*() weren't async function (e.g. because doing CPU heavy work)? The Rust Code would look just like the Golang code, except you'll now need the wrapper code do make them async.
• zaphar 6y ago
  The colors are important if you are going to do async. They tell you significant details about the functions you are using. The issues most people have with colored functions aren't that they are have colors. It's that the colors don't compose. The languages that deal the best with this pick a single color and commit to it. Go and any BEAM language make every function async effectively and handle the coordination for you. They succeed because their runtimes will do a better job than you will 99% of the time. For the rest they give you some tools to reason about it. Rust however doesn't come with it's own runtime it's meant for the kind of development where you are building your own runtime or bringing one. For that reason the colors (i.e. types) are surfacing all of that complexity for you and helping you handle it. The bigger problem as I see it is that the rush for async has left a lot of crates choosing either sync or async only and so finding the crate that for your particular use case can be hit or miss. Half the db libraries may be sync only and half of the http libraries may be async only and then you have to figure out how to marry the two. If you can live in one or the other world only you are fine. It's when you have to live in both simultaneously that things get hairy.
• kardianos 6y ago
  On one hand I agree that being explicit about what is going on is important. That is why in Go all code is written the same, no matter if it is called async or sync. Then the caller gets to decide how it should be invoked. So yes, let's be explicit and separate out the concerns. Thus red/green functions are not as good as single colored language. Good premise. Your conclusion doesn't follow.
  • azth 6y ago
    golang is a colored language since "context.Context" needs to be passed to all asynchronous calls. This is explained very well in the Zig talk: https://kristoff.it/blog/zig-colorblind-async-await/ https://kristoff.it/blog/zig-colorblind-async-await/
• pron 6y ago
  There are two problems with this: 1. Languages with "coloured functions" usually don't inform you of effects (except for Haskell and languages like it). They allow expressing equivalent semantics with the other colour, which doesn't distinguish effects. 2. Expressing anything in the type system always has a cost. That it can express something doesn't mean that it should. The only way to know whether expressing this difference -- assuming you could (which, again, most languages with coloured functions don't do) -- is to empirically show the benefits outweigh the cost, not just hypothesise that they do, which just amounts to "I like it this way." "I like it" is perfectly fine, but it's not the same as "it's a good thing."
• mlatu 6y ago
  Previously on HN: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ https://vorpus.org/blog/notes-on-structured-concurrency-or-g... This author claims there shouldn't be red and blue functions at all, instead concurrent paths should be handled by a Nursery object, so that no background tasks can ever be dropped by the runtime, but always return to the nursery.
• clarge1120 6y ago
  What I learned from this article was: Go is all asynchronous. Problem solved.
• jerf 6y ago
  I half agree and half disagree. Having "colored functions" can be a useful design feature in a language, and indeed, perhaps ought to be spread further than they are. However, those colors should arise from fundamental differences, and be chosen deliberately, not from accidental considerations chosen by expediency. This post elides the idea of coloring functions for effect vs. coloring them for asynchronousness. Those are not the same thing! Having coloration of your functions based on their synchronousness is an accident, either because of optimization choices (like Rust) or because your language was previously incapable of the new color and it just wasn't practical to simply put it into the language (Javascript). It is not necessary for the synchronousness of a function to be in the color, as proved by the languages where it isn't, and having used both I'm not convinced it's a good reason for a function to have a color. You get better code in languages that lack that color distinction, even if perhaps at a slight runtime cost. Having it based on "effects" is probably a good idea. But even in Haskell, the current reigning champion of a usable language that cares about effects (as opposed to Idris or whatever Coq renames itself to, which are interesting but not really "usable" in a software engineering sense)... asynchronousness is not an effect! You can use asynchronousness in whatever pure computation you like. There are things that are treated like effects that are related to concurrency like STM, but those are built on top of the asynchronousness that is not related to the effects machinery. (What is actually the "effect" in STM is the updating of the Transactional Memory, not whether or not you ran things concurrently or in parallel to develop your pure commit.) Concurrency, on its own, is not an effect. You may concurrently try to run multiple effects at once, and create new levels of problems with effects, but that's not the creation of the concurrency itself. Whether or not a given bit of code has "effects" is unrelated to whether or not you run it "concurrently". (Given the usual simplifying assumptions we make about not running out of memory, not having CPU starvation, etc., which are generally valid because we also simplify those away for non-concurrent code too.) So, coloration is probably a good thing in programming, but "spending" your coloration on the accidental detail of explaining to the compiler/runtime exactly how to schedule your code is not a good place to spend it. Compilers/runtimes ought to be able to work that out on their own without you having to specify it by hand all over the place.
• ItsMonkk 6y ago
  The link to hexagonal architecture at the bottom is an important one. Colored functions are a good thing. In fact, just splitting based on red/blue for sync/async doesn't go far enough. You also need to split based on pure/impure. Many times async are impure, but the reverse(sync being pure) is not at all guarenteed and the distinction is there. The colored pain point that the original topic pointed out IS a good thing, if you don't have a direct dependency between the code that ran before the await and after the await, they should not be nested into the same layer. The thing that I think I've come to understand is that the typical call stack has way to many layers. Hexagonal Architecture together tracking async/sync and pure/impure leads to a pipeline of many functions at the top of the module, and this leads to better code.
• cdaringe 6y ago
  I disagree, slightly. The association with "effect" and "async" is erroneous when discussing the topic at this level. Is a heap-read an effect? Is an L2 cache write an effect? Of course they are, but the machine abstracts these from us. We consider effects like `fetch(...)` an effect because of its associativity of remoteness and propensity to fail or change, but that's the incorrect lens. Instead, the lens should be purity, including purity of passed routines. If the network was rock solid (like mem reads) and we read a fixed document, suddenly that effect wouldn't feel like an effect. The color of the fn would go back to whichever is the pure form, and all would be well. We deem some effects effects because of programming constructs and performance, and other effects we don't even consider because we trust the underlying implementation of the machine. Consequently, I fundamentally don't fully align with the article's claim. While I do generally agree that the color information is communicative, I think function names and input/output types are responsible for that type communication. When OCaml effects land in 5.x, I look forward to using the effect keyword to achieve this goal. For better or worse, I plan to use it for much more as well--providing general implementations for _most things_, simply s.t. providing test doubles in integration tests becomes a piece of cake, with zero need for fancy test provisions (module mocking, DI, ...<insert usual suspects>)
• Jakobeha 6y ago
  My take on this: Red and blue functions, and async / await, are a good thing when implemented well. It's much better than implicit every function can be async (e.g. python's gevent). Because it 1) makes it clear what is and isn't async, and 2) lets the programmer convert async to a callback when necessary. The real issue is that some languages have bad implementations of async / await. Specifically Rust, where afaik traits can't have async and that makes using async really convoluted. And most languages don't have async-polymorphism, and maybe other features I can't think of, which ultimately create the problem "I have to call an async function in a sync function without blocking". With a good implementation of async / await and half-decent architecture you shouldn't have that problem.
  • pornel 6y ago
    The "traits can't have async" thing you're referring is at best a minor inconvenience or tiny optimization left on the table. This feature was intentionally postponed. Traits can use async code, just not without boxing/dynamic dispatch. Given that other languages always box all their Promises/Tasks, Rust needing to do it in a couple of cases doesn't even put it at a disadvantage. To use async code in Rust you don't need to define any traits. For users of async traits the issue is invisible. For library authors that want their code to be both async and higher-order at the same time, the problem boils down to needing to add a macro to "polyfill" the feature for now, until "native" support comes later.
• afarrell 6y ago
  Here is one benefit I imagine to naming the functions according to colors: expanding the Working Memory you have available to reason about code. The brain’s working memory hardware has a few pieces and among them are the visuospatial sketchpad and phonological loop [1]. Code is language and when you hold parts of a codebase in your head, it occupies space in the phonological loop. By making some data about code chromatic/visual, we can hold that in our heads more easily. [1] https://en.m.wikipedia.org/wiki/Working_memory https://en.m.wikipedia.org/wiki/Working_memory
• gsmecher 6y ago
  This is a good time to mention a Python experiment [1] that "covers up" Python 3.x's coloured asyncio with a synchronous wrapper that makes it more user-friendly. It's been posted here before [2]. I've pushed ahead with it professionally, and I think it's actually held up much better than I expected. This idiom worked well in Python 2.x with Tornado. Python 3's batteries-included async unfortunately broke it partway (event loops are not reentrant [3], deliberately). However, in this Rust post's "turn-that-frown-upside-down" spirit, the Python 3.x port of tworoutines enforces a more deliberate coding style since it doesn't allow you to remain ignorant of asynchronous calls within an asynchronous context. This (arguably) mandates better code. I don't claim Python is superior to Rust, or that hiding red/blue functions is a great idea everywhere. I did want to point out that dynamic languages allow some interesting alternatives to compiled languages. This is worth musing over here because of the author's stated bias for compiled languages (I think I agree), and his claim that red/blue unification would only be enabled by sophisticated compilers (not necessarily). [1]: http://threespeedlogic.com/python-tworoutines.html http://threespeedlogic.com/python-tworoutines.html [2]: https://news.ycombinator.com/item?id=18516121 https://news.ycombinator.com/item?id=18516121 [3]: https://bugs.python.org/issue22239 https://bugs.python.org/issue22239
• cube2222 6y ago
  Being used to Go, I must say the lack of colorless, always implicitly async functions everywhere is probably the main thing I miss whenever I write in a different language. It really is liberating, when you can write plain, straightforward, imperative, blocking code, and spawn almost-free async tasks wherever you might want to. Having experienced a plethora of more-or-less functional monadic approaches, python with its GIL, Rusts async approach (for which I understand the trade-offs were necessary) it makes life so much easier. People bash Go for staying in the Stone age with respect to some features, but this is why I always really feel like going back to the stone age when writing in other programming languages.
• yongjik 6y ago
  Maybe it's me, but I don't see the benefits of async functions that much. The one big advantage vs. threads is that you don't have to worry about locks if everything is single-threaded, but if you have multiple levels of async functions then you still have to worry about variables changing between them. Another advantage would be that it's much cheaper to keep a million callbacks in memory than a million thread stacks, which is true, but unless your job is writing a load balancer or DB engine, few people need that much concurrency. I've seen production services that take several concurrent requests on average. I feel that old-fashioned thread pools can be a more straightforward and manageable solution for most use cases.
  • pornel 6y ago
    In Rust I get two nice things: • Instant, reliable cancellation. I can drop a Future at any time, and it won't do any more work, and cleans everything up. Racing multiple things is easy. • Ability to mix network-bound and CPU-bound tasks, and have both resources fully utilized at the same time. With threads if I sized threadpool for CPU cores, it'd be underutilizing CPU when doing I/O. If I sized it for connections, it'd hammer the CPU. An async scheduler has CPU threadpool + any number of connections it wants.
  • estebank 6y ago
    > unless your job is writing a load balancer or DB engine, few people need that much concurrency. I've seen production services that take several concurrent requests on average. But Rust is aimed at exactly that space and embedded, so it needs a way to express this kind of code in an efficient way. It is true that you can go use threads instead and it might be easier to get something working, but if async/await already exists, you might as well take advantage of it. This is why you will see libraries that don't need to be async, written using async.
• rosstex 6y ago
  >By avoiding effect aware functions a language hobbles engineers and makes programs sloppier than they could be. This is a nice garden path sentence.
• ziml77 6y ago
  I hate that my first reaction to seeing that examples are in Rust was "No! People are going to take this as Rust evangelism and disagree for that reason alone!"
• cryptonector 6y ago
  You can have convenience, or you can have fast. Pick one. Convenience in this context == threads, which means you're not compressing program state very much, which means your program will have a larger memory and cache footprint. Fast in this context == compressing state by expressing it explicitly rather than implicitly in the call stack. You can compress the most by writing directly in CPS style. You can get the most convenience by using threads. Everything else is somewhere in the middle. Now, choose your poison.