7 ms·

Show HN: Rill – Composable concurrency toolkit for Go

• pajeetz 2y ago
  what sort of environment do you need to be in to have to compose concurrency like this instead of relying on native go's scaling?
  • dougbarrett 2y ago
    Batching is a pattern I’ve had to manually build in the past to push large amounts of analytic data to a database. I’d push individual events to be logged, map reduce those in batches and then perform insert on duplicate update queries on the database, otherwise the threshold of incoming events was enough to saturate the connection pool making the app inoperable. Even optimizing to where if an app instance new it ran the inert on update for a specific unique index by storing that in a hash map and only running updates from there on out to increase the count of occurrences of that event was enough to find significant performance gains as well.
  • jerf 2y ago
    The same sort of environment in which one uses such abstractions like "functions" instead of relying on the language's native ability to run sequential instructions. It's generally good for languages to provide relatively low-level functionality and let libraries be able to build on top of it, because as the programming language development world has now learned many times over, the hardest code to change is the code in the language and its standard library. It isn't the job of the language itself to provide every possible useful iteration on the base primitives it provides.
• gregwebs 2y ago
  Thanks for sharing what is working for you in production. I made a somewhat similar library that also does batching (docs are sparse, but I am updating docs on my libraries this week) [1]. I would call this parallelism rather than concurrency. The main issue I have with this library's implementation is how errors are handled. Errors are retrieved rather than assigned- but assignment is preferable because it gets verified by tools. In my library I used a channel for errors- that gives ultimate flexiblitiy- it can be converted to wait and collect them to a slice or to perform a cancellation on first error. [1] https://github.com/gregwebs/go-parallel https://github.com/gregwebs/go-parallel
  • destel 2y ago
    Thank you for the feedback. My design decision is of course a tradeoff. When multiple channels are exposed to the users (not encapsulated inside the lib), this forces them to use "select". And this is very error prone in my experience
    • gregwebs 2y ago
      I never needed to use select on an error channel for my use cases because at the point I operate on the error channel I want to block for completion. And I provide helpers for the desired behavior for the channel so I don't even directly receive from it. I see that some of Rill is designed to operate on continuous streams, and in that light the design decision makes sense. For my use cases though the stream always had an end.
• Groxx 2y ago
  Hmmm. Some stuff to like, but I do feel like this should have a big, noticeable cautionary note that it does not wait if you end early (e.g. via Err). Any pending actions continue running and draining in background goroutines, and are potentially VERY delayed if internal/core.Delay is ever exposed or your funcs sleep. I've seen that kind of pattern lead to TONS of surprise race conditions in user-code, because everyone always thinks that "it returned" means "it's done". Which is reasonable, nothing else is measurable on the caller side - changing that won't be noticeable by callers and may violate their expectations and cause crashes.
  • destel 2y ago
    Thank you for the feedback. I agree with your point. The current solution is to make pipeline stages context-aware (which is often happens automatically) and cancel the context before returning. This is the responsibility of the user and can lead to problems you described. I haven't yet found a better solution to this. On the other hand, exact same thing happens if your function spawns a goroutine and returns. That goroutine would run until done, unless it's context aware. Regarding the "Delay" and "infiniteBuffer" functions - these are part of the work on adding support for feedback loops to Rill. I haven't yet found a reliable and user friendly way to do it, so this work is on hold for now.
• icar 2y ago
  This reminds me of rxjs.
• pezo1919 2y ago
  How does it compare to RxGo?
  • destel 2y ago
    I have to be honest - I haven't ever heard about it. Just checked and found it's very mature and popular, though it seems to have had no commits in the last 2 years. After quick scanning, I'd highlight these 2 differences: - Rill is type safe and uses generics - Rill does not try to hide channels, it just provides primitives to transform them
• kermatt 2y ago
  Name overlap with another Go based package: https://github.com/rilldata/rill https://github.com/rilldata/rill
  • taffit 2y ago
    This also came to my mind when I heard `rill`, coming from https://www.rilldata.com/ https://www.rilldata.com/ .
• linhns 2y ago
  Looks super neat! Just a small question: Why did you chose the name Try instead of Result?
  • destel 2y ago
    Thank you. I "stealed" the name from scala. They have the similar value+error type. Maybe in context it rill the better name could have been "Item"
• destel 2y ago
  Hi everyone. Posting on HN for the first time. I'd like to share Rill - a toolkit for composable channel-based concurrency, that makes it easy to build concurrent programs from simple, reusable parts Example of what it looks like: // Convert a slice into a channel ids := rill.FromSlice([]int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, nil) // Read users from API with concurrency=3 users := rill.Map(ids, 3, func(id int) (*User, error) { return api.GetUser(ctx, id) }) // Process users with concurrency=2 err := rill.ForEach(users, 2, func(u *User) error { if !u.IsActive { u.IsActive = true return api.SaveUser(ctx, u) } return nil }) // Handle errors fmt.Println("Error:", err) Key features: - Makes concurrent code composable and clean - Works for both simple cases and complex pipelines - Built-in batching support - Order preservation when needed - Centralized error handling - Zero dependencies The library grew from solving real concurrency challenges in production. Happy to discuss the design decisions or answer any questions.
  • minus7 2y ago
    Hey, I found your library a few weeks ago when I was annoyed by nothing like this being built into the standard library. It’s been a breeze to use so far. A neat trick I found to gauge bottlenecks in pipelines is using Buffers between steps and running a goroutine that periodically prints `len(buffered)/cap(buffered)`
    • destel 2y ago
      Thank you very much for the feedback. I thought about something similar some time ago. Buffer of size one, then measure the average time each item spends in the buffer. But for debugging your approach is simpler and more practical.
  • fillskills 2y ago
    Very intuitive API. Thanks!
  • limit499karma 2y ago
    Is there an underlying assumption that the channels are containers and not streams?
• hu3 2y ago
  Hi! Looks great. I might use this to fan out/in my RSS reader HTTP calls. How would I implement timeout? In case a HTTP call takes too long?
  • lsaferite 2y ago
    Based on the examples and documentation, rill doesn't manage context for you. You'd simply set the client timeout or give each http call a timeout context.
  • mariusor 2y ago
    You might be interested by something that has been designed specifically for this problem. I created a state machine library for Go on top of which I mapped retry[1] and some other patterns. And funnily enough one of the first applications I implemented is an RSS reader[2] [1] https://pkg.go.dev/git.sr.ht/~mariusor/ssm#example-Retry https://pkg.go.dev/git.sr.ht/~mariusor/ssm#example-Retry [2] https://git.sr.ht/~mariusor/frankenjack/tree/master/item/sources/feeds/states.go#L107 https://git.sr.ht/~mariusor/frankenjack/tree/master/item/sou...
  • destel 2y ago
    For now, the library is context-agnostic by design. For HTTP timeouts, you'd use Go's standard approaches: either set the HTTP client timeout or pass a context with timeout to each request. Please let me know more about your use case - I'll let you know if Rill isn't a good fit.
• lspears 2y ago
  This is great. I am working on a robotics application and this seems like a better abstraction than alternatives such local messaging servers. How do you deal with something like back pressure or not keeping up with incoming data?
  • destel 2y ago
    The lib is based on channels and inherits the channel behavior in terms of backpressure. Simply put if no-one reads on one side of the pipeline, it wouldn't be possible to write anything on the other side. Still, it's possible to add buffering at arbitrary points in the pipeline using the rill.Buffer function.
• lyxell 2y ago
  The API looks really nice and intuitive! What motivated you to build this?
  • destel 2y ago
    Thank you! There are two pieces of motivation here. The first one is removing boilerplate of spawning goroutines that read from one channel and write to another. Such code also needs wait/err group to properly close the output channel. I wanted to abstract this away as "channel transformation" with user controlled concurrency. Another part is to provide solutions for some real problems I had. Most notably batching, error handling (in multi stage pipelines) and order preservation. I thought that they are generic enough to be the part of general purpose library.
• c4pt0r 2y ago
  Very handy!
• noctane 2y ago
  It looks great. What are other existing tools? And how do they compare to them?
  • Scaevolus 2y ago
    Sourcegraph Conc is broadly similar in providing pool helpers, but doesn't provide the same fine grained batching options: https://github.com/sourcegraph/conc https://github.com/sourcegraph/conc Uber CFF does code generation, and has more of a focus on readability and complex dependency chains: https://github.com/uber-go/cff https://github.com/uber-go/cff
    • noctane 2y ago
      Thanks for sharing!
• qaq 2y ago
  any plans to add context support ?
  • destel 2y ago
    I've just pushed few small changes to the readme that better explain context usage
  • destel 2y ago
    I am thinking on it. To be honest, the current design works fine for my use cases: simply put, the function that defines a pipeline should have context.WithCancel() and defer cancel() calls. I need a feedback on this. What kind of builtin context support would work for you? Do you need something like errgroup's ability to automatically cancel the context on first error?
• tschellenbach 2y ago
  this is a real problem in go, very easy to have bugs when working with channels and the way it handles errors etc.
  • latchkey 2y ago
    If you write comprehensive unit tests, it is not easy to have bugs in golang. Especially as things change over time. A library like this isn't going to protect you from having bugs. TIL: HN doesn't like writing tests. The downvotes on this are hilarious. "Job security" ¯\_(ツ)_/¯.
    • tmoertel 2y ago
      I think you're getting downvoted for the unsupported assertion that "If you write comprehensive unit tests, it is not easy to have bugs in golang." Probably because you made that assertion in the context of a discussion of channels, widely believed to have underlying concurrency semantics that are subtle and easy to misunderstand, making "write comprehensive unit tests" seem like a strategy that's apt to let real-world problems slip through (because a programmer's belief that their tests are "comprehensive" is likely to be mistaken).
      • steve_adams_86 2y ago
        Go makes it easier to write concurrent code, but it's a serious chore to iron out all of the kinks in more complex tasks. I've missed some weird stuff over the years. I don't blame Go. It's an inherently difficult problem space. As a result, testing isn't a trivial job either. I wish it was.
        • latchkey 2y ago
          It is not a chore, it is our job. This is what we do. We write code. Of course you've missed stuff, we all have. Tests help alleviate the missed stuff. Even better is that they protect us over time, especially when we want to refactor things or find bugs in production. How do you fix a production bug without breaking something else? You write tests so that you know your code works. Again with the HN downvotes, hilarious. People really hate the truth.
• jbendotnet21 2y ago
  Looks good, similar to https://github.com/sourcegraph/conc https://github.com/sourcegraph/conc which we've been using for a while. Will give this a look.
  • alpb 2y ago
    There are also libraries like https://github.com/Jeffail/tunny https://github.com/Jeffail/tunny or https://pkg.go.dev/go.uber.org/goleak https://pkg.go.dev/go.uber.org/goleak or https://github.com/fatih/semgroup https://github.com/fatih/semgroup to help deal with concurrency limits and goroutine lifecycle management. As the author of https://github.com/ahmetb/go-linq https://github.com/ahmetb/go-linq, it's hard to find adoption for libraries offering "syntactic sugar" in Go, as the language culture discourages those kind of abstractions and keeping the code straightforward.
• izolate 2y ago
  The batching concept is a cool idea and could be useful in the right context. That said, this feels like a JavaScript engineer's take on Go. Abstractions like Map and ForEach don't align with Go's emphasis on simplicity and explicitness. The lack of context.Context handling also seems like an oversight, especially when considering concurrency. Judging by the praise, I'm probably in the minority, but as a code reviewer, I’d much rather see straightforward loops, channels, and Go's native constructs over something like Rill.
  • hnlmorg 2y ago
    I don’t agree with your comment about Map and ForEach, just by virtue of the fact that sync.Map exists in Go’s standard library. But your point about the lack of contexts is definitely a deal breaker for me personally too.
    • akshayshah 2y ago
      The "map" under discussion here is very different from sync.Map. The discussion here is focused on the "map" primitive from functional programming - transforming a collection by applying a function to each element. sync.Map is a concurrency-safe hash map. Same name, totally different thing.
  • ARandomerDude 2y ago
    > Map and ForEach don't align with Go's emphasis on simplicity and explicitness I've never paid my bills with Go, but `Map` and `ForEach` don't seem all that different than `for _, u := range Users` to me. Yes, the former is "functional" but only mildly.
    • emseetech 2y ago
      In that case there's no particular reason to use them. As far as Go's philosophy goes.
      • ARandomerDude 2y ago
• deleted 2y ago
  [deleted]
• purpleidea 2y ago
  Nice! I do a lot of concurrency work with DAG's in https://github.com/purpleidea/mgmt/ https://github.com/purpleidea/mgmt/ and I would love to swap out some of those concurrency runners with a lib if possible. I was wondering if this could be it... Any thoughts in that direction, please let me know!
  • destel 2y ago
    Thank you! I took a quick look at mgmt, it's quite a large and complex project. I'd need to better understand your DAG-based concurrency patterns to say if Rill would be a good fit. Could you share some examples of the concurrent runners you're thinking about? This would help me understand if/how Rill might be useful for your case.
    • purpleidea 2y ago
      So we have two concurrency "engines" in mgmt: (1) One for running the function graph, and (2) one for running the resource graph. (1) https://github.com/purpleidea/mgmt/tree/master/lang/funcs/dage https://github.com/purpleidea/mgmt/tree/master/lang/funcs/da... (2) https://github.com/purpleidea/mgmt/tree/master/engine/graph https://github.com/purpleidea/mgmt/tree/master/engine/graph TBQH, I think I'm decent at concurrency, but I never considered it my passion or expertise and while both of those _work_ I do know that I have some concurrency bugs hanging out, and I'd love real professional help here. In the odd chance you'd be willing to hack on things with me, I'd be particularly elated. There might be a possible symbiosis! I'm @purpleidea on Mastodon, Matrix (where we have an #mgmtconfig room) and a few other places in case you'd like to chat more!
• fidotron 2y ago
  I think it is time to face the fact CSP style channels are a bad idea if you don’t also have the occam semantics for channel scope. (I know there are other people here that understood that sentence). The problem in golang is the channel cleanup, particularly. is a mess. In occam they come and go simply by existing as variables in seq or par blocks. Occam is very static though, so the equivalent to goroutines are all allocated at build. (Occam pi attempted to resolve this iirc). https://en.m.wikipedia.org/wiki/Occam_(programming_language) https://en.m.wikipedia.org/wiki/Occam_(programming_language) Some of the patterns in this library are reminiscent of constructs occam has as basic constructs, such as the for each block, although the occam one must have the number of blocks known at build time. The fact so many people in golang reach for mutexes constantly is a sign things are not all well there.
  • 0x696C6961 2y ago
    Channels are just another synchronization primitive in your toolbox. They do make some things much simpler, but there's no reason to reach for one if a mutex does the job. The usage of mutexes doesn't make channels "bad" for the same reason that usage of atomics doesn't make mutexes bad.
  • dboreham 2y ago
    +1 for understanding the sentence.
• 0x696C6961 2y ago
  I think that using iterators in the public API would have been better than channels.
  • destel 2y ago
    Rill might look like it tries to be a replacement for iterators, but it's not the case. It's a concurrency library, that's why it's based on channels
    • 0x696C6961 2y ago
      I disagree that using channels is necessary for concurrency. Consider the following iterator based signature for your Map function: func Map[A, B any](in iter.Seq[Try[A]], n int, f func(A) (B, error)) iter.Seq[Try[B]]
      • eweise 2y ago
        that makes Scala look easy.
        • 0x696C6961 2y ago
          For context, here's the existing Map signature from the linked library: func Map[A, B any](in <-chan Try[A], n int, f func(A) (B, error)) <-chan Try[B] Are you suggesting that this channel based signature is significantly easier to understand than the one I shared?
          • eweise 2y ago
            No, it was more of a general comment that once Go added support for generics, doing functional style programming starts to look as complex (or more actually) than languages that built support from the beginning.
            • 0x696C6961 2y ago
• dangoodmanUT 2y ago
  Love the idea, some weirdness though: > Here's a practical example: finding the first occurrence of a specific string among 1000 large files hosted online. Downloading all files at once would consume too much memory, processing them sequentially would be too slow, and traditional concurrency patterns do not preserve the order of files, making it challenging to find the first match. But this example will process ALL items, it won't break when a batch of 5 finds something?
  • destel 2y ago
    I've also just pushed few small changes to the readme that clarify this things.
  • destel 2y ago
    It will. Otherwise the example wouldn't make sense. There's one important detail I haven't clarified enough in that part of the readme. For proper pipeline termination the context has to be cancelled. So it should have been be Like: func main() { ctx, cancel := context.WithCancel(context.Background()) defer cancel() urls := rill.Generate(func(send func(string), sendErr func(error)) { for i := 0; i < 1000 && ctx.Err() == nil; i++ { send(fmt.Sprintf("https://example.com/file-%d.txt", i)) } }) ... One of the reasons I've ommited context cancellation in this and some other examples is because everything's happening inside the main function. I'll probably add cancellations to avoid confusion.
• deleted 2y ago
  [deleted]
• fatih-erikli-cg 2y ago
  := for assinging a variable sounds and looks weird to me
  • steve_adams_86 2y ago
    It's the walrus operator. Pascal and Python use it as well. You get used to it pretty quickly. Pascal: https://www.freepascal.org/docs-html/ref/refse104.html#x224-24800015.3 https://www.freepascal.org/docs-html/ref/refse104.html#x224-... Python: https://docs.python.org/3/whatsnew/3.8.html#assignment-expressions https://docs.python.org/3/whatsnew/3.8.html#assignment-expre...
    • fatih-erikli-cg 2y ago
      It was something introduced like a 1 april joke. Clean implementation of any programming language won't have that.
      • kortex 2y ago
        That ship has long sailed. ALGOL 1958 used := and Pascal popularized it. https://en.m.wikipedia.org/wiki/Assignment_(computer_science) https://en.m.wikipedia.org/wiki/Assignment_(computer_science...
        • fatih-erikli-cg 2y ago
          I think they scan the code by two characters because one is not enough for <= and => so what is why assignment is := or =:. Probably + is ++ too.
      • hnlmorg 2y ago
        I actually think it’s more readable because it makes the distinction between assignment and equivalence very clear. bugs originating from the similarity of == vs = has probably cost the industry millions over the last 3 decades.
  • indulona 2y ago