5 ms·

Go package: fanout – make writing parallel code even easier

42 points by sunfmin 12y ago

hide goroutines and channels from developer.

zzzcpan 12y ago
Seems to be way too trivial for a library: http://play.golang.org/p/U0URukpeO3 http://play.golang.org/p/U0URukpeO3 Although I like the idea of having some kind of API to spread the work across multiple machines, since channels are slow anyway.
StavrosK 12y ago
I'm not sure what the usefulness of this is (apart from saving you from writing one or two functions yourself), isn't it just a parallel queue?
- sunfmin 12y ago
  I guess it is a parallel queue, It saves the trouble to worry about goroutines and channels and synchronizations, encapsulated the logic that talked in http://blog.golang.org/pipelines http://blog.golang.org/pipelines last part. To make use of that pattern easier. I have quite a few use cases myself: 1. Say there is a article on the web, that contains dozens of images embed in the html. Which I want to crawl the article, and Also the images, and put the image into S3 for faster serve. I can use the fanout concurrency pattern to put upload all the images simultaneously to S3 to make it faster. 2. Say I have 20 mysql node sharded data, I want to fetch top 50 latest created data from all of them, and sort it to and show it to user after.
  - StavrosK 12y ago
    Oh, yes, there are lots of use cases for parallel queues, for sure.
- latch 12y ago
  it's a small abstraction, but I could see it being useful. My gripe, and I don't know how to say this without people saying I'm whining about generics, is the interface{} return. Such a pain.
  - aikah 12y ago
    Go has generics,you just can't define new ones,which is the heart of the problem. It's not about whinning.There is a need for that yet no real will to implement user defined one.
bradhe 12y ago
Why does this need to be a dependency that you bring on? This is a trivial pattern.
TheDong 12y ago
Oh, hey, it's a generic package that means you lose type safety and saves you from writing roughly 10 lines of code. Buffered channels + goroutines + WaitGroup already allow you to implement this trivially, and because channels are builtin generic, you can do it without the nasty type casts. Really, []interface{} is a terrible type to work with.
- sunfmin 12y ago
  Yes, I often find myself lost in this trivial channels + goroutines + WaitGroup combination. For my self I'd prefer cast interface{}, :-)
  - chimeracoder 12y ago
    interface{} is different from []interface{}. You can do a type assertion (not cast - these are different concepts) from interface{} to another type. You cannot do a type assertion on []interface{}; you have to iterate over each element and assert individually. I agree with GP - the built-in primitives for synchronization, along with sync from the stdlib, are much cleaner once you know how to use them, and they're more idiomatic.
- coldtea 12y ago
  >Oh, hey, it's a generic package that means you lose type safety and saves you from writing roughly 10 lines of code. Or actually, it's not a generic (as in generics) package, which is why you lose type safety.
  - chimeracoder 12y ago
    > Or actually, it's not a generic (as in generics) package, which is why you lose type safety. Can we please skip just one opportunity to begin this endless flamewar on a HN post about Go? Just one? Please? This topic has been beaten to death and beyond, and is not relevant to TFA at all, as the word "generic" can be used in many contexts, and it's more than clear which context OP meant.
drtse4 12y ago
>How does it make the program run faster? Sunfmin, wrong answer, you'd have a 20x improvement with 20 workers only with 20 cores (ignoring the limited overhead), regardless of the number of workers your queue of jobs will be consumed in number_of_jobs*single_job_duration/GOMAXPROCS.
- sunfmin 12y ago
  Thanks, but there will be I/O waiting stuff right? For that whois command example, I find the CPU usage is zero if I am not running it parallel.
  - drtse4 12y ago
    Yup, if your jobs are i/o heavy the cpu time is wasted waiting if you execute them in a strictly sequential way. With your worker pool the cpu intensive part of the jobs is instead being executed concurrently GOMAXPROCS jobs at a time.
goykasi 12y ago
I wrote a similar Go package for running work loads in parallel, but I used beanstalkd for job/result transport. This allows me a bit more freedom to spread the workers/requesters across my network. It's a bit rough around the edges and could use some refactoring, but it works well for my uses. https://github.com/eramus/worker https://github.com/eramus/worker
- sunfmin 12y ago
  Bookmarked and Stared!
AYBABTME 12y ago
Write yourself the 20 lines needed before pulling an extra 3rd party deps just for that. The cost of having another 3rd party deps is greater than the seconds you save from using this.
deleted 12y ago
[deleted]
jws 12y ago
Just yesterday I was staring at some of my go code thinking that channels are the "goto" of concurrency. You can make just about anything with them, but to understand code you have to read it all, hold it in your head, and reason about all possible outcomes. In the '60s that's how flow of control was done. As the '70s went by "structured programming" came in and exotic things like while loops, switch statements, and functions that you could only enter at the top (so limiting!) became the norm. This post proposes a level of abstraction to take a common 10 line idiom and abstract it to a word. I'd much rather read code with the abstraction. (In this case it is clean to read, but there are many complicated patterns in common use involving auxiliary chans for cancellation and timeouts.) Sadly, this is where it collides with the go language designers. Go is anti-abstraction by design. If you don't like that then you descend into interface{} hell and manual runtime type checking, or change languages, or just repeat yourself a lot and pray you get the fiddly bits right each time.
- zzzcpan 12y ago
  Oh, they use abstractions a lot, just not the flow control ones, mainly objects. Like sync.WaitGroup mentioned here is an object and is a completely brain dead choice for this kind of thing. Some form of parallel map() instead would be much easier to understand and to reason about, for example: http://play.golang.org/p/4I-uBJ1Tce http://play.golang.org/p/4I-uBJ1Tce Anyways, I mostly agree, just not generally over abstractions, but more in the context of proper flow control abstractions.
- namelezz 12y ago
  Is it easy to use channels for concurrency in UI development. I believe most UI developments use callbacks, such as onClick, handleClick...

42 points by sunfmin 12y ago

hide goroutines and channels from developer.

Seems to be way too trivial for a library: http://play.golang.org/p/U0URukpeO3 http://play.golang.org/p/U0URukpeO3 Although I like the idea of having some kind of API to spread the work across multiple machines, since channels are slow anyway.

StavrosK 12y ago

I'm not sure what the usefulness of this is (apart from saving you from writing one or two functions yourself), isn't it just a parallel queue?

sunfmin 12y ago
I guess it is a parallel queue, It saves the trouble to worry about goroutines and channels and synchronizations, encapsulated the logic that talked in http://blog.golang.org/pipelines http://blog.golang.org/pipelines last part. To make use of that pattern easier. I have quite a few use cases myself: 1. Say there is a article on the web, that contains dozens of images embed in the html. Which I want to crawl the article, and Also the images, and put the image into S3 for faster serve. I can use the fanout concurrency pattern to put upload all the images simultaneously to S3 to make it faster. 2. Say I have 20 mysql node sharded data, I want to fetch top 50 latest created data from all of them, and sort it to and show it to user after.
- StavrosK 12y ago
  Oh, yes, there are lots of use cases for parallel queues, for sure.
latch 12y ago
it's a small abstraction, but I could see it being useful. My gripe, and I don't know how to say this without people saying I'm whining about generics, is the interface{} return. Such a pain.
- aikah 12y ago
  Go has generics,you just can't define new ones,which is the heart of the problem. It's not about whinning.There is a need for that yet no real will to implement user defined one.

bradhe 12y ago

Why does this need to be a dependency that you bring on? This is a trivial pattern.

TheDong 12y ago

Oh, hey, it's a generic package that means you lose type safety and saves you from writing roughly 10 lines of code. Buffered channels + goroutines + WaitGroup already allow you to implement this trivially, and because channels are builtin generic, you can do it without the nasty type casts. Really, []interface{} is a terrible type to work with.

sunfmin 12y ago
Yes, I often find myself lost in this trivial channels + goroutines + WaitGroup combination. For my self I'd prefer cast interface{}, :-)
- chimeracoder 12y ago
  interface{} is different from []interface{}. You can do a type assertion (not cast - these are different concepts) from interface{} to another type. You cannot do a type assertion on []interface{}; you have to iterate over each element and assert individually. I agree with GP - the built-in primitives for synchronization, along with sync from the stdlib, are much cleaner once you know how to use them, and they're more idiomatic.
coldtea 12y ago
>Oh, hey, it's a generic package that means you lose type safety and saves you from writing roughly 10 lines of code. Or actually, it's not a generic (as in generics) package, which is why you lose type safety.
- chimeracoder 12y ago
  > Or actually, it's not a generic (as in generics) package, which is why you lose type safety. Can we please skip just one opportunity to begin this endless flamewar on a HN post about Go? Just one? Please? This topic has been beaten to death and beyond, and is not relevant to TFA at all, as the word "generic" can be used in many contexts, and it's more than clear which context OP meant.

drtse4 12y ago

>How does it make the program run faster? Sunfmin, wrong answer, you'd have a 20x improvement with 20 workers only with 20 cores (ignoring the limited overhead), regardless of the number of workers your queue of jobs will be consumed in number_of_jobs*single_job_duration/GOMAXPROCS.

sunfmin 12y ago
Thanks, but there will be I/O waiting stuff right? For that whois command example, I find the CPU usage is zero if I am not running it parallel.
- drtse4 12y ago
  Yup, if your jobs are i/o heavy the cpu time is wasted waiting if you execute them in a strictly sequential way. With your worker pool the cpu intensive part of the jobs is instead being executed concurrently GOMAXPROCS jobs at a time.

goykasi 12y ago

I wrote a similar Go package for running work loads in parallel, but I used beanstalkd for job/result transport. This allows me a bit more freedom to spread the workers/requesters across my network. It's a bit rough around the edges and could use some refactoring, but it works well for my uses. https://github.com/eramus/worker https://github.com/eramus/worker

sunfmin 12y ago
Bookmarked and Stared!

AYBABTME 12y ago

Write yourself the 20 lines needed before pulling an extra 3rd party deps just for that. The cost of having another 3rd party deps is greater than the seconds you save from using this.

deleted 12y ago

[deleted]

jws 12y ago

Just yesterday I was staring at some of my go code thinking that channels are the "goto" of concurrency. You can make just about anything with them, but to understand code you have to read it all, hold it in your head, and reason about all possible outcomes. In the '60s that's how flow of control was done. As the '70s went by "structured programming" came in and exotic things like while loops, switch statements, and functions that you could only enter at the top (so limiting!) became the norm. This post proposes a level of abstraction to take a common 10 line idiom and abstract it to a word. I'd much rather read code with the abstraction. (In this case it is clean to read, but there are many complicated patterns in common use involving auxiliary chans for cancellation and timeouts.) Sadly, this is where it collides with the go language designers. Go is anti-abstraction by design. If you don't like that then you descend into interface{} hell and manual runtime type checking, or change languages, or just repeat yourself a lot and pray you get the fiddly bits right each time.

zzzcpan 12y ago
Oh, they use abstractions a lot, just not the flow control ones, mainly objects. Like sync.WaitGroup mentioned here is an object and is a completely brain dead choice for this kind of thing. Some form of parallel map() instead would be much easier to understand and to reason about, for example: http://play.golang.org/p/4I-uBJ1Tce http://play.golang.org/p/4I-uBJ1Tce Anyways, I mostly agree, just not generally over abstractions, but more in the context of proper flow control abstractions.
namelezz 12y ago
Is it easy to use channels for concurrency in UI development. I believe most UI developments use callbacks, such as onClick, handleClick...