4 ms·

If you're willing to plumb through all the knobs imaginable then this is a fine approach. But if spider.Crawl just ran unbounded or with a fixed bound it could

If you're willing to plumb through all the knobs imaginable then this is a fine approach. But if spider.Crawl just ran unbounded or with a fixed bound it could trivially become a huge headache. There are many patterns in Go that are preferable to just starting a bunch of concurrent work. For example the pubsub client has options to disable batching and limit in flight connections.

disintegrator 3y ago
Indeed, it's down to the library API to give you the right knobs to control for concurrency. It says more about the quality of the library if it mismanages goroutines than it does about whether or not libraries should use goroutines at all.
twic 3y ago
> There are many patterns in Go that are preferable to just starting a bunch of concurrent work. I have not worked in Go for several years, but i remember that when i did, more experienced people told me that this was exactly what you were supposed to do in Go. That where in another language you might set up a queue and a threadpool and so on, in Go, you should just spawn a load of goroutines, and let the runtime sort it out. Is this no longer the canonical approach?
- bb88 3y ago
  It's also one thing if you want to provide SingleThreadedApp and a MultiThreadedApp instances in your package, but that should be left up to the user -- or made explicit in package docs. (We're spinning up 100 go routines!) Over a decade ago I was trying to debug why the UI written in Java was slow on solaris box. But that wasn't the primary reason. Turns out there were over 3000 java threads running at any given time. The people who wrote Java code wrapped an event class around a thread, then just started kicking off events willy-nilly. So after about 5 minutes the os had thousands of these things to deal with. There wasn't really any processing left over for anyone else.

disintegrator 3y ago

Indeed, it's down to the library API to give you the right knobs to control for concurrency. It says more about the quality of the library if it mismanages goroutines than it does about whether or not libraries should use goroutines at all.

twic 3y ago

> There are many patterns in Go that are preferable to just starting a bunch of concurrent work. I have not worked in Go for several years, but i remember that when i did, more experienced people told me that this was exactly what you were supposed to do in Go. That where in another language you might set up a queue and a threadpool and so on, in Go, you should just spawn a load of goroutines, and let the runtime sort it out. Is this no longer the canonical approach?

bb88 3y ago
It's also one thing if you want to provide SingleThreadedApp and a MultiThreadedApp instances in your package, but that should be left up to the user -- or made explicit in package docs. (We're spinning up 100 go routines!) Over a decade ago I was trying to debug why the UI written in Java was slow on solaris box. But that wasn't the primary reason. Turns out there were over 3000 java threads running at any given time. The people who wrote Java code wrapped an event class around a thread, then just started kicking off events willy-nilly. So after about 5 minutes the os had thousands of these things to deal with. There wasn't really any processing left over for anyone else.