3 ms·
It's kind of an "I know it when I see it" situation. To sit down and try to write a rigidly-specified set of rules on when it is and is not OK for a library to
by jerf 3y ago
It's kind of an "I know it when I see it" situation. To sit down and try to write a rigidly-specified set of rules on when it is and is not OK for a library to spin goroutines would be very difficult.
Yet the basic principle isn't that hard: Your code should generally be what is considered to be "sync", and it is up to the user to decide if they want that to be "async" by using a goroutine themselves.
This rule is primary for libraries that try to be "helpful" by, say, decoding an image unconditionally in a goroutine or something and providing a "promise" of some sort you can read the results from. Don't do that. If a Go programmer wants a "promise"-like behavior, any Go code can be so converted by an end-user at any time and the best thing the library can do in that case is just stay out of the way of the already-ever-present features that allow you to do that.
But on the flip side, I expect a library implementing a parallel map to have its own goroutines. As a parallel map user, I basically don't want to see them or have to think about them. At best, maybe the library has some knobs I can tune, but I don't want to be managing them. That would defeat the entire purpose of such a library. A deliberately recursive and parallel crawling library, documented to be as such as where that feature is its major utility, fits into this category. By calling ".Crawl" I am clearly asking for this functionality explicitly, by the nature of the contract of the library. Which is also a good use case for structured concurrency, which Go does not explicitly implement into its language but still makes for an easier and safer library than the alternative.
- skybrian 3y agoIt seems like it would be better if the concurrency were pluggable somehow. Maybe Crawl takes some kind of worker-starting interface, with a suitable default implementation? Then the job of the crawler is to find new units of work, not to schedule them. In theory it could be done single-threaded by pulling work from a queue.
- jerf 3y agoThat would be an inner platform: https://en.wikipedia.org/wiki/Inner-platform_effect https://en.wikipedia.org/wiki/Inner-platform_effect The Go scheduler is already taking units of work called goroutines and scheduling them. It's no big deal to ask the crawling system to have some limit on how many goroutines it'll use, the patterns for that are well-established, and also necessary because it's not all about the goroutines in this case. Crawling needs controls to limit how many requests/sec it makes to a given server, how deeply to recurse, what kind of recursion, etc. anyhow so it's not like it particularly sticks out to also have a concurrency parameter.
- skybrian 3y agoFair enough. But I'm not sure a wrapper around starting a goroutine counts as an inner platform, because it's not doing much work, and it's not really work that the Go SDK does. Choosing when to start goroutines and how many to start is an application concern. Depending on how it's done, it might be a decent way to structure a crawler?