4 ms·
I recently wrote some of the type of code that you would be upset to read, so maybe someone can sympathize with this a bit. The task at hand wasn’t easy at all
by corytheboyd 6y ago
I recently wrote some of the type of code that you would be upset to read, so maybe someone can sympathize with this a bit.
The task at hand wasn’t easy at all, it required both:
1. Making numerous network requests (some endpoints support batching, so do that as much as possible)
2. The entire main loop needs to be as fast as reasonably possible
The only real way to satisfy both, given that serially executing the network activity would waste tons of time, was some fairly tricky concurrent logic. I ended up using a few queue structures to collect day before making batch network calls, and stitching together all of the results at the end (block on all work threads, collect, report, then start next iteration)
Every step along the way made sense, every addition of complexity felt justified. And the first person to come to me asking how the fuck it got so complicated is frustrated, so my knee jerk reaction is to get frustrated back saying well how else could we satisfy complex requirements with a complex solution.
We should always be challenging ourselves and others to keep solutions simple, but remember that the simplest solution may itself still be complex.
- ccccc0 6y agoCould you not have used a simple thread pool model in your case? That would be a trivial parallelization of your network requests with limiting on the resources (number of parallel connections) Otherwise when the logic gets complex I try to model it as independent threads each having a well-defined role (e.g. exclusive write access to SQLite) communicating with channels. Although I tend to code these things in Rust, the language who taught me that philosophy is Go.
- tleb_ 6y agoMaybe providing a diagram to explain the structure would help? Representing queues and workers as arrows between queues.
- lmm 6y agoIMO the key is always to be compositional. E.g. here you should be able to separate the batching up requests from the performing network calls, and then you can test and reason about both pieces separately. I find the iteratee model is really good for this.
- mrbald 6y agoAlways have a drawn picture, if you cannot draw it you will fail to explain it to others (and to see design gaps) and inevitably everyone will think it’s an over engineered complex shit. Design-wise: The IO multiplexing — event loop(s), the completion callbacks go to the "Processing". Processing — usually beneficial to model as a processing graph (tree/pipeline being specific examples of a graph without feedback loops). It's a clean, low overhead abstraction, easy to visualise (and thus discuss with others), easy to change structurally or in the "aspect oriented" way. Examples in JVM world: Netty pipelines, AKKA actors, streams, and graphs, in C++ world ASIO & its executors sub-framework. Queues/Threads is still a sufficiently low level to comfortably operate on.
- theelous3 6y agoSounds like you wrote all of the stuff that wasn't io bound in to your concurrent logic, rather than group -> make concurrent calls and wait -> sort which would have been very straightforward, and probably faster too because you don't let any one step interrupt the other