3 ms·
I need to do HTTP requests, database queries and queue processing. Mastodon has chosen a Service oriented architecture for this, and I'm happy with that decisi
by INTPenis 4y ago
I need to do HTTP requests, database queries and queue processing.
Mastodon has chosen a Service oriented architecture for this, and I'm happy with that decision because it means I can keep the web and DB queries running on minimal resources, while scaling the queue processing up and down depending on current activity.
One could say that incoming http requests from end users is a fairly predictable level of activity, while queue processing is a very unpredictable level of activity. Because it's tied into people's posts, their followers, the number of instances their followers are on, and so forth.
And with Mastodon v4 the resources required for end user http requests are even fewer, because they've shifted more processing onto the client with Javascript.
So I think the freedom to be able to scale the queue processing separately is fundamental to me. I can't imagine what it would be like to run a different software where I had to scale a monolithic service.
It doesn't really matter how fast the software is because queue processing is about blocking tasks, and they're blocking because they involve contact with other services and servers on the internet.
So yes I'd love to see a sidekiq replacement in Golang, Rust, or even Elixir, but I would not like to see it bundled in with other services. Mainly because queue processing deals with blocking tasks, and incoming HTTP requests should never be blocking.