5 ms·
This is kind of relevant now: Mastodon is seeing a lot of growth all the sudden, the popular server for it is in Rails, and scaling up quickly is a challenge fo
by twotwotwo 4y ago
This is kind of relevant now: Mastodon is seeing a lot of growth all the sudden, the popular server for it is in Rails, and scaling up quickly is a challenge for server operators used to the levels of traffic from two months ago.
Twitter did rewrite their frontend to JVM languages because it was enough of a bottleneck to be worth it. If you can get Mastodon substantially further by serving some views using something besides Rails, or even going around ActiveRecord in places, that is a pretty different short-term to-do list than if the immediate problems are in the data layer. (Long term you have to scale both of course!)
- kstrauser 4y agoUgh, tell me about it. I had an interesting weekend dealing with traffic loads that grew by about 7x in a few days. I wrote about my misadventures at https://blog.freeradical.zone/post/surviving-thriving-through-2022-11-05-meltdown/ https://blog.freeradical.zone/post/surviving-thriving-throug... . The gist of it is that a RoR “Sidekiq” task queue gets CPU bound after about 25-30 worker threads doing things like making REST API calls to remote servers, querying a database, insert status updates, etc. I can’t help but to think that the equivalent written in Go, or even Ruby-without-Rails, could handle many times the traffic with fewer CPU and RAM resources.
- devoutsalsa 4y agoSounds like a job for MUMPS.
- kstrauser 4y agoYeah, I mentioned Elixir in the post as an example of a better alternative.
- hamandcheese 4y agoDid you actually observe CPU at 100%, or just that your sidekiq workers were falling behind? Solid chance that you are still IO bound and could push past 25 workers. The main bottleneck is probably RAM.
- kstrauser 4y agoI actually saw them at 100% CPU with RAM to spare.
- Lukas_Skywalker 4y agoI assume that Mastodon is using ActiveJob to push tasks to the queue. According to the creator of Sidekiq, bypassing ActiveJob would lead to significantly lower resource usage: https://github.com/mperham/sidekiq/wiki/Active+Job#performance https://github.com/mperham/sidekiq/wiki/Active+Job#performan... I never tried it though. Just remembered the Wiki note.
- berkes 4y agoI'm experimenting with some alternatives. I prefer Rust (because I speak that) or Ruby (it's my main language) but the Go option gotosocial is my favorite. It lacks a frontend and admin UI, so not ready for prime-time, i guess. But good enough to glue onto a pinafore frontend. And to manage a small instance. The most promising rust alternative, rustodon, is ground to a halt, it seems. A lean, sinatra-based backend could work, but wouldn't solve the complexity of hosting caused by Ruby runtime and deps. That also doesn't exist to my knowledge.
- kstrauser 4y agoI don’t have the bandwidth to contribute code to anything right now, but I hope one of those makes progress. GoToSocial looks promising, especially with their plan to be able to import (or possibly even run on top of) existing Mastodon databases.
- rakoo 4y agoI use pleroma, written in Elixir and hosting everything properly inside pg, for a personal instance and it just works. No magic knobs to turn, no significant delays.
- tinus_hn 4y agoTwitter spent a billion a year running their service, surely a lot of that can be shaved off but I think it’s a bit simplistic to think people can just run a comparable service for free.
- kaba0 4y agoWhat’s the state of TruffleRuby? Last time I checked it was impressively fast. I know that it is really hard to port everything (and sometimes minute details can change behavior), but nowadays Truffle can even execute C code, so those FFI parts can also do as is.