11 ms·

Scaling Mastodon in the face of an exodus

213 points by NoraCodes 4y ago

Havoc 4y ago
Feels a bit like mastodon needs to come with load testing tools? If only to give smaller admins a quantifiable sense of what level their instance turns into a flaming ball
- capableweb 4y ago
  Or Mastodon admins needs to publish a bit more numbers about what kind of instances they run on what kind of hardware, so people know what to expect at what stages.
  - NegativeK 4y ago
    That's coming. The infosec.exchange admin has been tooting what's going on to some extent, but delayed until now the full documentation in favor of gaining headroom for the incoming growth.
  - WorldMaker 4y ago
    Also, Mastodon needs a better handle on its "range" of scale: a lot of the core contributors of Mastodon get focused on the performance of the two massive "flagship" instances (mastodon.social/mastodon.online). Those have far different scaling needs than most of the smaller instances and sometimes the admin work needed for smaller instances gets overlooked because the "flagship" scale handles it just fine.
- wmf 4y ago
  Mastodon should probably throttle signups by default so new users can't pile on to servers.
  - sandyarmstrong 4y ago
    You understand there is no central authority for this, though, right? Many instances are growing, and account creation is a per-instance concept. The instance in the article is relatively small but it's impacted by activity coming from other, larger instances.
  - BitPirate 4y ago
    I would rather encourage migration and let the admin provide links to similar instances.
  - egypturnash 4y ago
    This has happened. Lots of servers, including the ones run by the Mastodon project, are now closed to new users. If you want an account on a Mastodon server then you pretty much need to set one up yourself, or know someone crazy enough to think joining the Holy Order Of Social Plumbers sounds like a good thing.
mdaniel 4y ago
So this one is a fork, and infosec.exchange is a fork ( https://github.com/glitch-soc/mastodon https://github.com/glitch-soc/mastodon ); are the mastodon committers opposed to incorporating those changes, or this fork situation is "as it should be" while these experiments run their course? I'm all for experimentation, but since "the exodus" I've become acutely aware of how many instances are out there advertising their version number[1], so the first big rails (or mastodon itself) vuln is going to be some ooooonooooz 1: although in this case, it says a version but the "Source Code" link points to the wrong repo, so .. security by obscurity I guess?
- huimang 4y ago
  I'm not sure about Glitch, but Hometown has several different features [1] that Mastodon may never opt to include as it changes the behavior a bit. 1: https://github.com/hometown-fork/hometown/wiki https://github.com/hometown-fork/hometown/wiki
- sandyarmstrong 4y ago
  "By our best understanding, our major changes are not wanted by the Mastodon project, hence maintaining this fork instead of trying to commit the changes to Mastodon." https://github.com/hometown-fork/hometown#readme https://github.com/hometown-fork/hometown#readme
- jdp23 4y ago
  Mastodon lead developer has resisted including some of the glitch-soc improvements since 2017 -- like "local-only posts", which are valuable both from a privacy and anti-harassment (and are also in Hometown).
- Kye 4y ago
  I've gotten the impression Eugen has a very clear idea of what he wants Mastodon to be and isn't shy about rejecting features that he doesn't think serve that goal. Those forks are the model of good open source. Other people have strong ideas about how it should be, couldn't convince the main project's owner, and moved on to create their own version that suits their vision. It's great. This is how it should be. And it's a huge improvement over open source projects that languish for years for lack of strong leadership with no one forking it because there's no one to say no to features. As long as the possibility exists, people are wary of forking. Remove the possibility, and people can feel confident doing their own take. As for Mastodon, as far as I know all the major forks are maintained by experienced developers who incorporate security updates.
huimang 4y ago
I'll use this opportunity to mention Pleroma [1], my preferred alternative to Mastodon & others. It's much more lightweight than Mastodon. 1: https://pleroma.social/ https://pleroma.social/
- capableweb 4y ago
  That's neat. Do you happen to know if it's possible to migrate from Mastodon to Pleroma without having to recreate the account and setting up redirects? I already host my own Mastodon instance since a few years back, and don't want to do another redirect if I can help it.
  - huimang 4y ago
    To my knowledge you would have to create a new account and set up redirects, since it's a new server. Migrating is definitely a pain point in the fediverse. That's why I ended up going with a single-user instance that I selfhost.
    - capableweb 4y ago
      Yeah, I mean migrating the software of a single-user instance, not migrating between instances, sorry if that was unclear. So I currently run a single-user instance of Mastodon. Wondering if it's possible to start running a single-user instance of Pleroma instead, without having to change the username+domain?
      - jsmith45 4y ago
        My understanding is in theory yes. But in practice things might be more complicated. Like merely keeping the same name and domain is easy enough. Just shut down the old system, start the new one, create an account with the same name. Of course you would need to refollow everyone. I'm not sure what happens to your followers. Your new server will not know they are followers, so unless this triggers mastodon to try to automatically refollow, I think they are left in a half-broken state. You definitely lose all your old content. All in all, this is not really a great option. For a full migration where in theory everything works, you could try to follow the linked process to migrate to Pleroma 2.0.5, and then obviously just upgrade to latest following plemora documentation for upgrading. description of migration process: https://icyphox.sh/blog/mastodon-to-pleroma/ https://icyphox.sh/blog/mastodon-to-pleroma/ (This is based on the following migrator: https://gitlab.com/soapbox-pub/migrator https://gitlab.com/soapbox-pub/migrator). I have no idea if it works with newer mastodon minor releases.
deleted 4y ago
[deleted]
adultSwim 4y ago
These technical solutions are great, but the biggest need is still organizational. We need infrastructure with transparent open governance, a la Debian. Is anyone aware of an org that hosts an instance but is also putting work into the governance side?
- gnramires 4y ago
  From some non-exhaustive research, there are many: social.coop, chaos.social, social.tchncs.de, nerdica.de (friendica) -- seem to have governance. I believe, mastodon.xyz, fosstodon.org among others have transparent donations. (I believe mastodon.xyz uses the default source) I agree that you should seek more open governance, at least transparent finances and active open source, (technically that's covered by AGPL), because otherwise the server could go at any time, and it violates the spirit of federation in the first place. Here's a nice transparency report: https://hub.fosstodon.org/about/ https://hub.fosstodon.org/about/
squidbeak 4y ago
The most likely outcome here is that Twitter stabilises somehow, Mastodon increases its name recognition and even usage, but that its incoming migrants eventually skulk back to Twitter for its reach.
- Ambolia 4y ago
  Twitter seems to have peaked on usage the last days, not decreased, despite the migrations.
  - pornel 4y ago
    Twitter is very entertaining right now with everyone dunking on The Main Character and making cheap ($8) jokes. But at the same time I think it's eroding Twitter's seriousness, and soon Elon's Twitter may be as uncool as Zuck's Facebook.
    - Ambolia 4y ago
      I think Twitter has two big killer features: - Get quick press releases (from the real press, but also from other sources) - Be able to, let's call it politely, "interact" with politians and celebrities. If too many celebrities leave feature 2 could be killed, it remains to be seen if increased openness in feature 1 would be enough to compensate.
      - kzrdude 4y ago
        Corps use it for a clean and unobtrusive public precense. But I'm not sure we lose anything if that stops.
        deleted 4y ago
        [deleted]
    - gilmore606 4y ago
      > eroding Twitter's seriousness Oh no!
standardUser 4y ago
I was extremely bummed about their sign-up process. Mastodon doesn't have a snowflakes chance in hell of becoming the de facto Twitter replacement if they don't change that process to a normal, near-instant sign up process that users have come to expect from every other service. Make the "choosing server" nonsense come after account creation. Ffs. EDIT: Geez, I get it, there are a lot of Mastodon fanboys that have zero interest in the success of Mastodon as a Twitter replacement. Which is fine, since it never will be with it's current onboarding process.
- LukeHoersten 4y ago
  Email did alright with users choosing servers.
  - standardUser 4y ago
    Not sure what you are talking about. I can sign up for an email right now with a dozen different services and not one will make me complete a mysterious task like "choosing a server".
    - swidi 4y ago
      Those dozen different services are servers, that you chose from.
      - standardUser 4y ago
        Right, and that is atrocious UX that will put a hard cap on how popular Mastodon can become.
        pwinnski 4y ago
        Agreed! I, for one, am enjoying the small-town vibe of my chosen mastodon service provider, and I'm delighted that it can never become "to big to fail." You keep saying "users" when you should probably just say "I," but maybe you understand it a bit better at this point. The onboarding UX is atrocious, mostly because people want it to be something it's not, and partly because developers often don't do UX well. But the UX isn't going to change, so focusing on helping people understand how and why it works the way it does seems more important than cursing the design. You're welcome to stick with Twitter, Inc, if you want a single oligarch to control your chosen microblogging service and everything about it. Or you can venture out and pick a server that supports the mastodon service instead. And who knows? Maybe someday a big company will grow a giant mastodon server so that something like 30% of all mastodon traffic happens there, just as with Gmail and email traffic[0] today. 0. https://techjury.net/blog/gmail-statistics/ https://techjury.net/blog/gmail-statistics/
andix 4y ago
Shouldn’t you use a postgres connection pooler like pgbounce for such a scenario? AFAIK it can be switched to transaction mode, which allows to have way more connections from clients to pgbounce than available postgres connections.
dewey 4y ago
For those interested, the author of Sidekiq also gave some hints on Reddit where this link was posted: https://old.reddit.com/r/rails/comments/yse6no/_/ivz1clf https://old.reddit.com/r/rails/comments/yse6no/_/ivz1clf
killingtime74 4y ago
I work on sharding at a bigcorp. Why not simply limit sign ups to a single Mastodon server which is then considered a shard in The overall fediverse. Stops one/some servers from becoming to powerful too
- mdaniel 4y ago
  Well, according to that theory all servers are fungible, but having read over quite a few of the Mastodon /about pages over the past few days, they are for sure not. The audience and sometimes the rules vary wildly Also, while yes federated etc etc, I for sure get a lot more enjoyment from the "Local Timeline" in my server than trying to keep up with the absolute firehose of "Federated Timeline" which is another reason to pick one where the local content would be interesting
- OctopusLupid 4y ago
  That is indeed usually what happens if the admin doesn't want to deal with a big userbase. I run an instance and I intend to close it off if it reaches about 100 monthly active users.

213 points by NoraCodes 4y ago

Feels a bit like mastodon needs to come with load testing tools? If only to give smaller admins a quantifiable sense of what level their instance turns into a flaming ball

capableweb 4y ago
Or Mastodon admins needs to publish a bit more numbers about what kind of instances they run on what kind of hardware, so people know what to expect at what stages.
- NegativeK 4y ago
  That's coming. The infosec.exchange admin has been tooting what's going on to some extent, but delayed until now the full documentation in favor of gaining headroom for the incoming growth.
- WorldMaker 4y ago
  Also, Mastodon needs a better handle on its "range" of scale: a lot of the core contributors of Mastodon get focused on the performance of the two massive "flagship" instances (mastodon.social/mastodon.online). Those have far different scaling needs than most of the smaller instances and sometimes the admin work needed for smaller instances gets overlooked because the "flagship" scale handles it just fine.
wmf 4y ago
Mastodon should probably throttle signups by default so new users can't pile on to servers.
- sandyarmstrong 4y ago
  You understand there is no central authority for this, though, right? Many instances are growing, and account creation is a per-instance concept. The instance in the article is relatively small but it's impacted by activity coming from other, larger instances.
- BitPirate 4y ago
  I would rather encourage migration and let the admin provide links to similar instances.
- egypturnash 4y ago
  This has happened. Lots of servers, including the ones run by the Mastodon project, are now closed to new users. If you want an account on a Mastodon server then you pretty much need to set one up yourself, or know someone crazy enough to think joining the Holy Order Of Social Plumbers sounds like a good thing.

mdaniel 4y ago

So this one is a fork, and infosec.exchange is a fork ( https://github.com/glitch-soc/mastodon https://github.com/glitch-soc/mastodon ); are the mastodon committers opposed to incorporating those changes, or this fork situation is "as it should be" while these experiments run their course? I'm all for experimentation, but since "the exodus" I've become acutely aware of how many instances are out there advertising their version number[1], so the first big rails (or mastodon itself) vuln is going to be some ooooonooooz 1: although in this case, it says a version but the "Source Code" link points to the wrong repo, so .. security by obscurity I guess?

huimang 4y ago
I'm not sure about Glitch, but Hometown has several different features [1] that Mastodon may never opt to include as it changes the behavior a bit. 1: https://github.com/hometown-fork/hometown/wiki https://github.com/hometown-fork/hometown/wiki
sandyarmstrong 4y ago
"By our best understanding, our major changes are not wanted by the Mastodon project, hence maintaining this fork instead of trying to commit the changes to Mastodon." https://github.com/hometown-fork/hometown#readme https://github.com/hometown-fork/hometown#readme
jdp23 4y ago
Mastodon lead developer has resisted including some of the glitch-soc improvements since 2017 -- like "local-only posts", which are valuable both from a privacy and anti-harassment (and are also in Hometown).
Kye 4y ago
I've gotten the impression Eugen has a very clear idea of what he wants Mastodon to be and isn't shy about rejecting features that he doesn't think serve that goal. Those forks are the model of good open source. Other people have strong ideas about how it should be, couldn't convince the main project's owner, and moved on to create their own version that suits their vision. It's great. This is how it should be. And it's a huge improvement over open source projects that languish for years for lack of strong leadership with no one forking it because there's no one to say no to features. As long as the possibility exists, people are wary of forking. Remove the possibility, and people can feel confident doing their own take. As for Mastodon, as far as I know all the major forks are maintained by experienced developers who incorporate security updates.

huimang 4y ago

I'll use this opportunity to mention Pleroma [1], my preferred alternative to Mastodon & others. It's much more lightweight than Mastodon. 1: https://pleroma.social/ https://pleroma.social/

capableweb 4y ago
That's neat. Do you happen to know if it's possible to migrate from Mastodon to Pleroma without having to recreate the account and setting up redirects? I already host my own Mastodon instance since a few years back, and don't want to do another redirect if I can help it.
- huimang 4y ago
  To my knowledge you would have to create a new account and set up redirects, since it's a new server. Migrating is definitely a pain point in the fediverse. That's why I ended up going with a single-user instance that I selfhost.
  - capableweb 4y ago
    Yeah, I mean migrating the software of a single-user instance, not migrating between instances, sorry if that was unclear. So I currently run a single-user instance of Mastodon. Wondering if it's possible to start running a single-user instance of Pleroma instead, without having to change the username+domain?
    - jsmith45 4y ago
      My understanding is in theory yes. But in practice things might be more complicated. Like merely keeping the same name and domain is easy enough. Just shut down the old system, start the new one, create an account with the same name. Of course you would need to refollow everyone. I'm not sure what happens to your followers. Your new server will not know they are followers, so unless this triggers mastodon to try to automatically refollow, I think they are left in a half-broken state. You definitely lose all your old content. All in all, this is not really a great option. For a full migration where in theory everything works, you could try to follow the linked process to migrate to Pleroma 2.0.5, and then obviously just upgrade to latest following plemora documentation for upgrading. description of migration process: https://icyphox.sh/blog/mastodon-to-pleroma/ https://icyphox.sh/blog/mastodon-to-pleroma/ (This is based on the following migrator: https://gitlab.com/soapbox-pub/migrator https://gitlab.com/soapbox-pub/migrator). I have no idea if it works with newer mastodon minor releases.

deleted 4y ago

[deleted]

adultSwim 4y ago

These technical solutions are great, but the biggest need is still organizational. We need infrastructure with transparent open governance, a la Debian. Is anyone aware of an org that hosts an instance but is also putting work into the governance side?

gnramires 4y ago
From some non-exhaustive research, there are many: social.coop, chaos.social, social.tchncs.de, nerdica.de (friendica) -- seem to have governance. I believe, mastodon.xyz, fosstodon.org among others have transparent donations. (I believe mastodon.xyz uses the default source) I agree that you should seek more open governance, at least transparent finances and active open source, (technically that's covered by AGPL), because otherwise the server could go at any time, and it violates the spirit of federation in the first place. Here's a nice transparency report: https://hub.fosstodon.org/about/ https://hub.fosstodon.org/about/

squidbeak 4y ago

The most likely outcome here is that Twitter stabilises somehow, Mastodon increases its name recognition and even usage, but that its incoming migrants eventually skulk back to Twitter for its reach.

Ambolia 4y ago
Twitter seems to have peaked on usage the last days, not decreased, despite the migrations.
- pornel 4y ago
  Twitter is very entertaining right now with everyone dunking on The Main Character and making cheap ($8) jokes. But at the same time I think it's eroding Twitter's seriousness, and soon Elon's Twitter may be as uncool as Zuck's Facebook.
  - Ambolia 4y ago
    I think Twitter has two big killer features: - Get quick press releases (from the real press, but also from other sources) - Be able to, let's call it politely, "interact" with politians and celebrities. If too many celebrities leave feature 2 could be killed, it remains to be seen if increased openness in feature 1 would be enough to compensate.
    - kzrdude 4y ago
      Corps use it for a clean and unobtrusive public precense. But I'm not sure we lose anything if that stops.
      - deleted 4y ago
        [deleted]
  - gilmore606 4y ago
    > eroding Twitter's seriousness Oh no!

standardUser 4y ago

I was extremely bummed about their sign-up process. Mastodon doesn't have a snowflakes chance in hell of becoming the de facto Twitter replacement if they don't change that process to a normal, near-instant sign up process that users have come to expect from every other service. Make the "choosing server" nonsense come after account creation. Ffs. EDIT: Geez, I get it, there are a lot of Mastodon fanboys that have zero interest in the success of Mastodon as a Twitter replacement. Which is fine, since it never will be with it's current onboarding process.

LukeHoersten 4y ago
Email did alright with users choosing servers.
- standardUser 4y ago
  Not sure what you are talking about. I can sign up for an email right now with a dozen different services and not one will make me complete a mysterious task like "choosing a server".
  - swidi 4y ago
    Those dozen different services are servers, that you chose from.
    - standardUser 4y ago
      Right, and that is atrocious UX that will put a hard cap on how popular Mastodon can become.
      - pwinnski 4y ago
        Agreed! I, for one, am enjoying the small-town vibe of my chosen mastodon service provider, and I'm delighted that it can never become "to big to fail." You keep saying "users" when you should probably just say "I," but maybe you understand it a bit better at this point. The onboarding UX is atrocious, mostly because people want it to be something it's not, and partly because developers often don't do UX well. But the UX isn't going to change, so focusing on helping people understand how and why it works the way it does seems more important than cursing the design. You're welcome to stick with Twitter, Inc, if you want a single oligarch to control your chosen microblogging service and everything about it. Or you can venture out and pick a server that supports the mastodon service instead. And who knows? Maybe someday a big company will grow a giant mastodon server so that something like 30% of all mastodon traffic happens there, just as with Gmail and email traffic[0] today. 0. https://techjury.net/blog/gmail-statistics/ https://techjury.net/blog/gmail-statistics/

andix 4y ago

Shouldn’t you use a postgres connection pooler like pgbounce for such a scenario? AFAIK it can be switched to transaction mode, which allows to have way more connections from clients to pgbounce than available postgres connections.

dewey 4y ago

For those interested, the author of Sidekiq also gave some hints on Reddit where this link was posted: https://old.reddit.com/r/rails/comments/yse6no/_/ivz1clf https://old.reddit.com/r/rails/comments/yse6no/_/ivz1clf

killingtime74 4y ago

I work on sharding at a bigcorp. Why not simply limit sign ups to a single Mastodon server which is then considered a shard in The overall fediverse. Stops one/some servers from becoming to powerful too

mdaniel 4y ago
Well, according to that theory all servers are fungible, but having read over quite a few of the Mastodon /about pages over the past few days, they are for sure not. The audience and sometimes the rules vary wildly Also, while yes federated etc etc, I for sure get a lot more enjoyment from the "Local Timeline" in my server than trying to keep up with the absolute firehose of "Federated Timeline" which is another reason to pick one where the local content would be interesting
OctopusLupid 4y ago
That is indeed usually what happens if the admin doesn't want to deal with a big userbase. I run an instance and I intend to close it off if it reaches about 100 monthly active users.