7 ms·

Saw some comments on the protocol being fluffy and typical implementations resource hungry. This is an interesting guy to follow: https://universeodon.com/@sup

Saw some comments on the protocol being fluffy and typical implementations resource hungry. This is an interesting guy to follow: https://universeodon.com/@supernovae https://universeodon.com/@supernovae He's the admin of universeodon, a mastodon instance with 13K MAU. He recently shared that in a month's time, 3TB of text was transferred just in ActivityPub events. Images a multiple of it. I don't know what the bill is, but I was pretty shocked by the stats...for "just" 13K users. And the cruel thing is that it still doesn't work properly. Likes/boosts and replies do not properly synchronize.

still_grokking 3y ago
But people are saying that the Fediverse could replace Twitter and Facebook and Tiktok and Instagram and what not, don't they? How much hardware would you need for 100 million MAUs? (And that's just a fraction of the current social media users.) If it really "scales" like indicated in the parent post this tech will never provide any alternative to centralized social media sites just for technical reasons no matter what people want or do. Maybe someone experienced in effective distributed systems should start to design an alternative. Otherwise there won't be any viable alternative to the commercial silos no matter how bad people would like one.
- dahwolf 3y ago
  Mastodon absolutely does not scale that far and this should be considered fact. Far smaller instance owners sometimes complain about bills of hundreds of USD per month. So the only way to scale up "indefinitely" is by having many small/medium-sized instances, but not really. In a 100M+ network, instances will suffer due to the wasteful nature of federation plus social media being append-only. Costs will forever go up and this doesn't even mention the burden and liabilities of moderation. Amidst all our hate for big social, we've forgotten about all the things they do very well. They are (financially) free. They are reliable and scale up without you noticing. You do not have to generally worry about your entire account and content being gone because some mod gave up. If you don't do anything funny, all your content is preserved, forever. Moderation works reasonably well, even if never perfect. We've taken all that for granted. But it costs billions, an army of engineers, mods, legal, marketing, UX, top notch infrastructure to make it run and work this smoothly. The idea that a bunch of enthusiasts can replicate this, is misplaced.
  - saurik 3y ago
    > If you don't do anything funny, all your content is preserved, forever. Are you really not able to think of any dead social networks that deleted all of their content? Hell: Google alone has managed to delete at least two!
    - dahwolf 3y ago
      Obviously with forever I mean within the lifespan of the service, not literally forever as in a trillion years. So yes, your 2008 Facebook post is still going to be there.
      - saurik 3y ago
        Yet, my 2011 Google+ post is not still there, so this seems like a crap shoot: your argument is inconsistent at best, but frankly feels more like you've bought into a lie.
  - rektide 3y ago
    > Mastodon absolutely does not scale that far Maybe? > and this should be considered fact. I see absolutely nothing about ActivityPub that is inherently hard to scale. Adding more points of presence & smart fan-out seems like it could keep scaling indefinitely, from what I can see. > Costs will forever go up Let's simplify follower-ship and assume instead a simpler model of bi-directional friending. If you ahve M users each of which have N total friends, the naive fear is this: that costs will keep growing terrifyingly. But I expect it's actually more a sigmoid curve. As you grow into millions and especially many-millions of users, more and more you'll have your people following the same person - which reduces traffic - and more and more of your people followed by multiple people on another server - which reduces traffic. The actual growth curve here is more sigmoidal, with a big variability depending on the number of active instances fediverse instances out there. > You do not have to generally worry about your entire account and content being gone because some mod gave up. To get listed on joinmastadon you have to promise to give at least 3 months heads-up, during which time users have to be able to activate an account transfer to other systems. There was a case where mastodon.au nearly pulled the plug with a short notice, but someone else stepped up to take over the system. The potential for this to be a huge problem is absolutely super real, but in practice, I have almost no real concern over this and think it should be broadly disregarded. The server I was on shut down, but the mod gave us almost a full year of continued service, and another year of read-only service during which we can transfer. Trying to drum up fear & doubt over the instability of this system seems irresponsible & premature, given how well things have gone. > The idea that a bunch of enthusiasts can replicate this, is misplaced. I agree that there are huge advantages & I think we absolutely have taken much for granted. Right now you're looking at this through shit-colored glasses though, and I think it's naive to wish so hard for this all to fail. It's convenient & easy to say it'll never work. But there's so many strategies where we can start to turn the scale into something useful. If servers publish WebBundles of user's feeds, users can p2p distribute the signed http content among themselves, perhaps via WebTorrent. Rather than live in fear of our users and our scaling, we can rely on our users to help us tackle scaling. Yes we can, man. It's totally doable. Get off the ledge.
- robga 3y ago
  > How much hardware would you need for 100 million MAUs? Given an informed yardstick of $.02-.05 per MAU for a 10k instance, possibly $2-5M/month, though much depends on the profile of users per instance. The support, moderation, legal, administration, and corporate costs would dwarf this cloud/hardware cost.
MR4D 3y ago
Would love to see how this compares to git.
xvilka 3y ago
Because they should have designed more compact and less talkative protocol. It was a similar problem with XMPP.
latch 3y ago
3TB for the text of 13K user sound crazy, you're right. But for the bill, strictly speaking about bandwidth, a 10mbps unmetered connection gets you roughly that. And 10mbps is pretty uncommon now because it's so low. So I'd expect the bandwidth bill to essentially be free (i.e. included)
robga 3y ago
Universeodon uses Fastly as a CDN. My masto admin experience is 80-90% CDN caching though obviously that’s images and static content. Bandwidth is a function of the number of connected servers for remote followers. A popular person like George Takei makes a post and 15000 servers ask universeodon for the text. IMHO some sort of smarter fan out or message relay capability would help, much like email infrastructure. Having said that, you can run 10k Mastodon MAU on a $200pm cloud resource budget, though double that adds headroom, a dynamically scalable architecture, a staging instance, elastic search and translation, more admin tooling, etc. Yes, some instances spend multiples of that per 10k but if you budget $400pm you’ll sleep well. (Thing is, a 1k MAU would cost $100+ if you want to start with a scalable footprint). Universedon had 70k MAU after the November surge and AIUI is still easily scalable to 100k+. I don’t know about you, but a cloud running cost of $0.02-$0.10 per MAU sounds cheap to me. All large instances can cover it with donations. The “real” cost is moderation and administration labour. > And the cruel thing is that it still doesn't work properly. Likes/boosts and replies do not properly synchronize. It works exactly as designed. Personally I am fine that on someone else’s post I don’t need to see every reply across the world and fully synced like counts, but I can respect the POV that many users expect just that. This is not an activity pub limitation but rather a software choice by Mastodon. Many users, including @supernovae, push for functional changes around this.

still_grokking 3y ago

But people are saying that the Fediverse could replace Twitter and Facebook and Tiktok and Instagram and what not, don't they? How much hardware would you need for 100 million MAUs? (And that's just a fraction of the current social media users.) If it really "scales" like indicated in the parent post this tech will never provide any alternative to centralized social media sites just for technical reasons no matter what people want or do. Maybe someone experienced in effective distributed systems should start to design an alternative. Otherwise there won't be any viable alternative to the commercial silos no matter how bad people would like one.

dahwolf 3y ago
Mastodon absolutely does not scale that far and this should be considered fact. Far smaller instance owners sometimes complain about bills of hundreds of USD per month. So the only way to scale up "indefinitely" is by having many small/medium-sized instances, but not really. In a 100M+ network, instances will suffer due to the wasteful nature of federation plus social media being append-only. Costs will forever go up and this doesn't even mention the burden and liabilities of moderation. Amidst all our hate for big social, we've forgotten about all the things they do very well. They are (financially) free. They are reliable and scale up without you noticing. You do not have to generally worry about your entire account and content being gone because some mod gave up. If you don't do anything funny, all your content is preserved, forever. Moderation works reasonably well, even if never perfect. We've taken all that for granted. But it costs billions, an army of engineers, mods, legal, marketing, UX, top notch infrastructure to make it run and work this smoothly. The idea that a bunch of enthusiasts can replicate this, is misplaced.
- saurik 3y ago
  > If you don't do anything funny, all your content is preserved, forever. Are you really not able to think of any dead social networks that deleted all of their content? Hell: Google alone has managed to delete at least two!
  - dahwolf 3y ago
    Obviously with forever I mean within the lifespan of the service, not literally forever as in a trillion years. So yes, your 2008 Facebook post is still going to be there.
    - saurik 3y ago
      Yet, my 2011 Google+ post is not still there, so this seems like a crap shoot: your argument is inconsistent at best, but frankly feels more like you've bought into a lie.
- rektide 3y ago
  > Mastodon absolutely does not scale that far Maybe? > and this should be considered fact. I see absolutely nothing about ActivityPub that is inherently hard to scale. Adding more points of presence & smart fan-out seems like it could keep scaling indefinitely, from what I can see. > Costs will forever go up Let's simplify follower-ship and assume instead a simpler model of bi-directional friending. If you ahve M users each of which have N total friends, the naive fear is this: that costs will keep growing terrifyingly. But I expect it's actually more a sigmoid curve. As you grow into millions and especially many-millions of users, more and more you'll have your people following the same person - which reduces traffic - and more and more of your people followed by multiple people on another server - which reduces traffic. The actual growth curve here is more sigmoidal, with a big variability depending on the number of active instances fediverse instances out there. > You do not have to generally worry about your entire account and content being gone because some mod gave up. To get listed on joinmastadon you have to promise to give at least 3 months heads-up, during which time users have to be able to activate an account transfer to other systems. There was a case where mastodon.au nearly pulled the plug with a short notice, but someone else stepped up to take over the system. The potential for this to be a huge problem is absolutely super real, but in practice, I have almost no real concern over this and think it should be broadly disregarded. The server I was on shut down, but the mod gave us almost a full year of continued service, and another year of read-only service during which we can transfer. Trying to drum up fear & doubt over the instability of this system seems irresponsible & premature, given how well things have gone. > The idea that a bunch of enthusiasts can replicate this, is misplaced. I agree that there are huge advantages & I think we absolutely have taken much for granted. Right now you're looking at this through shit-colored glasses though, and I think it's naive to wish so hard for this all to fail. It's convenient & easy to say it'll never work. But there's so many strategies where we can start to turn the scale into something useful. If servers publish WebBundles of user's feeds, users can p2p distribute the signed http content among themselves, perhaps via WebTorrent. Rather than live in fear of our users and our scaling, we can rely on our users to help us tackle scaling. Yes we can, man. It's totally doable. Get off the ledge.
robga 3y ago
> How much hardware would you need for 100 million MAUs? Given an informed yardstick of $.02-.05 per MAU for a 10k instance, possibly $2-5M/month, though much depends on the profile of users per instance. The support, moderation, legal, administration, and corporate costs would dwarf this cloud/hardware cost.

MR4D 3y ago

Would love to see how this compares to git.

xvilka 3y ago

Because they should have designed more compact and less talkative protocol. It was a similar problem with XMPP.

latch 3y ago

3TB for the text of 13K user sound crazy, you're right. But for the bill, strictly speaking about bandwidth, a 10mbps unmetered connection gets you roughly that. And 10mbps is pretty uncommon now because it's so low. So I'd expect the bandwidth bill to essentially be free (i.e. included)

robga 3y ago

Universeodon uses Fastly as a CDN. My masto admin experience is 80-90% CDN caching though obviously that’s images and static content. Bandwidth is a function of the number of connected servers for remote followers. A popular person like George Takei makes a post and 15000 servers ask universeodon for the text. IMHO some sort of smarter fan out or message relay capability would help, much like email infrastructure. Having said that, you can run 10k Mastodon MAU on a $200pm cloud resource budget, though double that adds headroom, a dynamically scalable architecture, a staging instance, elastic search and translation, more admin tooling, etc. Yes, some instances spend multiples of that per 10k but if you budget $400pm you’ll sleep well. (Thing is, a 1k MAU would cost $100+ if you want to start with a scalable footprint). Universedon had 70k MAU after the November surge and AIUI is still easily scalable to 100k+. I don’t know about you, but a cloud running cost of $0.02-$0.10 per MAU sounds cheap to me. All large instances can cover it with donations. The “real” cost is moderation and administration labour. > And the cruel thing is that it still doesn't work properly. Likes/boosts and replies do not properly synchronize. It works exactly as designed. Personally I am fine that on someone else’s post I don’t need to see every reply across the world and fully synced like counts, but I can respect the POV that many users expect just that. This is not an activity pub limitation but rather a software choice by Mastodon. Many users, including @supernovae, push for functional changes around this.