3 ms·

Can someone explain how is it planning to achieve the federation of that massive amount of data and storage ?

throawayonthe 2y ago
[dead]
arccy 2y ago
I believe the plan is to not have a plan (right now): https://mastodon.social/@dansup/113887622931474663 https://mastodon.social/@dansup/113887622931474663
- boriskourt 2y ago
  There is now also: https://www.kickstarter.com/projects/pixelfed/pixelfed-foundation-2024-real-ethical-social-networks https://www.kickstarter.com/projects/pixelfed/pixelfed-found...
mg 2y ago
Isn't Pixelfed using the ActivityPub protocol? As far as I know, that makes it scale like email. When you post something, your host sends it to all recipients - everyone who subscribed to your posts. That does not sound like a lot of traffic. Even if one has 10k followers, as the post would only go once to each of their hosts. So if the 10k followers are on 100 hosts, that means 100 messages.
- hexagonwin 2y ago
  It is indeed using ActivityPub but most people are on the main instance and there's not much federation afaik.
  - aloisdg 2y ago
    right now. Let's make it grow
  - mg 2y ago
    Well, sure - if someone wanted to be the hoster for all mail traffic on planet earth, then they would need a lot of resources and a business model to finance it. But if that is their plan, why would they have used ActivityPub in the first place? So I guess they could just shut their door as soon as the traffic is too much for them to handle. And tell people to look for another host or host themselfes.
  - cship2 2y ago
    What is needed is an index for all the instances that are willing to be published. All server instances would have access to the public index via some gossip protocol. Also are there any other server instances other than the php server? Seems like they ought have used the matrix protcal instead of active pub. More server implementations in different langs, and it has e2e build I. Don't think actpub has that.
treyd 2y ago
In addition to the other points, there was some discussion about more directly p2p exchange of content (like WebTorrent or something) to share the load more evenly. Spritely is a more methodical approach to it but it could be layered into ActivityPub as an extension without changes to the core protocol.
- evbogue 2y ago
  Pixelfed could make a hash of every blob uploaded and then store that hash in the ActivityPub message. Then the clients could peer using WebTorrent or Trystero and randomly ask peers for that hash until it discovers a peer that has already cached the image. This would reduce load on dansup's server within his own ecosystem anyway.
boramalper 2y ago
> the federation of that massive amount of data and storage I'm not sure what you mean by that but the media blobs (photos and videos) are not "federated" (i.e. passed around instances) most likely, but hosted in one place (the instance of the author) and referenced by their URL.
- cess11 2y ago
  I was under the impression that federated nodes acts as caches. Is this not the case?
- refulgentis 2y ago
  Intuitively, it is unlikely there's 500K individual servers set up. We can then also observe other comments clarifying that no, there aren't 500K instances. The other comments provide...tweets? mastodons?...from the maintainer also clarify that in practice, there's 1 instance. People are questioning how that will scale, and the tweet from the maintainer was cited as part of that because the content of the tweet is that tl;dr there's a $2,600 month gap between Patreon and hosting costs.
- ruined 2y ago
  it varies by activitypub implementation. mastodon for example caches media per instance, pleroma simply hotlinks. briefly searching through github issues, i believe pixelfed does not cache remote media. discussion on this issue about remote media cacheing seems to indicate that pixelfed only caches avatars https://github.com/pixelfed/pixelfed/issues/4571 https://github.com/pixelfed/pixelfed/issues/4571
isodev 2y ago
Pixelfed works like any other Fediverse server. Content is stored on the node where it’s posted by its author. The nodes of people following the account have the option to cache a copy of the posts or just keep a reference depending on their implementation but Mastodon for example would cache it.
deleted 2y ago
[deleted]
buyucu 2y ago
Mastodon seems to be doing pretty good so far on scaling up. Pixelfed is smaller than Mastodon, so it will be fine.
- arccy 2y ago
  mastodon is text, pixelfed is images/videos
  - captainbland 2y ago
    As I understand it activitypub itself shouldn't matter there as the only thing going through it should be URL references to the image/video data which you can front with a CDN. Federating instances don't have to host each others' data in that context.

[dead]

I believe the plan is to not have a plan (right now): https://mastodon.social/@dansup/113887622931474663 https://mastodon.social/@dansup/113887622931474663

boriskourt 2y ago
There is now also: https://www.kickstarter.com/projects/pixelfed/pixelfed-foundation-2024-real-ethical-social-networks https://www.kickstarter.com/projects/pixelfed/pixelfed-found...

mg 2y ago

Isn't Pixelfed using the ActivityPub protocol? As far as I know, that makes it scale like email. When you post something, your host sends it to all recipients - everyone who subscribed to your posts. That does not sound like a lot of traffic. Even if one has 10k followers, as the post would only go once to each of their hosts. So if the 10k followers are on 100 hosts, that means 100 messages.

hexagonwin 2y ago
It is indeed using ActivityPub but most people are on the main instance and there's not much federation afaik.
- aloisdg 2y ago
  right now. Let's make it grow
- mg 2y ago
  Well, sure - if someone wanted to be the hoster for all mail traffic on planet earth, then they would need a lot of resources and a business model to finance it. But if that is their plan, why would they have used ActivityPub in the first place? So I guess they could just shut their door as soon as the traffic is too much for them to handle. And tell people to look for another host or host themselfes.
- cship2 2y ago
  What is needed is an index for all the instances that are willing to be published. All server instances would have access to the public index via some gossip protocol. Also are there any other server instances other than the php server? Seems like they ought have used the matrix protcal instead of active pub. More server implementations in different langs, and it has e2e build I. Don't think actpub has that.

treyd 2y ago

In addition to the other points, there was some discussion about more directly p2p exchange of content (like WebTorrent or something) to share the load more evenly. Spritely is a more methodical approach to it but it could be layered into ActivityPub as an extension without changes to the core protocol.

evbogue 2y ago
Pixelfed could make a hash of every blob uploaded and then store that hash in the ActivityPub message. Then the clients could peer using WebTorrent or Trystero and randomly ask peers for that hash until it discovers a peer that has already cached the image. This would reduce load on dansup's server within his own ecosystem anyway.

boramalper 2y ago

> the federation of that massive amount of data and storage I'm not sure what you mean by that but the media blobs (photos and videos) are not "federated" (i.e. passed around instances) most likely, but hosted in one place (the instance of the author) and referenced by their URL.

cess11 2y ago
I was under the impression that federated nodes acts as caches. Is this not the case?
refulgentis 2y ago
Intuitively, it is unlikely there's 500K individual servers set up. We can then also observe other comments clarifying that no, there aren't 500K instances. The other comments provide...tweets? mastodons?...from the maintainer also clarify that in practice, there's 1 instance. People are questioning how that will scale, and the tweet from the maintainer was cited as part of that because the content of the tweet is that tl;dr there's a $2,600 month gap between Patreon and hosting costs.
ruined 2y ago
it varies by activitypub implementation. mastodon for example caches media per instance, pleroma simply hotlinks. briefly searching through github issues, i believe pixelfed does not cache remote media. discussion on this issue about remote media cacheing seems to indicate that pixelfed only caches avatars https://github.com/pixelfed/pixelfed/issues/4571 https://github.com/pixelfed/pixelfed/issues/4571

isodev 2y ago

Pixelfed works like any other Fediverse server. Content is stored on the node where it’s posted by its author. The nodes of people following the account have the option to cache a copy of the posts or just keep a reference depending on their implementation but Mastodon for example would cache it.

deleted 2y ago

[deleted]

buyucu 2y ago

Mastodon seems to be doing pretty good so far on scaling up. Pixelfed is smaller than Mastodon, so it will be fine.

arccy 2y ago
mastodon is text, pixelfed is images/videos
- captainbland 2y ago
  As I understand it activitypub itself shouldn't matter there as the only thing going through it should be URL references to the image/video data which you can front with a CDN. Federating instances don't have to host each others' data in that context.