5 ms·

>Then there is stuff like outbox, which contains past messages, but for instance, Mastodon does not fetch old messages at all It's infuriating how little suppo

>Then there is stuff like outbox, which contains past messages, but for instance, Mastodon does not fetch old messages at all It's infuriating how little support activitypub outboxes have. It seems like the perfect way to make small instances work well.

mariusor 3y ago
I think the justification Eugen (Mastodon's main developer) gave for not loading a user's past activities is that it would actually harm smaller instances when a user goes viral and the whole network hammers their servers for their content. I am not convinced of the reasoning, and, in my opinion it's actually working against the idea of "federation", and even against the whole paradigm the internet is built upon: as a UserAgent you should retrieve content when the user requests it.
- worrycue 3y ago
  > I think the justification Eugen (Mastodon's main developer) gave for not loading a user's past activities is that it would actually harm smaller instances when a user goes viral and the whole network hammers their servers for their content. Maybe we need some kind of protocol that can spread the load where popular messages are duplicated across the network instead of being stuck on one node that can get overwhelmed.
  - ComodoHacker 3y ago
    Maybe we don't need to value virality so much.
    - mariusor 3y ago
      I doubt "valuing virality" is the real problem. But when you're exposed to a universal network whose numbers increase every day, there are moments when it happens no matter if you want it or not. The "slashdot effect" has been coined maybe 25 years ago, so it's obviously not a new thing. :P
    - charcircuit 3y ago
      Virality will happen regardless of your intentions. Your systems need to be designed to handle it.
  - konart 3y ago
    BitTorrent?
    - JamesLeonis 3y ago
      Not the hero we deserved but the hero we needed. The irony and tragedy of the Internet is BitTorrent solved Internet Scale at the protocol level, but is at odds with how content companies, ISPs, and Silicon Valley want to organize themselves.
      - mariusor 3y ago
        It's also at odds with how people consume content. If you cram every social interaction a network has in a massive common distributed store (which is what using a bittorrent like protocol would entail) you have every node requiring to consume gigabytes over gigabytes of content, despite being interested in a fraction of a fraction of it. I don't know if nostr or scuttlebutt resolve this issue, but if they don't ... how do you imagine a mobile app for keeping in touch with your grandma, requiring gigabytes of data transfer and storage would work? At this point people saying bittorrent is a solution for social networks are basically just the guys who know about hammers and seeing nothing but nails.
        JamesLeonis 3y ago
        I'm advocating it as the content distribution layer, not the social interaction layer. You don't even need all of the metadata, as there are BEP extensions to distribute that as an index [0] or an RSS-style feed [1]. Images are particularly painful, like the previews in Mastodon, because they hammer the origin server rather than looking around at what other peers have. This problem will only get worse as the federation grows in size, both in users and in servers. I say irony because we had all these scaling problems two decades ago during the peak P2P years, including federation and implicit DDoSing, and BitTorrent emerged as the lasting protocol because it could handle the scaling issues. I say tragedy because for this to work you do need some openness, which puts people at risk of content companies trolling for you, ISPs dropping you, and is completely antithetical to the control Silicon Valley demands over its users and data. [0]: http://bittorrent.org/beps/bep_0049.html http://bittorrent.org/beps/bep_0049.html [1]: http://bittorrent.org/beps/bep_0039.html http://bittorrent.org/beps/bep_0039.html
  - mariusor 3y ago
    Since ActivityPub is mostly composed (75%) of immutable json payloads, they are very suitable for caching. The existing infrastructure should be more than enough.
  - still_grokking 3y ago
    > Maybe we need some kind of protocol that can spread the load where popular messages are duplicated across the network instead of being stuck on one node that can get overwhelmed. You mean like any proper P2P protocol? Add to this multi-cast on internet level, and everybody could actually operate something like YouTube on their internet router… But of course economic interests of some (already ultra rich) minority will prevent that forever, even we could have it just right now form the technical standpoint.
- tedunangst 3y ago
  Some day computers will be fast enough to serve hundreds of bytes of json many times per second.
  - mariusor 3y ago
    They are already if you architect your service for the real purpose, which is to store and process ActivityPub data. Mastodon however is a CRUD application that has its ActivityPub bolted on a totally different data paradigm. It takes a lot of computing to make it run fast. I created an ActivityPub service which (coincidentally) uses a similar storage idea as HN - raw files on disk - and it runs orders of magnitude faster than a similar sized Mastodon server would. Sadly it's not as fast as HN, but I'm getting it there. :D
    - RobotToaster 3y ago
      I think the person you are replying to was being sarcastic.
      - vxNsr 3y ago
        they were, but the point was that mastadon is incredibly inefficient and this guy is making a more efficient version
    - vxNsr 3y ago
      I see your profile is full of a lot of the info about this, do you have a more detailed write-up expanding on how this all works together?
      - mariusor 3y ago
        There is some documentation about various things[1], but I'm not entirely sure what you mean about "how it all works together". [1] https://man.sr.ht/~mariusor/go-activitypub/ https://man.sr.ht/~mariusor/go-activitypub/
- jameson71 3y ago
  I don’t understand why other instances wouldn’t cache the older content and periodically look for updates

mariusor 3y ago

I think the justification Eugen (Mastodon's main developer) gave for not loading a user's past activities is that it would actually harm smaller instances when a user goes viral and the whole network hammers their servers for their content. I am not convinced of the reasoning, and, in my opinion it's actually working against the idea of "federation", and even against the whole paradigm the internet is built upon: as a UserAgent you should retrieve content when the user requests it.

worrycue 3y ago
> I think the justification Eugen (Mastodon's main developer) gave for not loading a user's past activities is that it would actually harm smaller instances when a user goes viral and the whole network hammers their servers for their content. Maybe we need some kind of protocol that can spread the load where popular messages are duplicated across the network instead of being stuck on one node that can get overwhelmed.
- ComodoHacker 3y ago
  Maybe we don't need to value virality so much.
  - mariusor 3y ago
    I doubt "valuing virality" is the real problem. But when you're exposed to a universal network whose numbers increase every day, there are moments when it happens no matter if you want it or not. The "slashdot effect" has been coined maybe 25 years ago, so it's obviously not a new thing. :P
  - charcircuit 3y ago
    Virality will happen regardless of your intentions. Your systems need to be designed to handle it.
- konart 3y ago
  BitTorrent?
  - JamesLeonis 3y ago
    Not the hero we deserved but the hero we needed. The irony and tragedy of the Internet is BitTorrent solved Internet Scale at the protocol level, but is at odds with how content companies, ISPs, and Silicon Valley want to organize themselves.
    - mariusor 3y ago
      It's also at odds with how people consume content. If you cram every social interaction a network has in a massive common distributed store (which is what using a bittorrent like protocol would entail) you have every node requiring to consume gigabytes over gigabytes of content, despite being interested in a fraction of a fraction of it. I don't know if nostr or scuttlebutt resolve this issue, but if they don't ... how do you imagine a mobile app for keeping in touch with your grandma, requiring gigabytes of data transfer and storage would work? At this point people saying bittorrent is a solution for social networks are basically just the guys who know about hammers and seeing nothing but nails.
      - JamesLeonis 3y ago
        I'm advocating it as the content distribution layer, not the social interaction layer. You don't even need all of the metadata, as there are BEP extensions to distribute that as an index [0] or an RSS-style feed [1]. Images are particularly painful, like the previews in Mastodon, because they hammer the origin server rather than looking around at what other peers have. This problem will only get worse as the federation grows in size, both in users and in servers. I say irony because we had all these scaling problems two decades ago during the peak P2P years, including federation and implicit DDoSing, and BitTorrent emerged as the lasting protocol because it could handle the scaling issues. I say tragedy because for this to work you do need some openness, which puts people at risk of content companies trolling for you, ISPs dropping you, and is completely antithetical to the control Silicon Valley demands over its users and data. [0]: http://bittorrent.org/beps/bep_0049.html http://bittorrent.org/beps/bep_0049.html [1]: http://bittorrent.org/beps/bep_0039.html http://bittorrent.org/beps/bep_0039.html
- mariusor 3y ago
  Since ActivityPub is mostly composed (75%) of immutable json payloads, they are very suitable for caching. The existing infrastructure should be more than enough.
- still_grokking 3y ago
  > Maybe we need some kind of protocol that can spread the load where popular messages are duplicated across the network instead of being stuck on one node that can get overwhelmed. You mean like any proper P2P protocol? Add to this multi-cast on internet level, and everybody could actually operate something like YouTube on their internet router… But of course economic interests of some (already ultra rich) minority will prevent that forever, even we could have it just right now form the technical standpoint.
tedunangst 3y ago
Some day computers will be fast enough to serve hundreds of bytes of json many times per second.
- mariusor 3y ago
  They are already if you architect your service for the real purpose, which is to store and process ActivityPub data. Mastodon however is a CRUD application that has its ActivityPub bolted on a totally different data paradigm. It takes a lot of computing to make it run fast. I created an ActivityPub service which (coincidentally) uses a similar storage idea as HN - raw files on disk - and it runs orders of magnitude faster than a similar sized Mastodon server would. Sadly it's not as fast as HN, but I'm getting it there. :D
  - RobotToaster 3y ago
    I think the person you are replying to was being sarcastic.
    - vxNsr 3y ago
      they were, but the point was that mastadon is incredibly inefficient and this guy is making a more efficient version
  - vxNsr 3y ago
    I see your profile is full of a lot of the info about this, do you have a more detailed write-up expanding on how this all works together?
    - mariusor 3y ago
      There is some documentation about various things[1], but I'm not entirely sure what you mean about "how it all works together". [1] https://man.sr.ht/~mariusor/go-activitypub/ https://man.sr.ht/~mariusor/go-activitypub/
jameson71 3y ago
I don’t understand why other instances wouldn’t cache the older content and periodically look for updates