3 ms·

I think there is an simple solution to this problem. It does require some changes, but they seem to be pretty straightforward: - When installing any app, user

by theamk 4y ago

I think there is an simple solution to this problem. It does require some changes, but they seem to be pretty straightforward: - When installing any app, user must always specify an arbitrary prefix. It is applied to all pages (and resources) except toplevel one. So you never have "hostname.com/posts/1234", instead it is "hostname.com/m1/posts/1234" - When switching to a new software, you use web spider/export functionality to convert the entire old prefix to static files. Static files are fast, and you can easily keep 100000 of them on average computer (and if not, you can store them in sqlite or some immutable k-v store). You then expose the static versions, either directly or via some trivial app. - New ActivityPub goes to different prefix, so new posts are "hostname.com/m2/post/1234" - The only thing that _might_ need redirects is users, to get following to work.. Those will need to be set up manually. But there are much fewer users than other objects, so this should be pretty doable. ---- Does this needs some changes to be feasible? yes. Is it easier than designing all-new scheme? Absolutely.

danpalmer 4y ago
This is a good idea that I hadn't fully considered. There are a few issues that make it tricky in the general case though – for existing implementations there may not be a prefix that can be used, and if we're talking about adding prefixes to all new systems, we could just use frameworks that have built-in support for this. Additionally, web scraping isn't perfect. It assumes static content, which may not be possible with ACLs or other behaviour. It also assumes that there's a single link entry point to the content, or that all the necessary link entry points are known. It could also be very wasteful – storing a tweet in a database is much smaller than storing the JSON response for that tweet, or the HTML of the page the tweet is in. This is an interesting idea to consider, and it could be simpler in some cases, but I'm not sure it's a better solution in all cases.
- mariusor 4y ago
  You already commit a big fallacy in my opinion implying that objects need to be related to database access. Once an object has been created on an instance - with very few exceptions - there's no need to tamper with it. Storing it as a JSON on disk is perfectly acceptable.
  - danpalmer 4y ago
    JSON on disk is a perfectly acceptable database format ;) More seriously, I used "database" as a proxy for looking up in storage of some sort. Whether that's JSON files on disk, relational data in a SQL database, or anything in between, is just down to the access, consistency, durability, etc, requirements of the system.
- vidarh 4y ago
  There's nothing that requires the post URLs to even be on the same hostname as the user address. Mine are not. So switching locations for where the content is served up from for a new install is always possible. My Mastodon/Fediverse address is @vidar@galaxybound.com, while my posts are all hosted on m.galaxybound.com, e.g. [1]. In this case this is a result of installing Mastodon on the latter, while the former just redirects the webfinger, but there's AFAIK also nothing preventing you from e.g. the address being @user@domain1, the profile being on domain2, and the canonical URI of the posts being on domain3. In terms of being wasteful, that might be a consideration for large shared hosts, but in reality media storage will outpace it by a large factor for most users. E.g. when I moved my account from mastodon.social, the export had 9.6MB of media I'd posted, and the uncompressed JSON for all my posts took up just 800K. That's despite being someone who posts relatively few pictures and very rarely any video. > Additionally, web scraping isn't perfect. It assumes static content, which may not be possible with ACLs or other behaviour. It also assumes that there's a single link entry point to the content, or that all the necessary link entry points are known. You don't need to scrape, as Mastodon supports export (but that will not be true for every type of fediverse software), but you can scrape as for federation to work your endpoints need to be accessible to all but instances that are defederated. You can also get at every posts trivially by looking up a user in webfinger to find their outbox, retrieve their outbox and retrieve the link to every post. This works for every piece of fediverse software that cares about ensuring federation works. [1] https://m.galaxybound.com/@vidar/109965336245254807 https://m.galaxybound.com/@vidar/109965336245254807
  - danpalmer 4y ago
    It's true there's a disconnection between the account domain and the hosting domain, I already use this, but the Activity Pub object IDs are on the _hosting domain_, and if you want to move that you do need the IDs to continue to resolve in order to not (partially) defederate. As far as I am aware there's no other good way to move that content between software on the same domain. Mastodon will import/export, but that data still assumes it'll be backed by Mastodon code with the Mastodon database schema, which is not correct if you're migrating to or from Mastodon. > but you can scrape as for federation to work your endpoints need to be accessible to all but instances that are defederated ...with Activity Pub. Mostly. This is not true for all linked data, and my post was mostly looking at the general case of linked data rather than sticking to Mastodon or Activity Pub in particular. Activity Pub can be authenticated in some cases, although I think that may be something Mastodon has implemented on top. Not sure.
mariusor 4y ago
> You then expose the static versions, either directly or via some trivial app. Most ActivityPub objects are already static and immutable, the sooner fediverse developers realize this and simplify the code paths between reading a URL and returning the static and immutable object that has that URL as an ID, the better the ecosystem will be. It's sad that everyone insists in wrapping costly database accesses and logic on top of what should be very, very simple logic.
- rakoo 4y ago
  Activities are immutable but objects are not. Mutability allows posts to be edited, for instance. The ids in ActivityPub also aren't URLs, they're random strings. EDIT: They actually are URLs, my mistake
  - danpalmer 4y ago
    Are you sure? From the spec, object IDs must be: > Publicly dereferencable URIs, such as HTTPS URIs, with their authority belonging to that of their originating server (or they can be null in some cases)

by theamk 4y ago

danpalmer 4y ago

This is a good idea that I hadn't fully considered. There are a few issues that make it tricky in the general case though – for existing implementations there may not be a prefix that can be used, and if we're talking about adding prefixes to all new systems, we could just use frameworks that have built-in support for this. Additionally, web scraping isn't perfect. It assumes static content, which may not be possible with ACLs or other behaviour. It also assumes that there's a single link entry point to the content, or that all the necessary link entry points are known. It could also be very wasteful – storing a tweet in a database is much smaller than storing the JSON response for that tweet, or the HTML of the page the tweet is in. This is an interesting idea to consider, and it could be simpler in some cases, but I'm not sure it's a better solution in all cases.

mariusor 4y ago
You already commit a big fallacy in my opinion implying that objects need to be related to database access. Once an object has been created on an instance - with very few exceptions - there's no need to tamper with it. Storing it as a JSON on disk is perfectly acceptable.
- danpalmer 4y ago
  JSON on disk is a perfectly acceptable database format ;) More seriously, I used "database" as a proxy for looking up in storage of some sort. Whether that's JSON files on disk, relational data in a SQL database, or anything in between, is just down to the access, consistency, durability, etc, requirements of the system.
vidarh 4y ago
There's nothing that requires the post URLs to even be on the same hostname as the user address. Mine are not. So switching locations for where the content is served up from for a new install is always possible. My Mastodon/Fediverse address is @vidar@galaxybound.com, while my posts are all hosted on m.galaxybound.com, e.g. [1]. In this case this is a result of installing Mastodon on the latter, while the former just redirects the webfinger, but there's AFAIK also nothing preventing you from e.g. the address being @user@domain1, the profile being on domain2, and the canonical URI of the posts being on domain3. In terms of being wasteful, that might be a consideration for large shared hosts, but in reality media storage will outpace it by a large factor for most users. E.g. when I moved my account from mastodon.social, the export had 9.6MB of media I'd posted, and the uncompressed JSON for all my posts took up just 800K. That's despite being someone who posts relatively few pictures and very rarely any video. > Additionally, web scraping isn't perfect. It assumes static content, which may not be possible with ACLs or other behaviour. It also assumes that there's a single link entry point to the content, or that all the necessary link entry points are known. You don't need to scrape, as Mastodon supports export (but that will not be true for every type of fediverse software), but you can scrape as for federation to work your endpoints need to be accessible to all but instances that are defederated. You can also get at every posts trivially by looking up a user in webfinger to find their outbox, retrieve their outbox and retrieve the link to every post. This works for every piece of fediverse software that cares about ensuring federation works. [1] https://m.galaxybound.com/@vidar/109965336245254807 https://m.galaxybound.com/@vidar/109965336245254807
- danpalmer 4y ago
  It's true there's a disconnection between the account domain and the hosting domain, I already use this, but the Activity Pub object IDs are on the _hosting domain_, and if you want to move that you do need the IDs to continue to resolve in order to not (partially) defederate. As far as I am aware there's no other good way to move that content between software on the same domain. Mastodon will import/export, but that data still assumes it'll be backed by Mastodon code with the Mastodon database schema, which is not correct if you're migrating to or from Mastodon. > but you can scrape as for federation to work your endpoints need to be accessible to all but instances that are defederated ...with Activity Pub. Mostly. This is not true for all linked data, and my post was mostly looking at the general case of linked data rather than sticking to Mastodon or Activity Pub in particular. Activity Pub can be authenticated in some cases, although I think that may be something Mastodon has implemented on top. Not sure.

mariusor 4y ago

> You then expose the static versions, either directly or via some trivial app. Most ActivityPub objects are already static and immutable, the sooner fediverse developers realize this and simplify the code paths between reading a URL and returning the static and immutable object that has that URL as an ID, the better the ecosystem will be. It's sad that everyone insists in wrapping costly database accesses and logic on top of what should be very, very simple logic.

rakoo 4y ago
Activities are immutable but objects are not. Mutability allows posts to be edited, for instance. The ids in ActivityPub also aren't URLs, they're random strings. EDIT: They actually are URLs, my mistake
- danpalmer 4y ago
  Are you sure? From the spec, object IDs must be: > Publicly dereferencable URIs, such as HTTPS URIs, with their authority belonging to that of their originating server (or they can be null in some cases)