5 ms·

Activity Pub vs. Web Frameworks

122 points by danpalmer 4y ago

alexvoda 4y ago
> This is in direct contrast to typical REST-ish APIs that may use arbitrary identifiers. In these systems, clients must understand the identifiers and how they compose into paths to be used as URIs to request content. > Unfortunately almost all web frameworks are designed for the REST-ish applications where the client constructs URIs, and have the concept of a router based on path segments. I would argue that most frameworks do not really use REST, they use a form of HTTP and JSON based RPC. This goes back to the discussion about HATEOAS. In this case we see a clear advantage HATEOAS has with regards to architecture flexibility and ability to decentralize. With a HATEOAS REST API you do not construct URIs, you just follow them.
- danpalmer 4y ago
  Absolutely, this is why I used the term "REST-ish" because while it's the term the industry have decided to use, it's exactly because it's not really REST that it causes these issues.
pacifika 4y ago
Take the link as a global id and the initial value for url then allow url to be updated. It means you have to trust the redirect chain.
marcus_holmes 4y ago
I love the idea of every route on my site being just a UUID :) I've never thought of that, despite being a huge fan of UUIDs. I might go and experiment a bit with this. Anyone done this before? Any learnings?
bembo 4y ago
Wouldn't ipfs be a good fit for this?
- tetris11 4y ago
  On the topic of IPFS, but also slightly tangential -- how has nobody written an Activity Pub client/server in Javascript and distributed it as a Browser Addon? If an IPFS node/server can be literally hosted via a Firefox Extension[1], how has nobody done the same for an arguably easier protocol like Activity Pub? 1: https://github.com/ipfs/js-ipfs https://github.com/ipfs/js-ipfs
  - RobotToaster 4y ago
    Ipfs is a pull protocol. Activitypub is normally implemented as a push one (despite the specification having outboxes, so in theory it should be able to do either)
- jeroenhd 4y ago
  The ideas behind IPFS definitely seem to reflect the unchangeable URL intentions of this article. However, IPFS is also more limited, as edits and updates require special handling and interactivity is simply not possible. Furthermore, IPFS is too slow to be usable in practice. I'd love it to become a serious alternative to the web in terms of documents and blog posts, but the amount of timeouts and 30+ second loading times you need to tolerate to get any IPFS page to load makes it very difficult to tolerate. At least with the NFT scams IPFS found a real-life purpose, something I hadn't expected to see. That's dying down now, I think, so who knows how long the most active pin servers will stick around.
  - fabrice_d 4y ago
    IPFS would still be a good candidate for media attachments in Mastodon, potentially saving quite a bit of resources in bandwidth and storage. The performance of ipfs gateways has improved a lot lately, so even without a native ipfs:// protocol handler that could work nicely in clients.
    - rakoo 4y ago
      Bittorrent has been there for decades and can solve the issue ofedia attachments way more easily than ipfs
skybrian 4y ago
Maybe making everything a URL isn't a great idea, particularly when a lot of people on Mastodon have privacy concerns and don't want to be found? If you want people find your articles via links from other websites and search engines, better to put it on a blog.
- college_physics 4y ago
  Posting on mastodon is called microblogging for a reason - cause its a public blog. One can host a single-person instance and quite literally have a mastodon based blog site. Some people seem to confuse it with private messaging. User confusion about what an application or service does or does not is not a good basis on which to base a technical design, though it obviously hints that the particular app was not exactly "self-documenting". One can only speculate about how the confusion came about, but privacy advocates posting... publicly on mastodon may have unwillingly contributed to a sense that this space is "different" and "better" in every possible way. Yes, the fediverse/mastodon is different and better but in specific ways: it doesnt use algorithmic timelines, it doesn't push ads, it doesnt use dark addiction patterns etc. (though all of those might be reversed for particular instances or new server types). Unbundling the different issues and educating the broad base of users about what is going on is not just healthy and honest towards users, it may also contribute to having more well defined and stable user requirements about different types of online platforms.
- Zak 4y ago
  I'm a bit confused by people using a protocol that pushes their content to a large number of other people's computers if they don't want to be found. ActivityPub is inherently very public, even more so than publishing content to a website.
- Dalewyn 4y ago
  >people on Mastodon have privacy concerns and don't want to be found? Maybe don't use social media or the internet? Especially with your real name and mugshot?
  - iudqnolq 4y ago
raggi 4y ago
All of your links breaking is the good case. Parse all the URIs out of your log, capture static copies, and then serve them alongside the new implementation before dropping to 404.
gobengo 4y ago
We need good patterns for ‘dereferencing’ dns-bound identifiers like many https://{dns} https://{dns} URLs into something like a multiformats/cid[1] [1]: https://github.com/multiformats/cid https://github.com/multiformats/cid
- mariusor 4y ago
  URLs/URIs/IRIs are already as federated as they can be, there's no need to add an abstraction on top of that. Why do you think differently?
theamk 4y ago
I think there is an simple solution to this problem. It does require some changes, but they seem to be pretty straightforward: - When installing any app, user must always specify an arbitrary prefix. It is applied to all pages (and resources) except toplevel one. So you never have "hostname.com/posts/1234", instead it is "hostname.com/m1/posts/1234" - When switching to a new software, you use web spider/export functionality to convert the entire old prefix to static files. Static files are fast, and you can easily keep 100000 of them on average computer (and if not, you can store them in sqlite or some immutable k-v store). You then expose the static versions, either directly or via some trivial app. - New ActivityPub goes to different prefix, so new posts are "hostname.com/m2/post/1234" - The only thing that _might_ need redirects is users, to get following to work.. Those will need to be set up manually. But there are much fewer users than other objects, so this should be pretty doable. ---- Does this needs some changes to be feasible? yes. Is it easier than designing all-new scheme? Absolutely.
- danpalmer 4y ago
  This is a good idea that I hadn't fully considered. There are a few issues that make it tricky in the general case though – for existing implementations there may not be a prefix that can be used, and if we're talking about adding prefixes to all new systems, we could just use frameworks that have built-in support for this. Additionally, web scraping isn't perfect. It assumes static content, which may not be possible with ACLs or other behaviour. It also assumes that there's a single link entry point to the content, or that all the necessary link entry points are known. It could also be very wasteful – storing a tweet in a database is much smaller than storing the JSON response for that tweet, or the HTML of the page the tweet is in. This is an interesting idea to consider, and it could be simpler in some cases, but I'm not sure it's a better solution in all cases.
  - mariusor 4y ago
    You already commit a big fallacy in my opinion implying that objects need to be related to database access. Once an object has been created on an instance - with very few exceptions - there's no need to tamper with it. Storing it as a JSON on disk is perfectly acceptable.
itake 4y ago
I wish OP explained the issues with GoToSocial
- danpalmer 4y ago
  Bugs, lack of functionality, incompatibility with clients. Also I realised that there wasn’t any need to - Mastodon will run for a single user on pretty much anything so why do I need something that runs twice as fast? Also, I completely understand maintaining open source software is hard, but GoToSocial’s approach to addressing this was to entirely disable all issues and pull requests for a month over the holiday period, and I think that’s the wrong approach. I didn’t report any of the bugs because I was unable to do so. I would have understood not receiving a response for a few weeks, or even at all, but to not even be able to post them was a strange choice. It turned me off the software a bit, what if someone wanted to report a vulnerability?
deleted 4y ago
[deleted]
simonw 4y ago
> Being content based implies database queries for every request as in the previously mentioned option (3), but by raising this functionality to the framework level, more optimisations may be implemented, and correctness ensured, in one place, likely resulting in a lower impact of those additional queries. I'm not at all convinced that the performance overhead of a database hit for every incoming request in order to resolve the URL is significant enough in any of the widely used frameworks to justify throwing them out and doing something different. Most database-backed web apps I've worked on in my career have had a dozen or more SQL queries executing per page rendered. Adding an indexed lookup against a URL table of some sort is a tiny additional load - and it's also usually quite straight forward to add a cache using memcache or Redis or Varnish if is starts to become a problem. URL lookups are a lot more likely to work as a simple TTL based cache than most other kinds of query.
- danpalmer 4y ago
  My perspective here was that 1 database query implemented by a framework with: good caching, perhaps canonicalisation to ensure small caches (e.g. query parameters ordering?), the right database indexes, the right response behaviour, etc, would be fine, but implementing this well in a middleware added by developers using a framework would be hard to do well and could be significantly higher cost. In my experience, implementing things like access control, requiring a single database query, in a middleware that applies to every request, does actually become a noticeable overhead and problem at some point. Sure many pages might be doing 20 queries so adding one more isn't a problem, but at some point there might be an API that needs to respond quickly, does <1 query on average with caching, and therefore has _double_ the cost with this feature. Because the URLs are supposed to be opaque in the general case, this still needs to be applied everywhere. This can definitely all be optimised, but I think doing this at the framework level is a much better solution.
supermatt 4y ago
> When migrating between systems, this difference presents a problem. How does it break anything? ActivityPub ids can be any URI. You dont need to migrate your original rest-style links to anything - just publish those as the ids... And if you are going in the other direction (and why would you), then I cant think of a single app router that ENFORCES you to use rest-style routes. It may be encouraged with many, but its certainly not enforced. > Can the framework function in most of the ways we’d expect from a modern web framework, but with every URI being a UUID? URIs should already be unique. You can use those... > These requirements suggest that the framework would likely be content based, rather than route based ActivityPub objects contain the type, so thats ALWAYS going to be the case to reliably use 3rd-party ActivityPub objects. As for locally-sourced objects, you can still use your rest-style URIs and use those URIs as the ActivityPub ids!
- nicky0 4y ago
  Say I migrate my ActivityPub server from ProductX to ProductY, but I want to keep all the usernames and external follow relationships intact. We need external people who follow @me@myserver.example.com seamlessly still follow at the new server, which has a different code an URL structure. Will it "just work"?
  - RobotToaster 4y ago
    If you mean changing domain names, I think in theory you can setup redirects with Webfinger.
    - vidarh 4y ago
      You're right, you can, and it works. E.g. my Mastodon id is @vidar@galaxybound.com, but my Mastodon server is hosted on m.galaxybound.com. The webfinger on galaxybound.com is just a redirect. In addition to redirecting the webfinger query, the webfinger results themselves can point to an inbox etc. elsewhere.
  - mariusor 4y ago
    Why does the new server need to have a new url structure? Why do the URLs need a "structure" anyway? That's what TFA and OP are actually saying. Mastodon - partly due to the fact that it predates federating through ActivityPub - doesn't have the concept of URL as a unique identifier for an object (in the ActivityPub sense), but uses numeric ids, or uuids in some places. If you really want "a structure", I wonder why nobody is implementing ActivityPub services[1] using the simplest and dumb way of mapping objects in storage using their ids. Ie: an actor has the ID https://example.com/user1 https://example.com/user1, therefore it's inbox/outbox should be https://example.com/user1/inbox|outbox https://example.com/user1/inbox|outbox. Then, an activity created by that user can be found at https://example.com/user1/outbox/{unique-activity-id} https://example.com/user1/outbox/{unique-activity-id}. If the activity is a Create (the only ones with a side effect of creating an object), then this object can have the id https://example.com/user1/outbox/{unique-activity-id}/object https://example.com/user1/outbox/{unique-activity-id}/object, samd. [1] My own fediverse software does this for multiple types of storage: filesystem hierarchy, k/v store, database tables, but I haven't seen anything else similar.

122 points by danpalmer 4y ago

> This is in direct contrast to typical REST-ish APIs that may use arbitrary identifiers. In these systems, clients must understand the identifiers and how they compose into paths to be used as URIs to request content. > Unfortunately almost all web frameworks are designed for the REST-ish applications where the client constructs URIs, and have the concept of a router based on path segments. I would argue that most frameworks do not really use REST, they use a form of HTTP and JSON based RPC. This goes back to the discussion about HATEOAS. In this case we see a clear advantage HATEOAS has with regards to architecture flexibility and ability to decentralize. With a HATEOAS REST API you do not construct URIs, you just follow them.

danpalmer 4y ago
Absolutely, this is why I used the term "REST-ish" because while it's the term the industry have decided to use, it's exactly because it's not really REST that it causes these issues.

pacifika 4y ago

Take the link as a global id and the initial value for url then allow url to be updated. It means you have to trust the redirect chain.

marcus_holmes 4y ago

I love the idea of every route on my site being just a UUID :) I've never thought of that, despite being a huge fan of UUIDs. I might go and experiment a bit with this. Anyone done this before? Any learnings?

bembo 4y ago

Wouldn't ipfs be a good fit for this?

tetris11 4y ago
On the topic of IPFS, but also slightly tangential -- how has nobody written an Activity Pub client/server in Javascript and distributed it as a Browser Addon? If an IPFS node/server can be literally hosted via a Firefox Extension[1], how has nobody done the same for an arguably easier protocol like Activity Pub? 1: https://github.com/ipfs/js-ipfs https://github.com/ipfs/js-ipfs
- RobotToaster 4y ago
  Ipfs is a pull protocol. Activitypub is normally implemented as a push one (despite the specification having outboxes, so in theory it should be able to do either)
jeroenhd 4y ago
The ideas behind IPFS definitely seem to reflect the unchangeable URL intentions of this article. However, IPFS is also more limited, as edits and updates require special handling and interactivity is simply not possible. Furthermore, IPFS is too slow to be usable in practice. I'd love it to become a serious alternative to the web in terms of documents and blog posts, but the amount of timeouts and 30+ second loading times you need to tolerate to get any IPFS page to load makes it very difficult to tolerate. At least with the NFT scams IPFS found a real-life purpose, something I hadn't expected to see. That's dying down now, I think, so who knows how long the most active pin servers will stick around.
- fabrice_d 4y ago
  IPFS would still be a good candidate for media attachments in Mastodon, potentially saving quite a bit of resources in bandwidth and storage. The performance of ipfs gateways has improved a lot lately, so even without a native ipfs:// protocol handler that could work nicely in clients.
  - rakoo 4y ago
    Bittorrent has been there for decades and can solve the issue ofedia attachments way more easily than ipfs

skybrian 4y ago

Maybe making everything a URL isn't a great idea, particularly when a lot of people on Mastodon have privacy concerns and don't want to be found? If you want people find your articles via links from other websites and search engines, better to put it on a blog.

college_physics 4y ago
Posting on mastodon is called microblogging for a reason - cause its a public blog. One can host a single-person instance and quite literally have a mastodon based blog site. Some people seem to confuse it with private messaging. User confusion about what an application or service does or does not is not a good basis on which to base a technical design, though it obviously hints that the particular app was not exactly "self-documenting". One can only speculate about how the confusion came about, but privacy advocates posting... publicly on mastodon may have unwillingly contributed to a sense that this space is "different" and "better" in every possible way. Yes, the fediverse/mastodon is different and better but in specific ways: it doesnt use algorithmic timelines, it doesn't push ads, it doesnt use dark addiction patterns etc. (though all of those might be reversed for particular instances or new server types). Unbundling the different issues and educating the broad base of users about what is going on is not just healthy and honest towards users, it may also contribute to having more well defined and stable user requirements about different types of online platforms.
Zak 4y ago
I'm a bit confused by people using a protocol that pushes their content to a large number of other people's computers if they don't want to be found. ActivityPub is inherently very public, even more so than publishing content to a website.
Dalewyn 4y ago
>people on Mastodon have privacy concerns and don't want to be found? Maybe don't use social media or the internet? Especially with your real name and mugshot?
- iudqnolq 4y ago

raggi 4y ago

All of your links breaking is the good case. Parse all the URIs out of your log, capture static copies, and then serve them alongside the new implementation before dropping to 404.

gobengo 4y ago

We need good patterns for ‘dereferencing’ dns-bound identifiers like many https://{dns} https://{dns} URLs into something like a multiformats/cid[1] [1]: https://github.com/multiformats/cid https://github.com/multiformats/cid

mariusor 4y ago
URLs/URIs/IRIs are already as federated as they can be, there's no need to add an abstraction on top of that. Why do you think differently?

theamk 4y ago

I think there is an simple solution to this problem. It does require some changes, but they seem to be pretty straightforward: - When installing any app, user must always specify an arbitrary prefix. It is applied to all pages (and resources) except toplevel one. So you never have "hostname.com/posts/1234", instead it is "hostname.com/m1/posts/1234" - When switching to a new software, you use web spider/export functionality to convert the entire old prefix to static files. Static files are fast, and you can easily keep 100000 of them on average computer (and if not, you can store them in sqlite or some immutable k-v store). You then expose the static versions, either directly or via some trivial app. - New ActivityPub goes to different prefix, so new posts are "hostname.com/m2/post/1234" - The only thing that _might_ need redirects is users, to get following to work.. Those will need to be set up manually. But there are much fewer users than other objects, so this should be pretty doable. ---- Does this needs some changes to be feasible? yes. Is it easier than designing all-new scheme? Absolutely.

danpalmer 4y ago
This is a good idea that I hadn't fully considered. There are a few issues that make it tricky in the general case though – for existing implementations there may not be a prefix that can be used, and if we're talking about adding prefixes to all new systems, we could just use frameworks that have built-in support for this. Additionally, web scraping isn't perfect. It assumes static content, which may not be possible with ACLs or other behaviour. It also assumes that there's a single link entry point to the content, or that all the necessary link entry points are known. It could also be very wasteful – storing a tweet in a database is much smaller than storing the JSON response for that tweet, or the HTML of the page the tweet is in. This is an interesting idea to consider, and it could be simpler in some cases, but I'm not sure it's a better solution in all cases.
- mariusor 4y ago
  You already commit a big fallacy in my opinion implying that objects need to be related to database access. Once an object has been created on an instance - with very few exceptions - there's no need to tamper with it. Storing it as a JSON on disk is perfectly acceptable.

itake 4y ago

I wish OP explained the issues with GoToSocial

danpalmer 4y ago
Bugs, lack of functionality, incompatibility with clients. Also I realised that there wasn’t any need to - Mastodon will run for a single user on pretty much anything so why do I need something that runs twice as fast? Also, I completely understand maintaining open source software is hard, but GoToSocial’s approach to addressing this was to entirely disable all issues and pull requests for a month over the holiday period, and I think that’s the wrong approach. I didn’t report any of the bugs because I was unable to do so. I would have understood not receiving a response for a few weeks, or even at all, but to not even be able to post them was a strange choice. It turned me off the software a bit, what if someone wanted to report a vulnerability?

deleted 4y ago

[deleted]

simonw 4y ago

> Being content based implies database queries for every request as in the previously mentioned option (3), but by raising this functionality to the framework level, more optimisations may be implemented, and correctness ensured, in one place, likely resulting in a lower impact of those additional queries. I'm not at all convinced that the performance overhead of a database hit for every incoming request in order to resolve the URL is significant enough in any of the widely used frameworks to justify throwing them out and doing something different. Most database-backed web apps I've worked on in my career have had a dozen or more SQL queries executing per page rendered. Adding an indexed lookup against a URL table of some sort is a tiny additional load - and it's also usually quite straight forward to add a cache using memcache or Redis or Varnish if is starts to become a problem. URL lookups are a lot more likely to work as a simple TTL based cache than most other kinds of query.

danpalmer 4y ago
My perspective here was that 1 database query implemented by a framework with: good caching, perhaps canonicalisation to ensure small caches (e.g. query parameters ordering?), the right database indexes, the right response behaviour, etc, would be fine, but implementing this well in a middleware added by developers using a framework would be hard to do well and could be significantly higher cost. In my experience, implementing things like access control, requiring a single database query, in a middleware that applies to every request, does actually become a noticeable overhead and problem at some point. Sure many pages might be doing 20 queries so adding one more isn't a problem, but at some point there might be an API that needs to respond quickly, does <1 query on average with caching, and therefore has _double_ the cost with this feature. Because the URLs are supposed to be opaque in the general case, this still needs to be applied everywhere. This can definitely all be optimised, but I think doing this at the framework level is a much better solution.

supermatt 4y ago

> When migrating between systems, this difference presents a problem. How does it break anything? ActivityPub ids can be any URI. You dont need to migrate your original rest-style links to anything - just publish those as the ids... And if you are going in the other direction (and why would you), then I cant think of a single app router that ENFORCES you to use rest-style routes. It may be encouraged with many, but its certainly not enforced. > Can the framework function in most of the ways we’d expect from a modern web framework, but with every URI being a UUID? URIs should already be unique. You can use those... > These requirements suggest that the framework would likely be content based, rather than route based ActivityPub objects contain the type, so thats ALWAYS going to be the case to reliably use 3rd-party ActivityPub objects. As for locally-sourced objects, you can still use your rest-style URIs and use those URIs as the ActivityPub ids!

nicky0 4y ago
Say I migrate my ActivityPub server from ProductX to ProductY, but I want to keep all the usernames and external follow relationships intact. We need external people who follow @me@myserver.example.com seamlessly still follow at the new server, which has a different code an URL structure. Will it "just work"?
- RobotToaster 4y ago
  If you mean changing domain names, I think in theory you can setup redirects with Webfinger.
  - vidarh 4y ago
    You're right, you can, and it works. E.g. my Mastodon id is @vidar@galaxybound.com, but my Mastodon server is hosted on m.galaxybound.com. The webfinger on galaxybound.com is just a redirect. In addition to redirecting the webfinger query, the webfinger results themselves can point to an inbox etc. elsewhere.
- mariusor 4y ago
  Why does the new server need to have a new url structure? Why do the URLs need a "structure" anyway? That's what TFA and OP are actually saying. Mastodon - partly due to the fact that it predates federating through ActivityPub - doesn't have the concept of URL as a unique identifier for an object (in the ActivityPub sense), but uses numeric ids, or uuids in some places. If you really want "a structure", I wonder why nobody is implementing ActivityPub services[1] using the simplest and dumb way of mapping objects in storage using their ids. Ie: an actor has the ID https://example.com/user1 https://example.com/user1, therefore it's inbox/outbox should be https://example.com/user1/inbox|outbox https://example.com/user1/inbox|outbox. Then, an activity created by that user can be found at https://example.com/user1/outbox/{unique-activity-id} https://example.com/user1/outbox/{unique-activity-id}. If the activity is a Create (the only ones with a side effect of creating an object), then this object can have the id https://example.com/user1/outbox/{unique-activity-id}/object https://example.com/user1/outbox/{unique-activity-id}/object, samd. [1] My own fediverse software does this for multiple types of storage: filesystem hierarchy, k/v store, database tables, but I haven't seen anything else similar.