6 ms·

Show HN: Fediverser Portal – Bring your subreddits to Lemmy

167 points by rglullis 3y ago

This is my attempt at helping those who are trying to ditch reddit but have not been satisfied with the content from Lemmy or haven't been able to find the corresponding communities. There are two sides to this project. The first one is that I have setup a Lemmy instance (alien.top) which is mirroring some of the reddit content from subreddits that I wanted to follow with the comments. The difference from most mirroring bots is that, instead of one single bot account mirroring all content, the system creates one account for each reddit user that is being mirrored. The other part of this idea which I believe is more interesting: reddit users can take over their own mirrored bot account on this Lemmy instance. The instance itself does not use the regular registration process, but instead authenticates via Reddit OAuth. If you login through through the "Portal", we can then grab your subscribed subreddits and (when it can) find the corresponding Lemmy communities and subscribe you to those automatically. At the moment there are not that Lemmy communities that are being mirrored because I've been the sole user, but hopefully if more people sign-up, it will help to create the network effects and more instance admins will be interested in hosting these "fediversed" communities. All of the code is open source (https://github.com/mushroomlabs/fediverser https://github.com/mushroomlabs/fediverser) and I'm more than willing to help people getting their own instances if they don't want to use alien.top itself. Questions and any type of feedback is always welcome!

rglullis 3y ago
There is actually a third point to this whole idea. Given that most Lemmy admins were not interested in being a home to the "mirrored" communities, I'm building a set of different Lemmy instances that will be focused on specific niches or interests, such as: - https://soccer.forum, https://nba.space, https://nfl.community, https://matchpoint.zone (sport news, fan groups, discussion, etc) - https://metacritics.zone, https://roll20.top (pop-culture, movies, TV shows, games) - blockchained.world (web3) - https://selfhosted.forum (devops, homelab) - https://style.land (brands, fashion, consumer items) - https://hi-fi.community (audio), https://viewfinder.pro (photography) - https://hardware.watch (tech reviews and news, pc enthusiasts, etc) - https://gearhead.town (cars) - https://netheads.online (voip) These lemmy instances are meant to be only destination hubs. They are not open for registration, but I am planning to let people decide what communities should be opened on those instances. I've created those because it seems that a lot of people are signing up to an instance and then fail at discovering the content.
- mdaniel 3y ago
  clickable version (sport news fan groups discussion etc) https://soccer.forum https://soccer.forum https://nba.space https://nba.space https://nfl.community https://nfl.community https://matchpoint.zone https://matchpoint.zone --- (pop-culture movies TV shows games) https://metacritics.zone https://metacritics.zone https://roll20.top https://roll20.top --- https://blockchained.world https://blockchained.world (web3) https://selfhosted.forum https://selfhosted.forum (devops, homelab) https://style.land https://style.land (brands, fashion, consumer items) https://hi-fi.community https://hi-fi.community (audio) https://viewfinder.pro https://viewfinder.pro (photography) https://hardware.watch https://hardware.watch (tech reviews and news, pc enthusiasts, etc) https://gearhead.town https://gearhead.town (cars) https://netheads.online https://netheads.online (voip)
  - kmlx 3y ago
    why do these sites all look like reddit?
    - rglullis 3y ago
      Because they are all different deployments of Lemmy [0]? [0] https://join-lemmy.org https://join-lemmy.org
    - gitgud 3y ago
      They’re mirroring reddit content
      - kmlx 3y ago
        ah, it makes sense now. thank you!
- Y_Y 3y ago
  Isn't it expensive to register so many domains? I guess you could host them all as subdomains of alien.top or something. Anyway you're doing a great service, I still see people manually "mirroring" stuff to Lemmy from Reddit. It's nice to have an extra filter, but this automating some of that makes much more sense to me.
superb_dev 3y ago
Nice work! I think this is a much more elegant solution
mdaniel 3y ago
For my curiosity, why "access_type=offline" as well as "scope=submit" for connecting accounts? I got the impression from your description that it just allowed "claiming" the user-id on portal.alien.top but your scopes seem to imply you're going to submit any content from portal.alien.top back into reddit as me
- rglullis 3y ago
  Two-way communication has been requested by some people and is the next item on my to-do list [0] [0]: https://github.com/orgs/mushroomlabs/projects/1 https://github.com/orgs/mushroomlabs/projects/1
  - mdaniel 3y ago
    I see https://github.com/mushroomlabs/fediverser/issues/9 https://github.com/mushroomlabs/fediverser/issues/9 but I would be much more likely to use it if the oauth2 dance didn't ask for aggressive perms versus a "ask for all the perms then revoke them later"
    - rglullis 3y ago
      The issue you are referring to is on the reddit -> lemmy side, but I see what you mean. I think I could make a separate application with less permissions and then just this one when two-way communication is ready and opt-in?
      - mdaniel 3y ago
        I would not claim to be a oauth2 expert, but to the best of my knowledge the scopes that one declares during the app registration on the reddit side are the upper bound of what said app is allowed to request, and thus if there was a checkbox on alien.top's side asking whether the user wanted bidirectional behavior, if unchecked the oauth2 handshake would omit the write scopes from the handshake and the token that comes back would be read only. I have mixed feelings about the "offline" part but if it was read only, I would be perfectly fine with alien.top using my access token to help spread the rate limiting load during any sync activities for the subreddits I was in Furthermore, again, to the best of my knowledge, if an alien.top user changed their mind and wanted to opt-in to the bidirectional behavior later, you could have that user just re authorize with the same client-id and this time ask for all the scopes. Much less complexity than managing two separate app registrations
VoidWhisperer 3y ago
I'm just getting 'no recommendation at the moment' even for subs that are main/frontpage subs such as 'technology' and 'awww'
- rglullis 3y ago
  That's because I'm still building the reddit-to-lemmy mapping, and for some cases in creating the lemmy communities themselves. There is also the fact that the largest Lemmy instances (lemmy.world) decided to consider alien.top as a mere "reddit repost" instance and decided to defederate with it, so I can not recommend instances from there as they will not be reachable.
  - brucethemoose2 3y ago
    This seems like a flaw of the federation system. Why should defederation be two sided. You should be able to recommend public posts from lemmy.world, and they should be able to stop any outgoing communication within lemmy.world to alien.top, if thats what they want.
    - rglullis 3y ago
      I somewhat agree, but the thing is, I can recommend them and users on alien.top would even be able to subscribe to the community, but lemmy.world would never send updates to us. I actually wrote to the LW mods to see if there will eventually reconsider the decision (https://communick.news/comment/674064 https://communick.news/comment/674064), let's hope for the best.
Daegalus 3y ago
I wonder if you could team up with https://sub.rehab/ https://sub.rehab/ and evolve it.
- rglullis 3y ago
  Wow, I remember seeing this site during the first reddit protests but I completely forgot about it. I was using redditmigration.com for my initial pass, but it seems that a lot of the communities or instances recommended there have been abandoned. I will definitely take a look sub.rehab again.
goodboyjojo 3y ago
cool. i've been wanting more people to move to the fediverse services so i have more people to talk to
bestouff 3y ago
So you want my reddit credentials ?
- rglullis 3y ago
  No. OAuth can solve this exact problem, you just need to authorize reddit to "login" to the portal website, and once you authenticate I can know that you really own the reddit account that you claim to have.
  - bestouff 3y ago
    Oh nice ! I'll try it then.
spdif899 3y ago
Very interesting. Will this cover all subreddits, or all above a certain size? I've switched fully off of reddit in protest, and have turned to HN and Lemmy as imperfect replacements. The main remaining thing I've noticed reddit excels at is researching things like product recommendations. Searching a keyword on BIFL, along with poking through various threads on whatever niche community might exist for the topic, has been one of the most reliable and useful ways to get a sense for what brands or models are best when considering a purchase. So far Lemmy is not close to replacing this so it would be compelling if your project could change that.
- rglullis 3y ago
  There is no 1:1 mapping between subreddits and lemmy instances, so even the very tiny ones can be mirrored. The main bottleneck is actually on the lemmy side, as I am having to create most of the communities, setting out rules, etc.
- eek2121 3y ago
  perfection is subjective IMO. I quit reddit when Apollo died. I use Lemmy, but no longer find myself getting sucked into a doomscrolling vacuum. For me, Lemmy IS the perfect replacement because I use it less.
SheinhardtWigCo 3y ago
Are you not concerned about being sued by reddit?
ajvs 3y ago
The main thing Lemmy is missing for adoption is a good desktop and/or mobile app. Since instances can defederate with each other it's often not possible to follow all your favourite communities from one account. So a good app needs to decouple this and enable follow any community without having an account, and instead store following data in the client.
- Obscurity4340 3y ago
  Do you think there's a technical fix to this or do you think its an immutable aspect of Lemmy's design? Like, is there something facile that the client applications (Apps used to "browse" and interact with the Lemmy ecosystem/instances) can do or is it impossible with fundamentally altering Lemmy itself?
  - ajvs 3y ago
    Yeah this is a design change that can only be fixed on the client application side because there will always be a reason created by an instance for why they don't want X content mirrored to their site.
    - Obscurity4340 3y ago
      Couldn't you just sign up on an instancewho's sole purpose is to remain federated with every other (major) instance and then "add" that account which the client app maintains along with one or all of your other accounts and wouldn't that super easily fix this issue even though it would be better for rhe app to handle all that without having an additional account to maintain? Edit: also, did I just roughly describe a "mirror" without even knowing it from the spirit of what I said?
      - rglullis 3y ago
        > did I just roughly describe a "mirror" without even knowing it from the spirit of what I said? Yeap, you just did. :)
ted_bunny 3y ago
Am I crazy to think this might raise legal alarm bells? Mirroring content is commonplace but I haven't heard of sites with an explicit path to take users from another site. One is grazing on their land, the other is an existential threat.
roshin 3y ago
How is it that you are able to scrape all of the content of Reddit, while 3rd party apps have almost stopped working.
- rglullis 3y ago
  - reddit still allows 600 requests per 10 minutes free of charge. - third-party apps need to make requests on behalf of all their users in real-time, this is making requests for a single user (the backend) and can be batched. Right now it it fetches data from 10 subreddits per request and it can be made to get even more at once. - I am not scraping all of reddit, only about 80 for now. It can be done every 3-4 minutes well within the rate limits. - if necessary, I could take the same approach used by the lemmit.online dev, and just request the JSON feeds (which do not require an API key) and parse the results on my own. - If more people run this service, we can make them cooperate so that we do a "MapReduce" on the scraping: the requests to different subreddits are spread between the fediverser instances, and they can gossip back the results among themselves.

167 points by rglullis 3y ago

rglullis 3y ago

There is actually a third point to this whole idea. Given that most Lemmy admins were not interested in being a home to the "mirrored" communities, I'm building a set of different Lemmy instances that will be focused on specific niches or interests, such as: - https://soccer.forum, https://nba.space, https://nfl.community, https://matchpoint.zone (sport news, fan groups, discussion, etc) - https://metacritics.zone, https://roll20.top (pop-culture, movies, TV shows, games) - blockchained.world (web3) - https://selfhosted.forum (devops, homelab) - https://style.land (brands, fashion, consumer items) - https://hi-fi.community (audio), https://viewfinder.pro (photography) - https://hardware.watch (tech reviews and news, pc enthusiasts, etc) - https://gearhead.town (cars) - https://netheads.online (voip) These lemmy instances are meant to be only destination hubs. They are not open for registration, but I am planning to let people decide what communities should be opened on those instances. I've created those because it seems that a lot of people are signing up to an instance and then fail at discovering the content.

mdaniel 3y ago
clickable version (sport news fan groups discussion etc) https://soccer.forum https://soccer.forum https://nba.space https://nba.space https://nfl.community https://nfl.community https://matchpoint.zone https://matchpoint.zone --- (pop-culture movies TV shows games) https://metacritics.zone https://metacritics.zone https://roll20.top https://roll20.top --- https://blockchained.world https://blockchained.world (web3) https://selfhosted.forum https://selfhosted.forum (devops, homelab) https://style.land https://style.land (brands, fashion, consumer items) https://hi-fi.community https://hi-fi.community (audio) https://viewfinder.pro https://viewfinder.pro (photography) https://hardware.watch https://hardware.watch (tech reviews and news, pc enthusiasts, etc) https://gearhead.town https://gearhead.town (cars) https://netheads.online https://netheads.online (voip)
- kmlx 3y ago
  why do these sites all look like reddit?
  - rglullis 3y ago
    Because they are all different deployments of Lemmy [0]? [0] https://join-lemmy.org https://join-lemmy.org
  - gitgud 3y ago
    They’re mirroring reddit content
    - kmlx 3y ago
      ah, it makes sense now. thank you!
Y_Y 3y ago
Isn't it expensive to register so many domains? I guess you could host them all as subdomains of alien.top or something. Anyway you're doing a great service, I still see people manually "mirroring" stuff to Lemmy from Reddit. It's nice to have an extra filter, but this automating some of that makes much more sense to me.

superb_dev 3y ago

Nice work! I think this is a much more elegant solution

mdaniel 3y ago

For my curiosity, why "access_type=offline" as well as "scope=submit" for connecting accounts? I got the impression from your description that it just allowed "claiming" the user-id on portal.alien.top but your scopes seem to imply you're going to submit any content from portal.alien.top back into reddit as me

rglullis 3y ago
Two-way communication has been requested by some people and is the next item on my to-do list [0] [0]: https://github.com/orgs/mushroomlabs/projects/1 https://github.com/orgs/mushroomlabs/projects/1
- mdaniel 3y ago
  I see https://github.com/mushroomlabs/fediverser/issues/9 https://github.com/mushroomlabs/fediverser/issues/9 but I would be much more likely to use it if the oauth2 dance didn't ask for aggressive perms versus a "ask for all the perms then revoke them later"
  - rglullis 3y ago
    The issue you are referring to is on the reddit -> lemmy side, but I see what you mean. I think I could make a separate application with less permissions and then just this one when two-way communication is ready and opt-in?
    - mdaniel 3y ago
      I would not claim to be a oauth2 expert, but to the best of my knowledge the scopes that one declares during the app registration on the reddit side are the upper bound of what said app is allowed to request, and thus if there was a checkbox on alien.top's side asking whether the user wanted bidirectional behavior, if unchecked the oauth2 handshake would omit the write scopes from the handshake and the token that comes back would be read only. I have mixed feelings about the "offline" part but if it was read only, I would be perfectly fine with alien.top using my access token to help spread the rate limiting load during any sync activities for the subreddits I was in Furthermore, again, to the best of my knowledge, if an alien.top user changed their mind and wanted to opt-in to the bidirectional behavior later, you could have that user just re authorize with the same client-id and this time ask for all the scopes. Much less complexity than managing two separate app registrations

VoidWhisperer 3y ago

I'm just getting 'no recommendation at the moment' even for subs that are main/frontpage subs such as 'technology' and 'awww'

rglullis 3y ago
That's because I'm still building the reddit-to-lemmy mapping, and for some cases in creating the lemmy communities themselves. There is also the fact that the largest Lemmy instances (lemmy.world) decided to consider alien.top as a mere "reddit repost" instance and decided to defederate with it, so I can not recommend instances from there as they will not be reachable.
- brucethemoose2 3y ago
  This seems like a flaw of the federation system. Why should defederation be two sided. You should be able to recommend public posts from lemmy.world, and they should be able to stop any outgoing communication within lemmy.world to alien.top, if thats what they want.
  - rglullis 3y ago
    I somewhat agree, but the thing is, I can recommend them and users on alien.top would even be able to subscribe to the community, but lemmy.world would never send updates to us. I actually wrote to the LW mods to see if there will eventually reconsider the decision (https://communick.news/comment/674064 https://communick.news/comment/674064), let's hope for the best.

Daegalus 3y ago

I wonder if you could team up with https://sub.rehab/ https://sub.rehab/ and evolve it.

rglullis 3y ago
Wow, I remember seeing this site during the first reddit protests but I completely forgot about it. I was using redditmigration.com for my initial pass, but it seems that a lot of the communities or instances recommended there have been abandoned. I will definitely take a look sub.rehab again.

goodboyjojo 3y ago

cool. i've been wanting more people to move to the fediverse services so i have more people to talk to

bestouff 3y ago

So you want my reddit credentials ?

rglullis 3y ago
No. OAuth can solve this exact problem, you just need to authorize reddit to "login" to the portal website, and once you authenticate I can know that you really own the reddit account that you claim to have.
- bestouff 3y ago
  Oh nice ! I'll try it then.

spdif899 3y ago

Very interesting. Will this cover all subreddits, or all above a certain size? I've switched fully off of reddit in protest, and have turned to HN and Lemmy as imperfect replacements. The main remaining thing I've noticed reddit excels at is researching things like product recommendations. Searching a keyword on BIFL, along with poking through various threads on whatever niche community might exist for the topic, has been one of the most reliable and useful ways to get a sense for what brands or models are best when considering a purchase. So far Lemmy is not close to replacing this so it would be compelling if your project could change that.

rglullis 3y ago
There is no 1:1 mapping between subreddits and lemmy instances, so even the very tiny ones can be mirrored. The main bottleneck is actually on the lemmy side, as I am having to create most of the communities, setting out rules, etc.
eek2121 3y ago
perfection is subjective IMO. I quit reddit when Apollo died. I use Lemmy, but no longer find myself getting sucked into a doomscrolling vacuum. For me, Lemmy IS the perfect replacement because I use it less.

SheinhardtWigCo 3y ago

Are you not concerned about being sued by reddit?

ajvs 3y ago

The main thing Lemmy is missing for adoption is a good desktop and/or mobile app. Since instances can defederate with each other it's often not possible to follow all your favourite communities from one account. So a good app needs to decouple this and enable follow any community without having an account, and instead store following data in the client.

Obscurity4340 3y ago
Do you think there's a technical fix to this or do you think its an immutable aspect of Lemmy's design? Like, is there something facile that the client applications (Apps used to "browse" and interact with the Lemmy ecosystem/instances) can do or is it impossible with fundamentally altering Lemmy itself?
- ajvs 3y ago
  Yeah this is a design change that can only be fixed on the client application side because there will always be a reason created by an instance for why they don't want X content mirrored to their site.
  - Obscurity4340 3y ago
    Couldn't you just sign up on an instancewho's sole purpose is to remain federated with every other (major) instance and then "add" that account which the client app maintains along with one or all of your other accounts and wouldn't that super easily fix this issue even though it would be better for rhe app to handle all that without having an additional account to maintain? Edit: also, did I just roughly describe a "mirror" without even knowing it from the spirit of what I said?
    - rglullis 3y ago
      > did I just roughly describe a "mirror" without even knowing it from the spirit of what I said? Yeap, you just did. :)

ted_bunny 3y ago

Am I crazy to think this might raise legal alarm bells? Mirroring content is commonplace but I haven't heard of sites with an explicit path to take users from another site. One is grazing on their land, the other is an existential threat.

roshin 3y ago

How is it that you are able to scrape all of the content of Reddit, while 3rd party apps have almost stopped working.

rglullis 3y ago
- reddit still allows 600 requests per 10 minutes free of charge. - third-party apps need to make requests on behalf of all their users in real-time, this is making requests for a single user (the backend) and can be batched. Right now it it fetches data from 10 subreddits per request and it can be made to get even more at once. - I am not scraping all of reddit, only about 80 for now. It can be done every 3-4 minutes well within the rate limits. - if necessary, I could take the same approach used by the lemmit.online dev, and just request the JSON feeds (which do not require an API key) and parse the results on my own. - If more people run this service, we can make them cooperate so that we do a "MapReduce" on the scraping: the requests to different subreddits are spread between the fediverser instances, and they can gossip back the results among themselves.