6 ms·

Open Source federated link aggregator in rust

106 points by modinfo 4y ago

fritigern 4y ago
lemmy is awesome. I'm on lemmy.ml and it basically replaced my reddit browsing.
Tmpod 4y ago
Lemmy is great! Despite the network still being fairly small, everything work really well already. Setting up instance was a breeze. The devs are also very accessible which is nice.
ZeroGravitas 4y ago
I was just thinking about something of much smaller scope that this might be able to do as a feature. Basically, you have a list of articles that you want to read, in Pocket or a similar service. And you have a collection of 'link aggregators' where you get topical news and/or insightful commentary. By combining the two, you can boost the signal of something you'd bookmarked to read later that has been resubmitted for discussion even if it's only received minor upvotes. This then provides the impetus to read it. This was partly inspired by the idea that breaking news/speculation is more addictive but less filling than content were people have had time to consider it.
- ItsMonkk 4y ago
  You seem to be describing LinkLonk[0]. [0]: https://news.ycombinator.com/item?id=28405643 https://news.ycombinator.com/item?id=28405643
icy 4y ago
I remember Lemmy’s devs being adamant on having a hard-coded slur filter (https://github.com/LemmyNet/lemmy/issues/622 https://github.com/LemmyNet/lemmy/issues/622). I think it’s been resolved now — not a great way to go about it though.
- capableweb 4y ago
  Damned if you do, damned if you don't? Or what do you mean with "not a great way to go about it though"? Seems like the following happened: - Developers implemented something well-intended, poorly-executed - People pushed back - Developers changed their mind and disabled the non-customizable slur-filter Seems that's exactly how you'd want things to go. What's the alternative here that would be better?
  - alwayslikethis 4y ago
    I'm not sure it is "well-intended, poorly-executed". They are well conscious of what they are doing -- sacrificing flexibility to make a political point.
    - PontifexMinimus 4y ago
      For "sacrificing flexibility" read "deliberately making the software harder to use". The slur filter isn't so much an issue in itself as a red flag -- it makes me wonder what other stunts are they likely to pull.
    - antifa 4y ago
      > I'm not sure it is "well-intended, poorly-executed" This is a clear unambiguous example of "well-intended, poorly-executed, then fixed"
- woojoo666 4y ago
  For those who want to see what the slur filter looked like, it's still in their example config [1]. But even though it's now not included by default, the issue runs a little deeper imo. From what I've seen, the developers of Lemmy are very political. They run and mod their own instances at lemmy.ml and communism.lemmy.ml, and their moderation has been pretty controversial [2] [3]. But worse, judging by the hard-coded slur filter, they seem to have no issue with mixing up their politics with the platform. Also note that it took quite a bit of pushback to get the slur filter removed, and the devs were not apologetic [4][5]. Who knows what they may add next as an "easy way to keep some groups of users off". While it's nice that it's federated and open source, the question then becomes, are people prepared to monitor the source code and fork it + maintain it themselves if the devs decide to turn the platform political again? [1]: https://github.com/LemmyNet/lemmy/blob/0.16.5/config/config.hjson https://github.com/LemmyNet/lemmy/blob/0.16.5/config/config.... [2]: https://old.reddit.com/r/RedditAlternatives/comments/czae8c https://old.reddit.com/r/RedditAlternatives/comments/czae8c [3]: https://old.reddit.com/r/Anarchism/comments/cza14y https://old.reddit.com/r/Anarchism/comments/cza14y [4]: https://github.com/LemmyNet/lemmy/issues/1224#issuecomment-714886485 https://github.com/LemmyNet/lemmy/issues/1224#issuecomment-7... [5]: https://github.com/LemmyNet/lemmy/pull/816#issuecomment-644694838 https://github.com/LemmyNet/lemmy/pull/816#issuecomment-6446...
xcambar 4y ago
Congrats and thank you for your hard work. Now for a contrarian comment: whenever I read "in Rust", or whatever language, it does not make it cool or interesting to me. Rather, it makes it sound like it is a POC or training material. Which is unfortunate because eventually, as a user, I only care for features, not for tech.
- capableweb 4y ago
  Not everything has to be cool or interesting to you. To be fair to the authors of Lemmy, the only mention of Rust on the project page is "Server is written in rust", while the submission for some reason contains Rust like it's some highlight of the software. Since we usually can't change too much of our environment, but we can change how we react to our environment, my recommendation would be to simply just ignore the "in X" in titles you see here, as what you say is true, it basically doesn't matter. If the software is shit, it's shit, no matter if it's shit in PHP or in Rust.
  - xcambar 4y ago
    I tried to be explicit that my comment was anecdotal, but apparently failed.
  - antifa 4y ago
    > "Server is written in rust" Unfortunately this makes it a "new reddit clone" (with websockets and react SPA) not an old reddit or hn clone.
- crabbygrabby 4y ago
  I like knowing what technologies were made with to some extent. It can give me a feel for what type of bugs I might run into or what the installation process looks like. Agreed to some extent though, once something is done, what it was written in - should be irrelevant to the end user. To be fair though, a few weeks ago there were people on here bashing rust claiming it's like impossible to use, async didn't work, the code is unreadable, shouldn't do web dev with it, etc. Meanwhile, here's yet another example of rust being just fine. So I'm glad they shared the technology they used.
geek_at 4y ago
Fun fact, earliy versions of lemmy were using my open source image hosting and postprocesing cdn (with S3,NFS,ftp.. storage) PictShare[1] until the devs raised an issue[2] about a huge memory leak when resizing this[3] 10mb PNG file which caused the memory to raise over 1.5 gigabytes and basically crashed the backend. So they ditched PictShare and wrote their own tool[4] only to realize it has the exact same problem because of the way some PNG files are stored basically any resize operation causes huge RAM spikes no matter which software or implementation is used. [1] https://github.com/HaschekSolutions/pictshare https://github.com/HaschekSolutions/pictshare [2] https://github.com/HaschekSolutions/pictshare/issues/120 https://github.com/HaschekSolutions/pictshare/issues/120 [3] https://commons.wikimedia.org/wiki/File:Pizigani_1367_Chart_10MB.jpg https://commons.wikimedia.org/wiki/File:Pizigani_1367_Chart_... [4] https://github.com/LemmyNet/lemmy/pull/807 https://github.com/LemmyNet/lemmy/pull/807
- ntoskrnl 4y ago
  That image is already 300MB of uncompressed pixel data. That's a hard floor on the memory usage unless you do something crazy like build a streaming image processing pipeline.
  - btdmaster 4y ago
    With ImageMagick convert, it does take up 1.5 gigabyte of RAM and take quite about 5 seconds. With ffmpeg, it only takes up 150MB and finishes about 10 times faster! I am really curious what makes swscaler from ffmpeg so memory efficient compared to other tools.
- antifa 4y ago
  Also PictShare supports S3 compatible backends, wasn't cool to see lemmy drop that so frivolously.
salmo 4y ago
This has a tech solution, but the same people problem. Look at the % that have “leftist” in the description. Again, politics come before interest just like the existing sites. I decided to still look. There seems to be no interest-based community. Just random posts likely interesting to an existing group of friends. It would be fun to stand one up for like gardening, fishing, sewing, or another of my random hobbies. But I’m admittedly lazy and don’t want to own that (both $ and effort managing a community). I enjoyed niche subreddits until each devolved into feuding, politics, or just repetitive “how do I start” questions killing the signal:noise. The commercialization of Reddit was well after my disinterest.
- alwayslikethis 4y ago
  I think the very project is devoted to promoting leftist ideas though, which is itself a people problem. For example, the authors of the project has been adamant on having a hard-coded "slur" filter (until recently, it appears) because they want to stop those with a different opinion/language from using the software. https://github.com/LemmyNet/lemmy/issues/622 https://github.com/LemmyNet/lemmy/issues/622
  - crabbygrabby 4y ago
    Honestly, looking at the censored words in the config.hjson as of oct28 2021, I don't see anything political. Mostly filtering literally racist and sexual slurs in the same way television/radio/the workplace does... Only in 2022 would we be discussing the n word and it's equivalents as politically charged... In my opinion, the words in the slur filter I see are effectively bottom of the barrel for reasonable discourse... If you wrote them in an email at work you'd be terminated quickly...
    - fiddlerwoaroof 4y ago
      The problem is the filter from Lemmy seems to suffer from the Scunthorpe Problem (https://en.m.wikipedia.org/wiki/Scunthorpe_problem https://en.m.wikipedia.org/wiki/Scunthorpe_problem )
  - 4y ago
closedloop129 4y ago
Where's the benefit of federation for link aggregators? If reddit.com/r/programming and HN would also implement the ActivityPub protocol, wouldn't that turn all participating aggregators into one because all submissions and comments are shared? How can a community maintain its identity in the Fediverse?
- llllllllkjjj111 4y ago
  This will be a controversial opinion on HN, but link aggregators, even the ones with comment sections, aren't communities. Yes, I use a throwaway handle now, because I used a real username here and on Reddit long enough to learn two things: 1) the danger of being doxxed if you say the wrong thing is real and "the community" will ruin your life with glee 2) although I have made numerous valuable relationships online, this has never occurred in a link aggregator comment section Considering these realities, I think link aggregators lack the repeated exposure to common members (excluding celebrities like dang) to create any real bond between individuals. No community exists because nobody ever gets to know each other. No, the issue in this case isn't the Fediverse. It's link aggregators.
khimaros 4y ago
also worth looking at nostr which has a different approach to decentralization and doesn't trap your identity with your home server https://github.com/nostr-protocol/nostr https://github.com/nostr-protocol/nostr there is a reddit clone and a twitter clone implemented with this protocol as well https://nvote.co/ https://nvote.co/ and https://nostr.com/ https://nostr.com/

106 points by modinfo 4y ago

lemmy is awesome. I'm on lemmy.ml and it basically replaced my reddit browsing.

Lemmy is great! Despite the network still being fairly small, everything work really well already. Setting up instance was a breeze. The devs are also very accessible which is nice.

ZeroGravitas 4y ago

I was just thinking about something of much smaller scope that this might be able to do as a feature. Basically, you have a list of articles that you want to read, in Pocket or a similar service. And you have a collection of 'link aggregators' where you get topical news and/or insightful commentary. By combining the two, you can boost the signal of something you'd bookmarked to read later that has been resubmitted for discussion even if it's only received minor upvotes. This then provides the impetus to read it. This was partly inspired by the idea that breaking news/speculation is more addictive but less filling than content were people have had time to consider it.

ItsMonkk 4y ago
You seem to be describing LinkLonk[0]. [0]: https://news.ycombinator.com/item?id=28405643 https://news.ycombinator.com/item?id=28405643

icy 4y ago

I remember Lemmy’s devs being adamant on having a hard-coded slur filter (https://github.com/LemmyNet/lemmy/issues/622 https://github.com/LemmyNet/lemmy/issues/622). I think it’s been resolved now — not a great way to go about it though.

capableweb 4y ago
Damned if you do, damned if you don't? Or what do you mean with "not a great way to go about it though"? Seems like the following happened: - Developers implemented something well-intended, poorly-executed - People pushed back - Developers changed their mind and disabled the non-customizable slur-filter Seems that's exactly how you'd want things to go. What's the alternative here that would be better?
- alwayslikethis 4y ago
  I'm not sure it is "well-intended, poorly-executed". They are well conscious of what they are doing -- sacrificing flexibility to make a political point.
  - PontifexMinimus 4y ago
    For "sacrificing flexibility" read "deliberately making the software harder to use". The slur filter isn't so much an issue in itself as a red flag -- it makes me wonder what other stunts are they likely to pull.
  - antifa 4y ago
    > I'm not sure it is "well-intended, poorly-executed" This is a clear unambiguous example of "well-intended, poorly-executed, then fixed"
woojoo666 4y ago
For those who want to see what the slur filter looked like, it's still in their example config [1]. But even though it's now not included by default, the issue runs a little deeper imo. From what I've seen, the developers of Lemmy are very political. They run and mod their own instances at lemmy.ml and communism.lemmy.ml, and their moderation has been pretty controversial [2] [3]. But worse, judging by the hard-coded slur filter, they seem to have no issue with mixing up their politics with the platform. Also note that it took quite a bit of pushback to get the slur filter removed, and the devs were not apologetic [4][5]. Who knows what they may add next as an "easy way to keep some groups of users off". While it's nice that it's federated and open source, the question then becomes, are people prepared to monitor the source code and fork it + maintain it themselves if the devs decide to turn the platform political again? [1]: https://github.com/LemmyNet/lemmy/blob/0.16.5/config/config.hjson https://github.com/LemmyNet/lemmy/blob/0.16.5/config/config.... [2]: https://old.reddit.com/r/RedditAlternatives/comments/czae8c https://old.reddit.com/r/RedditAlternatives/comments/czae8c [3]: https://old.reddit.com/r/Anarchism/comments/cza14y https://old.reddit.com/r/Anarchism/comments/cza14y [4]: https://github.com/LemmyNet/lemmy/issues/1224#issuecomment-714886485 https://github.com/LemmyNet/lemmy/issues/1224#issuecomment-7... [5]: https://github.com/LemmyNet/lemmy/pull/816#issuecomment-644694838 https://github.com/LemmyNet/lemmy/pull/816#issuecomment-6446...

xcambar 4y ago

Congrats and thank you for your hard work. Now for a contrarian comment: whenever I read "in Rust", or whatever language, it does not make it cool or interesting to me. Rather, it makes it sound like it is a POC or training material. Which is unfortunate because eventually, as a user, I only care for features, not for tech.

capableweb 4y ago
Not everything has to be cool or interesting to you. To be fair to the authors of Lemmy, the only mention of Rust on the project page is "Server is written in rust", while the submission for some reason contains Rust like it's some highlight of the software. Since we usually can't change too much of our environment, but we can change how we react to our environment, my recommendation would be to simply just ignore the "in X" in titles you see here, as what you say is true, it basically doesn't matter. If the software is shit, it's shit, no matter if it's shit in PHP or in Rust.
- xcambar 4y ago
  I tried to be explicit that my comment was anecdotal, but apparently failed.
- antifa 4y ago
  > "Server is written in rust" Unfortunately this makes it a "new reddit clone" (with websockets and react SPA) not an old reddit or hn clone.
crabbygrabby 4y ago
I like knowing what technologies were made with to some extent. It can give me a feel for what type of bugs I might run into or what the installation process looks like. Agreed to some extent though, once something is done, what it was written in - should be irrelevant to the end user. To be fair though, a few weeks ago there were people on here bashing rust claiming it's like impossible to use, async didn't work, the code is unreadable, shouldn't do web dev with it, etc. Meanwhile, here's yet another example of rust being just fine. So I'm glad they shared the technology they used.

geek_at 4y ago

Fun fact, earliy versions of lemmy were using my open source image hosting and postprocesing cdn (with S3,NFS,ftp.. storage) PictShare[1] until the devs raised an issue[2] about a huge memory leak when resizing this[3] 10mb PNG file which caused the memory to raise over 1.5 gigabytes and basically crashed the backend. So they ditched PictShare and wrote their own tool[4] only to realize it has the exact same problem because of the way some PNG files are stored basically any resize operation causes huge RAM spikes no matter which software or implementation is used. [1] https://github.com/HaschekSolutions/pictshare https://github.com/HaschekSolutions/pictshare [2] https://github.com/HaschekSolutions/pictshare/issues/120 https://github.com/HaschekSolutions/pictshare/issues/120 [3] https://commons.wikimedia.org/wiki/File:Pizigani_1367_Chart_10MB.jpg https://commons.wikimedia.org/wiki/File:Pizigani_1367_Chart_... [4] https://github.com/LemmyNet/lemmy/pull/807 https://github.com/LemmyNet/lemmy/pull/807

ntoskrnl 4y ago
That image is already 300MB of uncompressed pixel data. That's a hard floor on the memory usage unless you do something crazy like build a streaming image processing pipeline.
- btdmaster 4y ago
  With ImageMagick convert, it does take up 1.5 gigabyte of RAM and take quite about 5 seconds. With ffmpeg, it only takes up 150MB and finishes about 10 times faster! I am really curious what makes swscaler from ffmpeg so memory efficient compared to other tools.
antifa 4y ago
Also PictShare supports S3 compatible backends, wasn't cool to see lemmy drop that so frivolously.

salmo 4y ago

This has a tech solution, but the same people problem. Look at the % that have “leftist” in the description. Again, politics come before interest just like the existing sites. I decided to still look. There seems to be no interest-based community. Just random posts likely interesting to an existing group of friends. It would be fun to stand one up for like gardening, fishing, sewing, or another of my random hobbies. But I’m admittedly lazy and don’t want to own that (both $ and effort managing a community). I enjoyed niche subreddits until each devolved into feuding, politics, or just repetitive “how do I start” questions killing the signal:noise. The commercialization of Reddit was well after my disinterest.

alwayslikethis 4y ago
I think the very project is devoted to promoting leftist ideas though, which is itself a people problem. For example, the authors of the project has been adamant on having a hard-coded "slur" filter (until recently, it appears) because they want to stop those with a different opinion/language from using the software. https://github.com/LemmyNet/lemmy/issues/622 https://github.com/LemmyNet/lemmy/issues/622
- crabbygrabby 4y ago
  Honestly, looking at the censored words in the config.hjson as of oct28 2021, I don't see anything political. Mostly filtering literally racist and sexual slurs in the same way television/radio/the workplace does... Only in 2022 would we be discussing the n word and it's equivalents as politically charged... In my opinion, the words in the slur filter I see are effectively bottom of the barrel for reasonable discourse... If you wrote them in an email at work you'd be terminated quickly...
  - fiddlerwoaroof 4y ago
    The problem is the filter from Lemmy seems to suffer from the Scunthorpe Problem (https://en.m.wikipedia.org/wiki/Scunthorpe_problem https://en.m.wikipedia.org/wiki/Scunthorpe_problem )
- 4y ago

closedloop129 4y ago

Where's the benefit of federation for link aggregators? If reddit.com/r/programming and HN would also implement the ActivityPub protocol, wouldn't that turn all participating aggregators into one because all submissions and comments are shared? How can a community maintain its identity in the Fediverse?

llllllllkjjj111 4y ago
This will be a controversial opinion on HN, but link aggregators, even the ones with comment sections, aren't communities. Yes, I use a throwaway handle now, because I used a real username here and on Reddit long enough to learn two things: 1) the danger of being doxxed if you say the wrong thing is real and "the community" will ruin your life with glee 2) although I have made numerous valuable relationships online, this has never occurred in a link aggregator comment section Considering these realities, I think link aggregators lack the repeated exposure to common members (excluding celebrities like dang) to create any real bond between individuals. No community exists because nobody ever gets to know each other. No, the issue in this case isn't the Fediverse. It's link aggregators.

khimaros 4y ago

also worth looking at nostr which has a different approach to decentralization and doesn't trap your identity with your home server https://github.com/nostr-protocol/nostr https://github.com/nostr-protocol/nostr there is a reddit clone and a twitter clone implemented with this protocol as well https://nvote.co/ https://nvote.co/ and https://nostr.com/ https://nostr.com/