4 ms·

Lemmy has become one of the major "lifeboats" during the #RedditBlackout even this past week. This has lead to interesting performance issues and questions... a

Lemmy has become one of the major "lifeboats" during the #RedditBlackout even this past week. This has lead to interesting performance issues and questions... as well as human-level scaling issues. On the technical side, this https://programming.dev https://programming.dev thread (itself, a Lemmy instance) asks how Rust may have affected Lemmy. The ongoing discussion covers a wide variety of topics. The #1 post right now is frome snowe@programming.dev, the owner of this instance. He took disk-measurements of the server and published them, demonstrating that Lemmy's current bottleneck seems to be RAM (proven by the change in Swapping behavior / Disk-io before-and-after he upgraded RAM). -------- I do think you can see the benefits of federation in this manner. Users from many instances: https://kbin.social https://kbin.social, https://lemmy.world https://lemmy.world, and https://beehaw.org https://beehaw.org... and more (all different Lemmy instances) have joined in the discussion and have made a high-quality thread. --------- Note: I am bullish on federation / Fediverse after playing around with it this past week. I'm less bullish on Lemmy itself, which has proven to be a buggy, laggy mess. Still, there's good discussion happening on Lemmy now (albeit slower due to fewer users). Communities are springing up (and will be dying), and I'm sure instances will spin up and die too. But the idea of federation works, especially for bringing together the discussion. I think work (and play) needs to happen on Lemmy, kbin, Mastodon, and other Fediverse instances to figure out what is the best software, and best experience for users. Its not quite ready for prime-time, but the core feature set / Minimum-viable-product demo here is extremely exciting to me.

dybber 3y ago
What happens with the content if posted on an instance that shuts down? Will it be kept alive by other instances?
- dragontamer 3y ago
  This has already happened in practice, as https://beehaw.org https://beehaw.org and https://lemmy.world https://lemmy.world defederated due to some (hopefully temporary) issues. I don't wanna go into the drama / discussion here but lets just call it some nerd drama + lack of moderation tools that can solve the issue. (Beehaw.org is very confident that if Mastodon-like tools existed, they'd be able to refederate and get the things they want. The issue is being blamed right now on Lemmy's relatively new codebase that's missing some features that Mastodon users are familiar with) https://Lemmy.world https://Lemmy.world is continues to host the copies of the "old Beehaw.org" material, allowing Lemmy.world users to talk. But the "federation" has shut down, you can only talk with local users (because Beehaw.org serves as the "True copy". So upvotes / downvotes, and federated messages/comments are coordinated by the Beehaw.org server) In practice, this means that all the Beehaw.org communities has become kinda-sorta useless on https://Lemmy.world https://Lemmy.world. But, the copies of the conversations (up to the point of defederation) do exist and remain on your local (erm... more-local?) server.
- grumbel 3y ago
  It might still remain in the cache of another instance, but there is really no permanence build into the Fediverse. If an instance goes down, its content and user accounts go down too. There is no quick&easy way to migrate content to another host. There are no crypt-ids (e.g. PGP keys), everything is refered to as user@host, even message-ids aren't even globally unique, e.g. these two refer to the same post: * https://beehaw.org/post/598023 https://beehaw.org/post/598023 * https://feddit.de/post/881008 https://feddit.de/post/881008 Fediverse still has a lot of work to, right now it feels more like a quick hack than something that could significantly improve the Internet in the long run. If an instance goes malicious, you have all the same problems as you'd have with an unfederated server, after all federation is completely optional in the Fediverse.
tialaramex 3y ago
> Lemmy's current bottleneck seems to be RAM Rust seems like a sound choice in that regard because it's not garbage collected, so you're not wasting some RAM overhead just to avoid needing to do extra work, and it doesn't have C-style struct packing issues. If you don't specify the layout you need for a user defined type, Rust will assume you are OK with it re-arranging your data structure to be smaller / faster. If you're willing to put in some work you can often go further with this in Rust than in other "systems programming" languages too. For example C++ Strings have the "Small String optimisation", exactly how this works varies but e.g libc++ (on a modern 64-bit server) String is a 24 byte data structure which can hold up to 22 bytes of textual data inline without yet needing a heap allocation. So if you do a lot of work modifying small amounts of text, Rust's built-in String isn't as efficient as a libc++ String, but the CompactString type from a popular Rust crate is also a 24 byte data structure yet it holds a more impressive 24 bytes of text before needing heap allocation.
cbeach 3y ago
Just wanted to pick up on this minor point: > current bottleneck seems to be RAM RAM exhaustion is often a secondary side effect of the true underlying bottleneck. It could be that RAM used for temporarily processing an HTTP request, say, would be freed quicker if the database returned sooner. Maybe the database is the actual bottleneck. And the actual root cause is an indexing / partitioning issue.

dybber 3y ago

What happens with the content if posted on an instance that shuts down? Will it be kept alive by other instances?

dragontamer 3y ago
This has already happened in practice, as https://beehaw.org https://beehaw.org and https://lemmy.world https://lemmy.world defederated due to some (hopefully temporary) issues. I don't wanna go into the drama / discussion here but lets just call it some nerd drama + lack of moderation tools that can solve the issue. (Beehaw.org is very confident that if Mastodon-like tools existed, they'd be able to refederate and get the things they want. The issue is being blamed right now on Lemmy's relatively new codebase that's missing some features that Mastodon users are familiar with) https://Lemmy.world https://Lemmy.world is continues to host the copies of the "old Beehaw.org" material, allowing Lemmy.world users to talk. But the "federation" has shut down, you can only talk with local users (because Beehaw.org serves as the "True copy". So upvotes / downvotes, and federated messages/comments are coordinated by the Beehaw.org server) In practice, this means that all the Beehaw.org communities has become kinda-sorta useless on https://Lemmy.world https://Lemmy.world. But, the copies of the conversations (up to the point of defederation) do exist and remain on your local (erm... more-local?) server.
grumbel 3y ago
It might still remain in the cache of another instance, but there is really no permanence build into the Fediverse. If an instance goes down, its content and user accounts go down too. There is no quick&easy way to migrate content to another host. There are no crypt-ids (e.g. PGP keys), everything is refered to as user@host, even message-ids aren't even globally unique, e.g. these two refer to the same post: * https://beehaw.org/post/598023 https://beehaw.org/post/598023 * https://feddit.de/post/881008 https://feddit.de/post/881008 Fediverse still has a lot of work to, right now it feels more like a quick hack than something that could significantly improve the Internet in the long run. If an instance goes malicious, you have all the same problems as you'd have with an unfederated server, after all federation is completely optional in the Fediverse.

tialaramex 3y ago

> Lemmy's current bottleneck seems to be RAM Rust seems like a sound choice in that regard because it's not garbage collected, so you're not wasting some RAM overhead just to avoid needing to do extra work, and it doesn't have C-style struct packing issues. If you don't specify the layout you need for a user defined type, Rust will assume you are OK with it re-arranging your data structure to be smaller / faster. If you're willing to put in some work you can often go further with this in Rust than in other "systems programming" languages too. For example C++ Strings have the "Small String optimisation", exactly how this works varies but e.g libc++ (on a modern 64-bit server) String is a 24 byte data structure which can hold up to 22 bytes of textual data inline without yet needing a heap allocation. So if you do a lot of work modifying small amounts of text, Rust's built-in String isn't as efficient as a libc++ String, but the CompactString type from a popular Rust crate is also a 24 byte data structure yet it holds a more impressive 24 bytes of text before needing heap allocation.

cbeach 3y ago

Just wanted to pick up on this minor point: > current bottleneck seems to be RAM RAM exhaustion is often a secondary side effect of the true underlying bottleneck. It could be that RAM used for temporarily processing an HTTP request, say, would be freed quicker if the database returned sooner. Maybe the database is the actual bottleneck. And the actual root cause is an indexing / partitioning issue.