8 ms·

Welcome Lemmy.world

167 points by dragontamer 3y ago

dragontamer 3y ago
There were some Lemmy-related discussions going on this past weekend. I think this is the one tidbit everyone around Hacker News wants to know: > The current VPS couldn't be resized that much anymore, and load was going up with all the new users. So I bought the same server at Hetzner: a 32-core/64 thread 128GB RAM dedicated server. (For Mastodon, I doubled the RAM. For Lemmy I don't think it's needed yet.) I migrated the Lemmy software and database there, and moved over. This took 4 minutes of downtime. So 32-core/64-threads with 128GB of RAM is running Lemmy.world, which is supporting ~36.7k users at the moment. --------- A brief look at Hertzner comes up with: https://www.hetzner.com/dedicated-rootserver/ax161 https://www.hetzner.com/dedicated-rootserver/ax161 A 32-core/64 thread + 128GB RAM server in Germany for 142 EUR / month.
- gareve 3y ago
  with some basic extrapolation, at how many users they'll hit the extra expensive Hertzner servers? or in other words, at which point they'll need to improve their architecture?
  - zmmmmm 3y ago
    it'll be interesting to see if at that point, federation starts to become the way of scaling or not. If it was seamless, it wouldn't matter where you signed up and they could just host multiple lemmy instances on different servers. So far I have rather spotty experiences though with content sometimes making it to federated servers, sometimes not etc etc.
    - dragontamer 3y ago
      User scaling seems easy enough. Its "community" (on Lemmy), or "magazine" (on kbin) scaling that seems hard. But since each server has a local-copy of the community that its serving out, maybe the hardest part has already been solved by the Federation model. Each federated-instance is effectively a proxy / front-end for the users on that instance. --------- I guess Mastodon is way larger than Lemmy though and they haven't had issues yet.
  - rglullis 3y ago
    I honestly wish that the answer is "when we get so big to the point that a single machine can not handle it, we close registrations". Can we please drop the "number must go up" mentality? The whole point of federated systems is to avoid concentration of power in a handful of servers. I'm sure that the people doing there have good intentions, but why can't we just let things just a little bit dispersed?
    - ajsnigrutin 3y ago
      Sure, until you google some error, or some other random thing, find some thread somewhere, want to comment/ask/contribute, and you can't, since the registrations are locked.
      - Stephen304 3y ago
        I'm on kbin.social and participate in communities across ~20ish other instances regardless of whether they've turned off registrations temporarily or permanently. Disabling registrations only prevents new accounts on that instance, it doesn't stop people from posting and commenting from other instances.
        rglullis 3y ago
        Parent comment is the perfect example of how we got so used with dealing with shitty and user-hostile systems. We've been dealing with walled gardens for almost a generation now, it's like they don't even understand that it is possible to interact with a remote system without having everything in one centralized database.
        phs318u 3y ago
        People have forgotten how email works.
        FormerBandmate 3y ago
        Gmail isn't a walled garden but it doesn't have a 20k user limit
  - dragontamer 3y ago
    1TB RAM Hertzner servers are available, so at least 8x more scaling before that's a problem. 2TB RAM is common in commodity servers, albeit expensive ones ($1000ish/month). Somewhere between 4TB to 20TB RAM is the pragmatic limit (where costs for vertical scaling start to get far worse)
    - tedunangst 3y ago
      Is RAM still the limiting factor at that scale? Are you assuming 256 CPU cores?
      - dragontamer 3y ago
        Current theory is that Lemmy software is mostly RAM limited. No one knows for sure until we reach those caps.
    - fulafel 3y ago
      Interestingly it seems max memories have been going down or at least not increasing in commodity x86 servers. Vendors advertised 24 TB servers enabled by lots of (192?) DIMM sockets in 2018 or maybe even earlier.
- creamyhorror 3y ago
  32 cores and/or 128GB RAM - for how many concurrent users, 36.7k? It seems like Lemmy requires a fair bit of resources per user for a primarily text-based API, more than I'd expected. Wonder where the majority of the resource burden lies - uncached local database queries? Too many simultaneous database connections?
  - jochem9 3y ago
    Federation takes quite a toll. The way I read the article is that they didn't upgrade every time the load was (almost) too high, but instead opted immediately for the beefy dedicated server (considering the price it makes sense – it's on par with beefy VMs that have less power). They probably have resources to spare right now, but no need to go through the hassle of an upgrade soon (although 4 mins downtime sounds like they have things under control). Having said that, I'm sure there's plenty to improve in Lemmy, given it's used at a bigger scale now.
  - spaduf 3y ago
    Keep in mind a lot of instances are preparing for what they anticipate to be the largest migratory wave yet when third party apps shut down at the end of the month.
eigenspace 3y ago
I wish Lemmy's default design was more of a copy of old.reddit.com. For me, that design was better than both HackerNews' design and way better than the current default look of Lemmy instances.
- tester457 3y ago
  Same. However it seems most people don't like the old.reddit look, only 4% of redditors use it. Most people are content with the slow bloated new reddit. When redditors talked about what sites to migrate to most of them preferred the modern interfaces over text based ones like tildes.net.
  - Narishma 3y ago
    How many even know that old.reddit exists?
    - aerio 3y ago
      5%
  - tuchsen 3y ago
    I wonder how much this has to do with the devices people use to access Reddit. I always preferred the old.reddit site on the desktop. Although when I'm on my phone, it gave me eye strain to try and read the tiny font, and pinch/zooming all the time is super annoying. The newer Reddit web app was still incredibly annoying, given all the "install the desktop app" prompts, and it's general lagginess, but allowing me to just flick scroll through articles was actually better for me. Lemmy's the same way for me, hate the UI on the desktop but it's fine on my phone.
    - Gigachad 3y ago
      I don’t think there is anything visually wrong with the new reddit. My issue is with how slow and buggy it is. And all the dark patterns and engagement tricks it’s full of.
Vermyndax 3y ago
So, single server scaled vertically? Gross.
- anaganisk 3y ago
  Is it /s? If not, deploying a kubernetes cluster and over engineering it?
- toomuchtodo 3y ago
  HN runs on a single core. https://news.ycombinator.com/item?id=29927630 https://news.ycombinator.com/item?id=29927630
  - shagie 3y ago
    HN has a lot of optimizations and limitations on the user experience. For example, I don't see the scores of every other post on HN - and certainly don't see them updated in realtime. New comments and posts show up when I reload the page - not as I scroll. There's only one HN instance - without having 100 more instances trying to mirror the content (just one - firebase). ... and HN is an example of a single, centralized, unfederated server.
- spicyusername 3y ago
  The default Lemmy deployment docs don't have a lot of breathing room. It would be reasonable to break off the postgres server Lemmy uses to a separate server, but at the moment there is only an Ansible playbook or a docker compose file, both of which assume a single server. There's only so much time in the day and it's much easier to size up than reimplement the install process.
camdenlock 3y ago
Took a look at mastodon.world out of curiosity. Total conformity of belief on the entire front page timeline. Maybe it’s another one of those Mastodon instances where anti-capitalist social justice is the approved dogma, and dissenters are banned? Seems to be a lot of those…
- yosito 3y ago
  The beauty of Mastodon is that you can set up your own server if you don't like how someone else runs theirs!
  - Arn_Thor 3y ago
    A sort of “safe space”, if you will
  - xigoi 3y ago
    And then you lose the benefits of federation.
- kstrauser 3y ago
  I run such a Mastodon instance. Go build your own and compete with me for users if you think there’s a demand for it.
- jbm 3y ago
  Sorry to see you are being downvoted for making an observation. I have been looking across various Lemmy instances and I recall seeing one or two that was cyberpunk-based or technology-focused in their local streams, but far too many have the "smug American Liberal" [1] style comments that made me leave Reddit [2]. Simply spinning up another community is not a quick fix either. Concern trolling means Centrist communities simply do not last, as they eventually get overrun by extremists of one kind of another. It's far, far too easy to take a complex argument and have it dishonestly re-interpreted in a negative way as a a one line meme in public communities. Weeding these people out is far more of a challenge than banning violent remarks, racists and child abusers, as well-meaning people will inevitably misunderstand. The solution to this is small private communities and group chats with people you trust, personally. You won't get the hit from having constant content to consume, but it's not as bad for you; getting in the door is the real problem. [1] https://www.vox.com/2016/4/21/11451378/smug-american-liberalism https://www.vox.com/2016/4/21/11451378/smug-american-liberal... [2] Signing up for an account because the defaults are crazy is not a viable solution
FullstakBlogger 3y ago
lemmy.world, for whatever reason, absolutely guzzles CPU cycles. I can't imagine what it could possibly need to do that requires executing that much javascript. It's all minimized, so it's hard to follow in the profiler.
- alephaleph 3y ago
  Apparently it’s bc it uses websockets to live update vote counts or smth
pentagrama 3y ago
> I started paying for the Lemmy cloud servers from the mastodon.world funds. Did this person ask to the mastodon.world Patreon [1] and Open Collective [2] supporters that he was going to use money for other project? If not, seems disingenuous to me. [1] https://www.patreon.com/mastodonworld https://www.patreon.com/mastodonworld [2] https://opencollective.com/mastodonworld https://opencollective.com/mastodonworld
TX81Z 3y ago
Having worked in the tech legal and compliance, privacy, and security spaces for a very long time the idea of letting thousands of strangers upload binary files to a server I’m legally liable for is terrifying.
- brightlancer 3y ago
  Even "text" files can be encoded binary files -- it was a big shift in USENET. I had a heated discussion with some Nostr advocates who thought that by calling servers "avocados" or something, then they wouldn't be responsible for what users posted on the "avocados". I've not looked into ActivityPub or the Fediverse enough to know what is hosted locally and would (not) be protected by Sec 230, but it might drive some centralization toward corporations or other accountable organizations.
  - TX81Z 3y ago
    base64 is a helluva drug.
- dragontamer 3y ago
  In practice, I think only the NSFW Lemmy's have to be worried. This isn't straight binary file hosting. Its just text, image, and video. Images/Video can get into trouble but if you have a Safe-for-work only policy and defederate from NSFW instances, you're probably in the clear? And I'm pretty sure any large amount of data is going to be throttled. So not even "big" videos are going to be hosted, more like a few-second .mp4 memes at best.
7373737373 3y ago
Who runs this site?
MaxikCZ 3y ago
I want lemmy to succeed, but if it doesnt change, it never will. I got the idea of "Ill find some communities from reddit there". Typed Lemmy into google, first result was relevant - good there. Got to mainpage, decision to join or create server - well, I chose join coz that seemed more instant. But then it made me chose between several instances with no instant info about how will my choice affect what I see, and I just gave up there. I dont want to choose instance, I dont want to learn how it works. I want content first, and if I like it, then maybe I learn the inner workings to get more of it. But as it stands now theres just no way Lemmy will ever become mainstream.
- cwkoss 3y ago
  It's open source and user growth has recently exploded. It's going to be changing rapidly in the coming months.
- spaduf 3y ago
  It feels like there's an easy fix for this too. Instances already have an about section. Simply add a short form version that can be used in circumstances like this.
distcs 3y ago
> We're over 33k users now, having the second largest Lemmy server there is. It became the largest a few hours ago!

167 points by dragontamer 3y ago

There were some Lemmy-related discussions going on this past weekend. I think this is the one tidbit everyone around Hacker News wants to know: > The current VPS couldn't be resized that much anymore, and load was going up with all the new users. So I bought the same server at Hetzner: a 32-core/64 thread 128GB RAM dedicated server. (For Mastodon, I doubled the RAM. For Lemmy I don't think it's needed yet.) I migrated the Lemmy software and database there, and moved over. This took 4 minutes of downtime. So 32-core/64-threads with 128GB of RAM is running Lemmy.world, which is supporting ~36.7k users at the moment. --------- A brief look at Hertzner comes up with: https://www.hetzner.com/dedicated-rootserver/ax161 https://www.hetzner.com/dedicated-rootserver/ax161 A 32-core/64 thread + 128GB RAM server in Germany for 142 EUR / month.

gareve 3y ago
with some basic extrapolation, at how many users they'll hit the extra expensive Hertzner servers? or in other words, at which point they'll need to improve their architecture?
- zmmmmm 3y ago
  it'll be interesting to see if at that point, federation starts to become the way of scaling or not. If it was seamless, it wouldn't matter where you signed up and they could just host multiple lemmy instances on different servers. So far I have rather spotty experiences though with content sometimes making it to federated servers, sometimes not etc etc.
  - dragontamer 3y ago
    User scaling seems easy enough. Its "community" (on Lemmy), or "magazine" (on kbin) scaling that seems hard. But since each server has a local-copy of the community that its serving out, maybe the hardest part has already been solved by the Federation model. Each federated-instance is effectively a proxy / front-end for the users on that instance. --------- I guess Mastodon is way larger than Lemmy though and they haven't had issues yet.
- rglullis 3y ago
  I honestly wish that the answer is "when we get so big to the point that a single machine can not handle it, we close registrations". Can we please drop the "number must go up" mentality? The whole point of federated systems is to avoid concentration of power in a handful of servers. I'm sure that the people doing there have good intentions, but why can't we just let things just a little bit dispersed?
  - ajsnigrutin 3y ago
    Sure, until you google some error, or some other random thing, find some thread somewhere, want to comment/ask/contribute, and you can't, since the registrations are locked.
    - Stephen304 3y ago
      I'm on kbin.social and participate in communities across ~20ish other instances regardless of whether they've turned off registrations temporarily or permanently. Disabling registrations only prevents new accounts on that instance, it doesn't stop people from posting and commenting from other instances.
      - rglullis 3y ago
        Parent comment is the perfect example of how we got so used with dealing with shitty and user-hostile systems. We've been dealing with walled gardens for almost a generation now, it's like they don't even understand that it is possible to interact with a remote system without having everything in one centralized database.
        phs318u 3y ago
        People have forgotten how email works.
        FormerBandmate 3y ago
        Gmail isn't a walled garden but it doesn't have a 20k user limit
- dragontamer 3y ago
  1TB RAM Hertzner servers are available, so at least 8x more scaling before that's a problem. 2TB RAM is common in commodity servers, albeit expensive ones ($1000ish/month). Somewhere between 4TB to 20TB RAM is the pragmatic limit (where costs for vertical scaling start to get far worse)
  - tedunangst 3y ago
    Is RAM still the limiting factor at that scale? Are you assuming 256 CPU cores?
    - dragontamer 3y ago
      Current theory is that Lemmy software is mostly RAM limited. No one knows for sure until we reach those caps.
  - fulafel 3y ago
    Interestingly it seems max memories have been going down or at least not increasing in commodity x86 servers. Vendors advertised 24 TB servers enabled by lots of (192?) DIMM sockets in 2018 or maybe even earlier.
creamyhorror 3y ago
32 cores and/or 128GB RAM - for how many concurrent users, 36.7k? It seems like Lemmy requires a fair bit of resources per user for a primarily text-based API, more than I'd expected. Wonder where the majority of the resource burden lies - uncached local database queries? Too many simultaneous database connections?
- jochem9 3y ago
  Federation takes quite a toll. The way I read the article is that they didn't upgrade every time the load was (almost) too high, but instead opted immediately for the beefy dedicated server (considering the price it makes sense – it's on par with beefy VMs that have less power). They probably have resources to spare right now, but no need to go through the hassle of an upgrade soon (although 4 mins downtime sounds like they have things under control). Having said that, I'm sure there's plenty to improve in Lemmy, given it's used at a bigger scale now.
- spaduf 3y ago
  Keep in mind a lot of instances are preparing for what they anticipate to be the largest migratory wave yet when third party apps shut down at the end of the month.

eigenspace 3y ago

I wish Lemmy's default design was more of a copy of old.reddit.com. For me, that design was better than both HackerNews' design and way better than the current default look of Lemmy instances.

tester457 3y ago
Same. However it seems most people don't like the old.reddit look, only 4% of redditors use it. Most people are content with the slow bloated new reddit. When redditors talked about what sites to migrate to most of them preferred the modern interfaces over text based ones like tildes.net.
- Narishma 3y ago
  How many even know that old.reddit exists?
  - aerio 3y ago
    5%
- tuchsen 3y ago
  I wonder how much this has to do with the devices people use to access Reddit. I always preferred the old.reddit site on the desktop. Although when I'm on my phone, it gave me eye strain to try and read the tiny font, and pinch/zooming all the time is super annoying. The newer Reddit web app was still incredibly annoying, given all the "install the desktop app" prompts, and it's general lagginess, but allowing me to just flick scroll through articles was actually better for me. Lemmy's the same way for me, hate the UI on the desktop but it's fine on my phone.
  - Gigachad 3y ago
    I don’t think there is anything visually wrong with the new reddit. My issue is with how slow and buggy it is. And all the dark patterns and engagement tricks it’s full of.

Vermyndax 3y ago

So, single server scaled vertically? Gross.

anaganisk 3y ago
Is it /s? If not, deploying a kubernetes cluster and over engineering it?
toomuchtodo 3y ago
HN runs on a single core. https://news.ycombinator.com/item?id=29927630 https://news.ycombinator.com/item?id=29927630
- shagie 3y ago
  HN has a lot of optimizations and limitations on the user experience. For example, I don't see the scores of every other post on HN - and certainly don't see them updated in realtime. New comments and posts show up when I reload the page - not as I scroll. There's only one HN instance - without having 100 more instances trying to mirror the content (just one - firebase). ... and HN is an example of a single, centralized, unfederated server.
spicyusername 3y ago
The default Lemmy deployment docs don't have a lot of breathing room. It would be reasonable to break off the postgres server Lemmy uses to a separate server, but at the moment there is only an Ansible playbook or a docker compose file, both of which assume a single server. There's only so much time in the day and it's much easier to size up than reimplement the install process.

camdenlock 3y ago

Took a look at mastodon.world out of curiosity. Total conformity of belief on the entire front page timeline. Maybe it’s another one of those Mastodon instances where anti-capitalist social justice is the approved dogma, and dissenters are banned? Seems to be a lot of those…

yosito 3y ago
The beauty of Mastodon is that you can set up your own server if you don't like how someone else runs theirs!
- Arn_Thor 3y ago
  A sort of “safe space”, if you will
- xigoi 3y ago
  And then you lose the benefits of federation.
kstrauser 3y ago
I run such a Mastodon instance. Go build your own and compete with me for users if you think there’s a demand for it.
jbm 3y ago
Sorry to see you are being downvoted for making an observation. I have been looking across various Lemmy instances and I recall seeing one or two that was cyberpunk-based or technology-focused in their local streams, but far too many have the "smug American Liberal" [1] style comments that made me leave Reddit [2]. Simply spinning up another community is not a quick fix either. Concern trolling means Centrist communities simply do not last, as they eventually get overrun by extremists of one kind of another. It's far, far too easy to take a complex argument and have it dishonestly re-interpreted in a negative way as a a one line meme in public communities. Weeding these people out is far more of a challenge than banning violent remarks, racists and child abusers, as well-meaning people will inevitably misunderstand. The solution to this is small private communities and group chats with people you trust, personally. You won't get the hit from having constant content to consume, but it's not as bad for you; getting in the door is the real problem. [1] https://www.vox.com/2016/4/21/11451378/smug-american-liberalism https://www.vox.com/2016/4/21/11451378/smug-american-liberal... [2] Signing up for an account because the defaults are crazy is not a viable solution

FullstakBlogger 3y ago

lemmy.world, for whatever reason, absolutely guzzles CPU cycles. I can't imagine what it could possibly need to do that requires executing that much javascript. It's all minimized, so it's hard to follow in the profiler.

alephaleph 3y ago
Apparently it’s bc it uses websockets to live update vote counts or smth

pentagrama 3y ago

> I started paying for the Lemmy cloud servers from the mastodon.world funds. Did this person ask to the mastodon.world Patreon [1] and Open Collective [2] supporters that he was going to use money for other project? If not, seems disingenuous to me. [1] https://www.patreon.com/mastodonworld https://www.patreon.com/mastodonworld [2] https://opencollective.com/mastodonworld https://opencollective.com/mastodonworld

TX81Z 3y ago

Having worked in the tech legal and compliance, privacy, and security spaces for a very long time the idea of letting thousands of strangers upload binary files to a server I’m legally liable for is terrifying.

brightlancer 3y ago
Even "text" files can be encoded binary files -- it was a big shift in USENET. I had a heated discussion with some Nostr advocates who thought that by calling servers "avocados" or something, then they wouldn't be responsible for what users posted on the "avocados". I've not looked into ActivityPub or the Fediverse enough to know what is hosted locally and would (not) be protected by Sec 230, but it might drive some centralization toward corporations or other accountable organizations.
- TX81Z 3y ago
  base64 is a helluva drug.
dragontamer 3y ago
In practice, I think only the NSFW Lemmy's have to be worried. This isn't straight binary file hosting. Its just text, image, and video. Images/Video can get into trouble but if you have a Safe-for-work only policy and defederate from NSFW instances, you're probably in the clear? And I'm pretty sure any large amount of data is going to be throttled. So not even "big" videos are going to be hosted, more like a few-second .mp4 memes at best.

7373737373 3y ago

Who runs this site?

MaxikCZ 3y ago

I want lemmy to succeed, but if it doesnt change, it never will. I got the idea of "Ill find some communities from reddit there". Typed Lemmy into google, first result was relevant - good there. Got to mainpage, decision to join or create server - well, I chose join coz that seemed more instant. But then it made me chose between several instances with no instant info about how will my choice affect what I see, and I just gave up there. I dont want to choose instance, I dont want to learn how it works. I want content first, and if I like it, then maybe I learn the inner workings to get more of it. But as it stands now theres just no way Lemmy will ever become mainstream.

cwkoss 3y ago
It's open source and user growth has recently exploded. It's going to be changing rapidly in the coming months.
spaduf 3y ago
It feels like there's an easy fix for this too. Instances already have an about section. Simply add a short form version that can be used in circumstances like this.

distcs 3y ago

> We're over 33k users now, having the second largest Lemmy server there is. It became the largest a few hours ago!