4 ms·

> already upgraded the server as high as possible (16€ to 67€ per month) No judgement on you or the dev using "as high as possible" here, I realise it could be

> already upgraded the server as high as possible (16€ to 67€ per month) No judgement on you or the dev using "as high as possible" here, I realise it could be a simplification in this instance. However, I have seen a fairly strong pattern of fediverse servers being run by people with relatively few skills in system administration/web apps/operations/etc. It seems like there are a lot more people who are engaging with it on the same level as people renting multiplayer game servers, or installing Wordpress on a consumer-focused VPS provider. And I think this is kinda great. I want the fediverse to be accessible at that level or even below! It does lead to issues like this though. 67 euros per month is a small server, and a single server is not the best way to run a service that other people are using. It's different when it's your personal site, but a social network? I think the fediverse needs to figure out whether it wants to focus on making smaller servers easier or bigger servers easier. It's fine if most people are on million-user servers with teams running them, and it's fine if most people are on family servers that can go down without much negative impact, but this middle ground where people are on ~10k sized servers that individuals without the necessary skills are struggling to keep running on a volunteer basis seems to be a bad middle-ground.

tayo42 3y ago
I was digging through the lemmy perf issues a little and was surprised how they are doing things. Cool they got as far as they did though. Maybe I really underestimate how unique knowing how to scale things is
lolinder 3y ago
In theory shouldn't there be able to be both? Some apps should be able to optimize for large-scale usage while others optimize for ease of installation and maintenance in your spare time, but all should theoretically (barring the inevitable drama) be able to federate with all the others. I do wonder what the funding model is supposed to be for the large servers. Tiny servers can be funded out of pocket, but large platforms can't, and the resource requirements of fediverse apps seem to be much higher per-user than those of a centralized platform. (I'm not sure if the resource requirements are inevitable or just a sign of an immature app.)
- spaniard89277 3y ago
  With +1gbps fiber available many people could even host them at home, I guess.
  - 29083011397778 3y ago
    Hell, I currently don't even have gigibit fiber and self-host a Mastodon instance at home. Currently hosting Plex (meaning content-sharing with friends and family), Mastodon, Matrix, Airsonic, Wireguard, PiHole, and more. Everything runs off an HP EliteDesk with QNAP hosting all the media. The SFF PC is quiet and uses relatively little power. I am glad I have no data limit though - I apparently went through >3 terabytes (split 30/60 between up and down) last month alone, mostly due to Plex and Wireguard.
- danpalmer 3y ago
  Yes! Although I think that targeting both is unlikely to work in one app, which means we need interoperability and true federation between different codebases. Currently this leaves a lot to be desired. It sorta works for status-based updates with Mastodon and a few others, but it's not a great situation as Mastodon isn't the most standards compliant player, and the standards themselves are pretty loosely defined.
Semaphor 3y ago
You have to remember, that until 1-2 weeks that 16€ server was more than enough. This was a tiny community and most people (including me) had not heard about kbin, many not even about lemmy. My eventual plan is to host my own 1-user instance and rely purely on federation.
Saris 3y ago
Also from what I've heard about Mastodon, running it is surprisingly resource intensive, for whatever reason the server doesn't seem to be very efficient. I wonder if this has the same issue.
- jeromegv 3y ago
  Not sure about Lemmy. But for mastodon, there’s an incredibly high amount of servers to federate and keep a connection with. Then add all the storage for all the content being created. I can imagine it being heavy.
  - Semaphor 3y ago
    According to the docs [0]: > Lemmy uses roughly 150 MB of RAM in the default Docker installation. CPU usage is negligible. [0]: https://join-lemmy.org/docs/en/administration/administration.html https://join-lemmy.org/docs/en/administration/administration...
  - selectodude 3y ago
    I made my own server for mastodon, had to shut it down because it was pulling like 10GB per day. It was fun but that was insane.
Brian_K_White 3y ago
The entire problem of fediverse versions of twitter & reddit reminds me of bbses, which I just remembered I loved and miss. The negatives all just became positives, or at least drivers towards the positives.
croutonwagon 3y ago
I agree here. I looked into setting up a public instance. Realized quickly if it gained tracking a single box is not going to work and you are immediately in a scale out setup of postgres at least. Once you get to 3 postgres nodes, 2 LB's for the DB traffic, then Front ends with Load balancing, you are looking at 9 servers-ish minimum.... That monthly cost even in Linode is a bit more than I would be willing to front with no compensation. That said, i may just setup an instance of kbin I can control and rely heavily on the federated portion to interface with the various communities. If my understanding is correct, I can access other lemmy instances even with kbin, such as beehaw etc.
- JoeC3 3y ago
  Yeah, that seems to be the case. I'm on a kbin instance, can join magazines (subreddits) on other instances. Votes seem to propagate across instances.
fho 3y ago
Honest question ... How far can you push "one server"? Given that HN runs on one server (plus one mirror) and this apparently serves a not that small community well ... Given "1 million connections" are possible on one server ... Given that green threads are a thing these days ... Given that SQLite is so much faster than networked databases ... Given that a monolith is faster than Microservices ... Given that it's possible to use a compiled language instead of trying to speed up interpreted languages ... What's the limit here? HN is a great example of what's possible, but can a single beefy server scale to handle the traffic of a Reddit?
- robben1234 3y ago
  Serving a social network is not the case where you'd want just one server. Mainly due to reliability concerns, spikes in traffic and different patterns at different times. But if you like the idea of ignoring best practices, there are EC2 instances with 192 vCPUs that go for several thousands an hour. And taking as example Golang benchmarks https://github.com/smallnest/go-web-framework-benchmark https://github.com/smallnest/go-web-framework-benchmark where it's not uncommon to be able to serve 6 digits of 30ms requests per second with just 4 cores. Then the conclusion is, the limit is the sky, if you are stubborn enough.
  - fho 3y ago
    > But if you like the idea of ignoring best practice ... do you know me? ;-)
  - fho 3y ago
    Another dumb question ... for "several thousand an hour" I could buy a pretty decent server myself ... last time I checked (on https://yourdatafitsinram.net/ https://yourdatafitsinram.net/) the largest physical server there was like 10 grand. Combine that with the ability to actually throttle a physical server (ie you only pay what you actually use, not per hour) ... I guess I would propose the following scaling strategy: 1. get cheap physical server 2. get your software up and running 3. get a second physical server that is roughly twice as capable 4. use that second server as a "beta" and development server ... do a lot of profiling there 5. once your real world usage spike on the first server routinely reach 80-90%, switch production over to the bigger server 6. get a new development server that is twice as beefy as the former development server

tayo42 3y ago

I was digging through the lemmy perf issues a little and was surprised how they are doing things. Cool they got as far as they did though. Maybe I really underestimate how unique knowing how to scale things is

lolinder 3y ago

In theory shouldn't there be able to be both? Some apps should be able to optimize for large-scale usage while others optimize for ease of installation and maintenance in your spare time, but all should theoretically (barring the inevitable drama) be able to federate with all the others. I do wonder what the funding model is supposed to be for the large servers. Tiny servers can be funded out of pocket, but large platforms can't, and the resource requirements of fediverse apps seem to be much higher per-user than those of a centralized platform. (I'm not sure if the resource requirements are inevitable or just a sign of an immature app.)

spaniard89277 3y ago
With +1gbps fiber available many people could even host them at home, I guess.
- 29083011397778 3y ago
  Hell, I currently don't even have gigibit fiber and self-host a Mastodon instance at home. Currently hosting Plex (meaning content-sharing with friends and family), Mastodon, Matrix, Airsonic, Wireguard, PiHole, and more. Everything runs off an HP EliteDesk with QNAP hosting all the media. The SFF PC is quiet and uses relatively little power. I am glad I have no data limit though - I apparently went through >3 terabytes (split 30/60 between up and down) last month alone, mostly due to Plex and Wireguard.
danpalmer 3y ago
Yes! Although I think that targeting both is unlikely to work in one app, which means we need interoperability and true federation between different codebases. Currently this leaves a lot to be desired. It sorta works for status-based updates with Mastodon and a few others, but it's not a great situation as Mastodon isn't the most standards compliant player, and the standards themselves are pretty loosely defined.

Semaphor 3y ago

You have to remember, that until 1-2 weeks that 16€ server was more than enough. This was a tiny community and most people (including me) had not heard about kbin, many not even about lemmy. My eventual plan is to host my own 1-user instance and rely purely on federation.

Saris 3y ago

Also from what I've heard about Mastodon, running it is surprisingly resource intensive, for whatever reason the server doesn't seem to be very efficient. I wonder if this has the same issue.

jeromegv 3y ago
Not sure about Lemmy. But for mastodon, there’s an incredibly high amount of servers to federate and keep a connection with. Then add all the storage for all the content being created. I can imagine it being heavy.
- Semaphor 3y ago
  According to the docs [0]: > Lemmy uses roughly 150 MB of RAM in the default Docker installation. CPU usage is negligible. [0]: https://join-lemmy.org/docs/en/administration/administration.html https://join-lemmy.org/docs/en/administration/administration...
- selectodude 3y ago
  I made my own server for mastodon, had to shut it down because it was pulling like 10GB per day. It was fun but that was insane.

Brian_K_White 3y ago

The entire problem of fediverse versions of twitter & reddit reminds me of bbses, which I just remembered I loved and miss. The negatives all just became positives, or at least drivers towards the positives.

croutonwagon 3y ago

I agree here. I looked into setting up a public instance. Realized quickly if it gained tracking a single box is not going to work and you are immediately in a scale out setup of postgres at least. Once you get to 3 postgres nodes, 2 LB's for the DB traffic, then Front ends with Load balancing, you are looking at 9 servers-ish minimum.... That monthly cost even in Linode is a bit more than I would be willing to front with no compensation. That said, i may just setup an instance of kbin I can control and rely heavily on the federated portion to interface with the various communities. If my understanding is correct, I can access other lemmy instances even with kbin, such as beehaw etc.

JoeC3 3y ago
Yeah, that seems to be the case. I'm on a kbin instance, can join magazines (subreddits) on other instances. Votes seem to propagate across instances.

fho 3y ago

Honest question ... How far can you push "one server"? Given that HN runs on one server (plus one mirror) and this apparently serves a not that small community well ... Given "1 million connections" are possible on one server ... Given that green threads are a thing these days ... Given that SQLite is so much faster than networked databases ... Given that a monolith is faster than Microservices ... Given that it's possible to use a compiled language instead of trying to speed up interpreted languages ... What's the limit here? HN is a great example of what's possible, but can a single beefy server scale to handle the traffic of a Reddit?

robben1234 3y ago
Serving a social network is not the case where you'd want just one server. Mainly due to reliability concerns, spikes in traffic and different patterns at different times. But if you like the idea of ignoring best practices, there are EC2 instances with 192 vCPUs that go for several thousands an hour. And taking as example Golang benchmarks https://github.com/smallnest/go-web-framework-benchmark https://github.com/smallnest/go-web-framework-benchmark where it's not uncommon to be able to serve 6 digits of 30ms requests per second with just 4 cores. Then the conclusion is, the limit is the sky, if you are stubborn enough.
- fho 3y ago
  > But if you like the idea of ignoring best practice ... do you know me? ;-)
- fho 3y ago
  Another dumb question ... for "several thousand an hour" I could buy a pretty decent server myself ... last time I checked (on https://yourdatafitsinram.net/ https://yourdatafitsinram.net/) the largest physical server there was like 10 grand. Combine that with the ability to actually throttle a physical server (ie you only pay what you actually use, not per hour) ... I guess I would propose the following scaling strategy: 1. get cheap physical server 2. get your software up and running 3. get a second physical server that is roughly twice as capable 4. use that second server as a "beta" and development server ... do a lot of profiling there 5. once your real world usage spike on the first server routinely reach 80-90%, switch production over to the bigger server 6. get a new development server that is twice as beefy as the former development server