4 ms·

> A social network has to make money somehow because it has bills to pay. Hosts aren't free. Servers aren't free. Sure, but probably a FOSS social network woul

• krapp 2y ago
  That's basically Mastodon, which isn't free. Plenty of small instances that try the donation model or that are just funded out of pocket go under. And I may be wrong but I don't think the recommendation algorithms and other such features take up as much of the cost as you're claiming. I think a lot of the cost of something like Facebook is probably taken up by infrastructure and storage. Recommendation algorithms probably aren't that expensive.
  • derefr 2y ago
    The algorithms themselves aren’t expensive, no. Having extra entire complete copies of the relationship-plus-posts graph, denormalized in various ways (incl. in ways that inherently prevent use of easily map-reducible algorithms, and so require heavy vertical scaling) such that you can run the algorithms, is what’s expensive. And constantly feeding the data into those denormalized models, using specially-tuned realtime ETL technologies that themselves do distributed scaling to ensure no infinite queue backlogging from activity bursts, is also expensive.
    • photonthug 2y ago
      In general i would argue that it is bloat that makes it not feasible to fund as a nonprofit, although people may have different ideas about where / what that bloat is. Recommendation algorithms, or unnecessary product changes, etc. I don’t think it’s true that it’s intrinsically impossible for a public service to be self funding, and I think that not everything has to grow / change forever to remain relevant. We need to figure this kind of stuff out, I mean Wikipedia is nice and all but it’s really bad that humanity in general has to rely on megacorp for things as basic as maps while we say we’re living in the Information Age
  • noirscape 2y ago
    Do keep in mind that Mastodon is build on a tech stack that's mainly known for not being very efficient since it solved the scaling question with the answer "throw more computer at it". (In other words, it's a Rails app.) It's not very suitable for a free social media network since it's designed in a way to encourage large silos since that's the only way Rails scales from a financial perspective; you need more from a smaller core of users as opposed to having every user pay a tiny amount. There's other AP implementations that aren't a constant server hog like Mastodon is and can run on much weaker hardware (some of it can run on a raspberry pi). You don't need a full rails stack if your user count never exceeds 100 (which y'know, is the ideal state of AP - small communities who can remotely interact with each other).
    • neonsunset 2y ago
      Someone ought to rewrite it in something less embarrassing given you only need to be protocol compatible.
      • ebiester 2y ago
        The second system will always be playing catchup to Mastodon features or they will fork in features, meaning clients will have to support both or take sides.
      • Zak 2y ago
        Multiple people have written multiple compatible alternatives that are lighter weight. Pleroma and its forks (Akkoma looked good last I checked) are popular for single-user servers.
• lelanthran 2y ago
  > nstead of, say, a feed constantly rebuilt to drive engagement and rage-bait, you'd just get a simple chronological feed of what everyone you're following is posting; with maybe the ability to dial down the number of posts you see from any given person you're following ("show me the only the most-liked Nth percentile of posts from this user") without unfollowing. But even that kind of filtering — in fact, even the merging of followed users' feeds! — could all be done client-side. The whole "feeds" part could be as simple as a post-triggered static-site-generator pushing JSON into an object-storage bucket hiding behind an edge cache. Modern Usenet :-)
  • krapp 2y ago
    Ah Usenet, that bastion of civil, intellectual discourse where the most brilliant minds of the day mingled in the rarefied air of their own fart clouds.
• evanelias 2y ago
  > 99% of the server costs of something like Facebook or Twitter, go toward the backends, analytical DBs, and graphical / ML models used to power "features" that no user wants This is incorrect. From direct personal experience, I strongly believe the backend infrastructure and staff required just to operate the core product functionality of a successful large-scale social network massively exceeds what could be provided by donations. Just in terms of core product OLTP data, Tumblr hit 100 billion distinct rows on MySQL masters back in Oct 2012. At the time, after accounting for HA/replication, that required over 200 expensive beefy database servers. This db server number grew by ~10 servers per month, because Tumblr was getting 60-75 million posts/day at this time. Then add in a couple hundred cache and async queue servers, and over a thousand web servers. And employees to operate all this, although we kept it quite lean compared to other social networks. Again, this was all just the core product, not analytics or ML or anything like that. These numbers also don't include image/media storage or serving, which was a substantial additional cost. Although Tumblr had some mainstream success at that time, it was still more niche than some of the larger social networks. At that time, Facebook was more than 2 orders of magnitude larger than Tumblr.
  • derefr 2y ago
    Also speaking from direct personal experience. I have an API SaaS (which, yes, I realize, as a B2B product, has a much smaller audience than a B2C platform, and so probably much lower reads per second... but I want to just talk about writes here, since reads and aggregates-of-reads get cached on so many other levels.) We have 1.5 trillion distinct rows living on a PG cluster right now. Growing at a higher rate. And that requires... four DB servers (four shards). And no replicas — all our prod query traffic hits these servers directly, as a mixed workload with the writes from our queue-consumer ETL agents. (The servers are "beefy", sure — but they're not that beefy. They've got 128 cores and 1TB of RAM each. That's not even top-of-the-line these days.) Despite serving a whole ton of QPS, these DB servers are not CPU-bound or memory-bound or even IO-bound. Our primary horizontal-sharding factor, in fact, is PCIe lanes — we're constrained mainly by the number of NVMe drives we can keep connected performantly per CPU [1] — and thereby, generationally, by "fast storage capacity" per server platform. One thing that's perhaps "special" about our use-case, though, is that our data is inherently append-only. Which is pretty great for OLTP performance: there's very little write contention, as we just partition the data by insert time (which is also a natural partitioning key[2] for most queries our business layer does) and then only write to the newest partition, with all previous partitions becoming effectively immutable. Most workloads aren't like this... but they could be, if you model your data carefully, with temporal tables holding versioned records and so forth. You trade off more storage growth, for far less worry about write contention. ("Event streaming" is not the silver-bullet solution it would seem here — as you would still need to create an aggregate to query from; and writing into that aggregate would still be an OLTP write bottleneck. No, the true solution here is data-driven [schema] design — i.e. forcing your API engineers to invite the DBA to feature design meetings, and ensuring the DBA has the perogative and incentives to say "no" to a bad design. Or you could just hire database systems engineers to code your business layer, like we did.) My point here, is that if you're very careful in designing for not just "scalability" but economies of scale — and if your org is engineering-driven rather than product-driven (as a non-profit social network would likely be!), and so has "mechanical sympathy" — then you can achieve things with just the budget of an average B2B SaaS, that would rival a (smaller) social network. But with the budget of a benefit corp that nevertheless charges even $1 for each install of its social-network mobile app? Who knows what you could achieve! (See also: WhatsApp, pre-acquisition. They were just charging $1 for each install. With just 50 employees, that got them a lot of operating budget.) --- [1] And actually, that's part of why the DBs are not IO-bound. Put enough NVMe disks in RAID0 together, and you get something nearly functionally indistinguishable from memory. (Yes, RAID0 — because our DBs aren't the store-of-record; our data lake is.) And even where NVMe reads are not indistinguishable from memory reads, they're still complementary. Postgres, in relying on mmap(2)ing heap files and thereby on the the OS page-cache as a buffer pool, makes per-query serial page faults expensive, and causes multiple threads wanting the same cold data to bottleneck by repeatedly becoming coalesced awaiters on the same sequence of pages faulting in. But when you've got highly-concurrent queries [= many different OS threads] that are all page-faulting for different cold data, then you get deep IO queuing, and everything works out optimally. So the expensive cases don't really come up — the page-cache serves the head of the request distribution, while the deep IO queues of the RAID0-NVMe-pool are a perfect match for the long tail. This makes it almost irrelevant, at runtime, whether data is hot or cold. As long as you have sufficient memory to host the very hottest data (e.g. relatively-low-cardinality fact tables like users/blogs that get joined to everything, and esp. their indices), the rest can be read straight from disk with nearly no penalty. --- [2] Funny enough, given that we're doing heavy joins for some queries, the one thing we do sometimes run out of at runtime, is timeslices of the Postgres postmaster to coordinate lightweight locking of the global locks table. We not only have time-partitioned tables, but also something akin to tenant-partitioned schemas. This results in a lot of relations. (It takes hours for us to run vacuumdb --analyze-only.) And some OLAP-ier queries (that we nevertheless have to run, synchronously, in response to client requests!) need to touch many of the partitions and many of the schemas. That's O(MN) locks they need to take — which take time to write into the global locks table, even though those are only memory writes. Every once in a while, we have to "consolidate" our table partitions — not by rolling them up with aggregations to reduce row-count, but rather by just copying all the data into fewer, larger partitions — so that we can take fewer access-shared locks per query, so that transactions can spend less time waiting on a handle into the global locks table at startup just to write these O(MN) read locks into it. (In other words, we consolidate data to reduce O(MN) read locks per transaction, to O(N log M) read locks per transaction.) But that's a PG problem, not a resource problem per se. We've been considering writing a patch for PG to allow relations to be marked as "immutable", where an "immutable" relation doesn't have locks tracked for it (or its indices) at all — but also can't be written to, re-indexed, or even dropped. We'd then just apply this setting to all of our historical partitions. (If you're curious: given its semantics, this property could be enabled for a relation at runtime; but would need an instance restart for disabling it to take effect, as the instance would have no idea what read-locks "would be being held if the table would have been tracking locks", and so couldn't safely do any writes to the table without a barrier that drops all existing MVCC read-states — i.e. an instance restart.) As a bonus, such relations could also have "perfect statistics" calculated for them by ANALYZE; and those stats could then be marked as never needing to be re-calculated — allowing both VACUUM and ANALYZE to be skipped for that relation forevermore.