24 ms·

Nation-scale Matrix deployments will fail using the community version of Synapse

141 points by zaik 2y ago

Etheryte 2y ago
At the end of the day, the money has to come from somewhere. There's only a handful of companies that make the open source profit game work and last I checked, most of them make their money from B2B deals with support, SLAs, etc. If you want to deploy something at nation scale, surely you can afford to pay for it.
- emacsen 2y ago
  While it's absolutely true that businesses need to make money, the idea that this must therefore logically conclude with a FLOSS and non-FLOSS version is not correct. If you look at companies like Red Hat, and Nextcloud, neither of these companies sold proprietary versions of their software. Red Hat sold licensed versions of their software, but they were always FLOSS. Nextcloud's CEO talks about how selling a proprietary version of your code is bad for your customers and puts your company into a Catch 22 with itself. "Do we add this feature to the 'Community' or 'Enterprise' version?" or worse "What do we do when someone makes a FLOSS plugin that competes or is better than our internal version of the same functionality?" There's a huge need for a good chat system to compete with the likes of Slack, but Matrix has bitten me several times and I don't trust it. In an alternate universe, they could have made Matrix work well for all levels of users and then sold customization and B2B deals with their customers leveraging the community, which has always wanted Matrix to succeed.
  - thaumasiotes 2y ago
    > There's a huge need for a good chat system to compete with the likes of Slack, but Matrix has bitten me several times and I don't trust it. I thought this was Zulip?
    - usr1106 2y ago
      Zulip is excellent for geeks. I don't think it's for the broad public. We use it in our company with great success, but our marketing people still have not understood how topics are used (both by their own comments and by the evidence I see regularly).
      - crabmusket 2y ago
        That's a shame! What sort of problems arise?
        usr1106 2y ago
        Topics only help to keep things organized and better than other chat systems if they are reusable and long-term identifiable. If people create a new topic like e.g. "Wednesday meeting" or "The printer does not work" every time they want to say something you end up with same mess as in other chat systems. Also it's quite human that discussions get offtopic. Zulip has nice support to continue in or move the discussion to the right place. But to my experience only a couple of geeks uses them.
  - portaouflop 2y ago
    Would you be able to point me to the talk or blogpost of nextcloud that talks about their business model in detail? I would be very interested in how it works in practice, how they differentiate with their enterprise versions and so on. I agree that their business model sounds like having the best of both worlds, so any material on the details that you could share would help me a lot figuring out how to replicate it for other solutions.
    - Arathorn 2y ago
      Last I checked they ratelimit push notifications using a proprietary push gateway, unless you’re a paying customer.
      - 9283409232 2y ago
        The feature exist in both versions but you pay for additional capacity which is not what they are talking about when they said deciding whether the feature should be in community or enterprise.
    - emacsen 2y ago
      If you look up Frank Karlitschek and "I forked my own project and my own company", he's given this talk many, many times. As for it being the best of both worlds, it depends what best means. A company that wants to make a lot of money won't do well with this model. It's a slow growth model requiring a lot of hands on work that requires customers who are paying for features and custom installations. It's labor intensive work that's quite the opposite of the proprietary software model, which has traditionally had a lot in common with the publishing or entertainment industry. Write a book, get paid for it for years.
  - whatevaa 2y ago
    RedHat sold to IBM and neutered it's free CentOS variant recently, so not so sure it's a good example.
    - graemep 2y ago
      CentOS is no longer a drop in replacement for RHEL, but both are still entirely open source other than trademarked logos and the like AFAIK. More importantly, Red Hat was very profitable even when CentOS was a drop in replacement.
      - Arathorn 2y ago
        I think the only reason Red Hat’s model works for them is that Linux distributions are obviously massive sprawling complex things, and maintaining LTS distros which you then sell support for is an incredibly complex and valuable thing that no intermediary can plausibly claim they can do. Even if the underlying code happens to be FOSS. Matrix servers (and web servers and mail servers etc) are not as complex - and if anything you want incentives to make them less complex, not more. And the better job you do of making them easy to admin and support (ie the more scalable and stable they are), the less reason there is to pay the upstream for support but some system integrator instead.
        eredengrin 2y ago
        > Linux distributions are obviously massive sprawling complex things Just my perspective but I think this was also perhaps a product of the time when Red Hat really started growing, back when linux was a little more rough around the edges. These days it seems linux has enough polish that some (many?) corporations are happy to just use off the shelf ubuntu and call it a day and hope nothing breaks too badly.
        graemep 2y ago
        I think another thing that has changed is that big business is less suspicious of open source and are less likely to feel that all the software they use has to be licensed from someone. I think it has dawned on people that it is impossible to avoid using open source.
nightfly 2y ago
So matrix is commercialized now and the "real" software version is as expensive as Slack/Rocket? That's really disappointing...
- robertlagrant 2y ago
  Is there an alternative? Can you set up your own server and it's free (as in beer)?
  - dsr_ 2y ago
    Yes. The article is about how a particular open source Matrix server implementation isn't suitable for millions of simultaneous users, and you should pay for the non-open-source version. But if you are merely handling tens of thousands of users, no problem. And if you have the budget to handle millions of users, maybe you have the budget to pay development as well as operations staff?
  - tetrisgm 2y ago
    Side note: why do people still say free as in beer? Do people not assume free automatically means price, unless there’s a qualifier?
    - seanw444 2y ago
      Within the programmer circle, "free software" implies FOSS, which is something more than just no-cost. A normie would assume free in cost though.
      - lukan 2y ago
        But the world is made up of normies and has no idea about the free software movement. And I believe the free software movement is lamenting that? Maybe clear communication would work better? Use words in a meaning, people understand intuitivly?
        2y ago
gwerbret 2y ago
Perhaps this should point instead to the actual blog post this link references: https://element.io/blog/scaling-to-millions-of-users-requires-synapse-pro/ https://element.io/blog/scaling-to-millions-of-users-require...
- WhyNotHugo 2y ago
  Basically, they rewrote the slow parts in Rust, but that’s only included in the propertary version. The open source one is the one they know has terrible performance. I don’t get why people want to me part of a community where the steward treats them in such a shitty way.
  - graemep 2y ago
    Two reasons: 1. its preferable to a proprietary alternative. 2. its an open standard so there are alternative implementations 3. Very few people need the high performance alternative. It might not be ideal, but what is the better alternative.
  - Arathorn 2y ago
    We didn’t rewrite the slow parts in Rust, we rewrote the bits which let large deployments scale (workers) in rust. Normal Synapse perf work will continue as ever, if not faster given it might have more funding. See https://news.ycombinator.com/item?id=42752911 https://news.ycombinator.com/item?id=42752911
Multicomp 2y ago
The worst part is the article reads like someone is nervous that the nation states are busting hosting services from somebody doing horizontal scaling via multiple server instances rather than doing the Proper rust implementation. The classic tension of oss vs paid version.
wkat4242 2y ago
I don't like this. I only have a handful of family users but I could still benefit from the additional vulnerability proofing of Rust. It's not just about the scaling. My data security matters too. I have not exposed it to the internet for a while now because I already had my doubts. Also, as 0xC0ncord mentioned large federated rooms caused too much load so I wanted to move to standalone anyway. For that too I could have benefited from this. This really sounds like "haha you community members are stuck with a sucky version of our software and you're never getting the good one!". It's not exactly making me feel valued as a community member. And we shouldn't forget this is what put matrix on the map. This feels very bait-and-switch'ey. Also pointing out the flaws in one's own software doesn't inspire a lot of confidence. I think it would have been a sounder approach to paywall real enterprise features rather than the core product. Such as SSO, auditing, legal holds, that stuff. Basically all the stuff that Microsoft Teams has in Purview. The community won't miss any of that stuff. Ps I thought dendrite was supposed to be the next big thing? It feels like element is often jumping on new things like big rewrites such as element X instead of improving the existing products.
- kumiokun 2y ago
  Yeah, as a past contributor and long-running homeserver admin I feel pretty rugpulled here. This violates the expectations set in past communications. Synapse was something very different to Element. (This would at least for me have had a very different taste and reception if the private for-profit alternative would have been done under a different branding than "Synapse Pro" and not hijacking matrix.org community channels to promote it. They don't seem to be able to keep their hats and chairs properly separated)
  - eredengrin 2y ago
    > I feel pretty rugpulled here How? Synapse is still being developed, this is just a commercial addon for huge customers. > and not hijacking matrix.org community channels to promote it. They don't seem to be able to keep their hats and chairs properly separated matrix.org has never excluded mentioning commercial entities, past blogs have frequently included information about etke or beeper for example.
0xC0ncord 2y ago
This is incredibly disappointing to me. Performance has always been one of the most glaring issues with using Matrix. The article seems to be intentionally obfuscating the fact that performance and scalability issues stem from the activity a server participates in rather than just the number of users it has. If you're in a rather large room with hundreds of members and it's constantly active, you'll quickly notice your server struggling to keep up. As a Matrix user, I feel like I'm being rug-pulled because Element HQ decided to gatekeep their solution to this problem that has plagued the community for more than half a decade.
- zaroth 2y ago
  Whatever you think a “rug pull” is, this ain’t it. FOSS owes you literally nothing. Go build what you want for yourself, geez. Oh, and feel free to give it away to the world… OR NOT!
  - wkat4242 2y ago
    No ok but then let's drop matrix completely as the "next-gen" IRC / FOSS collaboration platform. There's a lot of moral support and ecosystem gain (eg bridges) from the community. Why bother with that anymore if we're just getting the "broken implementation"? I think part of the issue here is that they're trying to compete with the enterprise IM solutions. That doesn't really work as the others are way better funded and have benefits that element can't deliver (such as the wide Microsoft M365 + EntraID ecosystem). A Microsoft shop is never ever going to choose this over teams (horrible as the latter is).
    - Arathorn 2y ago
      > Why bother with that anymore if we're just getting the "broken implementation"? You’re not. All work (other than stuff which primarily benefits scalability for enormous deployments) should land on FOSS Synapse - particularly core performance improvements like faster state res. This is just trying to fund it. And no, this is not us trying to compete with Teams - this is us trying to force big Matrix deployments to actually route $ to Element to fund upstream dev, rather than use System Integrators who win the big tenders and then mysteriously fail to route any $ to us.
karussell 2y ago
I find this comment interesting: https://mastodon.matrix.org/@element/113848890455666110 https://mastodon.matrix.org/@element/113848890455666110 where IMO it is stated that Deutsche Telekom (with HessenConnect 2.0) did the free-riding and "directly contributed to layoffs at Element in 2022 and 2023". It sounds as if Telekom changed its mind after an initial contract or something. Or was Telekom one of many companies that did a similar thing? I know from my own experience that making money based on an open source project is extremely challenging. However, I don't understand why they frame their Pro version in this way and point to failure instead of success. I can't think of a better way (they will have to do the work :)). But it's already a good thing if they've won them as "Matrix believers", and they shouldn't scare them to get there.
- Arathorn 2y ago
  > Or was Telekom one of many companies that did a similar thing? This. > However, I don't understand why they frame their Pro version in this way and point to failure instead of success. It’s really trying to spell out the risks of freeriding, given the severity of the situation. But you’re right that it may well be too negative. > I can't think of a better way (they will have to do the work :)). Someone proposed a better positioning in the OP thread at: https://ioc.exchange/@troed/113843356547707182 https://ioc.exchange/@troed/113843356547707182
- JeremyNT 2y ago
  Why does anybody want to host large scale Matrix servers anyway? Aren't they already helping just by providing infrastructure to run the software at all? Is anybody anywhere - Deutsche Telekom included - actually making money here?
  - Arathorn 2y ago
    I'll try to explain the situation: Some massive organisation (government agency, healthcare provider, network of schools etc) says: "Wow, Matrix is great, we should run our own instance rather than be dependent on Teams/Slack/Workspace. I'll put out a public tender for suppliers to provide this for us!" Then, all the system integrators who are geared up to sell big IT projects to large organisations jump on the tender and say "Sure, we can do this for you, and we're best qualified to do so because we're an enormous trusted system integrator with loads of people in the right geography, got all the right compliance checkboxes, and we do this sort of thing all the time. It'll cost you $xM a year." The upstream developer (Element) then reaches out to the integrators and says: "Oh hi! Are you planning to bid on this contract? If you want this to be successful, you should use our commercial offering, as then we'll be able to use that $ to keep upstream development alive". The integrator then looks bewildered, and says: "But... we've been using your opensource distro off github and it seems fine? Why would we pay you guys anything? The additional cost will just make our bid too expensive and we won't win, or it'll kill our profit margins. Sorry, we'll go ahead without you and maintain the software ourselves if needed." Given the main paying customers for Matrix are big organisations wanting digital sovereignty, this means the available $ disappears into the integrators rather making is way to fund the actual upstream dev - which then ends up stagnating, stalled, and completely at risk - putting all of these projects at risk. facepalm. A variation on the problem is where the massive org decides to do the project in-house rather than via an integrator... but then still chooses to YOLO and freeride and given it'll cost less than using the commercial offering from the upstream. I can list at least 4 national governments who have taken this route. To solve it, options include: * The buyer could mandate that suppliers for the tender have to use the commercial offering from the upstream product rather than freeride on opensource. This is probably the easiest solution, but seems staggeringly rare - the only instance I know of who have taken this principled stand are ZenDiS in Germany (https://zendis.de https://zendis.de) and we are very thankful for it. * The upstream could make their commercial offering more valuable than the FOSS offering - e.g. additional domain-specific functionality: what we've done with Synapse Pro. * The upstream could make the FOSS licensing less permissive, to force integrators to contribute back. We already did this with AGPL Synapse in 2023: https://element.io/blog/element-to-adopt-agplv3 https://element.io/blog/element-to-adopt-agplv3, and it has helped a bit... but in practice, we just saw a fleet of major Matrix tenders go to system integrators who then apologetically cut Element out, despite AGPL - putting Element's funding for 2025 directly at risk. Which is why we're putting out brutal blog posts saying "Look, big organisations: if you try to freeride on FOSS your projects will fail" - and providing a quantified explanation of the value that the commercial offering provides... so that we can then take the $ and hopefully keep the primary FOSS project on the road.
xyst 2y ago
Nation scale is Synapse “Pro”. Next level is multi-nation scale at “Pro+”. Then global scale version available at “Pro Max”. Universe scale at Synapse “Event Horizon” edit: what’s to stop community from re-implementing the worker component in rust and contributing it to the “pro” version?
- yencabulator 2y ago
  > edit: what’s to stop community from re-implementing the worker component in rust and contributing it to the “pro” version? The essence of Open Core suck, which Matrix has here committed to, is actively discouraging community work related to the bits that differentiate the proprietary version. History says any pull request improving the FOSS version too much, in the wrong direction, will be left unmerged. This is how "communities" die.
  - throwaway48476 2y ago
    Basically zero communities have the growth needed to compete with commercial offerings. Eventually users switch to a different paid offering because it has better fratures/performance. That's how communities die.
I_am_tiberius 2y ago
So FOSS is essentially a Marketing tool for Matrix?
- brookst 2y ago
  Isn’t that true for most/all corporate FOSS projects? I don’t think there are many for-profit companies who have some kind of idealistic vision of FOSS. It’s always part of the business model. Would be weird otherwise.
- Arathorn 2y ago
  Matrix is an open protocol published by a non-profit foundation. It has always had both FOSS (eg Element) and proprietary (eg Beeper) implementations. You sound like you are getting Element mixed up with Matrix. Element releases almost all of its implementations as FOSS, and to help Matrix grow, will make those as good as possible for average server sizes. It’s only gigantic servers which actually require the new proprietary modules. Given everyone who uses FOSS Synapse can keep doing so, and given FOSS Synapse dev should hopefully improve and accelerate if actually funded by $ from Synapse Pro, it’s not fair to say that “FOSS is essentially a marketing tool for Element”
jchw 2y ago
I'm pretty sick of Element and Matrix at this point, and I'm pretty sure the enthusiasts are the only ones keeping Matrix relevant. This monetization scheme is not going to work. I think the Matrix experiment is coming to an end. I'm about to fork Dendrite at its last ASL 2.0 version, because to hell with signing a CLA for Element.
- fishgoesblub 2y ago
  I'm tired of all the issues with Matrix as well. I've never had a message get received or a notification pushed in less than 10 minutes. Went back to XMPP and messages are instant. I want to love Matrix. E2EE, proper federation, voice calls, But the experience is terrible.
  - tcfhgj 2y ago
    > I've never had a message get received or a notification pushed in less than 10 minutes. I receive messages < 10 seconds. What's your setup?
    - fishgoesblub 2y ago
      Element (Later ElementX) On android, and same on iOS. Tried various other clients like Fluffychat, and SchildiChat.
      - Arathorn 2y ago
        Presumably you’re using an overloaded server which is struggling because we have been completely undermanned on synapse maintenance and haven’t been able to do core optimisation work. This is precisely why we’re trying to fund our FOSS work by selling Synapse Pro.
        throwaway48476 2y ago
        Proprietary licenses don't align developer incentives with customers. Instead you should consider non commercial licenses that convert to open source after some time, eg 5 years. Customers won't pay for development unless forced to and it creates a tragedy of the commons. No large commercial deployment will accept 5 year out of date broken insecure software. Non paying users won't care that they can't use it commercially
Leo_Verto 2y ago
It's a shame it had to come to this but I can see why Element had to make this choice. To me it feels like they've ended up in a position similar to Docker Inc. where they've spent years of work and tons of resources building the standards and reference implementations, but missed out on the juicy integration and support contracts. Docker has turned into the OCI standard and most Kubernetes deployment don't even run containerd anymore. The company has massively rate limited Docker hub and recently increased the pricing again, which has had much bigger impact for end users than this pivot by Element. I really hope this works out as a funding source for future Matrix development, I don't see how its ecosystem can continue to be developed and healthy without Element.
- koito17 2y ago
  > most Kubernetes deployment don't even run containerd anymore I assume you mean moby, which was used via dockershim in Kubernetes until support was removed in v1.24. I can't find a reliable source, but I believe containerd, followed by CRI-O and alternatives, are the common choices of container runtime for Kubernetes. https://kubernetes.io/docs/setup/production-environment/container-runtimes/ https://kubernetes.io/docs/setup/production-environment/cont...
  - ffsm8 2y ago
    It was called plain docker originally, the Moby rebrand came much later and not everyone is even aware that it happened, because plain docker was already much less relevant when they did it (vs k8s, which was already becoming the default by then). https://www.docker.com/blog/introducing-the-moby-project/ https://www.docker.com/blog/introducing-the-moby-project/
- deleted 2y ago
  [deleted]
vfclists 2y ago
Matrix have always been suspect. Disappointed but not surprised. In short they are saying that if the nation states hadn't opted to use Matrix they would never have bothered developing this Rust version, which only shows that they were never sincere about developing a performant open source version, not to mention their involvement with some suspect Israeli comms companies and the phone home agendas. Interesting question - will their "nation state" level customers have the right to inspect the source and make changes to it? Just waiting to have this post flagged.
- ezst 2y ago
  > In short they are saying that if the nation states hadn't opted to use Matrix they would never have bothered developing this Rust version, which only shows that they were never sincere about developing a performant open source version At that point, it's nice of you to assume that they are even capable of making it performant. But it is indeed "remarkable" that the people who started this, and who couldn't get anywhere with it, for over a whole decade, are still behind it today.
throwaway519 2y ago
Element have a perfectly good implementation written in Go called Dendrite. I run it. It's good. But support is deprioritised in favour of Synapse (Python). Moving the goalposts with Synapse does not rejuvenate Synspse, it creates a fragmented system. Get used to being boring. I am not moving to an ever centralised Synapse only ecosystem - a single implementation is not a federated protocol. That the new Element mobile client only supports Synapse was a huge red flag. If diversity in implementation is not encouraged with action I'm back to XMPP (+ Jitsi).
- Arathorn 2y ago
  > That the new Element mobile client only supports Synapse was a huge red flag. This is not true. Element X requires Matrix 2.0, which is currently only in Synapse because it has to land somewhere first. But Dendrite is implementing it (best effort, thanks to an amazing community contrib) too: https://github.com/element-hq/dendrite/pull/3493 https://github.com/element-hq/dendrite/pull/3493 and at conduwuit (one of the conduit forks) is too: https://github.com/girlbossceo/conduwuit/pull/666 https://github.com/girlbossceo/conduwuit/pull/666 There is absolutely nothing trying to lock EX to synapse.
  - throwaway519 2y ago
    Great to see some community efforts.
NewJazz 2y ago
Google, Microsoft, Apple, and Facebook should be required to integrate XMPP into their messaging products. Matrix is a cool idea but we now have like 1 server implementation that isn't somewhat gimpy, and its closed source... XMPP is much more compelling.
- blackeyeblitzar 2y ago
  In general all these large companies should be required to use open standards or spin off proprietary protocols into an independent open source project.
- miohtama 2y ago
  Google/Facebook did XMPP federation back in a day and it was spam hell.
  - BoredPositron 2y ago
    Matrix has the exact same problem?
  - jauntywundrkind 2y ago
    I had a pretty visible email address / xmpp name all over the web and i don't recall even a mild inconvenience from spam.
- airhangerf15 2y ago
  They did. Facebook's XMPP implementation was shit and unreliable. Google's actually federated, but they silently killed federation .. and then the entire protocol. (This was ~10 years ago) Exactly who should "require" these companies to implement theses standards? Some daddy government? Who would be forced to implement it? Just the big sites? What makes a site big enough? If it were required, it would be used to lock out smaller players. Consumers could demand it, but we know the average consumer doesn't even know what a protocol is, or understand the different between a web browser and e-mail. Most don't even know how their phones work.
heybrendan 2y ago
Arathorn, should this message reach you, I want to voice my support of you and your team. Notwithstanding the naysayers that surface with peculiar predictability any time Matrix is discussed here, know there are people like me that are happy Synapse users (and operators, nearly 5 years now!) that are in your corner, rooting for you [and Matrix]. Keep fighting the good fight.
- mgrandl 2y ago
  Jup same here. There is always so much negativity on Hackernews. I have been running synapse without issue for years and it has been a joy!
- vaylian 2y ago
  Agreed. Open source is an ecosystem. And ecosystems need to be healthy to grow. Matrix is not a hobby project that a couple of hackers implement in their free time but a large project that requires full-time developers. But if these developers don't get paid, then we (the community) won't get improvements for matrix software. Synapse Pro is a compromise to keep the ship afloat. This is not a matter of greed, but a matter of keeping the lights on.
- Arathorn 2y ago
  Thank you - and to the sibling comments too; the support is enormously appreciated. For what it’s worth, we have been frantically trying to keep all Synapse dev FOSS for the last 10 years - and the FOSS version will always be the primary project, where everything new happens (other than mega-scalability work). At least for as long as we’re able to work on it. My hope is that Synapse Pro will be net positive for the ecosystem, as it will hopefully fund us to be able to work more on FOSS Synapse. It could also inspire more alternative homeserver work, which is no bad thing for Matrix as a whole.
- sneak 2y ago
  The good fight does not involve proprietary software.
benatkin 2y ago
The audacity of this: > tl;dr is more "need to be able to keep paying people to work on this". https://mastodon.matrix.org/@element/113844400165509238 https://mastodon.matrix.org/@element/113844400165509238 That's your problem. I only had paid attention to this project in the past because it was under a permissive open source license.
- camgunz 2y ago
  Do you have something against making money on FOSS? A weird thing has happened while I've been reading this thread where I've noticed some outsized cynical or negative comments and realized they're both your user, then I remembered your bad faith arguments in the thread about the license change. I'm too lazy to check; do you come to every Matrix/Element thread to be uselessly negative or just these two?
  - defrost 2y ago
    https://hn.algolia.com/?query=matrix%20by%3Abenatkin&sort=byDate&type=comment https://hn.algolia.com/?query=matrix%20by%3Abenatkin&sort=by...
  - deleted 2y ago
    [deleted]
throwaway48476 2y ago
Another case where an 'eventually open source' license would be beneficial.
- Arathorn 2y ago
  This could be an option here; we haven’t thought about it yet.
  - throwaway48476 2y ago
    Free software is just a hack of the copyright system. The point of which is to incentivize people to create works and be paid for them and after a period of time be added to the public domain. Eventually open source follows the same principle, just with source availability.
tonygiorgio 2y ago
I was a paying pro user for a <1000 person server years ago. They forced me off of it due to offerings they were no longer servicing. Told me I had to self host and export all my data. I attempted this and it never worked. I abandoned that server and my profile I used across many matrix instances (and somehow my matrix room continued to run without me hosting it, and without an admin running it). I will never use nor recommend them ever again. They clearly do not know how to operate a business nor an open source project.
- intelfx 2y ago
  > somehow my matrix room continued to run without me hosting it, and without an admin running it Surprise, this is probably the primary architectural feature of Matrix. This is precisely what it's designed to do.
ssivark 2y ago
I sincerely wish you good luck with building -- hope people find this new product useful. This bait-and-switch leaves a really bad taste for me though; I now feel silly for having advocated for Matrix, and will cease henceforth. You can lament the free-rider problem all you like, but suddenly making it a big deal now betrays that you do not take seriously enough your own promises, and were clueless about the fundamentals all this while. This is not meant to be vindictive (I wish Matrix success as a product; it's certainly useful), but simply making clear that the expectations and standards are different for a "community" -vs- simply a business that sells a product. It seems to me like this disqualifies Matrix from consideration for "public infrastructure" IMHO. It would have been better if the differentiation between the paid and free versions wasn't scaling, but some other features not critical for public use (or provided as a non-free plugin), but something that enterprises would consider useful.
- Arathorn 2y ago
  > It seems to me like this disqualifies Matrix from consideration for "public infrastructure" IMHO. Please don’t conflate Synapse with Matrix. Yes, if a govt has a “we must only spend money on pure open source products” policy then they won’t buy Synapse Pro. Perhaps they’ll end up scaling by running hundreds of separate smaller FOSS instances instead. Perhaps other server implementations will step up as FOSS (and probably promptly be undermined by the same freerider problem). > It would have been better if the differentiation between the paid and free versions wasn't scaling, but some other features not critical for public use (or provided as a non-free plugin), but something that enterprises would consider useful. We’ve tried that (https://element.io/server-suite https://element.io/server-suite) and it hasn’t been enough.
  - AAAAaccountAAAA 2y ago
    > We’ve tried that (https://element.io/server-suite https://element.io/server-suite) and it hasn’t been enough. Will this finally be enough, or will it end up so that even more features are going to be available only on the proprietary version?
    - 2y ago
DoItToMe81 2y ago
I hope this is poorly phrased or I am misunderstanding it, because it feels like it's implying that Matrix's terrible scaling issues will have their fixes paywalled. Even non-federated homeservers chew up RAM and CPU like candy when you get too many users. It's heavier than any other protocol or chat program I know.
- Arathorn 2y ago
  Most of that performance is nothing to do with whether workers are written in python or rust - see https://news.ycombinator.com/item?id=42752911 https://news.ycombinator.com/item?id=42752911. The hope is to fund maintenance for FOSS synapse, including core perf work, via Syn Pro stuff which makes the biggest impact for gigantic servers.
knallfrosch 2y ago
What surprised me most is how the community and pro versions use completely different core technologies. I guess licensing and FOSS don't work together. But then the pro version doesn't benefit from the open-source scrutiny of the community version, especially in regards to cryptography and security. What's the reasoning for this technology split?
- Arathorn 2y ago
  FOSS Synapse already is a hybrid of Python and Rust, and has been for years - with Rust used for the hot paths. Synapse Pro is just pushing that a bit further for worker processes, by having the host process itself being Rust-first rather than Python-first. But the majority of the app and architecture is the same; it's just swapping out some of the python-with-rust processes for pure-Rust alternatives. I think of it a bit like using a hardware accelerator (how I envy the fact that nobody complains about the fact that hardware accelerators cost money!)
whytevuhuni 2y ago
Happy to see Rust being used for this. I tried using it for some toy implementation of a Matrix federation server [0], and there's lots of zero-copy/zero-allocation optimization tricks Rust allows, if one is willing to spend that time on it. I'd really wish to see an official non-Python OSS version though, to make it easier to run on a Raspberry Pi or whatever other tiny homelabs people usually host things on. When I tried Synapse, it essentially ate the whole memory and was constantly making the fan turn on. [0] https://github.com/andreivasiliu/fluctlight https://github.com/andreivasiliu/fluctlight
- ezst 2y ago
  There's conduit as an alternative implementation in rust, but Matrix being what it is, you won't be able to use the newer clients with it, and federation will be glitchy. This ecosystem of a single vendor deciding the fate of the clients, server and protocol has been unhealthy for too long..
BoredPositron 2y ago
It might sound selfish, but I hope Matrix disappears sooner rather than later. It’s a poor product and a constant maintenance headache, whether you’re using the commercial or OSS version. Please, make way for something better. Every time someone includes Matrix in a chart of OSS options for common services as a chat solution, it does a disservice to the OSS community.
crimsoneer 2y ago
I hate to say it, but I understand this. I really tried to implement matrix in gov, but the only way to convince decision makers to use it rather than slack was "it's free if we just use the non pro stuff".
captaincrisp 2y ago
Apologies if my timings are off in this comment. They should be close enough to make my point, but it can be hard to find the actual dates. I have run a Synapse server for almost 8 years, by my reckoning. For most of that time I have been a $15/mo supporter. For several years, I had been hoping Dendrite would be released, along with a migration path to get my users over there. Synapse's resource usage is not great. I do run workers in order to improve performance. I'm waiting for an official migration path because I don't want to have to migrate my community again like I did when we moved from Slack to Matrix. It takes a lot of work just to move people over, and you always lose a couple people, which is a serious cost. Early last year I learned that they had put that on ice due to money issues. So there wasn't much hope of moving to a lower-cost Matrix implementation without a lot of headache. This makes sense. Building a homeserver implementation while maintaining an older one is expensive. For a year or more we've had quite a few blog posts saying that there's not enough money and that large organizations join the Matrix Foundation. This makes sense. Those organizations have enough money to keep it going, unlike my small monthly donation, which doesn't really matter all that much in the grand scheme of things. It's been quite a while since we've seen a new user-facing feature, and longer since we've had a selling point (which I could use to answer the question "why should I move to Matrix?"). It makes sense to prioritize functionality over new features, particularly when you've got a limited budget. But we still don't have some features which are very popular in Discord and Slack, like custom react images; these are implemented in other clients, like Cinny, but not Element. Last year they released Sliding Sync in Synapse, deprecating the Sliding Sync Proxy which I had been running to support clients who wanted to use Element X (a new client implementation). I personally haven't switched since Element X does not support Spaces. Moving Sliding Sync into Synapse saved me some resources supporting those clients. It was a little hard to tell when it was safe to remove the Sliding Sync Proxy; I had to track a couple Github issues. Matrix used to have a public roadmap, but it's no longer updated, so it's hard to keep up with the status of different features in development. After that they released Matrix Authentication Service (MAS), which is an additional service to deploy that moves the internal authentication functionality out of Synapse and interfaces with Synapse using OIDC. I haven't deployed it yet. They say it will eventually get rolled into Synapse, so I'm intending to wait for that. All in all, it does not feel like the things I want, and (assuming I'm not a completely unusual case) that the community wants, are high priority for the Element team. Donations the size of mine don't make a difference for their budget. They're spending what budget they do have on refactors like MAS, which don't seem to impact usability (though perhaps they do if you have a massive homeserver). They spend time and effort supporting new features which make Element X faster (Sliding Sync) but have not yet implemented all the core functionality (Spaces) so there's not much reason to move. I concluded a few months ago that our interests are not aligned any more and stopped donating. I know I'm not owed anything for my donations. I donated to support a project which I was excited about. This announcement, and that RAM graph which I will never see on my own server, makes me confident in discontinuing my support. I do not feel like Matrix/Element values its community any more.
dpc_01234 2y ago
I don't see a problem with that. A lot of FOSS users conflate freedom with "I get free shit in perpetuity forever" and then whine when they see something that they don't get for free, even if nothing was taken away from them. Best wishes to the Matrix project.
AAAAaccountAAAA 2y ago
IMO this is really unfortunate. Not only they are admitting that it is an exercise in futility to set up a medium-to-large Matrix deployment using an open source software stack. 100 000 people is not "nation-scale". It is small town scale. I am wondering if they intend Matrix ever to become truly mainstream instead of being confined to internal use of government organizations. And it only gets worse from there: https://element.io/blog/synapse-pro-slashes-costs-for-running-nation-scale-matrix-deployments/ https://element.io/blog/synapse-pro-slashes-costs-for-runnin... Synapse Pro was released only about a month ago. All these years they have been selling their solution to large organizations, despite it being hopelessly inadequate what it comes to scalability.
deleted 2y ago
[deleted]
AAAAaccountAAAA 2y ago
https://grapheneos.social/@GrapheneOS/113851478440207324 https://grapheneos.social/@GrapheneOS/113851478440207324 GrapheneOS dev account seems to disagree about the scalability issue being limited to very large instances. Due to the way how Matrix works, even activity on other servers can generate events at your server, particularly when you have joined busy channels. Not a software engineer, but a server software that collapses at only 100 events per second on modern hardware sounds pretty bad to me.
- Arathorn 2y ago
  There are two different performance issues here. One is the stuff that Graphene is talking about, which affects all server sizes, and we are trying to fund work on to fix in FOSS Synapse - see https://news.ycombinator.com/item?id=42752847 https://news.ycombinator.com/item?id=42752847. Entirely separately: there are bottlenecks you hit if you try to run a 100K+ user server. Synapse Pro fixes those bottlenecks, and the idea is the $ derived from that can then fund the broader FOSS improvements, which otherwise are starved of funding.
Arathorn 2y ago
Hi folks - based on the feedback here and on Mastodon, i've updated the original blog post to try to spell out that the point of Synapse Pro is not to hold back general performance work from open source Synapse, but just eliminate scaling bottlenecks for massive deployments: https://element.io/blog/scaling-to-millions-of-users-requires-synapse-pro#update https://element.io/blog/scaling-to-millions-of-users-require...
mickael 2y ago
I have written an article explaining the difference between Matrix and XMPP, and why it cannot be as scalable by design. You can read it here: https://www.process-one.net/blog/matrix-and-xmpp-thoughts-on-improving-messaging-protocols-part-1/ https://www.process-one.net/blog/matrix-and-xmpp-thoughts-on... TL;DR: Matrix protocol is based on document replication and synchronisation, while XMPP is based on message passing. Matrix requires to constantly replicate and merge the data. It is an intended feature, but as it does more, you cannot expect to scale to the same level. That said, we want ejabberd to be able to suit all types of messaging needs and we have already implemented 1-to-1 gateway with Matrix server inside the protocol. We are fine tuning our chatroom interop with Matrix. It means that we have already most of the code in place to support Matrix chat (as we need to implement the replication part). It means, the next step could be to implement full Matrix support for those who would rather use Matrix clients. Disclosure: I am ProcessOne Founder, XMPP developer since 1999 and working on ejabberd since 2002.

141 points by zaik 2y ago

At the end of the day, the money has to come from somewhere. There's only a handful of companies that make the open source profit game work and last I checked, most of them make their money from B2B deals with support, SLAs, etc. If you want to deploy something at nation scale, surely you can afford to pay for it.

emacsen 2y ago
While it's absolutely true that businesses need to make money, the idea that this must therefore logically conclude with a FLOSS and non-FLOSS version is not correct. If you look at companies like Red Hat, and Nextcloud, neither of these companies sold proprietary versions of their software. Red Hat sold licensed versions of their software, but they were always FLOSS. Nextcloud's CEO talks about how selling a proprietary version of your code is bad for your customers and puts your company into a Catch 22 with itself. "Do we add this feature to the 'Community' or 'Enterprise' version?" or worse "What do we do when someone makes a FLOSS plugin that competes or is better than our internal version of the same functionality?" There's a huge need for a good chat system to compete with the likes of Slack, but Matrix has bitten me several times and I don't trust it. In an alternate universe, they could have made Matrix work well for all levels of users and then sold customization and B2B deals with their customers leveraging the community, which has always wanted Matrix to succeed.
- thaumasiotes 2y ago
  > There's a huge need for a good chat system to compete with the likes of Slack, but Matrix has bitten me several times and I don't trust it. I thought this was Zulip?
  - usr1106 2y ago
    Zulip is excellent for geeks. I don't think it's for the broad public. We use it in our company with great success, but our marketing people still have not understood how topics are used (both by their own comments and by the evidence I see regularly).
    - crabmusket 2y ago
      That's a shame! What sort of problems arise?
      - usr1106 2y ago
        Topics only help to keep things organized and better than other chat systems if they are reusable and long-term identifiable. If people create a new topic like e.g. "Wednesday meeting" or "The printer does not work" every time they want to say something you end up with same mess as in other chat systems. Also it's quite human that discussions get offtopic. Zulip has nice support to continue in or move the discussion to the right place. But to my experience only a couple of geeks uses them.
- portaouflop 2y ago
  Would you be able to point me to the talk or blogpost of nextcloud that talks about their business model in detail? I would be very interested in how it works in practice, how they differentiate with their enterprise versions and so on. I agree that their business model sounds like having the best of both worlds, so any material on the details that you could share would help me a lot figuring out how to replicate it for other solutions.
  - Arathorn 2y ago
    Last I checked they ratelimit push notifications using a proprietary push gateway, unless you’re a paying customer.
    - 9283409232 2y ago
      The feature exist in both versions but you pay for additional capacity which is not what they are talking about when they said deciding whether the feature should be in community or enterprise.
  - emacsen 2y ago
    If you look up Frank Karlitschek and "I forked my own project and my own company", he's given this talk many, many times. As for it being the best of both worlds, it depends what best means. A company that wants to make a lot of money won't do well with this model. It's a slow growth model requiring a lot of hands on work that requires customers who are paying for features and custom installations. It's labor intensive work that's quite the opposite of the proprietary software model, which has traditionally had a lot in common with the publishing or entertainment industry. Write a book, get paid for it for years.
- whatevaa 2y ago
  RedHat sold to IBM and neutered it's free CentOS variant recently, so not so sure it's a good example.
  - graemep 2y ago
    CentOS is no longer a drop in replacement for RHEL, but both are still entirely open source other than trademarked logos and the like AFAIK. More importantly, Red Hat was very profitable even when CentOS was a drop in replacement.
    - Arathorn 2y ago
      I think the only reason Red Hat’s model works for them is that Linux distributions are obviously massive sprawling complex things, and maintaining LTS distros which you then sell support for is an incredibly complex and valuable thing that no intermediary can plausibly claim they can do. Even if the underlying code happens to be FOSS. Matrix servers (and web servers and mail servers etc) are not as complex - and if anything you want incentives to make them less complex, not more. And the better job you do of making them easy to admin and support (ie the more scalable and stable they are), the less reason there is to pay the upstream for support but some system integrator instead.
      - eredengrin 2y ago
        > Linux distributions are obviously massive sprawling complex things Just my perspective but I think this was also perhaps a product of the time when Red Hat really started growing, back when linux was a little more rough around the edges. These days it seems linux has enough polish that some (many?) corporations are happy to just use off the shelf ubuntu and call it a day and hope nothing breaks too badly.
        graemep 2y ago
        I think another thing that has changed is that big business is less suspicious of open source and are less likely to feel that all the software they use has to be licensed from someone. I think it has dawned on people that it is impossible to avoid using open source.

nightfly 2y ago

So matrix is commercialized now and the "real" software version is as expensive as Slack/Rocket? That's really disappointing...

robertlagrant 2y ago
Is there an alternative? Can you set up your own server and it's free (as in beer)?
- dsr_ 2y ago
  Yes. The article is about how a particular open source Matrix server implementation isn't suitable for millions of simultaneous users, and you should pay for the non-open-source version. But if you are merely handling tens of thousands of users, no problem. And if you have the budget to handle millions of users, maybe you have the budget to pay development as well as operations staff?
- tetrisgm 2y ago
  Side note: why do people still say free as in beer? Do people not assume free automatically means price, unless there’s a qualifier?
  - seanw444 2y ago
    Within the programmer circle, "free software" implies FOSS, which is something more than just no-cost. A normie would assume free in cost though.
    - lukan 2y ago
      But the world is made up of normies and has no idea about the free software movement. And I believe the free software movement is lamenting that? Maybe clear communication would work better? Use words in a meaning, people understand intuitivly?
      - 2y ago

gwerbret 2y ago

Perhaps this should point instead to the actual blog post this link references: https://element.io/blog/scaling-to-millions-of-users-requires-synapse-pro/ https://element.io/blog/scaling-to-millions-of-users-require...

WhyNotHugo 2y ago
Basically, they rewrote the slow parts in Rust, but that’s only included in the propertary version. The open source one is the one they know has terrible performance. I don’t get why people want to me part of a community where the steward treats them in such a shitty way.
- graemep 2y ago
  Two reasons: 1. its preferable to a proprietary alternative. 2. its an open standard so there are alternative implementations 3. Very few people need the high performance alternative. It might not be ideal, but what is the better alternative.
- Arathorn 2y ago
  We didn’t rewrite the slow parts in Rust, we rewrote the bits which let large deployments scale (workers) in rust. Normal Synapse perf work will continue as ever, if not faster given it might have more funding. See https://news.ycombinator.com/item?id=42752911 https://news.ycombinator.com/item?id=42752911

Multicomp 2y ago

The worst part is the article reads like someone is nervous that the nation states are busting hosting services from somebody doing horizontal scaling via multiple server instances rather than doing the Proper rust implementation. The classic tension of oss vs paid version.

wkat4242 2y ago

I don't like this. I only have a handful of family users but I could still benefit from the additional vulnerability proofing of Rust. It's not just about the scaling. My data security matters too. I have not exposed it to the internet for a while now because I already had my doubts. Also, as 0xC0ncord mentioned large federated rooms caused too much load so I wanted to move to standalone anyway. For that too I could have benefited from this. This really sounds like "haha you community members are stuck with a sucky version of our software and you're never getting the good one!". It's not exactly making me feel valued as a community member. And we shouldn't forget this is what put matrix on the map. This feels very bait-and-switch'ey. Also pointing out the flaws in one's own software doesn't inspire a lot of confidence. I think it would have been a sounder approach to paywall real enterprise features rather than the core product. Such as SSO, auditing, legal holds, that stuff. Basically all the stuff that Microsoft Teams has in Purview. The community won't miss any of that stuff. Ps I thought dendrite was supposed to be the next big thing? It feels like element is often jumping on new things like big rewrites such as element X instead of improving the existing products.

kumiokun 2y ago
Yeah, as a past contributor and long-running homeserver admin I feel pretty rugpulled here. This violates the expectations set in past communications. Synapse was something very different to Element. (This would at least for me have had a very different taste and reception if the private for-profit alternative would have been done under a different branding than "Synapse Pro" and not hijacking matrix.org community channels to promote it. They don't seem to be able to keep their hats and chairs properly separated)
- eredengrin 2y ago
  > I feel pretty rugpulled here How? Synapse is still being developed, this is just a commercial addon for huge customers. > and not hijacking matrix.org community channels to promote it. They don't seem to be able to keep their hats and chairs properly separated matrix.org has never excluded mentioning commercial entities, past blogs have frequently included information about etke or beeper for example.

0xC0ncord 2y ago

This is incredibly disappointing to me. Performance has always been one of the most glaring issues with using Matrix. The article seems to be intentionally obfuscating the fact that performance and scalability issues stem from the activity a server participates in rather than just the number of users it has. If you're in a rather large room with hundreds of members and it's constantly active, you'll quickly notice your server struggling to keep up. As a Matrix user, I feel like I'm being rug-pulled because Element HQ decided to gatekeep their solution to this problem that has plagued the community for more than half a decade.

zaroth 2y ago
Whatever you think a “rug pull” is, this ain’t it. FOSS owes you literally nothing. Go build what you want for yourself, geez. Oh, and feel free to give it away to the world… OR NOT!
- wkat4242 2y ago
  No ok but then let's drop matrix completely as the "next-gen" IRC / FOSS collaboration platform. There's a lot of moral support and ecosystem gain (eg bridges) from the community. Why bother with that anymore if we're just getting the "broken implementation"? I think part of the issue here is that they're trying to compete with the enterprise IM solutions. That doesn't really work as the others are way better funded and have benefits that element can't deliver (such as the wide Microsoft M365 + EntraID ecosystem). A Microsoft shop is never ever going to choose this over teams (horrible as the latter is).
  - Arathorn 2y ago
    > Why bother with that anymore if we're just getting the "broken implementation"? You’re not. All work (other than stuff which primarily benefits scalability for enormous deployments) should land on FOSS Synapse - particularly core performance improvements like faster state res. This is just trying to fund it. And no, this is not us trying to compete with Teams - this is us trying to force big Matrix deployments to actually route $ to Element to fund upstream dev, rather than use System Integrators who win the big tenders and then mysteriously fail to route any $ to us.

karussell 2y ago

I find this comment interesting: https://mastodon.matrix.org/@element/113848890455666110 https://mastodon.matrix.org/@element/113848890455666110 where IMO it is stated that Deutsche Telekom (with HessenConnect 2.0) did the free-riding and "directly contributed to layoffs at Element in 2022 and 2023". It sounds as if Telekom changed its mind after an initial contract or something. Or was Telekom one of many companies that did a similar thing? I know from my own experience that making money based on an open source project is extremely challenging. However, I don't understand why they frame their Pro version in this way and point to failure instead of success. I can't think of a better way (they will have to do the work :)). But it's already a good thing if they've won them as "Matrix believers", and they shouldn't scare them to get there.

Arathorn 2y ago
> Or was Telekom one of many companies that did a similar thing? This. > However, I don't understand why they frame their Pro version in this way and point to failure instead of success. It’s really trying to spell out the risks of freeriding, given the severity of the situation. But you’re right that it may well be too negative. > I can't think of a better way (they will have to do the work :)). Someone proposed a better positioning in the OP thread at: https://ioc.exchange/@troed/113843356547707182 https://ioc.exchange/@troed/113843356547707182
JeremyNT 2y ago
Why does anybody want to host large scale Matrix servers anyway? Aren't they already helping just by providing infrastructure to run the software at all? Is anybody anywhere - Deutsche Telekom included - actually making money here?
- Arathorn 2y ago
  I'll try to explain the situation: Some massive organisation (government agency, healthcare provider, network of schools etc) says: "Wow, Matrix is great, we should run our own instance rather than be dependent on Teams/Slack/Workspace. I'll put out a public tender for suppliers to provide this for us!" Then, all the system integrators who are geared up to sell big IT projects to large organisations jump on the tender and say "Sure, we can do this for you, and we're best qualified to do so because we're an enormous trusted system integrator with loads of people in the right geography, got all the right compliance checkboxes, and we do this sort of thing all the time. It'll cost you $xM a year." The upstream developer (Element) then reaches out to the integrators and says: "Oh hi! Are you planning to bid on this contract? If you want this to be successful, you should use our commercial offering, as then we'll be able to use that $ to keep upstream development alive". The integrator then looks bewildered, and says: "But... we've been using your opensource distro off github and it seems fine? Why would we pay you guys anything? The additional cost will just make our bid too expensive and we won't win, or it'll kill our profit margins. Sorry, we'll go ahead without you and maintain the software ourselves if needed." Given the main paying customers for Matrix are big organisations wanting digital sovereignty, this means the available $ disappears into the integrators rather making is way to fund the actual upstream dev - which then ends up stagnating, stalled, and completely at risk - putting all of these projects at risk. facepalm. A variation on the problem is where the massive org decides to do the project in-house rather than via an integrator... but then still chooses to YOLO and freeride and given it'll cost less than using the commercial offering from the upstream. I can list at least 4 national governments who have taken this route. To solve it, options include: * The buyer could mandate that suppliers for the tender have to use the commercial offering from the upstream product rather than freeride on opensource. This is probably the easiest solution, but seems staggeringly rare - the only instance I know of who have taken this principled stand are ZenDiS in Germany (https://zendis.de https://zendis.de) and we are very thankful for it. * The upstream could make their commercial offering more valuable than the FOSS offering - e.g. additional domain-specific functionality: what we've done with Synapse Pro. * The upstream could make the FOSS licensing less permissive, to force integrators to contribute back. We already did this with AGPL Synapse in 2023: https://element.io/blog/element-to-adopt-agplv3 https://element.io/blog/element-to-adopt-agplv3, and it has helped a bit... but in practice, we just saw a fleet of major Matrix tenders go to system integrators who then apologetically cut Element out, despite AGPL - putting Element's funding for 2025 directly at risk. Which is why we're putting out brutal blog posts saying "Look, big organisations: if you try to freeride on FOSS your projects will fail" - and providing a quantified explanation of the value that the commercial offering provides... so that we can then take the $ and hopefully keep the primary FOSS project on the road.

xyst 2y ago

Nation scale is Synapse “Pro”. Next level is multi-nation scale at “Pro+”. Then global scale version available at “Pro Max”. Universe scale at Synapse “Event Horizon” edit: what’s to stop community from re-implementing the worker component in rust and contributing it to the “pro” version?

yencabulator 2y ago
> edit: what’s to stop community from re-implementing the worker component in rust and contributing it to the “pro” version? The essence of Open Core suck, which Matrix has here committed to, is actively discouraging community work related to the bits that differentiate the proprietary version. History says any pull request improving the FOSS version too much, in the wrong direction, will be left unmerged. This is how "communities" die.
- throwaway48476 2y ago
  Basically zero communities have the growth needed to compete with commercial offerings. Eventually users switch to a different paid offering because it has better fratures/performance. That's how communities die.

I_am_tiberius 2y ago

So FOSS is essentially a Marketing tool for Matrix?

brookst 2y ago
Isn’t that true for most/all corporate FOSS projects? I don’t think there are many for-profit companies who have some kind of idealistic vision of FOSS. It’s always part of the business model. Would be weird otherwise.
Arathorn 2y ago
Matrix is an open protocol published by a non-profit foundation. It has always had both FOSS (eg Element) and proprietary (eg Beeper) implementations. You sound like you are getting Element mixed up with Matrix. Element releases almost all of its implementations as FOSS, and to help Matrix grow, will make those as good as possible for average server sizes. It’s only gigantic servers which actually require the new proprietary modules. Given everyone who uses FOSS Synapse can keep doing so, and given FOSS Synapse dev should hopefully improve and accelerate if actually funded by $ from Synapse Pro, it’s not fair to say that “FOSS is essentially a marketing tool for Element”

jchw 2y ago

I'm pretty sick of Element and Matrix at this point, and I'm pretty sure the enthusiasts are the only ones keeping Matrix relevant. This monetization scheme is not going to work. I think the Matrix experiment is coming to an end. I'm about to fork Dendrite at its last ASL 2.0 version, because to hell with signing a CLA for Element.

fishgoesblub 2y ago
I'm tired of all the issues with Matrix as well. I've never had a message get received or a notification pushed in less than 10 minutes. Went back to XMPP and messages are instant. I want to love Matrix. E2EE, proper federation, voice calls, But the experience is terrible.
- tcfhgj 2y ago
  > I've never had a message get received or a notification pushed in less than 10 minutes. I receive messages < 10 seconds. What's your setup?
  - fishgoesblub 2y ago
    Element (Later ElementX) On android, and same on iOS. Tried various other clients like Fluffychat, and SchildiChat.
    - Arathorn 2y ago
      Presumably you’re using an overloaded server which is struggling because we have been completely undermanned on synapse maintenance and haven’t been able to do core optimisation work. This is precisely why we’re trying to fund our FOSS work by selling Synapse Pro.
      - throwaway48476 2y ago
        Proprietary licenses don't align developer incentives with customers. Instead you should consider non commercial licenses that convert to open source after some time, eg 5 years. Customers won't pay for development unless forced to and it creates a tragedy of the commons. No large commercial deployment will accept 5 year out of date broken insecure software. Non paying users won't care that they can't use it commercially

Leo_Verto 2y ago

It's a shame it had to come to this but I can see why Element had to make this choice. To me it feels like they've ended up in a position similar to Docker Inc. where they've spent years of work and tons of resources building the standards and reference implementations, but missed out on the juicy integration and support contracts. Docker has turned into the OCI standard and most Kubernetes deployment don't even run containerd anymore. The company has massively rate limited Docker hub and recently increased the pricing again, which has had much bigger impact for end users than this pivot by Element. I really hope this works out as a funding source for future Matrix development, I don't see how its ecosystem can continue to be developed and healthy without Element.

koito17 2y ago
> most Kubernetes deployment don't even run containerd anymore I assume you mean moby, which was used via dockershim in Kubernetes until support was removed in v1.24. I can't find a reliable source, but I believe containerd, followed by CRI-O and alternatives, are the common choices of container runtime for Kubernetes. https://kubernetes.io/docs/setup/production-environment/container-runtimes/ https://kubernetes.io/docs/setup/production-environment/cont...
- ffsm8 2y ago
  It was called plain docker originally, the Moby rebrand came much later and not everyone is even aware that it happened, because plain docker was already much less relevant when they did it (vs k8s, which was already becoming the default by then). https://www.docker.com/blog/introducing-the-moby-project/ https://www.docker.com/blog/introducing-the-moby-project/
deleted 2y ago
[deleted]

vfclists 2y ago

Matrix have always been suspect. Disappointed but not surprised. In short they are saying that if the nation states hadn't opted to use Matrix they would never have bothered developing this Rust version, which only shows that they were never sincere about developing a performant open source version, not to mention their involvement with some suspect Israeli comms companies and the phone home agendas. Interesting question - will their "nation state" level customers have the right to inspect the source and make changes to it? Just waiting to have this post flagged.

ezst 2y ago
> In short they are saying that if the nation states hadn't opted to use Matrix they would never have bothered developing this Rust version, which only shows that they were never sincere about developing a performant open source version At that point, it's nice of you to assume that they are even capable of making it performant. But it is indeed "remarkable" that the people who started this, and who couldn't get anywhere with it, for over a whole decade, are still behind it today.

throwaway519 2y ago

Element have a perfectly good implementation written in Go called Dendrite. I run it. It's good. But support is deprioritised in favour of Synapse (Python). Moving the goalposts with Synapse does not rejuvenate Synspse, it creates a fragmented system. Get used to being boring. I am not moving to an ever centralised Synapse only ecosystem - a single implementation is not a federated protocol. That the new Element mobile client only supports Synapse was a huge red flag. If diversity in implementation is not encouraged with action I'm back to XMPP (+ Jitsi).

Arathorn 2y ago
> That the new Element mobile client only supports Synapse was a huge red flag. This is not true. Element X requires Matrix 2.0, which is currently only in Synapse because it has to land somewhere first. But Dendrite is implementing it (best effort, thanks to an amazing community contrib) too: https://github.com/element-hq/dendrite/pull/3493 https://github.com/element-hq/dendrite/pull/3493 and at conduwuit (one of the conduit forks) is too: https://github.com/girlbossceo/conduwuit/pull/666 https://github.com/girlbossceo/conduwuit/pull/666 There is absolutely nothing trying to lock EX to synapse.
- throwaway519 2y ago
  Great to see some community efforts.

NewJazz 2y ago

Google, Microsoft, Apple, and Facebook should be required to integrate XMPP into their messaging products. Matrix is a cool idea but we now have like 1 server implementation that isn't somewhat gimpy, and its closed source... XMPP is much more compelling.

blackeyeblitzar 2y ago
In general all these large companies should be required to use open standards or spin off proprietary protocols into an independent open source project.
miohtama 2y ago
Google/Facebook did XMPP federation back in a day and it was spam hell.
- BoredPositron 2y ago
  Matrix has the exact same problem?
- jauntywundrkind 2y ago
  I had a pretty visible email address / xmpp name all over the web and i don't recall even a mild inconvenience from spam.
airhangerf15 2y ago
They did. Facebook's XMPP implementation was shit and unreliable. Google's actually federated, but they silently killed federation .. and then the entire protocol. (This was ~10 years ago) Exactly who should "require" these companies to implement theses standards? Some daddy government? Who would be forced to implement it? Just the big sites? What makes a site big enough? If it were required, it would be used to lock out smaller players. Consumers could demand it, but we know the average consumer doesn't even know what a protocol is, or understand the different between a web browser and e-mail. Most don't even know how their phones work.

heybrendan 2y ago

Arathorn, should this message reach you, I want to voice my support of you and your team. Notwithstanding the naysayers that surface with peculiar predictability any time Matrix is discussed here, know there are people like me that are happy Synapse users (and operators, nearly 5 years now!) that are in your corner, rooting for you [and Matrix]. Keep fighting the good fight.

mgrandl 2y ago
Jup same here. There is always so much negativity on Hackernews. I have been running synapse without issue for years and it has been a joy!
vaylian 2y ago
Agreed. Open source is an ecosystem. And ecosystems need to be healthy to grow. Matrix is not a hobby project that a couple of hackers implement in their free time but a large project that requires full-time developers. But if these developers don't get paid, then we (the community) won't get improvements for matrix software. Synapse Pro is a compromise to keep the ship afloat. This is not a matter of greed, but a matter of keeping the lights on.
Arathorn 2y ago
Thank you - and to the sibling comments too; the support is enormously appreciated. For what it’s worth, we have been frantically trying to keep all Synapse dev FOSS for the last 10 years - and the FOSS version will always be the primary project, where everything new happens (other than mega-scalability work). At least for as long as we’re able to work on it. My hope is that Synapse Pro will be net positive for the ecosystem, as it will hopefully fund us to be able to work more on FOSS Synapse. It could also inspire more alternative homeserver work, which is no bad thing for Matrix as a whole.
sneak 2y ago
The good fight does not involve proprietary software.

benatkin 2y ago

The audacity of this: > tl;dr is more "need to be able to keep paying people to work on this". https://mastodon.matrix.org/@element/113844400165509238 https://mastodon.matrix.org/@element/113844400165509238 That's your problem. I only had paid attention to this project in the past because it was under a permissive open source license.

camgunz 2y ago
Do you have something against making money on FOSS? A weird thing has happened while I've been reading this thread where I've noticed some outsized cynical or negative comments and realized they're both your user, then I remembered your bad faith arguments in the thread about the license change. I'm too lazy to check; do you come to every Matrix/Element thread to be uselessly negative or just these two?
- defrost 2y ago
  https://hn.algolia.com/?query=matrix%20by%3Abenatkin&sort=byDate&type=comment https://hn.algolia.com/?query=matrix%20by%3Abenatkin&sort=by...
- deleted 2y ago
  [deleted]

throwaway48476 2y ago

Another case where an 'eventually open source' license would be beneficial.

Arathorn 2y ago
This could be an option here; we haven’t thought about it yet.
- throwaway48476 2y ago
  Free software is just a hack of the copyright system. The point of which is to incentivize people to create works and be paid for them and after a period of time be added to the public domain. Eventually open source follows the same principle, just with source availability.

tonygiorgio 2y ago

I was a paying pro user for a <1000 person server years ago. They forced me off of it due to offerings they were no longer servicing. Told me I had to self host and export all my data. I attempted this and it never worked. I abandoned that server and my profile I used across many matrix instances (and somehow my matrix room continued to run without me hosting it, and without an admin running it). I will never use nor recommend them ever again. They clearly do not know how to operate a business nor an open source project.

intelfx 2y ago
> somehow my matrix room continued to run without me hosting it, and without an admin running it Surprise, this is probably the primary architectural feature of Matrix. This is precisely what it's designed to do.

ssivark 2y ago

I sincerely wish you good luck with building -- hope people find this new product useful. This bait-and-switch leaves a really bad taste for me though; I now feel silly for having advocated for Matrix, and will cease henceforth. You can lament the free-rider problem all you like, but suddenly making it a big deal now betrays that you do not take seriously enough your own promises, and were clueless about the fundamentals all this while. This is not meant to be vindictive (I wish Matrix success as a product; it's certainly useful), but simply making clear that the expectations and standards are different for a "community" -vs- simply a business that sells a product. It seems to me like this disqualifies Matrix from consideration for "public infrastructure" IMHO. It would have been better if the differentiation between the paid and free versions wasn't scaling, but some other features not critical for public use (or provided as a non-free plugin), but something that enterprises would consider useful.

Arathorn 2y ago
> It seems to me like this disqualifies Matrix from consideration for "public infrastructure" IMHO. Please don’t conflate Synapse with Matrix. Yes, if a govt has a “we must only spend money on pure open source products” policy then they won’t buy Synapse Pro. Perhaps they’ll end up scaling by running hundreds of separate smaller FOSS instances instead. Perhaps other server implementations will step up as FOSS (and probably promptly be undermined by the same freerider problem). > It would have been better if the differentiation between the paid and free versions wasn't scaling, but some other features not critical for public use (or provided as a non-free plugin), but something that enterprises would consider useful. We’ve tried that (https://element.io/server-suite https://element.io/server-suite) and it hasn’t been enough.
- AAAAaccountAAAA 2y ago
  > We’ve tried that (https://element.io/server-suite https://element.io/server-suite) and it hasn’t been enough. Will this finally be enough, or will it end up so that even more features are going to be available only on the proprietary version?
  - 2y ago

DoItToMe81 2y ago

I hope this is poorly phrased or I am misunderstanding it, because it feels like it's implying that Matrix's terrible scaling issues will have their fixes paywalled. Even non-federated homeservers chew up RAM and CPU like candy when you get too many users. It's heavier than any other protocol or chat program I know.

Arathorn 2y ago
Most of that performance is nothing to do with whether workers are written in python or rust - see https://news.ycombinator.com/item?id=42752911 https://news.ycombinator.com/item?id=42752911. The hope is to fund maintenance for FOSS synapse, including core perf work, via Syn Pro stuff which makes the biggest impact for gigantic servers.

knallfrosch 2y ago

What surprised me most is how the community and pro versions use completely different core technologies. I guess licensing and FOSS don't work together. But then the pro version doesn't benefit from the open-source scrutiny of the community version, especially in regards to cryptography and security. What's the reasoning for this technology split?

Arathorn 2y ago
FOSS Synapse already is a hybrid of Python and Rust, and has been for years - with Rust used for the hot paths. Synapse Pro is just pushing that a bit further for worker processes, by having the host process itself being Rust-first rather than Python-first. But the majority of the app and architecture is the same; it's just swapping out some of the python-with-rust processes for pure-Rust alternatives. I think of it a bit like using a hardware accelerator (how I envy the fact that nobody complains about the fact that hardware accelerators cost money!)

whytevuhuni 2y ago

Happy to see Rust being used for this. I tried using it for some toy implementation of a Matrix federation server [0], and there's lots of zero-copy/zero-allocation optimization tricks Rust allows, if one is willing to spend that time on it. I'd really wish to see an official non-Python OSS version though, to make it easier to run on a Raspberry Pi or whatever other tiny homelabs people usually host things on. When I tried Synapse, it essentially ate the whole memory and was constantly making the fan turn on. [0] https://github.com/andreivasiliu/fluctlight https://github.com/andreivasiliu/fluctlight

ezst 2y ago
There's conduit as an alternative implementation in rust, but Matrix being what it is, you won't be able to use the newer clients with it, and federation will be glitchy. This ecosystem of a single vendor deciding the fate of the clients, server and protocol has been unhealthy for too long..

BoredPositron 2y ago

It might sound selfish, but I hope Matrix disappears sooner rather than later. It’s a poor product and a constant maintenance headache, whether you’re using the commercial or OSS version. Please, make way for something better. Every time someone includes Matrix in a chart of OSS options for common services as a chat solution, it does a disservice to the OSS community.

crimsoneer 2y ago

I hate to say it, but I understand this. I really tried to implement matrix in gov, but the only way to convince decision makers to use it rather than slack was "it's free if we just use the non pro stuff".

captaincrisp 2y ago

Apologies if my timings are off in this comment. They should be close enough to make my point, but it can be hard to find the actual dates. I have run a Synapse server for almost 8 years, by my reckoning. For most of that time I have been a $15/mo supporter. For several years, I had been hoping Dendrite would be released, along with a migration path to get my users over there. Synapse's resource usage is not great. I do run workers in order to improve performance. I'm waiting for an official migration path because I don't want to have to migrate my community again like I did when we moved from Slack to Matrix. It takes a lot of work just to move people over, and you always lose a couple people, which is a serious cost. Early last year I learned that they had put that on ice due to money issues. So there wasn't much hope of moving to a lower-cost Matrix implementation without a lot of headache. This makes sense. Building a homeserver implementation while maintaining an older one is expensive. For a year or more we've had quite a few blog posts saying that there's not enough money and that large organizations join the Matrix Foundation. This makes sense. Those organizations have enough money to keep it going, unlike my small monthly donation, which doesn't really matter all that much in the grand scheme of things. It's been quite a while since we've seen a new user-facing feature, and longer since we've had a selling point (which I could use to answer the question "why should I move to Matrix?"). It makes sense to prioritize functionality over new features, particularly when you've got a limited budget. But we still don't have some features which are very popular in Discord and Slack, like custom react images; these are implemented in other clients, like Cinny, but not Element. Last year they released Sliding Sync in Synapse, deprecating the Sliding Sync Proxy which I had been running to support clients who wanted to use Element X (a new client implementation). I personally haven't switched since Element X does not support Spaces. Moving Sliding Sync into Synapse saved me some resources supporting those clients. It was a little hard to tell when it was safe to remove the Sliding Sync Proxy; I had to track a couple Github issues. Matrix used to have a public roadmap, but it's no longer updated, so it's hard to keep up with the status of different features in development. After that they released Matrix Authentication Service (MAS), which is an additional service to deploy that moves the internal authentication functionality out of Synapse and interfaces with Synapse using OIDC. I haven't deployed it yet. They say it will eventually get rolled into Synapse, so I'm intending to wait for that. All in all, it does not feel like the things I want, and (assuming I'm not a completely unusual case) that the community wants, are high priority for the Element team. Donations the size of mine don't make a difference for their budget. They're spending what budget they do have on refactors like MAS, which don't seem to impact usability (though perhaps they do if you have a massive homeserver). They spend time and effort supporting new features which make Element X faster (Sliding Sync) but have not yet implemented all the core functionality (Spaces) so there's not much reason to move. I concluded a few months ago that our interests are not aligned any more and stopped donating. I know I'm not owed anything for my donations. I donated to support a project which I was excited about. This announcement, and that RAM graph which I will never see on my own server, makes me confident in discontinuing my support. I do not feel like Matrix/Element values its community any more.

dpc_01234 2y ago

I don't see a problem with that. A lot of FOSS users conflate freedom with "I get free shit in perpetuity forever" and then whine when they see something that they don't get for free, even if nothing was taken away from them. Best wishes to the Matrix project.

AAAAaccountAAAA 2y ago

IMO this is really unfortunate. Not only they are admitting that it is an exercise in futility to set up a medium-to-large Matrix deployment using an open source software stack. 100 000 people is not "nation-scale". It is small town scale. I am wondering if they intend Matrix ever to become truly mainstream instead of being confined to internal use of government organizations. And it only gets worse from there: https://element.io/blog/synapse-pro-slashes-costs-for-running-nation-scale-matrix-deployments/ https://element.io/blog/synapse-pro-slashes-costs-for-runnin... Synapse Pro was released only about a month ago. All these years they have been selling their solution to large organizations, despite it being hopelessly inadequate what it comes to scalability.

deleted 2y ago

[deleted]

AAAAaccountAAAA 2y ago

https://grapheneos.social/@GrapheneOS/113851478440207324 https://grapheneos.social/@GrapheneOS/113851478440207324 GrapheneOS dev account seems to disagree about the scalability issue being limited to very large instances. Due to the way how Matrix works, even activity on other servers can generate events at your server, particularly when you have joined busy channels. Not a software engineer, but a server software that collapses at only 100 events per second on modern hardware sounds pretty bad to me.

Arathorn 2y ago
There are two different performance issues here. One is the stuff that Graphene is talking about, which affects all server sizes, and we are trying to fund work on to fix in FOSS Synapse - see https://news.ycombinator.com/item?id=42752847 https://news.ycombinator.com/item?id=42752847. Entirely separately: there are bottlenecks you hit if you try to run a 100K+ user server. Synapse Pro fixes those bottlenecks, and the idea is the $ derived from that can then fund the broader FOSS improvements, which otherwise are starved of funding.

Arathorn 2y ago

Hi folks - based on the feedback here and on Mastodon, i've updated the original blog post to try to spell out that the point of Synapse Pro is not to hold back general performance work from open source Synapse, but just eliminate scaling bottlenecks for massive deployments: https://element.io/blog/scaling-to-millions-of-users-requires-synapse-pro#update https://element.io/blog/scaling-to-millions-of-users-require...

mickael 2y ago

I have written an article explaining the difference between Matrix and XMPP, and why it cannot be as scalable by design. You can read it here: https://www.process-one.net/blog/matrix-and-xmpp-thoughts-on-improving-messaging-protocols-part-1/ https://www.process-one.net/blog/matrix-and-xmpp-thoughts-on... TL;DR: Matrix protocol is based on document replication and synchronisation, while XMPP is based on message passing. Matrix requires to constantly replicate and merge the data. It is an intended feature, but as it does more, you cannot expect to scale to the same level. That said, we want ejabberd to be able to suit all types of messaging needs and we have already implemented 1-to-1 gateway with Matrix server inside the protocol. We are fine tuning our chatroom interop with Matrix. It means that we have already most of the code in place to support Matrix chat (as we need to implement the replication part). It means, the next step could be to implement full Matrix support for those who would rather use Matrix clients. Disclosure: I am ProcessOne Founder, XMPP developer since 1999 and working on ejabberd since 2002.