8 ms·
How we hosted FOSDEM 2021 on Matrix
- kitkat_new 6y agoI wonder if there has ever been an online conference better than this one. It was amazing! I really hope there will be an online-opt-in for next year
- Arathorn 6y agoFeel free to pose any questions, thoughts or feedback on how FOSDEM went, and how we might be able to do better in future if the need arises!
- znpy 6y agoI was curious, how much computing infrastructure did this required? How many machines were needed (max autoscale peak)? How much bandwitdh did you consume at peak ?
- Arathorn 6y agoWe overprovisioned the hardware a lot as we really didn't want to hit any hardware limits. In practice, I think this meant there were 16 medium sized EC2 instances for the various Synapse processes; RDS for DB; 6x largish boxes in Upcloud for Jitsis, and ~70 machines in the automatic scaling group for the Jibris. I'm guessing peak bandwidth was about 100Mbps but obviously all the bulk HTTP (HLS etc) was behind CDNs.
- rakoo 6y agoDo you believe such a model is viable for any future conferences ? Does it need a lot of money and/or system administrators to run compared to the number of participants ?
- Arathorn 6y agoIt's definitely viable. I'm waiting to see what the bill was for the hardware costs, but I'm expecting it'll be a few $K. We had a lot of sysadmin on hand to try to keep it happy over the weekend though (but having learnt from the misadventures, subsequent events should be much easier).
- znpy 6y agoNice! Can you share about how much did it cost to host the conference then, even approximately? I wonder if your stack can become a pattern to host large-ish to large conferences on-line!
- dsdsdsds 6y agoI attended FOSDEM'21 as a speaker and the overall experience was very pleasant & smooth. No technical issues that occurred during the talk or q/a. Very well done to everyone involved!
- jeltz 6y agoMy main issue was that the widget stopped playing when switching conversations so I had to break out the videos I listened to.
- Arathorn 6y agoYeah, we hooked up PIP for broadcasters but not viewers - given it's not obvious when you're switching between rooms whether you want to switch video or not. Instead, you have pop-out button to let you view the video widget separately if you want, which sounds like what you were using. The UI could be clearer.
- detaro 6y agoI have no idea if that's FOSDEM infra, element.im or my home server (which is a fairly standard synapse install provided by a local tech group, so it shouldn't be out of the ordinary) to blame, but starting Saturday by joining a bunch of the relevant channels and getting told multiple times "Too many Requests!" and having to spread out joining 10-12 channels over 20 minutes wasn't fun and surprising, especially since the rooms weren't even set to provide backlog! I generally use Matrix quite a bit, but I didn't find FOSDEM a convincing example. I guess some people like having everything squeezed into one tab, but it didn't work well for me, other than that it didn't have much of a value proposition over any other chat system for me personally, and as initially described even the experience as a chat system wasn't great.
- driminicus 6y agoYou ran in to rate limiting, which can be configured in synapse.
- Arathorn 6y agoAll the rooms had history visibility enabled, so it sounds like something might have been going wrong there on your homeserver. In terms of membership changes being ratelimited - yes, this is an anti-abuse mechanism, and the default synapse config is overly conservative. We don't have a way of overriding that globally though, and the FOSDEM use case of loads of rooms flying around meant some people fell over it a bunch. Sorry you got bitten by this.
- detaro 6y agoRegarding the history, seems that maybe it just was very slow to sync - I can now indeed scroll back past when I joined. shrug
- rakoo 6y agoI didn't expect anything and was pleasantly surprised to be able to follow all talks if I wanted to, all with the room for questions and such. Congratulations, and thank you very much ! There were a few pain points, given the special nature of the rooms. Here's my workflow for joining a room: - I have my own homeserver and am using Element Desktop - When I'm interested in a talk, for a room I'm not already in, I need to get the room. I can't simply copy-paste it, because there's a link to https://matrix.to https://matrix.to and I don't want to fill my own homeserver every time. I need to click on that first link _then only_ I can extract the "#room:matrix.org" I'm interested in. I wish there was just a "matrix://" or something I could click and that my system could understand. - I joined many rooms just for FOSDEM and I didn't want to "pollute" my usual rooms. So I used the Communities thing from Element but it's counter-intuitive: in all other applications (Discord, Slack, I assume others) that button is a "namespace" at the top of a usual hierarchy (namespace/channel). In Element it's more like a tag: when I put the room in the community, I'm actually saying that the community is a view that lists all rooms with the corresponding tag. I don't know yet if I prefer this way of working or not, but maybe the elements should have a different appearance than other tools - Linked to the above, there's no "community" for the default view, which is "all of them" if I understood correctly; I have to know that I need to click after the list, below the "+" - Adding a room to a community is not exactly straightforward as well. I need to go in the Community, then add the room... but it must already be added beforehand, because #room:server.org doesn't seem to work. All in all it's a lot of manual manipulations for being able to use Element the way we're expected to be able to use it. Surely your backlog already has some items for the points above, so I'm going to wait patiently :)
- DenseComet 6y agoMatrix.to should remember your homeserver, but a "matrix://" url scheme is being worked on [1]. Work on that is caught up a bit in the backlog however, but there are folks in the ecosystem pushing that along in 3rd party clients. Communities are going away in favor of spaces (relatively?) soon. That was the topic of Matthew's talk at FOSDEM [2], and the entire experience should be vastly better. [1] https://github.com/matrix-org/matrix-doc/pull/2312 https://github.com/matrix-org/matrix-doc/pull/2312 [2] https://www.youtube.com/watch?v=TzUfS08lMek https://www.youtube.com/watch?v=TzUfS08lMek
- guerby 6y agoI've seen that FOSDEM chat are archived, does element web interface allow user to save their chat room to text files on their computer ? https://github.com/vector-im/element-web/issues/2630 https://github.com/vector-im/element-web/issues/2630 This issue has 149 thumbs up, as I noted I don't know of any other chat software without the ability to save discussion to a local file. Thanks!
- Arathorn 6y agoMatrix itself is a conversation store, which means we haven't prioritised exporting to text file (or HTML file or whatever) in Element. A workaround for now is https://github.com/russelldavies/matrix-archive https://github.com/russelldavies/matrix-archive, which works really well.
- guerby 6y agoThanks for the pointer. I installed a debian 11 VM, installed then launched the matrix-archive.py script --all-rooms with my not-for-profit ISP matrix+element credentials. I follow mostly my not-for-profit ISP channels (quite small) and may be 4 or 5 rooms on matrix.org, fosdem.org and mozilla.org. After 30 minutes it has written 878 MBytes of archives and it's not done yet. I don't see an option to resume / append to an existing archive. So I assume I will have to restart from scratch each day? I wonder: will it scale if everyone uses this script?
- Arathorn 6y agoSo yes, that tool is intended for one-off archives. And I'm not surprised that the archives are large, given Matrix rooms live forever by default, so if you're in some room like #matrix:matrix.org the full history goes back to Aug 2014 will be millions of messages by now. The 'right' solution is presumably to give Element a button to maintain ongoing archives on desktop (it already archives your encrypted rooms into an encrypted sqlite DB for indexing) - with a control to limit how far back in time it goes. Separately, having a "save the last N days/msgs of this room" button would be useful in Element Web. But both simply haven't got to the top of the todo list yet.
- belorn 6y agoSince the presentation part of the videos were prerecorded, why were they not imminently uploaded to the archive after the conference?
- markvdb 6y agoThe first videos are popping up on https://video.fosdem.org/2021 https://video.fosdem.org/2021 as we speak. P.S. The FOSDEM 2021 video team was three people. With day jobs. Humans. One of us with kids. Not perfect. Volunteering our time. During a pandemic. One of us of was at a funeral service on Saturday morning. A close friend had died of covid-19 . Another had professional obligations on Sunday afternoon. It was great fun though, so chances are you'll see us again at FOSDEM 2022...
- belorn 6y agoSorry, didn't mean to imply that you should have worked harder or faster, mostly just wondered why not just release the presentation part first since that was already made in advance and the Q/A later once the editing was done.
- markvdb 6y agoHey, no worries! Rereading my own comment, I realise I probably made you feel bad. That was not at all my goal. There is certainly room for improvement releasing videos early for future editions by working smarter. We'll certainly look into that!
- ssivark 6y agoThanks for sharing the setup! The process seems involved enough for non-experts (even with open source tools), and clear enough for y’all who’ve done it, that I wonder whether it might be worth productizing by Element — something to the effect of a turnkey solution for conference organizers (who are domain experts but would have a hard time figuring out and managing this infrastructure setup). Given the current pandemic situation, I wouldn’t be surprised if organizers across the board jump on to a solution of this kind. It would also serve to popularize Matrix and greatly expand its user base.
- Arathorn 6y agoYeah, we're thinking about it. The problem is that a successful conference is much more than just the tech stack - helping guide the organisers through how to get the most out of it and then being around to support the event is a lot of the effort. My hunch is that it might be time better spent focusing on making the core of Matrix & Element better.
- exyi 6y agoI did not have much time so I attended only a few talks. However, I was very pleasantly surprised how well it worked. Not only technically - it just works great having the video stream in a chat window where people can freely talk to the presenter. Maybe I'm even going to miss that on real-life FOSDEM which was inherently less interactive. Congratulations to the Matrix and FOSDEM teams.
- jasonjayr 6y agoHow much staff was required to make the behind the scenes event run smoothly? (Support, operations, helping speakers, helping attendees?)
- SubGenius 6y agoMatrix is not only an exciting ecosystem to build on top off, the overall community is pretty amazing too. Fossdem went great and I liked a lot of the talks. I'm beginning to build a couple of projects on matrix and am really hopeful for its long-term success.
- ognarb 6y agoSame I think the Matrix community is amazing and very welcoming. Element is really trying to build a community behind Matrix and I think they doing it with a great success. They regularly invite community members to talk in video interview (This Week in Matrix). Alternative clients are actively encouraged and promoted and small business build around the Matrix ecosystem too. Shameless plug to the Matrix client, I'm contributing too: https://invent.kde.org/network/neochat https://invent.kde.org/network/neochat
- SubGenius 6y agoUnfortunately I'm running a very minimal arch + i3 setup now, I wonder if I can install Neochat without pulling in a whole lot of KDE dependencies? Neochat looks great by the way!
- MayeulC 6y agoNheko is quite good too, I mostly use gomuks and nheko, though I regularly fire up neochat too :) Not sure which are packaged, though. You usually want the latest goodies, so AUR/flatpak it is for me.
- ognarb 6y agoNeoChat only needs Kirigami (for the UI), KCoreAddons (for the about screen), KNotifications/KWindowSystem (for the notifications), KConfig (for the configuration). Probably the KNotifications dependency could be made optional without too much ifdef but I'm not sure if this is something users want.
- gpanders 6y agoIf you use Weechat for IRC, there is a Matrix bridge plugin that would allow you to connect to Matrix from the command line.
- jononor 6y agoChaos Computer Congress on Matrix next?
- Arathorn 6y agoIf CCC were up for it, we'd be game. Could even go and embed Work Adventure into Matrix as a widget, for best of both worlds (hah). https://apfelkraut.org/2021/02/virtual-conference-experiences/ https://apfelkraut.org/2021/02/virtual-conference-experience... was an interesting side-by-side comparison of rC3 and FOSDEM.
- Buetol 6y ago+1 for Work Adventure !
- vaylian 6y agoI attended both conferences and both were great. The traditional tool for digital communication at CCC congresses is IRC. I personally still like IRC, but I think the matrix web client at FOSDEM made channel discovery and navigation a lot easier. I also think promoting Matrix could help with promoting open ecosystems in general. IRC works well, but it doesn't have the potential for mass adoption, which we need in order to curb WhatsApp's influence.
- exyi 6y agoIRC is easy to bridge to Matrix, so that could be fairly compatible with the more conservative guys at CCC(C) ;)
- markvdb 6y agoThe public matrix rooms at FOSDEM were bridged to irc. This was not by accident. We tried to give visitors the freedom to use their preferred tools to join FOSDEM. Whether you were using matrix with the latest Chromium or irssi in a tmux session on OpenBSD, you were able to speak to people at the conference. Same for the streaming. Inside the matrix channel would have worked, but mpv on some framebuffer without X or wayland too...
- Athas 6y agoI attended FOSDEM this year, and the platform worked quite well. Sure, there were the occasional technical hiccups, but bugs can always be fixed if the foundations are solid, and they appeared to be. Since FOSDEM's physical environment stopped scaling years ago, I hope this will make remote or mixed remote/physical participation increasingly viable.
- turbinerneiter 6y ago> mixed remote/physical I want this for all conferences :D
- robert_foss 6y agoThe Matrix P2P talk at FOSDEM 2021 was really interesting. https://fosdem.org/2021/schedule/event/matrix_pinecones/ https://fosdem.org/2021/schedule/event/matrix_pinecones/ https://www.youtube.com/watch?v=a7rHtQwpuus https://www.youtube.com/watch?v=a7rHtQwpuus
- emptysongglass 6y agoUnfortunately, their P2P demo doesn't work insofar as one can start a chat with you using your username or join link. https://p2p.riot.im https://p2p.riot.im
- robert_foss 6y agoYeaah, I think it's an old bit-rotted demo using their first p2p implementation. Still though, the progress they're making is great, and I think portable identities will be a huge deal.
- neilalexander 6y agoThe demo is still working for me on Chrome 88 (not incognito) seemingly. Which browser are you using?
- emptysongglass 6y agoIt works but you can't chat with anyone. I've tried both Vivaldi and Brave. Try and start a chat with your p2p user from a hosted instance. It will fail.
- neilalexander 6y agoThis is by design. The p2p.riot.im demo is a closed system and isn't connected to the rest of the Matrix federation. It speaks a different protocol.
- emptysongglass 6y ago
- Shish2k 6y ago> Jibri is essentially a Chromium which views the Jitsi webapp, running in a headless X server whose framebuffer and ALSA audio is hooked up to an ffmpeg process which livestreams to the appropriate destination So if I’m understanding this right, you have “private video conference -> web browser -> desktop capture -> ffmpeg -> public video stream"… why not “private conference -> ffmpeg -> public stream”? Having a web browser in the middle of your livestream setup feels very weird :P
- Arathorn 6y agoThe reason is that the Jitsi Meet conferencing UI exists only as a webapp. In other words, you have to render the "private conference" somehow - which means compositing together the various WebRTC streams, switching to the active talker, highlighting them, showing their names etc, rendering screenshares etc. Jitsi is an SFU (Stream Forwarding Unit), which means that the compositing is done clientside in the webapp with a bunch of HTML+JS+CSS and WebRTC video streams - so if you want to livestream a conference, you need to have a client somewhere to do that rendering. Which means needing headless chromium + ffmpeg to capture it. The alternative would be to use a system where the conference is rendered serverside in the first place - a so called MCU (Multipoint Conferencing Unit) like FreeSWITCH. But this has a different set of tradeoffs, and having integrated both with Matrix we currently default to Jitsi. Totally agreed that it feels unintuitive to spin up a farm of 50 headless chromiums in order to livestream a conference though!
- kabes 6y agoIt's not uncommon to do some webrtc video compositing (MCU) in a headless chrome, since it's usually the easiest solution and you can add some decorations with html on top. But it's definitely not very efficient. Especially since this is mostly ran on linux and chrome has almost no hardware video decode/encode capabilities on linux (starting to change).
- deleted 6y ago[deleted]
- barbs 6y agoCool to see Jitsi being a key piece of software for this. Hadn't seen it used in a conference-style broadcast context before, so that's awesome!