7 ms·
Real-time messaging
- bombolo 3y ago> They replace unhealthy CSs very quickly and efficiently Maybe fixing the bugs so that they don't mysteriously get ill would be a thing to pursue?
- paxys 3y agoSilly engineers, implementing things like failovers and redundancy. Why can't they just ensure that their servers never crash?
- bombolo 3y agoMore like, silly engineers relying on the fact that they have failover, so they don't bother to fix the actual issue. But sure, you get a medal for being a smartass online :)
- 0xCAP 3y agoWho suggested they're not doing that already?
- AlotOfReading 3y agoI'm sure they have whole teams dedicated to that, but even perfect software would still need these sorts of health checks at the scale slack operates. Sometimes you just have random failures in hardware or the OS, or you need to migrate software around the datacenter without bringing down the service. This is a generic way to route around all of that.
- matlin 3y agoKind of funny timing because I've noticed my messages in Slack have been quite a bit more delayed than usual for the last few months, especially when moving between devices. Maybe they need more "channel servers"
- TeMPOraL 3y agoThey're preparing you to the real-time messaging experience when you're working remote from Mars.
- cj 3y agoNoticed this today, too. I turned on a computer I haven’t logged in to in 2 months, and clicking a channel took at least 5 seconds to open/load. And then it worked normally after that.
- stavros 3y agoSame here, all channels took 5 seconds to load. I've used Slack on that computer recently though.
- nkozyra 3y agoI've noticed a big performance drop in the last few weeks as well. Everything has gotten more sluggish.
- muglug 3y agoThere was a performance issue with channel loading, documented here: https://status.slack.com/2023-06-23 https://status.slack.com/2023-06-23
- 8organicbits 3y agoAgreed, something changed that made Slack usage across multiple devices much worse. Loading a channel tends to sputter as various state and messages load, messages I already read on different devices change state, it's pretty jarring to watch. The "unread messages" badge is often out of sync. E: if you reach out to support, they can reset stuck state for you. They were very helpful but didn't sound sure it would work.
- jw1224 3y agoThis seems complex, and not necessarily in a good way
- eikenberry 3y agoWhy would an asynchronous messaging platform worry about real-time messaging? It seems rather counter to what the platform is meant for.
- nomel 3y ago> It seems rather counter to what the platform is meant for. Well that's just silly, if you've ever used used slack. It's corporate chat rooms, with real-time chat being a fundamental feature of it. > counter to what the platform is meant for. The platform is meant for communication. Async communication is one (probably smallish, at least in my experience) slice of that.
- pc86 3y agoNo you don't understand, the GP uses Slack asynchronously so if you don't use Slack asynchronously you're doing it wrong.
- candiddevmike 3y agoI wonder if slack engineering and it's users would benefit from tick/queue based messaging vs real time. Say messages are only exchanged every minute.
- nomel 3y ago> Say messages are only exchanged every minute. One of the primary uses for slack, in our org, is live support. Having people sitting around, waiting for timers to expire, could never be a benefit for us.
- pc86 3y agoWhy would that be at all beneficial? "We have this very successful, fairly technically advanced as-real-time-as-is-practicable messaging system that most of our users use for real-time communication. We spent large amounts of engineering resources, time, and money to make it not real-time."
- rmf11048 3y ago[dead]
- mabbo 3y agoA dumb pet peeve I have is the use of the term "real time". What on earth do you even mean by that? Talk to a controls engineer about real time operating systems and PLC programming, and you get a very solid definition. Talk to a software developer and it means something like "as fast as we can, without any purposeful delays or buffers". Maybe I'm just a pedant, but the entire article has graphs with no numbers, no defined service level objectives, and vaguely mentions 500ms at the end. What is real time?
- Wingy 3y agoI’d say anything that users perceive as instantaneous.
- paxys 3y ago> Talk to a software developer and it means something like "as fast as we can, without any purposeful delays or buffers" No, "real time" in a software context means having a persistent stateful TCP/UDP connection open between a client and a server (or two clients in case of WebRTC and the like) to exchange data, skipping over paradigms like having to establish new HTTP connections or being blocked on database fetches. This is a well-established definition, and has nothing to do with any specific millisecond threshold.
- pipe_connector 3y agoLots of people being negative about this, but if you've ever implemented anything that works in near-real-time at wide scale, most of this design makes sense and it works great. One thing interested me: Why the difference in pathing between events and messages? I think the event flow makes sense, but why not have messages also go through your gateway server instead of through webapp? Surely there is needless latency there when you already have an active websocket open to gateway? I thought perhaps it was because your gateway was for egress-only, but then the events section made it clear you also handle ingress there.
- deleted 3y ago[deleted]
- klabb3 3y agoMy guess: it’s persisted mutations that need their own retry- and deduplication logic, as well as user facing error handling. Since it hits the db in the main region anyway latency is similar.
- jhgg 3y agoYeah it's a pretty cool design. Discord's real time system operates on a similar (although much more complicated system) on top of hash rings as well.
- tekkk 3y agoInteresting architecture. Asynchronous message passing at scale becomes quite a PITA very quickly. Wonder how scalable the code base really is. Probably Java isn't that bad of a choice considering the tooling although it's not my personal favourite. Seems quite the ideal use-case for Elixir/Erlang but since haven't used it myself, don't know would it be that much better. Especially training/developer pool -wise. And Vitess/MySQL for persistence? No Cassandra or something similar?
- candiddevmike 3y agoWhat makes you recommend Cassandra here over the other?
- roughly 3y agoMySQL can scale extremely well if you tune and use it correctly - Twitter was still backed primarily on MySQL at least through Obama’s second election, and I’m pretty sure for some time after that.
- OkayPhysicist 3y agoDiscord uses Elixir. They're fans. https://elixir-lang.org/blog/2020/10/08/real-time-communication-at-scale-with-elixir-at-discord/ https://elixir-lang.org/blog/2020/10/08/real-time-communicat... Notably, with regards to the hiring pool: > None of the chat infrastructure engineers had experience with Elixir before joining the company. They all learned it on the job.
- xwowsersx 3y agoIt is definitely an area in which Elixir excels. My personal feeling, however, at this advanced age of late 30s, is that I just can't do dynamically typed languages for large projects anymore. A statically typed Elixir would be a dream I think. There is https://gleam.run/ https://gleam.run/, but I don't know how mature it is.
- regulation_d 3y agoThere is active work being done currently to add set-theoretic types to the core language. https://www.youtube.com/watch?v=gJJH7a2J9O8 https://www.youtube.com/watch?v=gJJH7a2J9O8
- Solvency 3y agoIs there anything fundamentally unique about this architecture? AOL chatrooms in 1995 too were blazingly real-time even on dialup. Anyone ever use progz/scrollers? With an OH (overhead) account you could flood rooms with thousands of messages a second. Rich text, not plaintext. And it'd appear in super fast "real time" to everyone in the chat.
- paxys 3y agoThe biggest difference is that if you shoved a million active users in a single chatroom back then it would have crashed all of AOL. The same problem can have vastly different solutions at different scales.
- Solvency 3y agoHuh? Where have you ever seen a Slack channel with a million active users?
- paxys 3y ago100K – https://slack.com/blog/news/top-u-s-retailer-h-e-b-rolled-out-slack-to-100k-employees-in-72-hours https://slack.com/blog/news/top-u-s-retailer-h-e-b-rolled-ou... 350K – https://www.theverge.com/2020/6/4/21280829/slack-amazon-aws-partnership-amazon-chime-voice-video-calls https://www.theverge.com/2020/6/4/21280829/slack-amazon-aws-... The Kubernetes Slack has close to 200K users right now. Amazon has 1.5 million employees, and I'm sure the majority are on Slack. Forget 1995, no service other than Slack can run group chats at this scale even today.
- berkle4455 3y ago> I’m sure the majority are on Slack 80% of Amazon’s employees are warehouse and drivers.
- paxys 3y agoWhich still leaves >300K corporate employees.
- roughly 3y agoI’m curious about the actual message passing implementation(s) here - there’s a lot of “CS passes a message to all the GSs” - what’s that look like? Is it a general pubsub system? Did they roll their own? What’s the actual machinery here?
- nyanpasu64 3y agoThis reminds me, a few months ago, a "real-time" chat app appeared on HN which would display your typing to a public chat room before you hit Enter. And people found ways to inject XSS into usernames, including images and JS popups. Does anyone remember the name and URL of the site? I think it would make an intriguing alternative to voice calls, which is more synchronous/immediate/interactive than a conventional IM app where the other person can only start reading a sentence once you've finished typing it.
- robotresearcher 3y ago> I think it would make an intriguing alternative to voice calls 'talk' was an everyday tool back in the 1980s and early 90s that sent characters as they were typed. I just checked and it's installed on my Mac today. It's person-to-person, not group chat, but there's something magical about seeing every keystroke in near-real-time. You can feel the person at the other end.
- akvadrako 3y agoThat was a standard chat feature back in the day, certainly with AIM and probably ICQ. Its fine if you don't care enough to edit what you say, but if you do it requires composing the sentence first in your head.
- sunnybeetroot 3y agoLike speaking?
- jongjong 3y agoMeh. Realtime pub/sub at scale is easy as pie. I wrote a WebSocket client/server toolkit which can handle unlimited users with automatic channel sharding and auto-scaling on Kubernetes as a weekend project. It was too easy. I made it open source and for some reason it only ended up being used by some crypto companies and a few adult chat websites though one of them said they were handling 120K concurrent users with a couple of instances.
- esafak 3y agoSounds fun. Is the repo still up?
- immibis 3y agoDid you actually test it with unlimited users?
- jongjong 3y agoI tested it with hundreds of thousands of concurrent connections running on multiple hosts yes.
- immibis 3y agoThousands is nothing. A single thread can handle thousands. You don't need any of this sharding or Kubernetes stuff. I'd wager 120K connections can be handled by one fast instance. Above that, you have basic sharding techniques like assigning each channel to an instance. This should work up to some thousands of moderately active chatters in a single channel - at which point nobody can read all the messages anyway so it's not working for non-technical reasons. Slack is trying to solve the problem of having thousands of channels with millions of people, as well as millions of channels with thousands of people. In other words, about 6 orders of magnitude bigger than your load test.
- jongjong 3y agoI was limited by time and budget there. My solution is open source after all and embarrassingly parallel so can potentially handle sharding across thousands of hosts. If I remember correctly, my calculations suggested that a cluster with 2k hosts could handle 40 million concurrent users if you push a message to every user every 5 to 10 seconds with some randomness. The per-channel limit at that rate (1 message to every users every 5 to 10 seconds) was 20K concurrent connections but the number of channels is unlimited since you can add more capacity by adding more hosts and the sharding of channels across available hosts is automatic. The only limit was number of hosts since there is only one coordinator instance in the cluster though the cluster can continue to operate without any downtime while the coordinator instance is down. The coordinator instance only needs to be up while the cluster is scaling up or down. That said it should be able to handle up to 5k hosts, potentially even much higher.
- immibis 3y agoI'm seeing a problem where gateway servers have to subscribe to channels therefore they need to know which users are in which channels, but they're not segregated per channel so this load might not distribute well. Gateway servers could have been made more stateless if the channel server would send a list of which user IDs to broadcast a message to.