10 ms·
How Facebook Keeps Messenger from Crashing on New Year's Eve
- lostmsu 8y agoMine crashes stably on launch right now on two different machines. Therefore not so good :-D
- gerdesj 8y agoNew Years Day hasn't happened yet. Your problems indicate that you should turn it off and on again.
- snazz 8y agoTurn two different machines off and on again? Doesn't sound like a coincidence to me, unless some other shared hardware (network, etc) is causing the problem.
- lostmsu 8y agoWell, the problem exists for quite a while now. But what's the point of the backend working, if the app does not launch. Says something about quality, when a 300MB IM app won't start.
- sluggg 8y agoon mobile? are you running the most up to date version? are you on a web browser? are any of your extensions mucking things up?
- lostmsu 8y agoBoth machines are Windows, the app installed from the store. When it does not crash on launch, it takes forever to start.
- mliker 8y agoHow do you know if your problem is a client app problem vs a backend issue? :P
- sjroot 8y agoAs someone who typically works on front-end projects, this was a very interesting read. I particularly loved the discussion of “graceful degradation.” That’s the kind of collaboration across the stack that makes a service like Messenger very pleasant to use.
- robertAngst 8y agoInteresting how they decide to remove features like 'seen' and 'online' in an effort to prioritize the actual message. Offtopic- I hate those features to begin with, but I know I'm the product not the customer, and those features are to keep people on the app.
- oscar_wong67 8y agoI also despise those features, but I agree wholeheartedly that these decisions push the money forward
- bcherny 8y agoOut of curiosity, why do you not like those features? To me, "seen" is a feedback mechanism. When I send a message to you, I don't always need a response, but I do usually want to know that you saw the message -- sort of an ACK.
- tuesdayrain 8y agoBecause I like spending a long time coming up with the optimal message to send, and it's awkward if the other person knows how long it took me
- hn_throwaway_99 8y agoI think at its very base it's a privacy issue, which is not surprising that FB has chosen to ignore it. Just because you sent me a message, why should you then be privy to see if I've read it or not without me explicitly doing something, or at least explicitly having the option to enable or disable this.
- moreira 8y agoBecause a lot of people are not like you. If someone saw that I read their message they might immediately expect a response, and be left wondering why it took hours to get back to them (and react negatively to the delay). Not everyone treats messaging as the async thing it is.
- ngngngng 8y agoI didn't realize message queues were used for this type of task. I'm assuming you would then also use autoscaling pods that respond to the number of messages in the queue. How do you scale pods fast enough for a messaging application or anything else trying for 100ms or less per operation?
- jcmi 8y agoMessaging queues are a core part of a lot of high scale distributed systems (source - Twitter) You want enough queue space to handle the expected volume and then some. Assuming you have that, you don't need to instantly scale instances out to match the amount of messages, you just need to catch up before the queue space runs out.
- underwater 8y agoFacebook runs its own hardware. How would "autoscaling" help them?
- tybit 8y agoDynamically allocating hardware to different services makes sense for self managed DCs too. Breaking down cost per service of team is useful, as is seeing how much hardware is necessary for peak and trough I imagine. I wouldn’t be surprised if on NYE services like chat take priority and get extra resources from background jobs that can wait a few hours.
- gmmeyer 8y agoMessage queues (or similar things like Kafka, which isn't quite a proper "message queue") are used for basically everything at this scale. Messages are being passed indirectly. An event happens, it gets popped on a queue, and then the recipients do something with it.
- latch 8y agoMaybe you'll consider this pedantic, or maybe even wrong, but I think you meant to say "asynchronous" as opposed to "indirectly". I think anyone googling to learn more will get better results.
- moron4hire 8y agoBy losing lots of users over data privacy scandals in the month before?
- webo 8y agoFacebook get its share amount of bad PR (some are well deserved), but we shouldn’t dismiss amazing engineering work because of those. This is a technical piece that highlights solutions to problems not many out there get to solve.
- adjkant 8y agoThe thing is, this is clearly for marketing purposes in the end. The approach is complex and sophisticated, but certainly not novel. A good architect in any decent interview should get to a similar solution. This must be read through the lines: this is an attempt to try and change their bad PR cycle. I'm not saying the parent comment is absolutely essential, but I'd rather see that instead of people using this to change the conversation from central flaws in Facebook's data practices, business model, and ethics in general. This may sound a bit harsh, but at a certain point you have to really consider what giving publicity to a group gives them versus the inherent value of the paper.
- blattimwind 8y agoYou can't disconnect the technical aspects from their sociopolitical environment (I say that and I'm neither left nor SJW, not by Internet's standards anyway). It'd be like marvelling at Nazi engineering without considering the context.
- porpoisely 8y agoEssentially what everyone else does - distributed systems with load balancing, load balancing and more load balancing. And if that goes awry, triage - where they prioritize messages and simply timeout and drop the lower priority messages. Of course the Messenger team is lucky in that they can drop messages since your family and friends missing a "Happy New Years" message isn't the end of the world. Other systems ( such as finance ), aren't so lucky. Drop a few transactions or apply them out of order and it is the end of the world. Was an interesting read, though it would have been nice if there were more specifics but I guess Facebook wouldn't approve that.
- groestl 8y agoFrom one of my projects (a MMORPG) I've learned that the required accuracy in non-financial transactions is often underestimated, while, on the other hand, financial transactions are often less critical that initially assumed. After all, compensation in financial transactions is often straightforward to calculate and apply. But the damage done through dropped/failed non-financial transactions is often hard to assess, and it's also more involved to find appropriate compensation.
- linkmotif 8y agoI don’t know, a missing “Happy New Years” isn’t missing dollars, but it’s definitely not cool to drop such a greeting —or any message—in my opinion. It should definitely be possible to at least store and then deliver these messages late. The baseline should be 100% deliverability and anything less than that should be subject to intense scrutiny. I mean, how big of a Kafka cluster do you need to make this happen?
- arusahni 8y agoDo you have any references/resources for plussing up on scaling transactions w.r.t. finance? It sounds like an intriguing set of challenges...
- freddie_mercury 8y ago> Of course the Messenger team is lucky in that they can drop messages since your family and friends missing a "Happy New Years" message isn't the end of the world. Other systems ( such as finance ), Messenger is (also) a finance system. You can send money via Messenger. You can purchase products directly in Messenger. It's had all that for more than two years now.
- dominotw 8y agoCant wait for akward new yr wishes from ppl I haven't heard from since last new year.
- mrdickbig 8y agoDeleted my FB 2 years ago... no regrets. You should try
- secabeen 8y agoInteresting that the messenger team is ~40 people, as compared to WhatsApp having 32 engineers at the time of their sale to FB.
- traek 8y agoThe photo caption says that's just the infrastructure team. I'd imagine the product team is much larger given how many features are crammed into Messenger.
- adtac 8y agoI wish it was the other way around. We'd have fewer, but faster features.
- marksomnian 8y agoSeems like Messenger is getting WeChat-ized - in China, WeChat [0] is your one-stop shop to getting anything done - chatting, following celebrities, booking housekeeping, payments, even "mini-programs". And now many Western messaging apps are starting to take inspiration from this [1]. [0]: https://en.wikipedia.org/wiki/WeChat https://en.wikipedia.org/wiki/WeChat [1]: https://blog.ycombinator.com/lessons-from-wechat/ https://blog.ycombinator.com/lessons-from-wechat/
- markkanof 8y agoIs it working though? Just because Facebook keeps adding features doesn’t necassarily mean that people are using those features at the same scale as WeChat. I avoid Facebook messenger entirely, but are others using it for all these ancillary things like sending friends money?
- fwn 8y agoMy (non-technical) friends & family, as anecdotal evidence, seem almost universally happy about switching to Facebooks Lite Messenger. (An app that only provides messaging functionality.) They perceive it as less confusing.
- mrdickbig 8y agoI read the title and instantly thought "Erlang"
- ckwalsh 8y agoFun fact: FB Chat was originally implemented in Erlang https://www.facebook.com/notes/facebook-engineering/chat-stability-and-scalability/51412338919/ https://www.facebook.com/notes/facebook-engineering/chat-sta...
- openloop 8y agohttps://docs.docker.com/engine/swarm/ingress/ https://docs.docker.com/engine/swarm/ingress/
- sn41 8y agoI found the concept of affinity used for bunching messages to be a new concept. The other concepts are not surprising, they are the typical engineering solutions used. One thing I am curious about: how does it compare to how the old postal systems used to handle Christmas and new year loads? On a lighter note: perhaps you can predictively generate and cache messages at the receiver's end based on their contacts and their style of communication. When a sender actually sends a message, just send one bit across, and the local cache gets flushed and displayed :)
- kaustyap 8y agoIt may be a good idea to provide range of fixed templates for standard messages and send just selective message Id along with recipient name over the network.
- zimpenfish 8y ago> I found the concept of affinity used for bunching messages to be a new concept. I'm reasonably sure that UUCP + things like INN were "batching messages per destination" for efficiency a long time ago. Nowhere near the same scale, obviously, but the same kind of concept, no?
- Jedd 8y agoActually, I'd expect the scale would be similar. The raw numbers (users, bandwidth, latency) would be quite different.
- ehsankia 8y agotake the 16 most common messages, normalize them a bit, and encode them in 4 bit, and I would guess that would probably cover over 50% of the messages sent, maybe even close to 80%.
- fhrow4484 8y agoWhile it's a nice compression technique, I doubt you'd gain anything from this at the message level. Even by reducing a message content to close to 0 bits, I doubt you would get that much gains. Most common messages would probably be things like "lol" or an emoji, which in utf-8 is 24 to 32 bits. But There'll always be metadata associated with each message: message id, sender, receiver, timestamps, etc. which are unremovable. If those first 3 are typical GUIDs, this is already 3*128 bits. This will put a minimum on the entire message size , and more importantly the processing time to simply route the messages is what is costly in terms of CPU time. Then there's the different receipts mentioned in the article, adding a lot more traffic.
- hellofunk 8y agoOne thing I find very manipulative about Facebook is how, when someone sends you a message, the email notification has a link to open messenger and it says that messenger is the only way you can read that message, even if you don’t have messenger installed. They are trying everything they can do to have everyone install that app. Yet, of course you can just read and respond directly on their website without any app, but they don’t link to that or mention it.
- groestl 8y agombasic.facebook.com for anyone reading this and looking for a way to work around Facebook's dark UI pattern on mobile, where a click on the "messages" button wants you to install the app.
- Jaruzel 8y agoBack in the day when SMS text message was the way to send messages to each other over the mobile phone network (in the UK), people would jump the gun by sending 'Happy NY!' messages 5 minutes before midnight, because the moment 12am hit, any messages sent then could be queued for hours as the mobile networks struggled to cope with the massive uptick in messages being sent at the same time.
- Semaphor 8y agoI used to use bulk sending tools (yeah, they existed as J2ME apps) to send 10-50 SMS (it's been a while, not sure how many it was) and some would only arrive on January 1st quite a while into the day. At some point, it changed and everything would arrive just a few minutes after sending. I think that was when we had iPhone and Android already and I'm not sure if it was because of messengers (were there any back then?) or because the German telcos finally upgraded their infrastructure enough.
- Markoff 8y agothis certainly feel like paid article to bring some good image to Facebook
- John_KZ 8y agoThis is a 2018 internet-connected app, not a 1985 GSM network. 1 billion 100 Byte messages sums up to an almost trivial 100GB. This might be a technical challenge for the neighborhood's web admin but not for any real company.
- notacoward 8y agoWhat kind of tyro thinks GB/s instead of ops/s is the right measure here?
- John_KZ 8y agoThis is a text message. Unless you're doing it horribly wrong, ops/s should be very close to the number of messages.
- notacoward 8y agoThat wasn't the point at all. You were sneering at others because you had converted the numbers into GB/s and those numbers looked small, but the only foolishness was doing the conversion in the first place and the only person doing it was you. A billion messages of any size is nothing to sneeze at. Have you ever worked on a system that handled that many messages in its entire lifetime, let alone a single day? P.S. There's no such thing as a 1985 GSM network. That phrase just makes your comment look even sillier.