4 ms·

Essentially what everyone else does - distributed systems with load balancing, load balancing and more load balancing. And if that goes awry, triage - where the

Essentially what everyone else does - distributed systems with load balancing, load balancing and more load balancing. And if that goes awry, triage - where they prioritize messages and simply timeout and drop the lower priority messages. Of course the Messenger team is lucky in that they can drop messages since your family and friends missing a "Happy New Years" message isn't the end of the world. Other systems ( such as finance ), aren't so lucky. Drop a few transactions or apply them out of order and it is the end of the world. Was an interesting read, though it would have been nice if there were more specifics but I guess Facebook wouldn't approve that.

groestl 8y ago
From one of my projects (a MMORPG) I've learned that the required accuracy in non-financial transactions is often underestimated, while, on the other hand, financial transactions are often less critical that initially assumed. After all, compensation in financial transactions is often straightforward to calculate and apply. But the damage done through dropped/failed non-financial transactions is often hard to assess, and it's also more involved to find appropriate compensation.
linkmotif 8y ago
I don’t know, a missing “Happy New Years” isn’t missing dollars, but it’s definitely not cool to drop such a greeting —or any message—in my opinion. It should definitely be possible to at least store and then deliver these messages late. The baseline should be 100% deliverability and anything less than that should be subject to intense scrutiny. I mean, how big of a Kafka cluster do you need to make this happen?
arusahni 8y ago
Do you have any references/resources for plussing up on scaling transactions w.r.t. finance? It sounds like an intriguing set of challenges...
freddie_mercury 8y ago
> Of course the Messenger team is lucky in that they can drop messages since your family and friends missing a "Happy New Years" message isn't the end of the world. Other systems ( such as finance ), Messenger is (also) a finance system. You can send money via Messenger. You can purchase products directly in Messenger. It's had all that for more than two years now.
- therein 8y ago
  Then don't drop the transactions pertaining sending money. They are not necessarily treated the same way as messages in the backend.
- tozeur 8y ago
  That’s not the proper comparison. The difference is that payments sent through Messenger aren’t critical (think paying your friend for lunch), whereas for most financial transactions (eg trades of stocks, derivatives, swaps etc) it’s absolutely critical that they execute immediately.
- dmix 8y ago
  The OP mentioned a prioritization system for a reason...
radicality 8y ago
Actual messages with content are never dropped. Only ‘meta-messages’ like read receipts - it’s not critical if on a group chat with many participants the state of ‘who’s seen the last message” is not 100% correct on New Year’s Eve.

groestl 8y ago

From one of my projects (a MMORPG) I've learned that the required accuracy in non-financial transactions is often underestimated, while, on the other hand, financial transactions are often less critical that initially assumed. After all, compensation in financial transactions is often straightforward to calculate and apply. But the damage done through dropped/failed non-financial transactions is often hard to assess, and it's also more involved to find appropriate compensation.

linkmotif 8y ago

I don’t know, a missing “Happy New Years” isn’t missing dollars, but it’s definitely not cool to drop such a greeting —or any message—in my opinion. It should definitely be possible to at least store and then deliver these messages late. The baseline should be 100% deliverability and anything less than that should be subject to intense scrutiny. I mean, how big of a Kafka cluster do you need to make this happen?

arusahni 8y ago

Do you have any references/resources for plussing up on scaling transactions w.r.t. finance? It sounds like an intriguing set of challenges...

freddie_mercury 8y ago

> Of course the Messenger team is lucky in that they can drop messages since your family and friends missing a "Happy New Years" message isn't the end of the world. Other systems ( such as finance ), Messenger is (also) a finance system. You can send money via Messenger. You can purchase products directly in Messenger. It's had all that for more than two years now.

therein 8y ago
Then don't drop the transactions pertaining sending money. They are not necessarily treated the same way as messages in the backend.
tozeur 8y ago
That’s not the proper comparison. The difference is that payments sent through Messenger aren’t critical (think paying your friend for lunch), whereas for most financial transactions (eg trades of stocks, derivatives, swaps etc) it’s absolutely critical that they execute immediately.
dmix 8y ago
The OP mentioned a prioritization system for a reason...

radicality 8y ago

Actual messages with content are never dropped. Only ‘meta-messages’ like read receipts - it’s not critical if on a group chat with many participants the state of ‘who’s seen the last message” is not 100% correct on New Year’s Eve.