3 ms·

FOQS: Making a distributed priority queue disaster-ready

55 points by el_duderino 5y ago

metadat 5y ago
Sounds cool, almost like competition for Kafka if it solves global multi-DC replication and has a scaling story. Is FB / “MarkZ The Generous”* planning to open-source this valuable sounding thing? I wasn’t able to find a mention about sharing it publicly in TFA. Would be a nice gesture, given how much $FB has benefited from running the entirety of the worlds largest social graph on top of MySQL. * lol
- vosper 5y ago
  > given how much $FB has benefited from running the entirety of the worlds largest social graph on top of MySQL. They've contributed quite a lot back to the MySQL community, I thought? MyRocks comes to mind http://myrocks.io/ http://myrocks.io/
  - metadat 5y ago
    Has BookFace contributed back commensurately to the degree Señor Mark Fuckerberg has benefitted? That would be truly impressive and challenging, to the tune of tens or hundreds of billions of dollars. MySQL is truly the unsung hero of FB, the whole operation would not have taken off and become a smash hit in the first place without riding atop the back good ol' MySQL. Unfortunately, Postgres wasn't such hot shit 15 years ago. Today Postgres is AMAZING, it is truly a great day to be alive and a programmer. Now our best choices for relational DBs are Postgres, MySQL, or Oracle 21c (lol, Oracle <3's $$$ but at least Larry is straightforward and honest about it :)
    - uvdn7 5y ago
      The same argument can be applied to Linux and apply to virtually all companies having a backend?
      - metadat 5y ago
        What does this change? Few companies have the resources making it even potentially feasible that they even could give back at the scale Faebook could.
    - evanelias 5y ago
      It's a two-way street. Facebook has made significant contributions to MySQL, including commits (they're literally one of the top third-party contributors), bug reports, and overall roadmap. Several of FB's top DB engineers previously worked for MySQL AB, Sun, and/or Oracle.
    - Proven 5y ago
      [dead]
- hn_throwaway_99 5y ago
  > Would be a nice gesture, given how much $FB has benefited from running the entirety of the worlds largest social graph on top of MySQL. There are plenty things to shit on Facebook for, but "not contributing to open source" is definitely not one of them. React, GraphQL, Cassandra, etc. are all hugely successful and influential projects.
- nemothekid 5y ago
  >Is FB / “MarkZ The Generous” planning to open-source this valuable sounding thing?* FB usually open sources many projects - but I wouldn't be surprised if this never leaves FB just on practicality. Anytime there's "global data-center scale" state management thingy, it's usually deeply integrated with whatever flavor of Zookeeper/Chubby/Service Discovery/AWS/Global metadata store/GPS Clock/Intern's cron job that makes it a pain to decouple, even worse to actually test against an open source version of these stacks.
- derekperkins 5y ago
  I mentioned this in a top level comment, but YouTube already open sourced their version of it in Vitess, with almost identical characteristics.
dimgl 5y ago
It's interesting that Facebook has created another queuing system with priority rules. I believe beanstalkd originated as a result of Facebook's use cases. Seems like a really cool piece of tech.
- evanelias 5y ago
  Beanstalkd was created by a company that ran an early third-party Facebook app (Causes), not Facebook itself. Facebook used their in-house "async" system, described in the post, for many years. Prior to that (over a decade ago) I believe they used Gearman for some things, but I'm not certain, might be misremembering. I don't know if they ever used beanstalkd early on.
  - dimgl 5y ago
    Cool! I didn't know this!
derekperkins 5y ago
You can already get almost the exact same functionality via Vitess Messaging, also built on MySQL and HA across regions, which has been tested at YouTube scale. Vitess doesn't support ordered messages, but otherwise the architecture is almost identical. https://vitess.io/docs/12.0/reference/features/messaging/ https://vitess.io/docs/12.0/reference/features/messaging/

55 points by el_duderino 5y ago

Sounds cool, almost like competition for Kafka if it solves global multi-DC replication and has a scaling story. Is FB / “MarkZ The Generous”* planning to open-source this valuable sounding thing? I wasn’t able to find a mention about sharing it publicly in TFA. Would be a nice gesture, given how much $FB has benefited from running the entirety of the worlds largest social graph on top of MySQL. * lol

vosper 5y ago
> given how much $FB has benefited from running the entirety of the worlds largest social graph on top of MySQL. They've contributed quite a lot back to the MySQL community, I thought? MyRocks comes to mind http://myrocks.io/ http://myrocks.io/
- metadat 5y ago
  Has BookFace contributed back commensurately to the degree Señor Mark Fuckerberg has benefitted? That would be truly impressive and challenging, to the tune of tens or hundreds of billions of dollars. MySQL is truly the unsung hero of FB, the whole operation would not have taken off and become a smash hit in the first place without riding atop the back good ol' MySQL. Unfortunately, Postgres wasn't such hot shit 15 years ago. Today Postgres is AMAZING, it is truly a great day to be alive and a programmer. Now our best choices for relational DBs are Postgres, MySQL, or Oracle 21c (lol, Oracle <3's $$$ but at least Larry is straightforward and honest about it :)
  - uvdn7 5y ago
    The same argument can be applied to Linux and apply to virtually all companies having a backend?
    - metadat 5y ago
      What does this change? Few companies have the resources making it even potentially feasible that they even could give back at the scale Faebook could.
  - evanelias 5y ago
    It's a two-way street. Facebook has made significant contributions to MySQL, including commits (they're literally one of the top third-party contributors), bug reports, and overall roadmap. Several of FB's top DB engineers previously worked for MySQL AB, Sun, and/or Oracle.
  - Proven 5y ago
    [dead]
hn_throwaway_99 5y ago
> Would be a nice gesture, given how much $FB has benefited from running the entirety of the worlds largest social graph on top of MySQL. There are plenty things to shit on Facebook for, but "not contributing to open source" is definitely not one of them. React, GraphQL, Cassandra, etc. are all hugely successful and influential projects.
nemothekid 5y ago
>Is FB / “MarkZ The Generous” planning to open-source this valuable sounding thing?* FB usually open sources many projects - but I wouldn't be surprised if this never leaves FB just on practicality. Anytime there's "global data-center scale" state management thingy, it's usually deeply integrated with whatever flavor of Zookeeper/Chubby/Service Discovery/AWS/Global metadata store/GPS Clock/Intern's cron job that makes it a pain to decouple, even worse to actually test against an open source version of these stacks.
derekperkins 5y ago
I mentioned this in a top level comment, but YouTube already open sourced their version of it in Vitess, with almost identical characteristics.

dimgl 5y ago

It's interesting that Facebook has created another queuing system with priority rules. I believe beanstalkd originated as a result of Facebook's use cases. Seems like a really cool piece of tech.

evanelias 5y ago
Beanstalkd was created by a company that ran an early third-party Facebook app (Causes), not Facebook itself. Facebook used their in-house "async" system, described in the post, for many years. Prior to that (over a decade ago) I believe they used Gearman for some things, but I'm not certain, might be misremembering. I don't know if they ever used beanstalkd early on.
- dimgl 5y ago
  Cool! I didn't know this!

derekperkins 5y ago

You can already get almost the exact same functionality via Vitess Messaging, also built on MySQL and HA across regions, which has been tested at YouTube scale. Vitess doesn't support ordered messages, but otherwise the architecture is almost identical. https://vitess.io/docs/12.0/reference/features/messaging/ https://vitess.io/docs/12.0/reference/features/messaging/