3 ms·
Oh wow, some interesting stuff in here. Looks like they use MySQL as a queue for scheduling the ORC Peons? Would have loved to hear more about why they did that
by elvinyung 10y ago
Oh wow, some interesting stuff in here. Looks like they use MySQL as a queue for scheduling the ORC Peons? Would have loved to hear more about why they did that.
- hueving 10y agoBecause it works perfectly fine? I tend to write anyone off who scoffs at simple database-as-queue designs without understanding what the scaling requirements are. You can use a database as a job queue for 10s of thousands of jobs per day without any sweat.
- OJFord 10y agoScoffing? GP said it was "interesting stuff" and that they'd "love to hear more". What's wrong with that?
- deleted 10y ago[deleted]
- elvinyung 10y agoPlease forgive me, but I don't understand your hostility. I understand that there are perfectly legitimate reasons for using a database as a queue. If you frequently need to look at and rearrange your jobs while they're in flight, chances are a pure FIFO structure probably doesn't work that well for you anyways. If the enqueue is contingent on a transaction committing, probably makes a lot of sense for the job to be in the same database. You don't need to tell me -- I've seen more than a few in production. But an actual message queue also "works perfectly fine" based on the information provided, and I would imagine that a company like Facebook probably already has a few of those lying around. It would have been a conscious choice to use MySQL as a queue. I swear I'm actually, seriously just curious!
- jivid 10y agoAuthor here. We use it because it works well for us. We've put a lot of work into making MySQL scale for us to the point where it's a very well supported system and one of the main choices for a lot of storage decisions. We even use MySQL as a queue for Facebook Messenger. More details about this: https://www.youtube.com/watch?v=eADBCKKf8PA https://www.youtube.com/watch?v=eADBCKKf8PA https://code.facebook.com/posts/820258981365363/building-mobile-first-infrastructure-for-messenger https://code.facebook.com/posts/820258981365363/building-mob...
- elvinyung 10y agoThanks! Standardizing on MySQL because of internal expertise is definitely a great reason.
- hamandcheese 10y agoGitHub also migrated persistent data out of Redis and in to MySQL, with their expertise in MySQL as one of the motivating factors. https://githubengineering.com/moving-persistent-data-out-of-redis/ https://githubengineering.com/moving-persistent-data-out-of-...