3 ms·

Just about everything you've said here is factually incorrect. There are far, far fewer than 100 people working on MySQL at Facebook. I say this as a former me

Just about everything you've said here is factually incorrect. There are far, far fewer than 100 people working on MySQL at Facebook. I say this as a former member of the team, and I am still in close contact with several team members. Additionally, every engineer at FB has the ability to issue ad hoc SQL queries, and many teams do use special-case MySQL DBs that they interact with directly. I was the lead on FB's DBaaS so trust me that I know what I'm talking about here. Over 1000 engineers had used the DBaaS at the time I left the company. There is no legacy issue involved. FB is changing storage engines in MySQL right now; they have the resources to completely move away from MySQL if there was a compelling reason. Your statement of "If they had it to do over again I think that no one at Facebook would select MySQL at this time" is completely incorrect. Even if you ignore FB, a much larger percentage of the largest sites on the internet use MySQL than Postgres.

evgen 10y ago
As a former manager on the ops side who worked closely with the managers of the db teams, and the hadoop/hbase teams, and the presto folks, wormhole, scuba, a few other teams that may not be public knowledge, etc. I can say with absolute certainty that the only people in the company who thought it was not a sad joke that we used MySQL down in the bottom layers of the stack were the people on the db team. And why wouldn't that team like MySQL? Knowledge of its intricacies and bugs and how to work around its problems is what got them this high-paying gig. I will simply state that having also spent a long time inside FB I disagree with your assessment regarding the high regard in which you think MySQL is held within the company.
- evanelias 10y ago
  > I can say with absolute certainty that the only people in the company who thought it was not a sad joke You can say with absolute certainty that you know the opinion of thousands of engineers at Facebook? This is clearly hyperbole. It does not appear you are interested in having a meaningful discussion on this topic. > I disagree with your assessment regarding the high regard in which you think MySQL is held Where did I say it's held in high regard? I simply said there is no compelling reason to move away from it, and also said your numbers were way off base in terms of how many people are working on MySQL at FB, as well as how many people directly interact with MySQL at FB. I also greatly question how closely you were interacting with the managers of the db teams if you think there were 100 people on these teams! [EDIT: It's also possible that we just have very different definitions of people "on the db team". For people "working on MySQL" I would only include MySQL Eng, DB Client, MySQL Infra PE, and the majority of Data Performance PE. Arguably you could include some RocksDB people at this point, though their work is used in many non-MySQL ways too, and even then that brings the total to still way under 100 people. Overall I would not include teams like Wormhole or TAO as "working on MySQL", though even if you add them in, still significantly under 100.]
evanelias 10y ago
In retrospect my tone should have been softer and less combative here. My apologies. But to give some perspective on why this hits a nerve: By greatly overstating the number of engineers who worked on something, you're directly undermining the effort and ingenuity of the engineers who worked on it. This stings coming from a former coworker. In your second reply (sibling to this one) you seem to state that Facebook's db engineers are motivated by self-preservation (re: staying with MySQL), and you imply they are over-paid. If self-preservation was a motivation at all, FB wouldn't be migrating major data sets from InnoDB to MyRocks, which has a completely different set of performance characteristics and administrative/operational/automation concerns. As for pay, yes the db teams are very well-compensated because they are literally among the top experts in the field, working on an absolutely massive billion-daily-user OLTP system and all of the insane challenges that entails. Put yourself in my shoes. You worked in security, yes? If I posted some factually incorrect comments about your team, and the engineering motivations and talents of your team, on a public forum like this, how would you react?
- evgen 10y ago
  I actually started at FB by managing SRO prior to joining the security team, so I am quite sure that I was over-estimating my total number of people by including the teams in Dublin and probably everyone who needed to hop on to a udb directly for some reason. As you can imagine, from that starting-point in my introduction to MySQL at FB it was all about dealing with various problems and slapping down hundreds of db replication error and similar alarms every hour in the pre-FBAR/pre-db-automation world. I was also probably out of line with the 'it is about job security' bits, sorry about that. Actually, scratch 'probably' from that apology, but having passed through the ops side, infra eng, and security roles at FB I ended up seeing all of the problems and a lot of the solutions developed there seemed in my view to be rather expensive ways to fix problems that either didn't manifest in Postgres or would be easier to fix in a different db. I still maintain that while MySQL was the right choice in 2006, if you were starting FB today it would be hard to convince someone that MySQL is a better choice than Postgres.
  - evanelias 10y ago

evgen 10y ago

As a former manager on the ops side who worked closely with the managers of the db teams, and the hadoop/hbase teams, and the presto folks, wormhole, scuba, a few other teams that may not be public knowledge, etc. I can say with absolute certainty that the only people in the company who thought it was not a sad joke that we used MySQL down in the bottom layers of the stack were the people on the db team. And why wouldn't that team like MySQL? Knowledge of its intricacies and bugs and how to work around its problems is what got them this high-paying gig. I will simply state that having also spent a long time inside FB I disagree with your assessment regarding the high regard in which you think MySQL is held within the company.

evanelias 10y ago
> I can say with absolute certainty that the only people in the company who thought it was not a sad joke You can say with absolute certainty that you know the opinion of thousands of engineers at Facebook? This is clearly hyperbole. It does not appear you are interested in having a meaningful discussion on this topic. > I disagree with your assessment regarding the high regard in which you think MySQL is held Where did I say it's held in high regard? I simply said there is no compelling reason to move away from it, and also said your numbers were way off base in terms of how many people are working on MySQL at FB, as well as how many people directly interact with MySQL at FB. I also greatly question how closely you were interacting with the managers of the db teams if you think there were 100 people on these teams! [EDIT: It's also possible that we just have very different definitions of people "on the db team". For people "working on MySQL" I would only include MySQL Eng, DB Client, MySQL Infra PE, and the majority of Data Performance PE. Arguably you could include some RocksDB people at this point, though their work is used in many non-MySQL ways too, and even then that brings the total to still way under 100 people. Overall I would not include teams like Wormhole or TAO as "working on MySQL", though even if you add them in, still significantly under 100.]

evanelias 10y ago

In retrospect my tone should have been softer and less combative here. My apologies. But to give some perspective on why this hits a nerve: By greatly overstating the number of engineers who worked on something, you're directly undermining the effort and ingenuity of the engineers who worked on it. This stings coming from a former coworker. In your second reply (sibling to this one) you seem to state that Facebook's db engineers are motivated by self-preservation (re: staying with MySQL), and you imply they are over-paid. If self-preservation was a motivation at all, FB wouldn't be migrating major data sets from InnoDB to MyRocks, which has a completely different set of performance characteristics and administrative/operational/automation concerns. As for pay, yes the db teams are very well-compensated because they are literally among the top experts in the field, working on an absolutely massive billion-daily-user OLTP system and all of the insane challenges that entails. Put yourself in my shoes. You worked in security, yes? If I posted some factually incorrect comments about your team, and the engineering motivations and talents of your team, on a public forum like this, how would you react?

evgen 10y ago
I actually started at FB by managing SRO prior to joining the security team, so I am quite sure that I was over-estimating my total number of people by including the teams in Dublin and probably everyone who needed to hop on to a udb directly for some reason. As you can imagine, from that starting-point in my introduction to MySQL at FB it was all about dealing with various problems and slapping down hundreds of db replication error and similar alarms every hour in the pre-FBAR/pre-db-automation world. I was also probably out of line with the 'it is about job security' bits, sorry about that. Actually, scratch 'probably' from that apology, but having passed through the ops side, infra eng, and security roles at FB I ended up seeing all of the problems and a lot of the solutions developed there seemed in my view to be rather expensive ways to fix problems that either didn't manifest in Postgres or would be easier to fix in a different db. I still maintain that while MySQL was the right choice in 2006, if you were starting FB today it would be hard to convince someone that MySQL is a better choice than Postgres.
- evanelias 10y ago