5 ms·

Gotta love how far MySQL + k/v store can get ya. I have no experience with zippyDB, but MySQL and ScyllaDB certainly feels like having an M1 Abrams and a B2 Spi

Gotta love how far MySQL + k/v store can get ya. I have no experience with zippyDB, but MySQL and ScyllaDB certainly feels like having an M1 Abrams and a B2 Spirit in my back pocket. I’m constantly pushing the “relationships from MySQL, data from Cassandra” narrative. It sounds like Facebook has a similar approach - although this post doesn’t make it clear where message data actually resides. MySQL, ScyllaDB, and RabbitMQ. A stack made in pragmatist heaven by devops angels. Now to find some developers who don’t hate it ;)

neon_electro 3y ago
Can you share your take on why MySQL and not Postgres?
- erulabs 3y ago
  For me it's mostly about known-knowns. I've been responsible for pretty large/company-critical MySQL clusters and have managed to sleep soundly at night. I tend to be the "database guy" at every startup, and those startups tended to use MySQL. I have to say, I do love PostGIS. But for "this data absolutely cannot be lost or its game-over", I'd go MySQL every time. I can and have debugged MySQL replication on zero hours of sleep. It's entirely possible Postgres 16 is the better choice for new projects today, but I haven't had the time to try it out in production yet.
  - brightball 3y ago
    > But for "this data absolutely cannot be lost or its game-over", I'd go MySQL every time. Man...times have seriously changed if that's the case. Good for MySQL to improve enough to get that recommendation.
  - raffraffraff 3y ago
    It amazes me how much better MySQL has been in this regard for at least a decade, and it's also amazing that it's still not that well-known today. Back in 2015 I worked at a fast-growing unicorn that had badly implemented basically everything because they started with a tiny ops team of grads and developers. Very little was being monitored, there were only a handful of metrics being graphed (mostly network stuff in Cacti). Our DB issues were all caused by stupid stuff : * undetected hard disk in array fails * battery in array controller fails * disk fills up * dubious backups, with no point-in-time recovery * extremely poorly written SQL queries * poory configured MySQL (in oh-so-many ways) The top three (at least) would lastly cause replication lag, which would eventually trigger an alert. ... And yet we never lost a cluster. (And we far a lot of them!) My team sweated blood improving processes and tooling, and then I spent a 6 month stint on database clusters (switching to GTID based replication and rewriting the ops config code so that they were all consistently configured and monitored). Occasionally we'd get a new senior hire insist that PostgreSQL was a necessity, so we'd stand back and let them produce a proof of concept that stood up to the types of failures our MySQL clusters dealt with regularly, without waking oncall up at night. And it was always a bit of a joke by comparison.
- lamroger 3y ago
  I haven't worked in big tech but I imagine their tooling is really good for this particular stack. Internal expertise too for running into big scale edge cases.
hnbad 3y ago
From having talked to Facebook engineers (I suspect Instagram/Threads is a very different beast for historical reasons), Facebook is only built on "PHP and MySQL" in the sense that the statement is technically correct but not helpful. The way Facebook apparently uses MySQL is not "MySQL + k/v store" but "MySQL is the k/v store". Forget SQL, they treat it as a key-value lookup and it's good enough for that. This also explains a lot of oddities coming out of FB engineering open source that at times look like they're trying to roll their own database on top of ... well, a database. I wouldn't go to Facebook for architecture advice unless you have both the scale of Facebook and the technological baggage of Facebook. I'm not sure how transferrable this writeup about Threads is either given that Threads is so strongly tied to Instagram (although EU legislation seems to have motivated Meta at least to no longer require users to delete their Instagram account in order to delete their Threads profile).
- phyrex 3y ago
  > look like they're trying to roll their own database on top of ... well, a database. That’s exactly what fb is doing. MySQL isn’t used as a k/v store per se but as the storage layer of the Tao graph database (which is also using memcache. So fb builds a database on top of 2 databases, technically). Source: https://engineering.fb.com/2013/06/25/core-infra/tao-the-power-of-the-graph/ https://engineering.fb.com/2013/06/25/core-infra/tao-the-pow...
- evanelias 3y ago
  This is not correct. Facebook has a large number of distinct MySQL tiers, which serve different purposes and have wildly different workloads. None of them are traditional "key-value lookup" workloads, not even the core user database (entities/associations); this k/v claim is often repeated here, but has never been an accurate representation of the architecture. The TAO graph access pattern absolutely requires range scans over indexes, which is not compatible with a k/v access pattern. source: I'm a former member of one of Facebook's MySQL teams, and among other things I worked heavily on the managed database-as-a-service tier. This was like an internal-only RDS, used by thousands of engineers for many different purposes, most of which involved SQL.
evanelias 3y ago
Facebook actually moved off of Cassandra well over a decade ago. Not sure about other Meta properties though like Instagram, but the original article here doesn't mention it anywhere.

neon_electro 3y ago

Can you share your take on why MySQL and not Postgres?

erulabs 3y ago
For me it's mostly about known-knowns. I've been responsible for pretty large/company-critical MySQL clusters and have managed to sleep soundly at night. I tend to be the "database guy" at every startup, and those startups tended to use MySQL. I have to say, I do love PostGIS. But for "this data absolutely cannot be lost or its game-over", I'd go MySQL every time. I can and have debugged MySQL replication on zero hours of sleep. It's entirely possible Postgres 16 is the better choice for new projects today, but I haven't had the time to try it out in production yet.
- brightball 3y ago
  > But for "this data absolutely cannot be lost or its game-over", I'd go MySQL every time. Man...times have seriously changed if that's the case. Good for MySQL to improve enough to get that recommendation.
- raffraffraff 3y ago
  It amazes me how much better MySQL has been in this regard for at least a decade, and it's also amazing that it's still not that well-known today. Back in 2015 I worked at a fast-growing unicorn that had badly implemented basically everything because they started with a tiny ops team of grads and developers. Very little was being monitored, there were only a handful of metrics being graphed (mostly network stuff in Cacti). Our DB issues were all caused by stupid stuff : * undetected hard disk in array fails * battery in array controller fails * disk fills up * dubious backups, with no point-in-time recovery * extremely poorly written SQL queries * poory configured MySQL (in oh-so-many ways) The top three (at least) would lastly cause replication lag, which would eventually trigger an alert. ... And yet we never lost a cluster. (And we far a lot of them!) My team sweated blood improving processes and tooling, and then I spent a 6 month stint on database clusters (switching to GTID based replication and rewriting the ops config code so that they were all consistently configured and monitored). Occasionally we'd get a new senior hire insist that PostgreSQL was a necessity, so we'd stand back and let them produce a proof of concept that stood up to the types of failures our MySQL clusters dealt with regularly, without waking oncall up at night. And it was always a bit of a joke by comparison.
lamroger 3y ago
I haven't worked in big tech but I imagine their tooling is really good for this particular stack. Internal expertise too for running into big scale edge cases.

hnbad 3y ago

From having talked to Facebook engineers (I suspect Instagram/Threads is a very different beast for historical reasons), Facebook is only built on "PHP and MySQL" in the sense that the statement is technically correct but not helpful. The way Facebook apparently uses MySQL is not "MySQL + k/v store" but "MySQL is the k/v store". Forget SQL, they treat it as a key-value lookup and it's good enough for that. This also explains a lot of oddities coming out of FB engineering open source that at times look like they're trying to roll their own database on top of ... well, a database. I wouldn't go to Facebook for architecture advice unless you have both the scale of Facebook and the technological baggage of Facebook. I'm not sure how transferrable this writeup about Threads is either given that Threads is so strongly tied to Instagram (although EU legislation seems to have motivated Meta at least to no longer require users to delete their Instagram account in order to delete their Threads profile).

phyrex 3y ago
> look like they're trying to roll their own database on top of ... well, a database. That’s exactly what fb is doing. MySQL isn’t used as a k/v store per se but as the storage layer of the Tao graph database (which is also using memcache. So fb builds a database on top of 2 databases, technically). Source: https://engineering.fb.com/2013/06/25/core-infra/tao-the-power-of-the-graph/ https://engineering.fb.com/2013/06/25/core-infra/tao-the-pow...
evanelias 3y ago
This is not correct. Facebook has a large number of distinct MySQL tiers, which serve different purposes and have wildly different workloads. None of them are traditional "key-value lookup" workloads, not even the core user database (entities/associations); this k/v claim is often repeated here, but has never been an accurate representation of the architecture. The TAO graph access pattern absolutely requires range scans over indexes, which is not compatible with a k/v access pattern. source: I'm a former member of one of Facebook's MySQL teams, and among other things I worked heavily on the managed database-as-a-service tier. This was like an internal-only RDS, used by thousands of engineers for many different purposes, most of which involved SQL.

evanelias 3y ago

Facebook actually moved off of Cassandra well over a decade ago. Not sure about other Meta properties though like Instagram, but the original article here doesn't mention it anywhere.