10 ms·
Call Me Maybe: MariaDB Galera Cluster
- no1youknowz 11y agoI flirted with MariaDB Galera Cluster. This was on Amazon EC2. Frequently mariadb would just go down, not the server mind you. Frequently the whole cluster would go down as well. To bring it back up, I'd have to bootstrap the whole cluster. Essentially, what a pain in the backside to use. This did not fill me with confidence and thankfully did not go into production and later on went with Postgres/CitusDB. The difference is day and night!
- ZeWaren 11y agoHas a similar experience a year ago. The only difference being that I did go into production. Man, what a mistake.
- shaftoe 11y agoWe've been using PXC, which is still Galera but a Percona variant of MySQL instead. No problems at all. I've been strongly considering MariaDB on Galera for encryption-at-rest. Is there something about MariaDB that was not working well with Galera?
- eeeeeeeeeeeee 11y agoI had similar issues, but this was a few years ago. It was not something I was comfortable deploying in any production environment because it fell apart so many times in dev. Luckily I caught these issues before it was used in production.
- ohitsdom 11y agoI wish these "Call Me Maybe" blog titles would die already...
- DannoHung 11y agoIt's a series on testing durability claims in distributed computing workloads. It's a good name.
- mistaken 11y agoI wouldn't have guessed that from the title. What I got was something security related to MariaDB. And that's because I got used to the funny gifs and the `hacking is cool` mentality of the security industry...
- kasey_junk 11y agoIt's a fairly long running and well known series. The first one was published a couple of years ago and was themed around the Carly Rae Jepsen song titled "Call Me Maybe" that was popular at the time. From that a testing harness named Jepsen was also developed.
- mistaken 11y agoSure, no problem with that. I just don't understand why we can't call e.g. a buffer overflow in software X a buffer overflow in software X instead of "Adventures in Memoryland!!! The bug, that 0wnzors your boxorz!!".
- yid 11y ago> I just don't understand why we can't call e.g. a buffer overflow in software X a buffer overflow in software X instead of "Adventures in Memoryland!!! The bug, that 0wnzors your boxorz!!". Since you appear to be sincere, it's because sometimes a lot of people like to interact in human ways, which includes a bit of storytelling to relieve the monotony of just the facts.
- helper 11y agoWe? Who is we? Aphyr can call it whatever he wants.
- teraflop 11y agoI don't find it surprising that so many distributed databases fail at ensuring consistency; it's a very hard problem. What really gets me is that they deliberately, knowingly sacrifice consistency for performance, and then claim the opposite in their documentation/marketing. See https://github.com/codership/galera/issues/336#issuecomment-136635018 https://github.com/codership/galera/issues/336#issuecomment-... (linked from the article)
- chralieboy 11y agoThat's really been the most enlightening part of the series for me. Reinforces that the hardest part in engineering is rarely the technical problem. Distributed databases are really f*ing hard, but infinitely harder if the people can't work together or don't open up to faults.
- bbrazil 11y agoFrom experience I think some of the problem is that not everyone appreciates the importance of correctness in this sort of system. At the very least you should clearly documenting your expected failure modes, so that it's possible to build correct things on top of it. In part it's hard to convince everyone of the importance of spending time on what looks like an unlikely corner case until after the outage - when it also becomes much harder to fix as you usually need to build this into the core design of your system.
- acconsta 11y ago>I don't find it surprising that so many distributed databases fail at ensuring consistency; it's a very hard problem It's not hard — it's slow. Sending reads and writes through Paxos or Raft gives you sequential consistency. But not surprisingly, touching a quorum of nodes for every operation is too slow to be practical for many workloads. And that's usually fine — most data aren't bank account balances.
- simoncion 11y agoThough, it's not even remotely okay if your documentation and marketing material indicates that consistency isn't being sacrificed for performance. :) I suspect that no experienced programmer expects to get data safety for free. We do expect that the documentation doesn't lie to us when describing the level of data safety provided.
- steveklabnik 11y agoI wish more companies supported people like Stripe does for Aphyr. Research is so valuable, but there's so little incentive to be in academia. There's a certain irony that undergrad left me with so much debt I couldn't really consider joining a research university...
- yid 11y ago> I wish more companies supported people like Stripe does for Aphyr. Research is so valuable, but there's so little incentive to be in academia. If you're talking about incentives, don't forget that Kyle's research is some of the most commercially useful research out there at the moment, and Kyle is also skilled at implementing his ideas (a rare combination among researchers). I'm actually quite pleasantly surprised that he's putting so much effort into these blog posts rather than writing a string of journal papers.
- sanderjd 11y agoI know zero about academia – could he also publish this work in a string of journal papers and reap the rewards (whatever they are) of both a successful and widely useful blog series and more formal academic recognition?
- yid 11y agoAbsolutely. But I doubt he'd find as much value for his time by writing journal papers at the moment, given that Stripe will essentially pay him a decent salary to continue his open-access research. There's always time for a retrospective book :)
- panic 11y agoIs there a point to writing journal papers if you're not playing the academic publication / citation game? Web sites are easier to publish and have wider distribution than academic journals.
- Jweb_Guru 11y ago
- gregwebs 11y agoWhat about AWS Aurora? I have not used it, but I definitely would if I was using MySQL.
- edutechnion 11y agoAWS Aurora seems to go at the problem differently. They spread the disk out over three AZs in 10GB chunks. So if the instance powering the Aurora db engine crashes, a new one spins up quickly and begins pulling from the same disks. https://aws.amazon.com/rds/aurora/faqs/ https://aws.amazon.com/rds/aurora/faqs/
- samstave 11y agoI predict in 5 years... this will be the standard. Its not so much about wanting to bolt onto amazon's eco for stuff -- but just allowing an org to focus. AWS allows for an ops team to completely have no concern for hardware. Lovely. I also don't want an ops team doing DB cluster mgmt. I want to deploy and delve into my data. I want my data/eng/dev/ops teams to be pummeling the shit out of my data without concern for the instances/hw/cluster. aurora adds to the cloud fabric. One thing amazon is moving to is to define compute capacity vs defining instances at all. Their vision of abstraction is amazing.
- ranman 11y agoI certainly think this will be the trend for startups but I believe it will take much longer for established companies to get out of the buying hardware business. Often times running locally can be drastically less costly than running on Amazon -- especially if you already have the staff dedicated to that sort of role.
- samstave 11y agoI agree -- but I am curious to know the diff in opex/capex when you dont need to have staff to handle said HW/Roles etc... I am sure this has been looked at by people -- anyone know of a report on it?
- mysql_cass_dba 11y agoThanks to aphyr for the testing. Certainly it will help focus efforts on improving things. I am a MariaDB contributor and operate Percona Cluster in production, so I can talk a bit about Galera. It's recommended that writes go to one master, rather than be distributed across the nodes. That will help with isolation issues. Also, some commenters have complained about year-old releases. PXC has improved significantly in the past year regarding manageability, so you may want to try again. For example, the startup script has a bootstrap option now. For most people, vanilla async MySQL replication works best, esp. 5.6. But Galera gives you another option when you need something else. Having said that, it takes 5-10 years for a database or filesystem to mature, so anybody using Galera now is an early adopter.
- Erwin 11y agoDespite the reliability, that product sounds nice -- is there anything for Postgres with similar ease of setup for replication etc. that someone can recommend? E.g. EnterpriseDB ?
- bsaul 11y agoEverytime i read one of those post (or other about mongodb failures), i keep thinking about this talk http://youtu.be/4fFDFbi3toc http://youtu.be/4fFDFbi3toc And why noone tried to repeat their strategies for building a robust db system : start by building an extremely robust failure simulation and testing facility. Then build your product. Actually, i think what those guys at foundationdb did was so exceptional, that by buying the company and killing the product, Apple harmed the software industry for the next 10 years. The fact the foundationdb is mentionned in OP as the only distributed db system one could recommend makes me more confident making that statement.
- lostcolony 11y agoOnly distributed SQL store. You want non-distributed SQL, there are options, you want distributed NoSQL, there are options.
- rspeer 11y agoAphyr's been testing more than SQL stores. "NoSQL" is not a magic bullet: concurrency is still hard when you skip the SQL.
- lostcolony 11y agoI know. Sorry, I should have been clearer; what I meant was Aphyr has had some fine results for SQL DBs, provided you avoided distribution. And he has had some fine results for NoSQL DBs, when distributed, provided you preferred availability to consistency (generally in certain scenarios, or with certain caveats). What he has not had, and so presumably was lamenting, was any distributed DB that offered transactions that passed any sort of muster, except for FoundationDB, which was made defunct. That, or any DBs he could recommend that met every claim their documentation made (as that's largely what he's testing for). I meant that with this being the first distributed SQL store, he was saying there was nothing he could recommend that actually offered similar guarantees to what this was claiming. That is, ACID style (well, technically --ID I believe) DB transactions in a distributed environment. FoundationDB did (though it was NoSQL), hence his mentioning it. But that's different than there is nothing he can recommend, at large; he can't recommend similar solutions, but for a given problem he could likely recommend a compelling, different, solution.
- StavrosK 11y agoIs there any data store that has come out of these tests looking good? Every Aphyr post I've read had a pretty big failure at some point, even for datastores I considered solid, like Riak or Cassandra.
- iyn 11y agoThat's the questions that I ask myself every time I see "Call me maybe" posts. Is there a good solution?
- tedunangst 11y agoI think the takeaway, about a dozen posts back, was that your clustered database is shit and you shouldn't rely on it. Sharding at the client level may be the best approach. Sometimes the old ways are best.
- mjb 11y agoThose are fundamentally different things. Sharding is a scaling strategy for reads, writes and data size. It doesn't address durability (which many distributed databases do). It doesn't address the discoverability problem of how you know which piece of data is where (easy if you have a well-distributed natural key, potentially difficult if you have a poorly distributed natural key, potentially obscenely difficult if you don't have a natural key). It doesn't address data locality (it can, but then you need to migrate keys, and end up with a second-order 'which key is where?' problem). Most crucially, it doesn't address what you need to do if you have to do transactions (of any kind) across shards. You can try invent something to do that. You may get it right (probably using consensus or 2PC), or you may get it fundamentally wrong. Either way, it's a tricky problem. Sharding is good, for sure. It's not a replacement for this kind of technology. Aphyr is doing great work keeping vendors honest, and making sure they live up to their marketing. I don't think any of this says that clustered databases are inherently shit.
- tedunangst 11y agoThat's fair. I may have been a little cavalier with the use of the word shit. Though I note auto sharding is a headline feature for many systems tested. And as for durability, many seem to fall short of rsync running out of cron.
- cpr 11y agoIt's no mystery that the kinds of systems with massive reach--Google search, Facebook, Twitter--are really not data-consistency-critical applications. Who'll ever know if a search result isn't perfectly up to date or perfectly accurate? Who'll ever know if you missed a Facebook feed entry because it "wasn't relevant" or simply wasn't seen due to DB vagaries? And who'll ever know about a few tweets going astray here or there? In all cases, they're all likely "eventually consistent" (or close to it), but it's no accident that it doesn't ultimately matter in those massive scale examples. And maybe that's the secret to massive scale--it can't ultimately matter.
- vidarh 11y agoOne of the startups I worked on was a classifieds aggregation engine pulling data from external feeds. We basically queued all retrieved items for processing with no attempts at avoiding data loss whatsoever - including using in memory queues for lots of things. Our reasoning was that if a machine crashed, worst case was that a few listings would take up to 24 hours to update, but generally much less (we adapted crawling rate for our sources based on change frequency; so large sources of listings would get re-indexed far more frequently, so if a feed didn't update or 24 hours it'd be because it wasn't a source of much data), and we could force refreshes of the data. Some people were horrified at the approach because the idea of ensuring consistency and not losing data is so ingrained. But the reality is that you need to measure the cost of consistency up against the value it provides. And often it's not very valuable, especially when there is an authoritative source of the data to recover from and when the data will be outdated quickly anyway. A lot of the time any notion of consistency is an illusion anyway - by the time the page has returned, the results are outdated - and what matters is maintaining the illusion (e.g. ensure that if a user makes an update it's reflected in the page that's returned). The key is you need to know the tradeoffs and apply them consciously rather than get caught out by tradeoffs components you rely on makes without telling you..
- baudehlo 11y agoAbsolutely. I wrote a system to provide real time signature updates to Symantec's (then MessageLabs) global anti spam infrastructure. It used UDP to send them. It worked amazingly well and distributed signatures across thousands of servers worldwide in milliseconds and if a packet got dropped oh well. It's about choosing the right tool for the job. Game servers use the same technique.
- Karunamon 11y agoCan I point out that I'm utterly impressed and astounded by Kyle's ability to remain detached and professional in the faces of what appear to be outright falsehoods stated by some of these companies? If I had gone to his lengths to critically evaluate the safety of a database system, and then it comes out that the marketing materials or the words of the developers were.. significantly misleading at best, my first response is likely to be a profanity laden rant, not a cool recounting of how and why they're wrong.