9 ms·
For the kind of stuff Uber stores, they may actually be doing it wrong (given what that Postgre mailing list post says) because that is one hell of an ugly use
by DiabloD3 10y ago
For the kind of stuff Uber stores, they may actually be doing it wrong (given what that Postgre mailing list post says) because that is one hell of an ugly use case for any DB.
I would have tried solving it by loading a dual E5v4 server full of 3TB and a slew of SSDs for L2ARC+ZIL under ZFS: more SSDs > bigger SSDs because the absolute worst case SSD performance that any and all SSDs suffer from is random reads (not writes) can slow to 200MB/sec even on those insanely fast "enterprise" PCI-E SSDs that do 2-4GB/sec continuous reads.
Given that, there are only four dbs worth using: Postgre, Oracle, DB2, and MS SQL. Unless you're doing something that is truly not suited for SQL (or SQL is insanely overkill), or you're Google and have a database in the hundreds or thousands of TB and literally have to write your own database (they went from almost inventing mapreduce, to going full circle back to full scale distributed RDBMS SQL with F1, which I wish they'd open source), any other database is just going to be a pain in the ass, buggy, and full of gotchas and undiscovered corner cases simply because it doesn't have over a decade of development and millions of users behind it.
Also, I have not included MySQL for obvious reasons: fun for toy SQL DBs where sqlite isn't a good fit, but not for enterprise use by any means. Uber switching to MySQL over Postgre is rather scary, I wouldn't want to be a Uber investor right now.
- nubela 10y agoYou wouldn't wanna be an Uber investor because choice of DBMS? As a programmer, I think this is the problem with developers thinking that technical problems are a bigger deal than it is.
- viraptor 10y agoMysql eats and corrupts data by design. For a company responding to real time events in physical world, that can be a big issue. I know they're trying to improve their defaults lately, but a lot of weird behaviour remains. And you don't have to be an expert DBA to know that choosing a technology known for silent data corruption is risky.
- deleted 10y ago[deleted]
- mixedCase 10y ago>Mysql eats and corrupts data by design. I'd like to have a source on that, would help shutdown a lot of MySQL discussions if true.
- matwood 10y agoYou can Google for silent truncation for a quick example. To be fair, MySql > 5.6 has fixed some of these issues and it also has some flags that can be set to help prevent them. The by design part is referring to early versions of mysql and discussions around it purposely did not care about ACID. Speed was the number one driver.
- dkersten 10y agoI was recently advised by a DB consultant whose area of expertise is MySQL that 5.7 is still too new and risky and that he would advise against upgrading for at least another 6 months or more. He feels that the releases come out much, much too unstable and unpolished and that it typically takes at least a year since release before he's comfortable running it in production. I don't know enough about MySQL to know if that's true or not. We are now investigating switching to MariaDB instead. (I'd personally love to move to Postgres, but that's not likely to happen any time soon)
- pizza234 10y agoI can confirm that the query optimizer introduced a rather serious bug (significantly suboptimal plan for queries involving low cardinality indices), which caused serious issues in our system. This, in addition to the fact that index merging has been broken in MySQL 5.6 for more than an year now (in some cases it will cause empty resultsets to be returned), and that it is still broken on MySQL 5.7
- 10y ago
- DiabloD3 10y agoUber relies entirely on their database. If the database is slow, or cannot be considered reliable under extreme load, or can lose data in ways that are hard to recover, then Uber can potentially lose a lot of money especially during peak hours. Yes, I'm aware there are systems for MySQL to handle failure, but I'm also aware of the systems for Postgre, and Postgre's failure handling seem to be far saner and easier to recover from. Defense in depth against failure is easier in Postgre from my experience. This would be like Elon Musk blogging about how "oh yeah, sometimes the brand new Tesla factory shuts down completely because sometimes the power goes out; but we're using really popular well known power distribution systems, so we're industry compliant, so everything is okay." If Elon Musk blogged that, HN would go apeshit, and rightfully so.
- sciurus 10y agoHere is an Uber engineer's talk about their worst outage ever. 16 hours of downtime for their API as they repeatedly try and fail to promote a new postgresql master and reparent slaves to it. https://www.youtube.com/watch?v=bNeZYVIfskc&t=26m54s https://www.youtube.com/watch?v=bNeZYVIfskc&t=26m54s https://surge.omniti.com/2015/images/presentations/MattRanney_Surge.pdf https://surge.omniti.com/2015/images/presentations/MattRanne... (Slides 39-63)
- GrinningFool 10y agoOn the other hand, it's an internal thing. If it is a poor choice in the end, they'll change it -- that's what investors have confidence in.
- jamespo 10y agoMaybe Facebook investors should bail out too by that reasoning?
- sneak 10y agoGoogle, too.
- devy 10y agoGoogle's hosted CloudSQL (a hosted &customized MySQL solution) had tons of connection drop issues. Last incident it took them 72+hrs to resolve and they had no idea what and why caused it (the resolution was a side effect from another issue that resolved for some other custom complains.)
- brianwawok 10y agoThat was 1st gen CloudSQL right? 2nd Gen CloudSQL seems much more similar to running your own MySQL server. (Both the pros and cons of that)
- devy 10y agoThose were 2nd Gen CloudSQL instances. :( We jumped onto the 2nd gen instances even when they were in beta.
- dijit 10y agoFacebook are using MySQL as an object store with no transactions, complex queries or even joins. I could be wrong but most databases are pretty competent with this style of workload.
- evanelias 10y agoThat isn't fully accurate. It's true that the large majority of FB's mysql fleet is dedicated to the main product data, essentially an object-graph store with a restricted access pattern. The queries are indeed relatively simple. I can't remember for certain but I think that simple joins and transactions are actually used there. However, the rest of the fleet supports an extremely diverse set of workloads -- keep in mind that MySQL is the primary data store of Facebook, and this includes product, ad serving, payments, async task persistence, internal tooling, many many other things. Countless different query patterns are in use. And although this is a minority of FB's mysql fleet, it's still many many thousands of machines supporting these other workloads, substantially larger than the vast majority of Postgres installations in the world. (Source: I worked on MySQL automation at FB, and was lead dev on their RDS-like DBaaS, used by a large portion of the company's engineering teams in one way or another.) I will readily admit that Postgres is a very good database, and definitely a better choice for OLAP workloads than MySQL (which has problems with complex queries) / InnoDB (problems with long-running read queries impacting old-row purge behavior). However, for extremely high-volume, large-scale OLTP workloads, MySQL/InnoDB is absolutely an excellent choice, imo better than Postgres for reasons of performance, ecosystem, and number experienced engineers who have worked at extreme scale.
- stonewhite 10y agoGithub works on MySQL and they got pretty good at it. Ubers engineers made a respectable decision by switching to a lower-hyped database purely on excellent engineering.
- dijit 10y agoMitigating against your database going mental is scary and should not be encouraged but it's the default in most MySQL shops. Whether you /can/ coerce it into doing the right thing is not the issue, Uber are doing their own thing regardless of if its right anyway so they're unlikely to change in that regard- what makes you sure they will do MySQL right at all?
- popmystack 10y ago>Uber switching to MySQL over Postgre is rather scary, I wouldn't want to be a Uber investor right now. That is extremely hyperbolic.
- vegabook 10y ago> Given that, there are only four dbs worth using: Postgre, Oracle, DB2, and MS SQL This is total bullshit. Even a modest size database (3-5 terabyte) for a small company like mine, chokes a relational database bigtime on time series. Especially the modern use cases where you are ingesting fast. You need Cassandra or Hbase which are very serious pieces of technology and are definitely "worth using", and will crush any of the four relational dinosaurs you mention in that space at comparable cost, and with embarrassingly better scalability. And by the way, that space is not some corner case, it is probably the major growth area of the next computing era.
- Roboprog 10y agoSuggested edit: s/This is total bullshit/Perhaps you overlooked something:/. Otherwise, this would have been a good comment, I think.
- Grishnakh 10y agoThis is one thing I like about HN's system: it lets me edit posts for a limited period. I'll sometimes bang out a post like the above, with a rather negative opening line like that and hit "submit". Then a minute later I'll realize I need to tone that down some, so I'll go edit it, as you suggest here. Unfortunately some shitty web forums don't allow you to edit posts at all, but I do appreciate ones which do allow editing stuff like this before anyone has a chance to read it.
- chc 10y agoHN also lets you set a delay before a comment appears, if you're like me and tend to catch mistakes more easily on the page than in the composition box. It's called "delay" on your settings page.
- jamespo 10y agoI think there's something to be said for the Linus style
- ansible 10y ago... or you're Google and have a database in the hundreds or thousands of TB and literally have to write your own database (they went from almost inventing mapreduce, to going full circle back to full scale distributed RDBMS SQL with F1, which I wish they'd open source), You may be interested in CockroachDB, which was inspired by Google's F1 and Spanner implementations. https://www.cockroachlabs.com/ https://www.cockroachlabs.com/ Another contender was FoundationDB, but Apple bought them and stopped selling it as a product.
- assface 10y ago> Another contender was FoundationDB, No, it was not. FoundationDB's SQL layer was a joke. The system was horribly slow. Every DBA that I talked to that tried it said that it did not deserve the hype that it got.
- SmellTheGlove 10y agoMaybe I've been in big companies for too long, but why not Teradata? Yeah it's expensive for the amount of storage Uber would need, but it's going to scale for them and perform well.
- spriggan3 10y ago> Also, I have not included MySQL for obvious reasons: fun for toy SQL DBs where sqlite isn't a good fit, but not for enterprise use by any means. Uber switching to MySQL over Postgre is rather scary, I wouldn't want to be a Uber investor right now. I've been running a few MySQL setups for a decade, why do you think it is not a good fit for enterprise apps? InnoDB solves most of the previous limitations. I also think MySQL is slightly easier to deploy/scale than Postgres. I believe Facebook is still running MySQL for instance.
- jsight 10y ago> Uber switching to MySQL over Postgre is rather scary, I wouldn't want to be a Uber investor right now. I guess you wouldn't have wanted to be a GOOG investor either? Most of their early revenue was from AdWords on MySQL.
- DiabloD3 10y agoI actually know engineers who work for Google for that. Google maintains their own MySQL fork internally and also know exactly how MySQL fucks up and has planned for MySQL failure in depth. Google is maybe the only company I trust to know what they're doing when it comes to databases.
- jayflux 10y agoWhat about facebook? As I'm sure I've read they've forked MySQL too
- Thaxll 10y agoFacebook is a massive MySQL user so you wouldn't be a Facebook investor as well?
- gpapilion 10y agoThis was my first thought, and generally what I always think when I hear about update heavy use cases. Typically they represent some issue in the object model, and often driven by an ORM layer. I always ask is there a need for absolute correctness, and would this be better handled by a batch processing method. I'm not sure any hardware would have solved their issue though. It may not have actually been disk speed, but issues in the mvcc design itself.
- otterley 10y agoYou realize that Google, Facebook, Yahoo, and many, many other shops have successfully run on MySQL for well over a decade, right? You can armchair engineer all you like, but at the end of the day, the proof of the engineering is in the working. And if you have Uber stock you'd like to sell just because you think they're making a questionable engineering decision around databases (despite all the real world evidence that it is adequate to serve large scale businesses), I know a few investors who would gladly take it off your hands.
- dberhane 10y agoIt is like saying I would not invest in Facebook as it is using PHP.