6 ms·

I think its worth mentioning again that what Uber ended up using has no resemblence to an RDBMS (single table, manual indexes). So regardless of whether their

I think its worth mentioning again that what Uber ended up using has no resemblence to an RDBMS (single table, manual indexes). So regardless of whether their complaints are justified or not, it should not be taken as an endorsment of mySQL over postgres, but rather of an endorsement of NoSQL over RDBMS . Which is really just what every company at these scales do (except for google and f5 if whitepapers are to be considered).

leftnode 10y ago
Can you expand on what you mean by a single table with manual indexes? How does a frequently used table with indexes not resemble an RDBMS?
- Illniyar 10y ago
  the secondary indexes are saved in a different table and (I can only assume) the application layer (or one layer above mySql) is responsible for keeping that table up to date. If you use only 1 table, then it's by no means "relational", so I don't see why you'll need a database system designed from start to finish to support a relational model.
  - pas 10y ago
    The "relationalness" is/was not really important here. It's all about MVCC and how storage engines handle it. Postgres is lacking in these scenarios, whereas a particular fine tuned version (or fork) of InnoDB (or MyRocks or whatever they end up chosing) handles this better. See Facebook's "mysql-5.6" branch, that has hundreds of patches piled on to support especially these taxing workloads.
    - Illniyar 10y ago
      Well, for that matter, as I understood it, they also have no transactions nor atomicity and are basically eventually consistant , though these I'm simply inferring from their posts, it was not stated as far as I can remember. So MVCC also has almost zero bearing on what they are doing.
      - spotman 10y ago
        Just because you may not always use the database with the BEGIN/COMMIT semantics, that absolutely does not mean that "MVCC has almost no bearing on what they are doing". MVCC and how it has been implemented in the two storage engines in discussion here absolutely has a lot to do with it.
  - laszlokorte 10y ago
    > If you use only 1 table, then it's by no means "relational" [...] FYI The term "relational" in "relational database" does not refer to the relation between tables but to the mathematical concept of a relation which is some set of tuples ie a single table.
    - Illniyar 10y ago
      Well, the relational model does not describe or contain a notion of table really, so I guess you could model it in any way you want, even within a single table. But I think that's a mute point, I doubt they kept any part of the relational model inside their schemaless database.
      - slavik81 10y ago
        Probably just an auto-correct mistake, but you mean "that's a moot point".
        hinkley 10y ago
        Autocorrect, don't take it for granite.
  - deleted 10y ago
    [deleted]
fauigerzigerk 10y ago
If it was just about NoSQL vs RDBMS they could have used Postgres as a key-value store instead of mySQL. So yes they absolutely did endorse mySQL over Postgres. Also, what is the scale of Uber that you think is unsuitable for RDBMS? Uber does a million rides per day but these rides are very predictably local. I don't know enough to make any firm claims here but at first sight this doesn't look like the sort of scale that is unachievable on an RDBMS in principle.
- Illniyar 10y ago
  "Also, what is the scale of Uber that you think is unsuitable for RDBMS" Never said it's unsuitable, I have no idea. Just said the move from relational to NoSQL is what most billion dollar companies with mass-market apps or websites do. Many of them also do sharding on normal RDBMS.
- marcosdumay 10y ago
  Well, Uber claims they are using MySQL because it's a better NoSQL database than Postgres. From the context, I don't think they ever evaluated Postgres NoSQL features. But I don't think they would get a different conclusion if they did. MySQL trades some consistency guarantees for speed, and it looks like Uber doesn't need those extra guarantees anyway.
- duaneb 10y ago
  They endorsed MySQL over PostgreSQL for a type of structure and access very few people use. I don't see how the query language plays into the recommendation at all.
mranney 10y ago
Uber runs a lot of databases in production. We stopped using Postgres a while ago for new applications that talk SQL, and this article explains some of the reasoning. We still have many applications that talk directly to MySQL, and we still have our original API monolith that talks directly to Postgres. All new applications are being built using distributed databases like our in-house Schemaless system which happens to be backed by MySQL, and we also have Riak and Cassandra in production.
- andy_ppp 10y ago
  Will Schemaless ever be released open source? It sounds great...
- im_down_w_otp 10y ago
  I would love to see a similarly detailed write-up on your guys' transition from Riak to Cassandra. Especially considering you would have been really, really deep into the details and guts of operating Riak at the red-line from your time at Voxer.
- amaks 10y ago
  Does Uber still use Python for backend services?
threeseed 10y ago
> Which is really just what every company at these scales do No they aren't. You need to understand that databases are being used in a whole range of different ways than you think. Batch/streaming analytics are typically done with HDFS, Cassandra, HBase, MongoDB etc. Event aggregation often with a time series database like Druid, InfluxDB etc. Web API serving layers are still the domain of lightweight SQL databases like MySQL, PostgreSQL. Customer data warehouses are still dominated by Teradata, Oracle, SAP etc. What you misconstrue as a single platform is often a multi headed beast with various architectural components. Each comprising their own technology stacks.
- ahachete 10y ago
  While I mostly agree with your comment, some NoSQL databases are not suitable for analytics. Indeed, they are terrible at them. Unstructured information is, surprise, unstructured, which means it is harder to query / analyze, since the structure needs to be fully scanned and parsed to perform the computation. I have seen many cases where MongoDB is used for this and suffers from really bad performance.
  - brianwawok 10y ago
    I think the trap is simple POCS. Do a simple POC where you ram 1 million rows in a MongoDB and then make a webapp to do some basic analytics. Look, it works and get responses within a second. Cool! Then real data comes in at 12 billion rows and your analytics take 3 hours to run. So you try to do the sharding thing, and realize that it works for a while, except now every analytics query needs to hit every shard....
    - AndyNemmity 10y ago
      Absolutely true, and the scale keeps getting larger > 12 billion rows, and comes up with even harder challenges. Worked on a 1.3 trillion row (for 1 table, others varying below this) database this year for predictive analytics, and it was mind boggling the hoops to jump through to try to get that thing to run anything in a manageable time frame. Any POC level show of a database would be about meaningless at that scale. You have to do a real test to view the challenges around it.
kyrra 10y ago
For scalable SQL products, I'd consider Google's Spanner to be in that boat. There is actually a class of products trying to do this under the name of NewSQL[0]. CockroachDB seems to be mimicking spanner's design principles and deliver a similar product[1]. [0] https://en.wikipedia.org/wiki/NewSQL https://en.wikipedia.org/wiki/NewSQL [1] https://github.com/cockroachdb/cockroach#design https://github.com/cockroachdb/cockroach#design

leftnode 10y ago

Can you expand on what you mean by a single table with manual indexes? How does a frequently used table with indexes not resemble an RDBMS?

Illniyar 10y ago
the secondary indexes are saved in a different table and (I can only assume) the application layer (or one layer above mySql) is responsible for keeping that table up to date. If you use only 1 table, then it's by no means "relational", so I don't see why you'll need a database system designed from start to finish to support a relational model.
- pas 10y ago
  The "relationalness" is/was not really important here. It's all about MVCC and how storage engines handle it. Postgres is lacking in these scenarios, whereas a particular fine tuned version (or fork) of InnoDB (or MyRocks or whatever they end up chosing) handles this better. See Facebook's "mysql-5.6" branch, that has hundreds of patches piled on to support especially these taxing workloads.
  - Illniyar 10y ago
    Well, for that matter, as I understood it, they also have no transactions nor atomicity and are basically eventually consistant , though these I'm simply inferring from their posts, it was not stated as far as I can remember. So MVCC also has almost zero bearing on what they are doing.
    - spotman 10y ago
      Just because you may not always use the database with the BEGIN/COMMIT semantics, that absolutely does not mean that "MVCC has almost no bearing on what they are doing". MVCC and how it has been implemented in the two storage engines in discussion here absolutely has a lot to do with it.
- laszlokorte 10y ago
  > If you use only 1 table, then it's by no means "relational" [...] FYI The term "relational" in "relational database" does not refer to the relation between tables but to the mathematical concept of a relation which is some set of tuples ie a single table.
  - Illniyar 10y ago
    Well, the relational model does not describe or contain a notion of table really, so I guess you could model it in any way you want, even within a single table. But I think that's a mute point, I doubt they kept any part of the relational model inside their schemaless database.
    - slavik81 10y ago
      Probably just an auto-correct mistake, but you mean "that's a moot point".
      - hinkley 10y ago
        Autocorrect, don't take it for granite.
- deleted 10y ago
  [deleted]

fauigerzigerk 10y ago

If it was just about NoSQL vs RDBMS they could have used Postgres as a key-value store instead of mySQL. So yes they absolutely did endorse mySQL over Postgres. Also, what is the scale of Uber that you think is unsuitable for RDBMS? Uber does a million rides per day but these rides are very predictably local. I don't know enough to make any firm claims here but at first sight this doesn't look like the sort of scale that is unachievable on an RDBMS in principle.

Illniyar 10y ago
"Also, what is the scale of Uber that you think is unsuitable for RDBMS" Never said it's unsuitable, I have no idea. Just said the move from relational to NoSQL is what most billion dollar companies with mass-market apps or websites do. Many of them also do sharding on normal RDBMS.
marcosdumay 10y ago
Well, Uber claims they are using MySQL because it's a better NoSQL database than Postgres. From the context, I don't think they ever evaluated Postgres NoSQL features. But I don't think they would get a different conclusion if they did. MySQL trades some consistency guarantees for speed, and it looks like Uber doesn't need those extra guarantees anyway.
duaneb 10y ago
They endorsed MySQL over PostgreSQL for a type of structure and access very few people use. I don't see how the query language plays into the recommendation at all.

mranney 10y ago

Uber runs a lot of databases in production. We stopped using Postgres a while ago for new applications that talk SQL, and this article explains some of the reasoning. We still have many applications that talk directly to MySQL, and we still have our original API monolith that talks directly to Postgres. All new applications are being built using distributed databases like our in-house Schemaless system which happens to be backed by MySQL, and we also have Riak and Cassandra in production.

andy_ppp 10y ago
Will Schemaless ever be released open source? It sounds great...
im_down_w_otp 10y ago
I would love to see a similarly detailed write-up on your guys' transition from Riak to Cassandra. Especially considering you would have been really, really deep into the details and guts of operating Riak at the red-line from your time at Voxer.
amaks 10y ago
Does Uber still use Python for backend services?

threeseed 10y ago

> Which is really just what every company at these scales do No they aren't. You need to understand that databases are being used in a whole range of different ways than you think. Batch/streaming analytics are typically done with HDFS, Cassandra, HBase, MongoDB etc. Event aggregation often with a time series database like Druid, InfluxDB etc. Web API serving layers are still the domain of lightweight SQL databases like MySQL, PostgreSQL. Customer data warehouses are still dominated by Teradata, Oracle, SAP etc. What you misconstrue as a single platform is often a multi headed beast with various architectural components. Each comprising their own technology stacks.

ahachete 10y ago
While I mostly agree with your comment, some NoSQL databases are not suitable for analytics. Indeed, they are terrible at them. Unstructured information is, surprise, unstructured, which means it is harder to query / analyze, since the structure needs to be fully scanned and parsed to perform the computation. I have seen many cases where MongoDB is used for this and suffers from really bad performance.
- brianwawok 10y ago
  I think the trap is simple POCS. Do a simple POC where you ram 1 million rows in a MongoDB and then make a webapp to do some basic analytics. Look, it works and get responses within a second. Cool! Then real data comes in at 12 billion rows and your analytics take 3 hours to run. So you try to do the sharding thing, and realize that it works for a while, except now every analytics query needs to hit every shard....
  - AndyNemmity 10y ago
    Absolutely true, and the scale keeps getting larger > 12 billion rows, and comes up with even harder challenges. Worked on a 1.3 trillion row (for 1 table, others varying below this) database this year for predictive analytics, and it was mind boggling the hoops to jump through to try to get that thing to run anything in a manageable time frame. Any POC level show of a database would be about meaningless at that scale. You have to do a real test to view the challenges around it.

kyrra 10y ago

For scalable SQL products, I'd consider Google's Spanner to be in that boat. There is actually a class of products trying to do this under the name of NewSQL[0]. CockroachDB seems to be mimicking spanner's design principles and deliver a similar product[1]. [0] https://en.wikipedia.org/wiki/NewSQL https://en.wikipedia.org/wiki/NewSQL [1] https://github.com/cockroachdb/cockroach#design https://github.com/cockroachdb/cockroach#design