3 ms·
I guess we should train the new generation better so they don't end up as Uber so... tip for any company who gets incredibly successfull: The first thing one h
by user5994461 10y ago
I guess we should train the new generation better so they don't end up as Uber so... tip for any company who gets incredibly successfull:
The first thing one has to do is to drop SQL databases as the main data source.
The usual choice is to move to Cassandra. It does have build-in sharding AND backup AND multi-master replication AND multi datacenter support AND performances scale linearly with the number of servers.
The [only] other option is ElasticSearch (which has slightly different properties regarding data format and data consistence).
You don't need to craft complex custom sharded distributed WTF software to abstract hundreds of cluster of hundreds of databases. Use the right tool for the job, that has that build in.
- pg314 10y agoDropping the RDBM and replacing it with something like Cassandra is fashionable, but sounds like bad advice to me. A quick google gives these stats for Uber[1]: - 8 million users - 160 thousand drivers - 1 million daily rides - 2 billion rides so far The biggest data, as stated in [2], is the trip info. They are storing the trip info as a 20kB JSON blob. If you add a custom data type [3] for a trip to PostgreSQL and encode it efficiently (binary, using deltas), you should be able to do much better than the ~3kB they get by using messagepack and zlib, say 1kB. All that data should easily fit into 4TB. At one million rides per day, one reasonably beefy RDBM server should be able to easily handle that. You can partition the trips table and move the trips that are older than say, 1 month, to a slower disk to save money on disks. That helps with schema migration too: new data uses the new schema, and you use a view on the old data to adapt to the new schema. I very much doubt you need anything much fancier than that. It would help if people really learned their tools (in this case PostgreSQL) instead of jumping to a different technology at the first problem they run into. [1] http://expandedramblings.com/index.php/uber-statistics/ http://expandedramblings.com/index.php/uber-statistics/ [2] https://eng.uber.com/trip-data-squeeze/ https://eng.uber.com/trip-data-squeeze/ [3] https://www.postgresql.org/docs/current/static/xtypes.html https://www.postgresql.org/docs/current/static/xtypes.html
- user5994461 10y agoAs a rule of thumbs, putting 5TB of data -that are growing exponentially- in a single box is always a terrible idea. You're gonna hit all kind of limits with the RDBMS software and the special hardware that will be required. Cassandra will handle the sharding automatically, it will have multiple instances with always one available for your applications, it will handle replications across datacenters all around the world, it will have dependable performances that can scale linearly with the specs you give it, it will allow you to do maintenance while online, it will let you add remove and refresh nodes. A bigger RDBMS has none of these qualities. It's a one trick pony that will die when either the hardware or the software will reach a limit, then your whole site will be down. Even if you know what needs to be done to avoid the disaster, you can't do it because the RDBMS is a SPOF and every maintenance you perform is called "downtime". Short term = Postgre/MySQL because it's easier and it gets the job done. Long term = Cassandra because it's dependable.
- pg314 10y ago> As a rule of thumbs, putting 5TB of data -that are growing exponentially- in a single box is always a terrible idea. Storage capacity is also growing exponentially: Samsung is shipping a 15TB SSD (albeit at $10000), Seagate has previewed a 60TB SSD. > You're gonna hit all kind of limits with the RDBMS software and the special hardware that will be required. The limits of RDBMs are well understood. You don't need any fancy hardware for this use case. A couple of Xeons, as much RAM as you can afford and a RAID of SSDs (or possible spinning disks, it doesn't look like Uber is doing anything too fancy). SPOF: you have slave replicas running that can take over if something goes wrong with the master. You don't have to take down an RDBM to do maintenance. DDL statements are transactional in PostgreSQL. The automatic sharding of Cassandra is nice, when you need it. Of course, Uber's use case seems like it lends itself to easy geographic sharding when you're using an RDBM, if needed. In the end, I'd rather deal with a mature well-understood technology like an RDBM compared to a 5 year old technology like Cassandra (release 1.0 in 2011). You obviously prefer the opposite. To each its own.
- user5994461 10y ago