3 ms·
I think we did "sharding" with relational databases... Back in the 80s. Then we got fast hash joins and partitioning and it turns out that the disadvantages of
by gaius 14y ago
I think we did "sharding" with relational databases... Back in the 80s. Then we got fast hash joins and partitioning and it turns out that the disadvantages of sharding just aren't worth it. The NoSQL crew will figure this out too around 2030 :-p
- taligent 14y agoI guess so will the likes of Facebook, Google, Twitter, eBay etc. Must be amazing to know more about databases than all of the world's biggest companies put together.
- gaius 14y agoWhy? The guys at FB or Google are just guys like you and me, don't believe the hype, they are not superheroes or gurus despite their much talked about interview process (the Google interviews I did were a walk in the park compared to GS, btw). Some will know more than me sure, but some'll know less.
- taligent 14y agoThose were only a few examples. I can keep listing companies that use sharding. The idea that it is some 1980's approach to scaling databases is laughable.
- PommeDeTerre 14y agoYou're right, it isn't a 1980s approach. I recall hearing about it in the mid 1970s, and even then it was probably originally devised in the 1960s.
- jeremyjh 14y agoJust guys like you or me except they build systems with billions of page views a month.
- gaius 14y agoAt my last job, we averaged 60k pages/sec, on what the NoSQL guys would call a "legacy" database, and there was plenty of headroom. It's not rocket science, just engineering. Companies like Google love to weave a mystique around what they do, it's in their interests after all, to convince their investors that what they do is magic. But I'm from back in the day when we were the same way about millions of pages per month, then per week, then per day...
- deleted 14y ago[deleted]
- deleted 14y ago[deleted]
- einhverfr 14y agoOn good hardware there were PostgreSQL instances running a billion queries a day back in the 9.0 days. Often these are actual accounting apps where reporting matters and so NoSQL would be a very poor fit. For example the French government uses it to distribute welfare program dispersements. The Wisconsin Courts also uses PostgreSQL at loads in the billion per day range. I know there are larger instances out there. Now a days we are talking about thousands of concurrent users and up to 350k reads/sec on high end machines.
- gaius 14y agoYes there is a lurking iceberg of things that aren't in the public eye like Facebook or Google, and that's where the really intense and interesting stuff is happening. I wonder what the guys at Visa make of all the hype around these websites, when they were doing these volumes all along... At said 60k pages/sec job, you know who we looked up to for databases? Starbucks. Walmart. McDonalds. 'Cos we'd seen what they do, and anyone who thinks Twitter is impressive, well it'd blow their minds.
- einhverfr 14y ago
- einhverfr 14y agoBe a little careful with this level of disdain. Sharding may have been available in the 1980's, but what it lead to in some products is quite amazing. Consider Teradata's clustering ability which is sort of like sharding your database but without the disadvantages typically associated with it. Postgres-XC now offers something similar as a semi-fork of PostgreSQL. Basically what we are talking about here is a two-tier database layer where storage and coordination are separated, and two phase commit is used between these two. Thus the coordination tier can enforce referential integrity between storage nodes if necessary and thus allow write-extensibility. This isn't something without uses. For high-end, high-write-load databases, serving very large amounts of traffic (think airline reservations), this has been a typical approach for quite some time. The fundamental problem though is that once you give up on local consistency over a given domain, you cannot have any guarantee of global consistency. The current relational approaches (Postgres-XC and Teradata) both enforce ACID compliance. BASE doesn't offer any consistency guarantee and therefore it is only good for throw-away data.
- gaius 14y agoOh absolutely, but what you're talking about there, people do with CICS today, and that's even older than the 80s. CICS is a technology I have a lot of respect for. But my point is - when I need to use something like that, I know that's what I'm using. I don't imagine that it's some new invention. Hell, a lot of what the NoSQL guys think they're inventing, IBM did back then too - IMS.
- einhverfr 14y ago"Those who don't understand Unix are condemned to reinvent it, poorly." – Henry Spencer