10 ms·
Google Spanner's Most Surprising Revelation: NoSQL is Out and NewSQL is In
- nolok 14y agoNewSQL ? Seriously ? Do we really need another low-quality buzzword for people to re-use everywhere ?
- mwexler 14y agoI don't know, I think it's helpful. It changes the conversation from being about the access language to being focused on how a technology processes data at scale. Well, almost... Notice that even with the shift from "NoSQL" to "NewSQL", mentions of joins continues to be conspicuously absent from many of these discussions. So, it's worth noting that many of these NewSQL things are "almost but not quite SQL", hence the value of a new word.
- grandalf 14y agoIt's as if someone just implemented a crude parser to map SELECT statements into a query for whatever the underlyig nosql system happens to be.
- bcoates 14y agoI don't think Spanner is one of these fake-relational databases, it's an ACID datastore that's used as underlying storage for the real RDBMS F1. From the F1 paper (http://research.google.com/pubs/pub38125.html http://research.google.com/pubs/pub38125.html) It looks like it's mostly intended as an improvement to sharded MySQL, by putting the sharding where it belongs, down at the physical storage level, instead of up at the client access level. Maybe a better buzzword would be NoMySQL (NOracle?)
- DanWaterworth 14y agoI think there should be a term for these new DBMSs, but the problem with NewSQL is that the thing they have in common is not SQL, but consistency and transactions.
- ChuckMcM 14y agoSo your saying its not ACID its more like Ecstasy?
- DanWaterworth 14y agoEcstasy might work: Eternally Consistent State, Transactions And Scale, Yes please.
- charlieflowers 14y agoVery well done, sir.
- akldfgj 14y agoIt is ACID. It is a DBMS. It's not a Relational DBMS.
- ChuckMcM 14y agoDude its a pun :-)
- nostrademons 14y agoWe just need to name a drug "DBMS" (preferably something that makes you feel emotionally close, hence "relational DBMS") to complete the pun.
- jbigelow76 14y agoI didn't really like the NewSQL thing either. I initially thought it must have meant transactional DBs with some first class schema-less features like Postgres has. Nope, it's another one of those phrases or ideas that ReadWriteWeb, the source of the linked to NewSQL definition, is trying to turn into a "big idea". They tried that previously with the Internet of Things (still don't know what the hell that is) as well as their own name, when they referred to the Web 2.0/API hype as the Read/Write Web. I actually like RWW over TC or Mashable but sometimes I have to tune out some of their pompous prognostication.
- chris_mahan 14y agoHow about ReSQL? Or SQLAgain? Or even BTS (Back to SQL)?
- mariuz 14y agoI like the OldSQL thanks , it is used in open source SQL databases
- peterwwillis 14y agoSo is Spanner basically just a distributed mostly-relational database? It seems like to find the record you want you just search the indices of nodes like a B-tree and in a couple queries are left at the node with the records you want. The downside is all that lost write performance but since it's synchronous you can probably mitigate some of the rebalancing by having many indices of indices, so only a couple nodes have to actually rebalance anything on a write.
- zaphar 14y agoDon't mistake Transactional with Relational. Just because a database has transactions doesn't automatically mean it is Relational too.
- kyt 14y ago"Maybe this time Open Source efforts should focus elsewhere, innovating rather than following Google?" There's a ton of innovative projects in the open source community, but it's difficult to convince people to use them. Developing a clone of a Google tech has a built in marketing advantage: "Google uses something like this."
- neovive 14y agoWith some of the best programming minds in the world, it's interesting to see that Google finds it more efficient for programmers to solve the performance issues vs reimplementing core db functionality, such as transactions. I like the direction of moving closer to original database theory concepts and allowing the creative energy to focus on solutions to performance problems at high-scale.
- dmix 14y ago> best programming minds They probably have a lot of great minds at the top of Google but there's still going to be a significant amount of average ones, just like in any company/structure.
- zaidf 14y agoI am in favor of any label that replaces "NoSQL" with something else. NoSQL always rubbed me the wrong way. Besides not being a huge fan of the "No", it really says nothing about what it is(if it isn't SQL).
- realrocker 14y agoImagine an amateur programmer walking into this whole debacle.
- wmf 14y agoIsn't that called Hacker News?
- realrocker 14y agoI guess I should clarify my statement. But I won't.
- debacle 14y agoI'd probably apologize for my clumsiness and keep walking.
- realrocker 14y agowell played sir.
- cgs1019 14y agoThis article comes across as really cynical and entirely lacking in the kind of rigor and detail I have previously found on highscalability. Spanner is really mind-blowingly cool tech. I thought this article was much more informative and worth the time to read: http://news.ycombinator.com/item?id=4562546 http://news.ycombinator.com/item?id=4562546
- efuquen 14y ago"Another complicating factor is that as Masters of Disk it’s not surprising Google ..." Masters of Disk? He seriously that that was a good line? One of the many things that annoyed me about that post. Thanks for the links.
- josephcooney 14y ago"He seriously that that was a good line?" Ironic.
- apu 14y agoMuphry's law
- Jare 14y agoI think the notes about transactions are quite valid as the point of a HS post. However the author didn't have anything else to add ("I look forward to more insightful commentary. There’s a lot to make sense of") yet felt the need to keep writing, and filled the article with style and attitude rather than substance.
- nostrademons 14y agoIt's really funny to watch tech journalists try to write about Google infrastructure from the outside, based only on one paper... Hell, it's usually really funny just to watch tech journalists try to write.
- neilk 14y agoThe part that amuses me is their surprise that Google sees some advantage to transactions. Journalists (of any kind) want to see things in terms of personality or philosophical conflict, e.g. SQL vs NoSQL. Because it's what their readers understand, and usually, it's all that they understand. They don't seem to consider that people might be making rational choices and difficult tradeoffs.
- seiji 14y agoIf that site contains one journalist, I'll eat my diamond tiara. Let's call them blogophiles instead. Or reposting aggragists. Adveraggrapostsummarizers. "One who reposts content from other sites while attaching ads and providing a veneer of quote-unquote insight."
- deleted 14y ago[deleted]
- bduerst 14y agoThis is exactly why I think the "News" market is ripe for some disruptive product. The only problem is that sources can send cease and desist letters to any company that effectively steals the sources' viewerbase.
- VonGuard 14y agoI can't really apologize for my brethren in the market, but yes, tech journalism tends to be terrible. The journalism market was already disrupted by Twitter, Facebook and blogs. Blogs in particular are great, because instead of reading what some idiot from journalism school has to say about, oh, I dunno.... uhm.... exceptions versus errors in programming languages (as touched off by that Python vs. Go post), instead of reading some wonk's interpretation, you can instead read a developer's opinion. In the meantime I just wanted to point out that everyone and their mother is making "announcements" about Spanner, and usually trying to make the world think they're better. Mostly this is from NoSQL companies.
- sigil 14y agoBuzzword headline aside, the Spanner paper is great and worth your time. As is the BigTable paper, the Dremel paper, and the Paxos Made Live paper. I read the Google whitepapers and wonder, is there anywhere else one can go to work on real solutions to distributed systems problems? At smaller scales you can cheat -- you don't need Paxos, you can get away with non-consensus-based master / slave failover. You can play the odds with failure modes. At Google's scale, you can't: behavior under normally unlikely failures matters, probability matters, CAP matters.
- chops 14y agoYou could go to Basho to work on Riak Enterprise. http://basho.com/products/riak-enterprise/ http://basho.com/products/riak-enterprise/
- neilk 14y agoLots of big sites roll their own solutions for this problem. Often there will even be multiple in-house technologies with different characteristics. Google, LinkedIn, and Facebook for sure.
- fintler 14y agoAt lanl.gov, we've been developing a ton of stuff related to distributed problems, recently we wrote a library for abstracting self-stabilization for single-mount petascale filesystem treewalking that extends Dijkstra's original design: https://github.com/hpc/libcircle https://github.com/hpc/libcircle We have a detailed paper on it coming out at SC12.
- equark 14y agoYou could come work at Sense (http://www.senseplatform.com http://www.senseplatform.com) or email me tristan@senseplatform.com. Not Google scale, but the same technological challenges without the legacy bagage.
- ckuijjer 14y agoI've downloaded them all, now I should really start reading them. Here are some direct pdf links of the papers: - Spanner: http://research.google.com/archive/spanner-osdi2012.pdf http://research.google.com/archive/spanner-osdi2012.pdf - BigTable: http://research.google.com/archive/bigtable-osdi06.pdf http://research.google.com/archive/bigtable-osdi06.pdf - Dremel: http://research.google.com/pubs/archive/36632.pdf http://research.google.com/pubs/archive/36632.pdf - Paxos made live: http://www.eecs.harvard.edu/cs262/Readings/paxosmadelive.pdf http://www.eecs.harvard.edu/cs262/Readings/paxosmadelive.pdf
- Dave_Rosenthal 14y agoDisclaimer: I'm a co-founder of a database company (FoundationDB) building a scalable, ACID database. I couldn't agree more with main quote that they pulled from the paper, expressing the difficulty of (even great) programmers having to "code around the lack of transactions." Ease of development is one of the biggest benefits of transactions. However, another huge benefit that didn't get much play in the article is the freedom that transactions afford you to build abstractions and other data models on top of whatever you are given. In our product's case, a low-level ordered K/V store is used for a storage layer and several different data models are exposed on top (see http://foundationdb.com/#layers http://foundationdb.com/#layers). I think the future of databases has a diversity of data models and query languages (including SQL, document, K/V, columnar, etc.). I also think the future of databases is ACID. It seems like more and more of the NoSQL early adopters (and creators) are coming to the same conclusion.
- dkhenry 14y agoI wonder how you see the publishing of the Spanner paper effects your product? Does this give validity to your product seeing as there is a production system with similar features to yours already in use, or is there a risk Google might provide this system as a service to other companies possibly competing with you.
- Dave_Rosenthal 14y agoWell, it doesn't seem like it's their ambition, but there's no doubt that if Google released all of their in-house data tools, including Spanner, that it would drastically change the market. We love to see this stuff, though. When we started building our product over three years ago, the idea of a distributed, ACID database was sort of laughed at. (I think the CAP theorem scared a lot of people off from building really useful products.) FoundationDB isn't the same as Spanner, but they share some of the same goals. We see that as a huge validation. Thanks for the question.
- akldfgj 14y agoWhat's not their ambition? http://cloud.google.com/products http://cloud.google.com/products /compute-engine.html /cloud-storage.html /big-query.html /more-products.html
- exabrial 14y agoSo... transactions are cool again? Thank goodness!
- jbigelow76 14y agoTransactions have always been cool, it's just that business that don't need to them have always been cooler in the in the world of tech journalism so they don't get much mention. If something goes wrong at Instagram nobody really cares if you they don't roll back posting the picture of your pastrami sandwich just because the #lolcatz tag got applied by mistake.
- rbranson 14y agoI'd love to have actual distributed transactions that could scale indefinitely and not create availability issues. We actually get a steady stream of user complaints about inconsistencies between counter caches and what appears in results. Worse is the inconsistencies that can happen between graph edges that you want to partition in two different manners (eg. following vs followers).
- jaylevitt 14y agoLikewise, consistency's more important in facebook-y scenarios than you'd think. The canonical example I've heard is that "defriend my boss" MUST always be seen before "post 'I'm quitting!'"
- aklein 14y agoHere is a google engineer giving a keynote on Spanner: http://vimeo.com/43759726 http://vimeo.com/43759726
- gbog 14y ago> few organizations need to support transactions on a global scale That doesn't ring true to me. My previous job was in the field of services to administrations, nothing very uncommon, and we needed global transactions very badly. I suppose any bank, insurance and even plane ticket trader would love to have global transactions.
- effinjames 14y agoi like the way elasticsearch use json to express query (query-DSL)
- effinjames 14y agowhat's with the downvote? do i really care? the thing is, json could be use as a replacement for sql, a safer sql.
- philthom 14y agoYeah Silver Bullets Are Always Plated
- cageface 14y agoSo the correct and responsible thing for Google to do now would be to patent the shit out of this and then sue back into the stone age anybody that implements anything even vaguely similar, right? I mean, the social fabric depends on companies protecting their innovations, right?
- mjs 14y ago"A complicating factor for an Open Source effort is that Spanner includes the use of GPS and Atomic clock hardware." (!) Can anyone explain why such an accurate clock is helpful? I can see that it's needed if you create a document on the East Coast at about the same time as you create a document on the West Coast, and you absolutely need to know which was created first, but for most applications can't you just go with whatever time the system that got the insert thinks it is??
- saraid216 14y ago> for most applications I suspect that part of the point is that this application does care.
- jpollock 14y agoDistributed state machines are all about deciding in what order things happened in. That A happened before B. If you had a clock that everyone agreed on, much of the problems would go away. The problem is that we don't have one that everyone can agree on. Protocols like Paxos allow us to guarantee an order will be agreed, but it can be slow. They seem to be using the clocks with an error bound to perform a first pass ordering to see if a collision is possible at all, or if they can skip a few steps. Caveat: I don't fully understand that part of the paper yet...
- acdha 14y ago> whatever time the system that got the insert is the key part: if you want to know what order to apply changes in, you need every system to have fairly close time synchronization so you can apply changes in the correct order. NTP will get you enough precision for many applications but if you get enough updates to the same resources, you're certain to start getting differences which are smaller than the reliable accuracy of your system clock. You can use other protocols - e.g. a CAS-style "change old-value to new-value" conditional update - but those have performance implications and require app support. Given that a GPS probably costs about an hour or two of engineer-time, trying to eke a little more precision out of the system clock seems like a potentially cheap win.
- mooneater 14y agoWhy does Google so often give details on their production infrastructure? I would think they could get a bigger lead by keeping quiet.
- novaleaf 14y agoa bit late, but bigtable has transactions... we use it a lot (but limits you to 5 objects at a time)