6 ms·
Ex-Googlers CockroachDB: A Scalable, Geo-Replicated, Transactional Datastore
- rabino 12y agoWhy the 'Ex-Googlers' in the title. Is it like a seal of approval or something?
- calpaterson 12y agoYeah, I feel a bit uncomfortable about using the brand of a previous employer so prominently to promote a different project
- quotemstr 12y agoHaven't you noticed the pedigree obsession of some in our field?
- wslh 12y agoIn this context it adds more signal than noise. There are zillions of open source projects but when you need to use them in production a very small subset of this universe is ready.
- sigzero 12y agoDoes it? I don't think ex-Googlers gives anything of value here.
- svisser 12y agoIt indicates that the project is worth a closer look rather than waiting or dismissing it outright.
- hueving 12y agoWhy? Being an ex-googler could just as easily imply they were incompetent and fired.
- melling 12y agoSo, you're assuming that half of all ex-googlers were fired? Otherwise, there aren't equal probabilities.
- BinaryIdiot 12y ago> It indicates that the project is worth a closer look rather than waiting or dismissing it outright. reply I'm curious, could you explain why? Google is an incredibly large company with many developers who never even touch their data systems so to me saying ex-Googler really doesn't mean anything beyond that they're probably a senior developer considering how rigorous (and honestly some old hat) their interview process is. But that doesn't change my viewpoint of the project at all.
- wslh 12y agoYou can take a look at the project author profiles.
- angrymouse 12y agoI think that refers to the Wired article http://www.wired.com/2014/07/cockroachdb/ http://www.wired.com/2014/07/cockroachdb/ With the tag being sign of pedigree. "A scalable DB from people who worked at a place with huge scale DBs". Whether it is very convincing or not is another matter, but I am sure it gets more clicks/attention
- eng_monkey 12y agoIt is a pretty strange title. Some might interpret it as they started this project after being fired from Google.
- bojo 12y agoThat's kind of how I read it.
- deleted 12y ago[deleted]
- melling 12y agoYes, it's called social proof. If you're observant you'll notice that people use it everywhere. Of course, social proof doesn't mean guarantees.
- deleted 12y ago[deleted]
- peterwwillis 12y agoNah, it's honor-by-association fallacy. Social proof is a behavioral thing, not false reasoning.
- datashovel 12y agoI don't think it matters how you try to promote an open source project. The code will ultimately be the determining factor whether the project can support a following. And everyone has access to the code to make that determination independently.
- acjohnson55 12y agoThat seems rather naive to me. No one is scouring GitHub looking for quality code and bringing unrecognized projects to light.
- datashovel 12y agoMy point is not that unfound projects will necessarily become found. No matter how good they are. Instead the idea is that no matter how a project is promoted, ultimately they only have their code-base to back their claims.
- acveilleux 12y agoA requirement of a successful project/product is visibility. Ex-Googlers does not imply much more then "used to work there" but if it drives more attention to them, it's good MarCom.
- kfcm 12y agoName-dropping has been common since the first salesman.
- Blackthorn 12y agoThis was based on an internal Google technology, so the "Ex-Googlers" label is actually quite relevant.
- ryandrake 12y agoI was kind of put off by this as well. Does where they worked in the past have anything to do with how useful the project is now or how talented the people are? What if they're doing it on their own because Google thought the project was useless? What if they were fired from Google, or only worked at Google for a month? How does the founders' past involvement with Google mark this project as any more interesting than any other random "Show HN" project?
- ultimape 12y agoI hope Ceph can takes cues from this and be able to do geo-replication at scale.
- srcmap 12y agoComparing to Facebook ' s TAO approach from yesterday post, I like the FB small set of api approach better. But that mainly focus on handling social graph with objects and associations. On the other hand, almost all my backend related features can be easily abstracted to those APIs.
- IgorPartola 12y agoSo I cannot tell if this is aiming to be CA or AP. Having beaten my head against the CAP wall for a while, how does it deal with partitions?
- pjc50 12y agoThe design doc https://docs.google.com/document/d/11k2EmhLGSbViBvi6_zFEiKzuXxYF49ZuuDJLe6O8gBU/edit https://docs.google.com/document/d/11k2EmhLGSbViBvi6_zFEiKzu... says, "TBD: how to avoid partitions? Need to work out a simulation of the protocol to tune the behavior and see empirically how well it works" .. so they've gone for CA and forgotten about P.
- e12e 12y agoGeoreplication that can't handle partitions? Sign me up! I'll have two (or more!)!
- rdtsc 12y agoUsually when building a distributed database, "TBD: how to avoid partitions?" is not what you want to see except on the initial whiteboard discussion before a single line of code is written. But what do I know, I am not an Ex-Googler.
- teraflop 12y agoYou're taking a sentence out of context and giving it the most uncharitable possible interpretation. They certainly haven't "forgotten" about network partitions because if you actually read the design document, instead of ctrl-F'ing for the word "partition", they talk about the mechanisms they use to ensure sequential consistency. The software is not yet at the point of being testable AFAIK, but clearly the intent is to build a CP system. The section you're quoting is discussing a separate gossip protocol that is used to lazily propagate node state information. It does not affect the consistency of actual data replicas.
- pjc50 12y agoI'll admit to aggressive use of ctrl-F, but only because I too felt they should be a lot clearer about what the intended properties are. From the intro page: "Cockroach is a distributed key/value datastore which supports ACID transactional semantics and versioned values as first-class features. The primary design goal is global consistency and survivability, hence the name. Cockroach aims to tolerate disk, machine, rack, and even datacenter failures with minimal latency disruption and no manual intervention." If we read 'survivability' as 'availability', then that would suggest they've gone for CA. Although closer inspection reveals that their architecture seems to be made of shards each of which is maintained with Raft/Paxos. An evaluation of this by the Cambridge Computer laboratory is here: http://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-857.pdf http://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-857.pdf That report makes two points relevant to this discussion. One is that a hard definition of C A and P can be difficult and that it's possible to achieve all three almost all of the time in real conditions. The other is from the conclusion: "In particular, a [Raft] cluster can be rendered useless by a frequently disconnected node or a node with an asymmetric partition"
- datashovel 12y agoHappy to see it's written in Golang.
- bojo 12y agoAs a Go developer I was happy as well. On the other hand this question immediately popped into my mind: What kind of overhead does the GC incur, and how does it affect processes like a database where low latency is desired?
- datashovel 12y agoIf I had to guess, I would say network and disk I/O will be the bottlenecks. Disk I/O less so because distributed system, and SSD. I imagine if anyone can solve GC issues, though, I would bet on Google :)
- natebrennand 12y agoTheir chosen solutions to the GC issues are in this roadmap. [0] [0]: https://docs.google.com/document/d/16Y4IsnNRCN43Mx0NZc5YXZLovrHvvLhK_h0KN8woTO4/preview?sle=true https://docs.google.com/document/d/16Y4IsnNRCN43Mx0NZc5YXZLo...
- ansible 12y agoWhat kind of overhead does the GC incur, and how does it affect processes like a database where low latency is desired? The eventual goal (golang v1.5 IIRC) is to have 40ms out of every 50ms available for actual processing. This is the kind of 'soft real time' that should provide good responsiveness for most clients most of the time.
- tedchs 12y agoHere in the Southeast, we call them "Palmetto Bugs". Maybe a rename to PalmettoDB? :)
- viiralvx 12y agoSouth Carolina? :P
- wuliwong 12y agoPalmetto bugs are generally the larger, flying "American Cockroach." "German Cockroaches" are smaller, don't fly but are generally the ones that cause infestations. I grew up in the Northeast and German Cockroaches are all I ever saw up there. I've been down in Atlanta for years now and I only see Palmetto bugs here. Unless this database can fly, I'd say they've chosen the right name. ;)
- JackC 12y agoAre there distributed data stores like this that are also resilient to intentional sabotage? I've been looking recently at long-term digital preservation systems -- tools designed to archive large amounts of data for decades. This is the Library of Alexandria problem -- how do we preserve all this data we're generating against once-in-a-century disasters? So this 2005 paper lists thirteen different threats to long-term archives: Media Failure; Hardware Failure; Software Failure; Communication Errors; Failure of Network Services; Media & Hardware Obsolescence; Software Obsolescence; Operator Error; Natural Disaster; External Attack; Internal Attack; Economic Failure; Organizational Failure.[1] Fault-tolerant distributed data stores are exciting, because they solve a bunch of those problems off the bat -- media failure, hardware failure, communication errors, failure of network services, hardware obsolescence, and natural disaster. They also help to address software failure, software obsolescence, and economic failure, because archival projects are always strapped for resources and it's great to rely on tools that exist for totally distinct, commercially-valuable reasons. But that still leaves operator error, external attack, and internal attack -- burning down the Library. Hence my original question: are there distributed data stores that can be configured to resist intentional destruction of data? [1] http://www.dlib.org/dlib/november05/rosenthal/11rosenthal.html http://www.dlib.org/dlib/november05/rosenthal/11rosenthal.ht...
- otoburb 12y agoJournaling or storing incremental backups (perhaps offline?) of validated/verified checkpoints may address this, although it sounds like something you wouldn't be happy with since it's not a 'built-in' feature but an additional backup & maintenance process that a system administrator would need to implement. I guess you're asking whether there exists a distributed fault-tolerant with a form of version control (similar to git/cvs/perforce) as part of the native feature set.
- imaginenore 12y ago> are there distributed data stores that can be configured to resist intentional destruction of data? Well, Git has checksums on everything.
- jchrisa 12y ago
- kul_ 12y agoawful name!
- eksith 12y agoI think the sentiment was "hard to kill" and/or become extinct due to its resiliency. But I agree, they should have picked something else.
- vram22 12y agoGood point about the likely reason. Hydra could have been another good choice :) http://en.wikipedia.org/wiki/Hydra http://en.wikipedia.org/wiki/Hydra See first entry at above Wikipedia page, about the many-header serpent.
- disputin 12y agoI think the name's great. A refreshing change from all the synthetic cutesy crap. Bubblegumlydb?
- Thaxll 12y agoMany things that I don't agree with: https://github.com/cockroachdb/cockroach/ https://github.com/cockroachdb/cockroach/ MySQL: Weak consistency Cassandra: No availability or weak consistency with datacenter failure
- deleted 12y ago[deleted]
- teraflop 12y agoMySQL provides strong consistency when run on a single machine, but that breaks down when you handle failover using asynchronous replication between DCs.
- eksith 12y agoTo add to this "Postgres: Limited scalability". I wish someone told us before we ticked over to 31TB... 5 Months ago.
- JSno 12y agothe name feels sick
- vishly 12y agoAwful name!!! Why Cockroach..??
- vishly 12y agoawful awful name... folks didnt get anything than cockroach..
- peterwwillis 12y agoIt's really strange to see networked databases go through this iterative design fad. It was file transfers back in the 90s/00s; everyone had their own distributed decentralized file transfer solution. Little known fact, Gentoo's Portage almost became an internet-wide distributed decentralized public file system. Thank god they abandoned that idea. Can you imagine trying to debug a file transfer error just to get an mp3 player to install on your machine? I can't wait until databases go back to being mainframes.
- anonbanker 12y ago> Little known fact, Gentoo's Portage almost became an internet-wide distributed decentralized public file system. Am I the only person really sad that this didn't happen? after using apt over Tahoe-LAFS (over I2P - KillYourTV's PPA is on clearnet and I2P), I wanted this to be the default behavior for apt.
- xedarius 12y agoAre there any details on how the distributed joins are achieved? (Sorry if that detail is in the design doc, my access to google drive is blocked from work).