38 ms·

CockroachDB beta-20161013

309 points by aphyr 10y ago

vonklaus 10y ago
Wow. Probably fundraising and getting out front of googles new consumer spanner service. I just started experimenting w/ cockroachdb and it sounds/seems great. They def are campaigning hard, landing a Jepsen test and a lot of posts/articles last couple. I am pulling for them over G on principle. I infer they know people at Stripe where the Rethink team & Aphyr are so hopefully they can learn from them and build an awesome product. G has lost a lot of battles on UX, Docs, attitude and just general product. They have won a lot of battles on sheer size & resources. Be nice to have this stick around.
- aphyr 10y ago
  Just as a headsup, I left Stripe in late 2015 to run Jepsen as an independent consultancy.
  - baq 10y ago
    i wanted to thank you for your work, your blog is singular on the web. chapeau bas.
  - seahckr 10y ago
    big fan of your work! Thank you
  - vonklaus 10y ago
    Hey man, didn't realize you left Stripe but did sort of notice Jepsen getting more serious. Jepsen has phenomenal reach: Interesting and hilarious coverage of technical complexity which assumes just enough knowledge to engage a keen novice while seemingly engaging seasoned experts in the field. Thanks for that. Your entertainment firm is a bit out if my wheelhouse, but it's such a crowded space and you end up getting really tied down. Maybe get acquihired for an Iditarod team, but I reckon this is more scaleable. Posts are great. Tganks
BrentOzar 10y ago
"For instance, on a cluster of five m3.large nodes, an even mixture of processes performing single-row inserts and selects over a few hundred rows pushed ~40 inserts and ~20 reads per second (steady-state)." Wow, that's...not fast.
- aphyr 10y ago
  I just want to emphasize that: a.) this is a beta product where the team has been focusing on completeness and correctness before performance b.) I was testing Cockroach from August through October--Cockroach Labs has put a good deal of work into performance in the last three months, but that work hasn't been merged into mainline yet. I expect things should be a good deal faster by 1.0. c.) Jepsen does some intentionally pathological things to databases. For instance, we choose extremely small Cockroach shard sizes to force constant rebalancing of the keyspace under inserts. Don't treat this as representative of a production workload.
  - qaq 10y ago
    They need to be at least 1000x to 10,000x faster on the read side though
    - aphyr 10y ago
      I agree that it can and should be faster, but the targets will depend on what you're going for. 20k linearizable reads/sec on EBS-backed instances is, uh, maybe a tad optimistic.
    - hinkley 10y ago
      I'm a little confused about how read speed is 2x slower than write speed. With respect to 'correctness', you're drifting into pyrrhic victory or 'not even wrong' territory at that point. When there are basic expectations of behavior that aren't being met, many of us would reject the idea that this code is 'correct'.
      - aphyr 10y ago
bluejekyll 10y ago
> Should the clock offset between two nodes grow too large, transactions will no longer be consistent and all bets are off. A concern I have is that there could be datacorruption in this context, no? It sounds like on a per cluster basis, i.e. Same Raft cluster, there is a strict serializability guarantee. But without linearalizability system wide, you could end up with data corruption on multi-keyspace writes.
- mixedCase 10y ago
  If I understood correctly, nodes get killed if they drift outside the max time offset tolerance.
- aphyr 10y ago
  You're correct--large clock anomalies (or VM pauses, message delays, etc) can cause serializability anomalies. I don't exactly know what will happen, because our tests explicitly focused on staying within clock skew bounds. To speculate: if clocks do go out of bounds, I don't... think... you can violate single-key serializability, because that goes through a single Raft cluster, but you can definitely see multi-key consistency anomalies. I think you could also get stale single-key reads, because the read path IIRC relies on a time-based lease. Might be other interesting issues if Cockroach is shuffling data from one Raft ensemble to another--I think the range info itself might have a time-based lease that could go stale as well. Perhaps a Cockroach Labs engineer could comment here? Note that the time window for these anomalies is limited to a few seconds-nodes will kill themselves when they detect clock skew.
  - bdarnell 10y ago
    (Cockroach Labs CTO) The major issue with clock offsets is stale reads due to the time-based read lease. Pretty much everything else is independent of the clocks. Stale reads are still enough to get you in trouble (some of the jepsen tests will show serializability violations if you disable the clock-offset suicide and mess with the clocks enough), but it's tricky to actually get bad data written into the database, as opposed to returning a stale-but-consistent view of the data.
exhilaration 10y ago
Like Spanner, Cockroach’s correctness depends on the strength of its clocks... Unlike Spanner, CockroachDB users are likely deploying on commodity hardware or the cloud, without GPS and atomic clocks for reference. I'm curious, do any of the cloud providers offer instances with high-precision clocks? Maybe this is something one of the smaller cloud providers might want to offer to differentiate themselves - instances with GSP clocks.
- Already__Taken 10y ago
  Why can't these datacenters be using a high precision gps source for their NTP source in the building?
  - tdrd 10y ago
    [Cockroach Labs employee] In addition to being extremely precise, TrueTime is explicit about error in its measurements. In other words, a call to TrueTime returns a lower and upper bound for the current time, which is important for systems like Spanner or CockroachDB when operations must be linearized. In effect, Spanner sleeps on each operation for a duration of [latest_upper_bound - current_lower_bound] to ensure that operations are strictly ordered. Your suggestion might improve typical clock synchronization, but still leaves defining the critical offset threshold to the operator, and importantly leaves little choice for software like CockroachDB other than to crash when it detects that the invariant is violated.
  - shin_lao 10y ago
    NTP isn't enough, you need PTP.
- mikecb 10y ago
  The spanner announcement mentioned offering direct access to the TrueTime api sometime in the not too distant future. Edit: Quote from Brewer's post: "Next steps For a significantly deeper dive into the details, see the white paper also released today. It covers Spanner, consistency and availability in depth (including new data). It also looks at the role played by Google’s TrueTime system, which provides a globally synchronized clock. We intend to release TrueTime for direct use by Cloud customers in the future."
deleted 10y ago
[deleted]
sroussey 10y ago
How does this compare to gundb.io?
- 010a 10y ago
  Gundb is garbage.
  - aphyr 10y ago
    To be a tad more concrete: GunDB can run in the browser as well as on servers; CockroachDB is a traditional, client/server database. GunDB is aiming for eventual consistency, but when I looked a year ago, it seemed like the algorithm they chose wasn't actually convergent--nodes could diverge because non-commutative updates could be applied in different orders on different nodes. CockroachDB, by contrast, is a serializable, single-key linearizable store. I don't think GunDB offers transparent sharding or SQL, both of which are important aspects of CockroachDB. Both depend on wall clocks.
    - marknadal 10y ago
      Aphyr, thanks! I appreciate seeing you here and replying. GUN Author here. First, to answer @sroussey - Kyle is correct, GUN is an eventually consistent graph database that runs browser/server with extremely high availability, while CockroachDB is trying to be a strongly consistent, linearizable key/value SQL store. To read about what we guarantee/don't, check out these articles: https://github.com/amark/gun/wiki/CAP-Theorem https://github.com/amark/gun/wiki/CAP-Theorem http://gun.js.org/distributed/matters.html http://gun.js.org/distributed/matters.html (tech talk on how the algorithms work) https://github.com/amark/gun/wiki/Conflict-Resolution-with-Guns https://github.com/amark/gun/wiki/Conflict-Resolution-with-G... The last year we've focused on performance, and can now do 25M+ reads/sec (no cache miss, disk I/O performance isn't particularly interesting to us). More on that here: https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2GB-Atom-CPU https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2... (we're not kidding about it being able to run in the browser) With regards to commutative / non-commutative, @Aphyr, slight correction: Different machines will converge to the same value, it is just that you'd need to use a commutative CRDT on top of gun (we have one here: https://github.com/amark/gun/wiki/snippets-(v0.3.x)#counter https://github.com/amark/gun/wiki/snippets-(v0.3.x)#counter ) for them to converge to their combined value. Example: GUN treats primitives as atomic, so if you try to `gun.get('some').path('math').put(5 + 2)` it will converge to the atomic value of 7. So if two machines do `gun.get('some').path('math').put(currentValue + 2)` at the same time, both machines will converge to the fixed atomic value, not a commutative value - you need the CRDT for that. I mention this in the "What could go wrong?" section of the talk (linked above), and it is easy to add the necessary CRDT when you need it - actually, this was inspired by you @aphyr, from our discussion a few years ago. Note: @Aphyr, we built a distributed testing framework ( https://github.com/gundb/panic-server https://github.com/gundb/panic-server ), and now that our "performance improvement" stage is done, in the near future we're going to start building distributed correctness tests (we're rolling out to a government client soon). We'll probably be in touch with you in the next half year or so. :) Great work, keep it up. You are always a huge inspiration. <3 Edit: With regards to sharding / SQL, we don't have SQL support yet but somebody is currently building a prototype for it. Sharding, not built in, but peers store the data they request, which acts as a natural shard, but nothing fancy - for more info check out this article: https://github.com/amark/gun/wiki/sharding https://github.com/amark/gun/wiki/sharding .
will_pseudonym 10y ago
CockroachDB might be the worst product name I've come across. I get it; cockroaches are extremely hardy, but they're also much more strongly associated with uncleanliness and disease. EDIT: Also, cockroaches are BUGS! Bugs cause problems in computer systems.
- radub 10y ago
  http://morgana249.blogspot.com/2014/10/images-of-most-beautiful-cockroaches.html http://morgana249.blogspot.com/2014/10/images-of-most-beauti...
- artursapek 10y ago
  I think it's the most memorable database name I've seen. So at least it has that going for it.
  - atmosx 10y ago
    I'm not a native English speaker and have troubles pronouncing "cockroach". In my view the naming is rather unfortunate.
    - artursapek 10y ago
      Do you find it easier to pronounce "postgres"?
      - mixedCase 10y ago
        Not him, but as a native spanish speaker "postgres" rolls of the tongue more easily than "cockroachdb" or just "cockroach".
      - atmosx 10y ago
        Yes but I'm not sure the comparison stands. PSQL was created in 1996 and it happens to be one of the 2-3 most common/famous databases ever made as far as I can tell. If you're postgres, it doesn't really matter because you're already there.
WhitneyLand 10y ago
Could the name actually slow the project's momentum? I only recently read some cool things about CDB and wondered if I could have been subconsciously skipping articles about it for deep seeded reasons like http://bbc.com/future/story/20140918-the-reality-about-roaches http://bbc.com/future/story/20140918-the-reality-about-roach... Of course anything good may eventually rise to success on its merits, but in case I'm not the only one with a subconscious aversion maybe the info below will help put some people one less click away. How about SurvivalDB? :) CDB is a distributed SQL database built on a transactional and strongly-consistent key-value store. Scales horizontally; survives disk, machine, rack, and datacenter failures with low latency disruption and no intervention; strongly-consistent ACID transactions; SQL API for querying data. Inspired by Google’s Spanner and F1. Completely open source. https://cockroachlabs.com/docs/frequently-asked-questions.html https://cockroachlabs.com/docs/frequently-asked-questions.ht...
- Perignon 10y ago
  Agree, the name is pretty off putting. Maybe they were trying to be funny? I don't know, but I would recommend changing it.
  - andrewchambers 10y ago
    the level of bikeshedding on the name is crazy. I think it is a good name btw.
    - nine_k 10y ago
      The amount of rejection this name evokes is considerable. Maybe they should think about branding, unless their target is to stay in the fringe.
    - deleted 10y ago
      [deleted]
- finder83 10y ago
  I agree completely. I love reading about databases, but apparently have been skipping reading about CockroachDB for that same reason subconsciously. I took a deep dive the other day and was amazed that I've seen the name dozens of times and have never bothered to ask what was special about it. It really is a fascinating database.
AsyncAwait 10y ago
I'd be interested in any comparisons of CockroachDB vs TIDB[1], especially when it comes to speed. 1 - https://github.com/pingcap/tidb https://github.com/pingcap/tidb
- placeybordeaux 10y ago
  Yeah I find it odd that most discussions of CDB don't reference TiDB, but in this case speed is not really the focal point.
ohstopitu 10y ago
CDB is a really cool db for the time I've been playing with it (just a few months I think). That said, I'd really appreciate if they'd launch a hosted DBaaS. By that I mean, more in line with dynamodb or documentdb instead of say MySQL on AWS for example. I know it's on their timeline, but dbs generally get selected during the start of the projects and unless absolutely necessary, companies don't want to change dbs.
- dianasaur323 10y ago
  CockroachDB employee here. DBaaS is definitely on our roadmap. There are a lot of moving pieces to building out DBaaS, so our focus first is on honing our core product. In the meantime, we are working on making it as easy as possible to deploy CockroachDB across different cloud vendors and with orchestration tools like Kubernetes.
irfansharif 10y ago
I'm disappointed with the commentary here, I for one would think the discussions around building a distributed system with the consistency guarantees claimed here would be far more interesting than the name of said system.
- tshannon 10y ago
  The name may just be low hanging fruit to comment on if one has nothing else to add to the conversation.
  - Gigablah 10y ago
    I think it's hilarious that people are up in arms about the name but apparently something like MongoDB is fine.
- justicezyx 10y ago
  People care about CDB raises one of the most prominent and easiest fixable problem. I see 0 problem of this.
  - p4lindromica 10y ago
    Just like you think you can fix my shed by painting it yellow, right?
    - justicezyx 10y ago
      What do you mean? I think the feedback is that the name has a negative impact. Very few comments claim that they know a better name.
dijit 10y ago
I wish everyone would stop bikeshedding the damn name. From my perspective I'm glad something that's trying to do this is coming into existence. a couple years ago I was given the arduous task of ensuring that we never lost data. One of the requirements was that: At any time, any server can (and will) fail.. if you acknowledge that you have data then you MUST never lose it. You cannot imagine how many database solutions that ruled out. Including basically _all_ noSQL data stores. We settled on postgresql, because, despite going into toast tables and having to implement sharding on top- it was not only the most sane it could be wrangled into basically doing the right thing in regards to fsyncing and ensuring data wasn't in VFS. So, Kudos to these guys, who, in a world of people who don't give a shit about consistency (except eventual consistency) are actually trying to further it.
- Perignon 10y ago
  It's not bikeshedding. Who the fuck wants to work for a company called Cockroach Labs? Seriously, sure your tech friends and recruiters might appreciate it on your resume but anyone outside of tech? Applying for an apartment, mortgage, car, lol good luck. And also not too sure that anyone outside the immediate tech/startup nerd scene would appreciate it other. That aside why the hell do I want to thing of cockroaches day in and day out when I am using their product? Seriously, is this someone's idea of a bad joke? The name has got to go.
  - dijit 10y ago
    Nobody is going to give a flying fuck what your company is named on a mortgage application. Honestly there are companies with _much_ worse names in active use like "Dick & Daughters"[0] or "Ismail ANUS"[1] who even got an appointment to the queen. Seriously, it's bikeshedding, and the point of the DB is that you don't think about it all day. I also don't think of women called Cassandra when I use the database of the same name. Bikeshedding. Seriously. [0] https://beta.companieshouse.gov.uk/company/08610385 https://beta.companieshouse.gov.uk/company/08610385 [1] https://beta.companieshouse.gov.uk/officers/5-sFnEsibrDnUR2-Y0_IgYEG2I0/appointments https://beta.companieshouse.gov.uk/officers/5-sFnEsibrDnUR2-...
deleted 10y ago
[deleted]
throwawasiudy 10y ago
May I suggest renaming the thing already? 90% of the business world can't touch it with a name like that. How about Duradero ... Spanish for durable. The name doesn't seem to be taken by any other database type software
- dijit 10y ago
  I'm going to be the dissenting voice here and say: Use whatever name you like as long as it isn't something like cuntbagDB. CockroachDB accurately and immediately conveys meaning. It'll survive when nuclear war happens just like a cockroach. And I might work in the games industry but I would never have a hard time selling this. I would basically force them to use it and explain the benefits. If your company wont adopt technology because it uses a bug as it's name then your company has much greater problems than technology adoption.
  - throwawasiudy 10y ago
    It's not my company or me, it's my clients. Here's a sort of long winded story explaining what I mean that the corporate world can't use anything with the name "cockroach". If you disagree after reading, I'm open for a friendly debate. Average project, some 60 year old VP signing off on 100k of custom software with 8 pages of lawyer speak contracts is NOT going to allow "his" project to run off of "cockroach database". VP's/CEO's like to take personal ownership of projects they feel will go well, and ones done with us usually do. None of these people want ANY chance of their project getting criticized internally, so nobody will use CockroachDb. It's just not going to happen. Having successful projects is an easy way for these people to gather promotions and respect. Having any part of it do with "cockroach" could result in the whole project being an embarrassment. Imagine the database goes down one day. One of their low level staff calls us(we have ongoing support contract). Problem gets rapidly escalated to development. One of our lead developers replies to the email thread, that by now has a third of the clients' company on it. "Yeah see the problem was the cockroach database that wasn't supposed to go down needed to be restarted" . Most of the staff think we're joking. Guarantee this becomes a big pun around their office, every time anything breaks in the app they say "maybe the cockroaches on the back end died again". It's made worse by the fact that issues in software systems are commonly called "bugs". Well, our system doesn't have general bugs anymore, it has "cockroaches". Do you think this is an unlikely scenario? Because it's not. Besides the word "cockroach" this is exactly how urgent issues tend to be reported and fixed. Replace "cockroach" with "Microsoft" and I've probably violated my NDA. After that incident, whoever championed the project, probably our strongest ally in the company, resents us for making such a stupid mistake that resulted in his project becoming a joke. Maybe we aren't as great of a dev shop as he once thought? Maybe it's better to do open bidding next time so whoever wins he can't be blamed directly for what happens. You could say their company culture sucks, maybe it does? But that doesn't change that I can't use it on any client projects because of the chance we'll lose hundreds of thousands over a stupid name. There's plenty of other names that convey meaning but don't make lawyers nervous. Your attitude of "oh, well if you can't use it your company culture sucks" is EXACTLY why nobody will rename it and why corporate american won't touch it. Nobody cares about how cool it is to be edgy when millions of dollars are on the line. If you plan on forcing clients to use it you might as well hold a going away party for any of your clients with more than 5M revenue.
ysleepy 10y ago
So I assume this means cockroachDB currently (probably) meets its promised consistency levels. How does it compare to the usual default settings of PostgreSQL for example? - I think you get SERIALIZABLE, so the behaviour should be very reasonable. I understand the txn/s numbers are in a pathological scenario, but will these transactions block the whole db from moving faster or have unrelated transactions still normal throughput? I'm just glad we get linearizable single keys, I had to switch to HBase because it was the only choice at the time. Cassandra did not even give me read-your-own-writes. HBase also had issues while testing on the same host, delete+write lead to deletes being reordered after the writes because they had the same timestamp. I'm a bit disappointed from HN in this post, everyone only talks about the name. But 90% probably did not even read the article. Aaanyway, heres my 2cents: https://cr.yp.to/cdb.html https://cr.yp.to/cdb.html is CDB for me :P - so better choose another acronym!
- jpgvm 10y ago
  I don't believe CockroachDB provides SERIALIZABLE consistency for the entire keyspace, rather for subsets of the keyspace that reside in the same Raft cluster. In practice this means no cross key serializability without including a serialization token in transactions. I may be mistaken however but that is my understanding from reading CockroachDB docs and Aphyrs post.
  - tschottdorf 10y ago
    Cockroach Labs employee here. We provide serializable consistency for the whole keyspace. It's just that it's easier when you stay in one Raft ensemble as less can go wrong.
- ak4g 10y ago
  > How does it compare to the usual default settings of PostgreSQL for example? - I think you get SERIALIZABLE, so the behaviour should be very reasonable. Not clear if you're referring to PostgreSQL or CockroachDB, but for the record, the PostgreSQL default transaction isolation level is READ COMMITTED, not SERIALIZABLE. https://www.postgresql.org/docs/current/static/sql-set-transaction.html https://www.postgresql.org/docs/current/static/sql-set-trans...

309 points by aphyr 10y ago

Wow. Probably fundraising and getting out front of googles new consumer spanner service. I just started experimenting w/ cockroachdb and it sounds/seems great. They def are campaigning hard, landing a Jepsen test and a lot of posts/articles last couple. I am pulling for them over G on principle. I infer they know people at Stripe where the Rethink team & Aphyr are so hopefully they can learn from them and build an awesome product. G has lost a lot of battles on UX, Docs, attitude and just general product. They have won a lot of battles on sheer size & resources. Be nice to have this stick around.

aphyr 10y ago
Just as a headsup, I left Stripe in late 2015 to run Jepsen as an independent consultancy.
- baq 10y ago
  i wanted to thank you for your work, your blog is singular on the web. chapeau bas.
- seahckr 10y ago
  big fan of your work! Thank you
- vonklaus 10y ago
  Hey man, didn't realize you left Stripe but did sort of notice Jepsen getting more serious. Jepsen has phenomenal reach: Interesting and hilarious coverage of technical complexity which assumes just enough knowledge to engage a keen novice while seemingly engaging seasoned experts in the field. Thanks for that. Your entertainment firm is a bit out if my wheelhouse, but it's such a crowded space and you end up getting really tied down. Maybe get acquihired for an Iditarod team, but I reckon this is more scaleable. Posts are great. Tganks

BrentOzar 10y ago

"For instance, on a cluster of five m3.large nodes, an even mixture of processes performing single-row inserts and selects over a few hundred rows pushed ~40 inserts and ~20 reads per second (steady-state)." Wow, that's...not fast.

aphyr 10y ago
I just want to emphasize that: a.) this is a beta product where the team has been focusing on completeness and correctness before performance b.) I was testing Cockroach from August through October--Cockroach Labs has put a good deal of work into performance in the last three months, but that work hasn't been merged into mainline yet. I expect things should be a good deal faster by 1.0. c.) Jepsen does some intentionally pathological things to databases. For instance, we choose extremely small Cockroach shard sizes to force constant rebalancing of the keyspace under inserts. Don't treat this as representative of a production workload.
- qaq 10y ago
  They need to be at least 1000x to 10,000x faster on the read side though
  - aphyr 10y ago
    I agree that it can and should be faster, but the targets will depend on what you're going for. 20k linearizable reads/sec on EBS-backed instances is, uh, maybe a tad optimistic.
  - hinkley 10y ago
    I'm a little confused about how read speed is 2x slower than write speed. With respect to 'correctness', you're drifting into pyrrhic victory or 'not even wrong' territory at that point. When there are basic expectations of behavior that aren't being met, many of us would reject the idea that this code is 'correct'.
    - aphyr 10y ago

bluejekyll 10y ago

> Should the clock offset between two nodes grow too large, transactions will no longer be consistent and all bets are off. A concern I have is that there could be datacorruption in this context, no? It sounds like on a per cluster basis, i.e. Same Raft cluster, there is a strict serializability guarantee. But without linearalizability system wide, you could end up with data corruption on multi-keyspace writes.

mixedCase 10y ago
If I understood correctly, nodes get killed if they drift outside the max time offset tolerance.
aphyr 10y ago
You're correct--large clock anomalies (or VM pauses, message delays, etc) can cause serializability anomalies. I don't exactly know what will happen, because our tests explicitly focused on staying within clock skew bounds. To speculate: if clocks do go out of bounds, I don't... think... you can violate single-key serializability, because that goes through a single Raft cluster, but you can definitely see multi-key consistency anomalies. I think you could also get stale single-key reads, because the read path IIRC relies on a time-based lease. Might be other interesting issues if Cockroach is shuffling data from one Raft ensemble to another--I think the range info itself might have a time-based lease that could go stale as well. Perhaps a Cockroach Labs engineer could comment here? Note that the time window for these anomalies is limited to a few seconds-nodes will kill themselves when they detect clock skew.
- bdarnell 10y ago
  (Cockroach Labs CTO) The major issue with clock offsets is stale reads due to the time-based read lease. Pretty much everything else is independent of the clocks. Stale reads are still enough to get you in trouble (some of the jepsen tests will show serializability violations if you disable the clock-offset suicide and mess with the clocks enough), but it's tricky to actually get bad data written into the database, as opposed to returning a stale-but-consistent view of the data.

exhilaration 10y ago

Like Spanner, Cockroach’s correctness depends on the strength of its clocks... Unlike Spanner, CockroachDB users are likely deploying on commodity hardware or the cloud, without GPS and atomic clocks for reference. I'm curious, do any of the cloud providers offer instances with high-precision clocks? Maybe this is something one of the smaller cloud providers might want to offer to differentiate themselves - instances with GSP clocks.

Already__Taken 10y ago
Why can't these datacenters be using a high precision gps source for their NTP source in the building?
- tdrd 10y ago
  [Cockroach Labs employee] In addition to being extremely precise, TrueTime is explicit about error in its measurements. In other words, a call to TrueTime returns a lower and upper bound for the current time, which is important for systems like Spanner or CockroachDB when operations must be linearized. In effect, Spanner sleeps on each operation for a duration of [latest_upper_bound - current_lower_bound] to ensure that operations are strictly ordered. Your suggestion might improve typical clock synchronization, but still leaves defining the critical offset threshold to the operator, and importantly leaves little choice for software like CockroachDB other than to crash when it detects that the invariant is violated.
- shin_lao 10y ago
  NTP isn't enough, you need PTP.
mikecb 10y ago
The spanner announcement mentioned offering direct access to the TrueTime api sometime in the not too distant future. Edit: Quote from Brewer's post: "Next steps For a significantly deeper dive into the details, see the white paper also released today. It covers Spanner, consistency and availability in depth (including new data). It also looks at the role played by Google’s TrueTime system, which provides a globally synchronized clock. We intend to release TrueTime for direct use by Cloud customers in the future."

deleted 10y ago

[deleted]

sroussey 10y ago

How does this compare to gundb.io?

010a 10y ago
Gundb is garbage.
- aphyr 10y ago
  To be a tad more concrete: GunDB can run in the browser as well as on servers; CockroachDB is a traditional, client/server database. GunDB is aiming for eventual consistency, but when I looked a year ago, it seemed like the algorithm they chose wasn't actually convergent--nodes could diverge because non-commutative updates could be applied in different orders on different nodes. CockroachDB, by contrast, is a serializable, single-key linearizable store. I don't think GunDB offers transparent sharding or SQL, both of which are important aspects of CockroachDB. Both depend on wall clocks.
  - marknadal 10y ago
    Aphyr, thanks! I appreciate seeing you here and replying. GUN Author here. First, to answer @sroussey - Kyle is correct, GUN is an eventually consistent graph database that runs browser/server with extremely high availability, while CockroachDB is trying to be a strongly consistent, linearizable key/value SQL store. To read about what we guarantee/don't, check out these articles: https://github.com/amark/gun/wiki/CAP-Theorem https://github.com/amark/gun/wiki/CAP-Theorem http://gun.js.org/distributed/matters.html http://gun.js.org/distributed/matters.html (tech talk on how the algorithms work) https://github.com/amark/gun/wiki/Conflict-Resolution-with-Guns https://github.com/amark/gun/wiki/Conflict-Resolution-with-G... The last year we've focused on performance, and can now do 25M+ reads/sec (no cache miss, disk I/O performance isn't particularly interesting to us). More on that here: https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2GB-Atom-CPU https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2... (we're not kidding about it being able to run in the browser) With regards to commutative / non-commutative, @Aphyr, slight correction: Different machines will converge to the same value, it is just that you'd need to use a commutative CRDT on top of gun (we have one here: https://github.com/amark/gun/wiki/snippets-(v0.3.x)#counter https://github.com/amark/gun/wiki/snippets-(v0.3.x)#counter ) for them to converge to their combined value. Example: GUN treats primitives as atomic, so if you try to `gun.get('some').path('math').put(5 + 2)` it will converge to the atomic value of 7. So if two machines do `gun.get('some').path('math').put(currentValue + 2)` at the same time, both machines will converge to the fixed atomic value, not a commutative value - you need the CRDT for that. I mention this in the "What could go wrong?" section of the talk (linked above), and it is easy to add the necessary CRDT when you need it - actually, this was inspired by you @aphyr, from our discussion a few years ago. Note: @Aphyr, we built a distributed testing framework ( https://github.com/gundb/panic-server https://github.com/gundb/panic-server ), and now that our "performance improvement" stage is done, in the near future we're going to start building distributed correctness tests (we're rolling out to a government client soon). We'll probably be in touch with you in the next half year or so. :) Great work, keep it up. You are always a huge inspiration. <3 Edit: With regards to sharding / SQL, we don't have SQL support yet but somebody is currently building a prototype for it. Sharding, not built in, but peers store the data they request, which acts as a natural shard, but nothing fancy - for more info check out this article: https://github.com/amark/gun/wiki/sharding https://github.com/amark/gun/wiki/sharding .

will_pseudonym 10y ago

CockroachDB might be the worst product name I've come across. I get it; cockroaches are extremely hardy, but they're also much more strongly associated with uncleanliness and disease. EDIT: Also, cockroaches are BUGS! Bugs cause problems in computer systems.

radub 10y ago
http://morgana249.blogspot.com/2014/10/images-of-most-beautiful-cockroaches.html http://morgana249.blogspot.com/2014/10/images-of-most-beauti...
artursapek 10y ago
I think it's the most memorable database name I've seen. So at least it has that going for it.
- atmosx 10y ago
  I'm not a native English speaker and have troubles pronouncing "cockroach". In my view the naming is rather unfortunate.
  - artursapek 10y ago
    Do you find it easier to pronounce "postgres"?
    - mixedCase 10y ago
      Not him, but as a native spanish speaker "postgres" rolls of the tongue more easily than "cockroachdb" or just "cockroach".
    - atmosx 10y ago
      Yes but I'm not sure the comparison stands. PSQL was created in 1996 and it happens to be one of the 2-3 most common/famous databases ever made as far as I can tell. If you're postgres, it doesn't really matter because you're already there.

WhitneyLand 10y ago

Could the name actually slow the project's momentum? I only recently read some cool things about CDB and wondered if I could have been subconsciously skipping articles about it for deep seeded reasons like http://bbc.com/future/story/20140918-the-reality-about-roaches http://bbc.com/future/story/20140918-the-reality-about-roach... Of course anything good may eventually rise to success on its merits, but in case I'm not the only one with a subconscious aversion maybe the info below will help put some people one less click away. How about SurvivalDB? :) CDB is a distributed SQL database built on a transactional and strongly-consistent key-value store. Scales horizontally; survives disk, machine, rack, and datacenter failures with low latency disruption and no intervention; strongly-consistent ACID transactions; SQL API for querying data. Inspired by Google’s Spanner and F1. Completely open source. https://cockroachlabs.com/docs/frequently-asked-questions.html https://cockroachlabs.com/docs/frequently-asked-questions.ht...

Perignon 10y ago
Agree, the name is pretty off putting. Maybe they were trying to be funny? I don't know, but I would recommend changing it.
- andrewchambers 10y ago
  the level of bikeshedding on the name is crazy. I think it is a good name btw.
  - nine_k 10y ago
    The amount of rejection this name evokes is considerable. Maybe they should think about branding, unless their target is to stay in the fringe.
  - deleted 10y ago
    [deleted]
finder83 10y ago
I agree completely. I love reading about databases, but apparently have been skipping reading about CockroachDB for that same reason subconsciously. I took a deep dive the other day and was amazed that I've seen the name dozens of times and have never bothered to ask what was special about it. It really is a fascinating database.

AsyncAwait 10y ago

I'd be interested in any comparisons of CockroachDB vs TIDB[1], especially when it comes to speed. 1 - https://github.com/pingcap/tidb https://github.com/pingcap/tidb

placeybordeaux 10y ago
Yeah I find it odd that most discussions of CDB don't reference TiDB, but in this case speed is not really the focal point.

ohstopitu 10y ago

CDB is a really cool db for the time I've been playing with it (just a few months I think). That said, I'd really appreciate if they'd launch a hosted DBaaS. By that I mean, more in line with dynamodb or documentdb instead of say MySQL on AWS for example. I know it's on their timeline, but dbs generally get selected during the start of the projects and unless absolutely necessary, companies don't want to change dbs.

dianasaur323 10y ago
CockroachDB employee here. DBaaS is definitely on our roadmap. There are a lot of moving pieces to building out DBaaS, so our focus first is on honing our core product. In the meantime, we are working on making it as easy as possible to deploy CockroachDB across different cloud vendors and with orchestration tools like Kubernetes.

irfansharif 10y ago

I'm disappointed with the commentary here, I for one would think the discussions around building a distributed system with the consistency guarantees claimed here would be far more interesting than the name of said system.

tshannon 10y ago
The name may just be low hanging fruit to comment on if one has nothing else to add to the conversation.
- Gigablah 10y ago
  I think it's hilarious that people are up in arms about the name but apparently something like MongoDB is fine.
justicezyx 10y ago
People care about CDB raises one of the most prominent and easiest fixable problem. I see 0 problem of this.
- p4lindromica 10y ago
  Just like you think you can fix my shed by painting it yellow, right?
  - justicezyx 10y ago
    What do you mean? I think the feedback is that the name has a negative impact. Very few comments claim that they know a better name.

dijit 10y ago

I wish everyone would stop bikeshedding the damn name. From my perspective I'm glad something that's trying to do this is coming into existence. a couple years ago I was given the arduous task of ensuring that we never lost data. One of the requirements was that: At any time, any server can (and will) fail.. if you acknowledge that you have data then you MUST never lose it. You cannot imagine how many database solutions that ruled out. Including basically _all_ noSQL data stores. We settled on postgresql, because, despite going into toast tables and having to implement sharding on top- it was not only the most sane it could be wrangled into basically doing the right thing in regards to fsyncing and ensuring data wasn't in VFS. So, Kudos to these guys, who, in a world of people who don't give a shit about consistency (except eventual consistency) are actually trying to further it.

Perignon 10y ago
It's not bikeshedding. Who the fuck wants to work for a company called Cockroach Labs? Seriously, sure your tech friends and recruiters might appreciate it on your resume but anyone outside of tech? Applying for an apartment, mortgage, car, lol good luck. And also not too sure that anyone outside the immediate tech/startup nerd scene would appreciate it other. That aside why the hell do I want to thing of cockroaches day in and day out when I am using their product? Seriously, is this someone's idea of a bad joke? The name has got to go.
- dijit 10y ago
  Nobody is going to give a flying fuck what your company is named on a mortgage application. Honestly there are companies with _much_ worse names in active use like "Dick & Daughters"[0] or "Ismail ANUS"[1] who even got an appointment to the queen. Seriously, it's bikeshedding, and the point of the DB is that you don't think about it all day. I also don't think of women called Cassandra when I use the database of the same name. Bikeshedding. Seriously. [0] https://beta.companieshouse.gov.uk/company/08610385 https://beta.companieshouse.gov.uk/company/08610385 [1] https://beta.companieshouse.gov.uk/officers/5-sFnEsibrDnUR2-Y0_IgYEG2I0/appointments https://beta.companieshouse.gov.uk/officers/5-sFnEsibrDnUR2-...

deleted 10y ago

[deleted]

throwawasiudy 10y ago

May I suggest renaming the thing already? 90% of the business world can't touch it with a name like that. How about Duradero ... Spanish for durable. The name doesn't seem to be taken by any other database type software

dijit 10y ago
I'm going to be the dissenting voice here and say: Use whatever name you like as long as it isn't something like cuntbagDB. CockroachDB accurately and immediately conveys meaning. It'll survive when nuclear war happens just like a cockroach. And I might work in the games industry but I would never have a hard time selling this. I would basically force them to use it and explain the benefits. If your company wont adopt technology because it uses a bug as it's name then your company has much greater problems than technology adoption.
- throwawasiudy 10y ago
  It's not my company or me, it's my clients. Here's a sort of long winded story explaining what I mean that the corporate world can't use anything with the name "cockroach". If you disagree after reading, I'm open for a friendly debate. Average project, some 60 year old VP signing off on 100k of custom software with 8 pages of lawyer speak contracts is NOT going to allow "his" project to run off of "cockroach database". VP's/CEO's like to take personal ownership of projects they feel will go well, and ones done with us usually do. None of these people want ANY chance of their project getting criticized internally, so nobody will use CockroachDb. It's just not going to happen. Having successful projects is an easy way for these people to gather promotions and respect. Having any part of it do with "cockroach" could result in the whole project being an embarrassment. Imagine the database goes down one day. One of their low level staff calls us(we have ongoing support contract). Problem gets rapidly escalated to development. One of our lead developers replies to the email thread, that by now has a third of the clients' company on it. "Yeah see the problem was the cockroach database that wasn't supposed to go down needed to be restarted" . Most of the staff think we're joking. Guarantee this becomes a big pun around their office, every time anything breaks in the app they say "maybe the cockroaches on the back end died again". It's made worse by the fact that issues in software systems are commonly called "bugs". Well, our system doesn't have general bugs anymore, it has "cockroaches". Do you think this is an unlikely scenario? Because it's not. Besides the word "cockroach" this is exactly how urgent issues tend to be reported and fixed. Replace "cockroach" with "Microsoft" and I've probably violated my NDA. After that incident, whoever championed the project, probably our strongest ally in the company, resents us for making such a stupid mistake that resulted in his project becoming a joke. Maybe we aren't as great of a dev shop as he once thought? Maybe it's better to do open bidding next time so whoever wins he can't be blamed directly for what happens. You could say their company culture sucks, maybe it does? But that doesn't change that I can't use it on any client projects because of the chance we'll lose hundreds of thousands over a stupid name. There's plenty of other names that convey meaning but don't make lawyers nervous. Your attitude of "oh, well if you can't use it your company culture sucks" is EXACTLY why nobody will rename it and why corporate american won't touch it. Nobody cares about how cool it is to be edgy when millions of dollars are on the line. If you plan on forcing clients to use it you might as well hold a going away party for any of your clients with more than 5M revenue.

ysleepy 10y ago

So I assume this means cockroachDB currently (probably) meets its promised consistency levels. How does it compare to the usual default settings of PostgreSQL for example? - I think you get SERIALIZABLE, so the behaviour should be very reasonable. I understand the txn/s numbers are in a pathological scenario, but will these transactions block the whole db from moving faster or have unrelated transactions still normal throughput? I'm just glad we get linearizable single keys, I had to switch to HBase because it was the only choice at the time. Cassandra did not even give me read-your-own-writes. HBase also had issues while testing on the same host, delete+write lead to deletes being reordered after the writes because they had the same timestamp. I'm a bit disappointed from HN in this post, everyone only talks about the name. But 90% probably did not even read the article. Aaanyway, heres my 2cents: https://cr.yp.to/cdb.html https://cr.yp.to/cdb.html is CDB for me :P - so better choose another acronym!

jpgvm 10y ago
I don't believe CockroachDB provides SERIALIZABLE consistency for the entire keyspace, rather for subsets of the keyspace that reside in the same Raft cluster. In practice this means no cross key serializability without including a serialization token in transactions. I may be mistaken however but that is my understanding from reading CockroachDB docs and Aphyrs post.
- tschottdorf 10y ago
  Cockroach Labs employee here. We provide serializable consistency for the whole keyspace. It's just that it's easier when you stay in one Raft ensemble as less can go wrong.
ak4g 10y ago
> How does it compare to the usual default settings of PostgreSQL for example? - I think you get SERIALIZABLE, so the behaviour should be very reasonable. Not clear if you're referring to PostgreSQL or CockroachDB, but for the record, the PostgreSQL default transaction isolation level is READ COMMITTED, not SERIALIZABLE. https://www.postgresql.org/docs/current/static/sql-set-transaction.html https://www.postgresql.org/docs/current/static/sql-set-trans...