6 ms·

> if you want password-authentication, you must configure your cluster to communicate using certificates. You can start each node in "insecure" mode, but then c

> if you want password-authentication, you must configure your cluster to communicate using certificates. You can start each node in "insecure" mode, but then can't use password authentication. There's a issue about this. Like the people posting there, our network is already secured, so this requirement is just an unnecessary nuisance. Eh, I'm not sure how I feel about this. Obviously you should secure your network as best you can, but how confident are you really that a bad actor will never find a foothold anywhere in your network? I think I would advocate for your services to all communicate securely (TLS et al), even internally, and if your database supports mutual client/server auth like it sounds like Cockroach does then you should use that as well. Particularly if you're depending on at-rest encryption handled transparently by the DBMS to protect your users' data - that won't do you much good if someone can just sniff/MITM network traffic and wait until a bunch of your data has been queried.

kccqzy 7y ago
I actually have a similar concern. In a previous job I tried to push for internal services communicating over TLS with a custom certificate authority. It was ultimately not adopted as too much complexity, including operationally when something goes wrong the developer should always be able to issue unencrypted requests from an internal network. It seemed to boil down to removing as many hoops as possible when something terrible happens and automated tools fail and manual recovery is needed.
jupp0r 7y ago
I came here to mention this, too. It's been shown from numerous leaks that defense in depth is a much more effective approach than simply relying on network security. You mention that query response data can be sniffed by an attacker in case networks are compromised. The much worse scenario is that the passwords themselves can be intercepted, giving attackers full access.
- fulafel 7y ago
  Relying on network security is usually a bad idea even if you don't have defense in depth. (Ie it's less bad to rely on some other one thing)
manigandham 7y ago
CockroachDB annoyingly requires manual certificate creation and distribution to each node, and if you don't set those up then you can't use any authorization at all. It's strange and user-hostile to disable auth completely just because you don't use certs. Also some environments like Kubernetes might already have network encryption (provided by a service mesh) and don't need another layer. Certificates serve both encryption and authentication, but encryption can be done with self-signed certs automatically generated by the nodes upon joining the cluster and authentication can be handled in the joining process by using a token or some other bootstrap value that's easier to manage.
- knz42 7y ago
  > CockroachDB annoyingly requires manual certificate creation and distribution to each node, and if you don't set those up then you can't use any authorization at all. It's strange and user-hostile to disable auth completely just because you don't use certs. This comment is not any more true as of CockroachDB 20.1 (Upcoming in Q2 2020). This version will enable you to set up your own transport-level security with clients using password authentication to crdb.
  - deleted 7y ago
    [deleted]
latch 7y ago
All of our internal communication already is secured via wireguard and communication can only flow through the wireguard interface. Does adding cockroach's own encryption on top of this add anything?
- thenewnewguy 7y ago
  I am unfamiliar with how setups like these usually work, but it could an active attacker MITM your network?
  - vsareto 7y ago
    You wouldn't be able to decrypt the wireguard packets unless you had each party's private keys
- fulafel 7y ago
  What happens when someone compromises your wireguard overlay network? The cert business is not just about encryption, it's also about authentication over an untrusted network.
  - tridentlead 7y ago
    Not much if it’s set up in a standard way because each node will only communicate directly with the node it’s expecting too. The communications are secured with the other node’s public key.
    - fulafel 7y ago
      So if you have 10 boxes interconnected in a full mesh, and one of those boxes gets compromised, then it's all fine even if you have been relying on the wireguard vpn being secure?
fulafel 7y ago
Yep, and besides these concerns, there's the difficulty in modeling the trust relationships and dependencies in this kind of "secure internal network" world view. The relationships are complex and you have low likelihood of your mental model matching the reality, because the relationships are not explicitly configured.
jrockway 7y ago
I am a huge fan of mTLS but it is quite difficult to set up and I'm sort of taking a "wait and see" approach, especially with Google pushing ALTS, which sounds pretty sane. My dream is that you can ignore "VPCs" (i.e. trusted networking, firewalls, etc.) and let applications decide whether or not to allow traffic. "You have presented an incorrect TLS certificate, no response for you." The problem I have is that I'm not sure what method you want to use to inject certificates. The current systems seem to be "something will figure it out for you at runtime, and your application will be none the wiser". (This is things like Istio and Linkerd.) I am not sure I'm convinced this adds much security. Anyone that can talk to the apiserver that injects certificates can probably convince the system to give you a certificate for your rogue container. And, I'm not sure that applications have enough visibility into the networking layer to really add any trust. I'd like for each incoming request to have some provenance information attached with dimensions like "this container was built from trusted source code and each commit that it contains was code reviewed", "this container was launched from a manifest written and applied by a trusted engineer from a trusted machine", and of course "this request was authorized by a human user whose data is being accessed". Right now, I feel like you have to hand-wave to get the first two. Sure, maybe Istio thinks the container is legit, perhaps because it asked a container notary (very early stages of support), and so now it can open a TLS connection to anything in the cluster. And, adding a request-scoped JWT is easy enough to provide the "here is the human user that made the request". I just don't think you can hide this from the application with a service mesh; every application needs to be aware of what's going on so it can say no to suspicious requests. And for that, I don't think we're really there yet. We had this at Google and it worked well, but I haven't found anything that works well in the outside world yet. With that in mind, I totally understand why people trust the "well, it's all behind a firewall" model. It's not very good, but it is easy. I hope that in the next year or two we can get to where I want to be, but I'm also not sure that anyone cares. "Good is the enemy of great" and a private network seems to fit people's mental model pretty well. The only thing that automatic mTLS gets you right now is protection against "SSL added and removed here ;-)", and it's only the NSA that's really doing that. If you aren't a target of the NSA, your data is being exfiltrated by employees with access, or by simple application level bugs (XSRF, SQL injection). This is all very rambly and I'm not sure where I'm going with this... but I wish that someone made application-level transport security work and it was trivial to set up.
bionsystem 7y ago
See, I do infrastructure work in companies that often don't even expose their services. They just want a database and a frontend for a very small internal app that will never get out. And yet, a lot of the tools are designed "for the web" and, rightfully so, enforce https, ssl, certs, you name it. Now the issue is that they are a real PITA to implement in some of those networks, where you have to use tons of private registries and repositories, and to get everything working with certificates. Because those companies are not used to issue certs, often times it take them months to do so, as they have a very rigorous process, so everyone wonders why this tiny app never goes to "production" and why we are blocked by such restrictions. Now I'm not even accounting for the fact that you sometimes actually need to go online to install some of those stuff. Recent example was node-sass, dependancy from a composer package, which you would expect to just install from the composer registry ; but no, it has a setup script that goes to nodejs.org or something, which of course you cannot fake because it will have to check certs as well. So again, something designed for the web that'll never work seemlessly elsewhere. (There are workaround of course but I'd love to spend my time elsewhere). To give you an idea, the cloud I'm working on right now is pretty much entirely offline, for both egress and ingress. There is a nexus mirror-proxy to dockerhub and that's it, even this is in the LAN. So really, having to configure cockroachdb certs (or anything else for that matter, like etcd...) is useless, time consuming, frustrating, and counterproductive.
- dkersten 7y ago
  > very small internal app Is cockroachdb the right DB for this use case, though? A more traditional Postgres/MySQL or evening sqlite (depending on needs) is often plenty good for a small internal app without requiring this extra security > the cloud I’m working on right now is pretty much entirely offline I fee like “cloud” has lost all meaning it may have once had
  - bionsystem 7y ago
    > Is cockroachdb the right DB for this use case, though? A more traditional Postgres/MySQL or evening sqlite (depending on needs) is often plenty good for a small internal app without requiring this extra security It's not, but the app I described is just an example. Some use cases involve bigger dataset and resilience. It's not cockroachdb specifically but any of those "web" tools that, used internally in a closed environment, become a PITA to manage. Language tooling, repositories, dbs... Etcd for example is used as an internal storage for tenants (for example to store DNS information) and the team that manages it cannot upgrade right now because they have to figure out first how to implement ssl. > I fee like “cloud” has lost all meaning it may have once had Well, how would you call an on-prem install of OpenStack ? It's their internal cloud. Cloud has not much to do with the web or with the fact that it is publicly exposed, it's a tool to manage infrastructure. And to be perfectly honest, "cloud" isn't a proper term to begin with. Years ago when the word came out one of my professor said they basically renamed "grid" to "cluster" then "cloud". They used to administer hundreds+ of machines and multiples of that of jobs using pssh. They didn't wait for kube, and they didn't need ssl between each and every API and services. Now don't get me wrong, the public cloud provides a great service and tooling coming out now should be web ready and web safe. Just, please, pretty please, give us a --insecure and --offline even if it means having it all over my ansible code, so we can get the job done without having to spend my days working around those tools.
tyingq 7y ago
One part where it makes sense to me is that CRDB often talks about deploying their product in K8S as a good idea. In that scenario, it's very easy to end up in a situation where CRDB communications are encrypted twice, with separate certificate management systems. Not just the client->db connections, but also the node->node ones. Especially with things like Istio becoming more popular.
pc86 7y ago
I roll my eyes so hard at people who say "well our network is secured, so it can insecure on the inside" I'm surprised I don't have a repetitive stress injury by now. It shows a shocking lack of foresight and is honestly insanely unprofessional. This is the exact same kind of opinions that lead to just about every major data breach in recent memory. Secure everything, and assume that everything is compromised. Anything less is downright negligent.
- wwarner 7y ago
  With all due respect, this is like saying the more locks you have, the more secure your building is. I mean, yes you have to lock some doors to have security, but on the other hand putting a lock on the bathroom door, the light switch and the flush handle isn't increasing security, and could be reducing security. I feel somewhat the opposite -- there should be a very good reason to add security overhead, and without a compelling story the default should be open. This is why i've been so happy reading about wireguard in the kernel recently. This makes it possible to have a very secure, low hassle default that is completely independent of applications.
  - bradstewart 7y ago
    > putting a lock on the bathroom door, the light switch and the flush handle isn't increasing security It certainly is if the thing you're securing is the flush handle itself. Like all security questions, understanding exactly what you need/want to secure and the threat models surrounding it is extremely important.
  - yjftsjthsd-h 7y ago
    We put valuables in a safe even if the door locks.
    - wwarner 7y ago
      The key work is "valuables". You probably wouldn't put your pants in a safe (though I acknowledge that depends on your roommates). I'm saying that going through the motions doesn't make something secure. How many passwords are in slack channels or github repos? Creating a password barrier, but neglecting a secure mechanism for rotating them and distributing them doesn't buy any security.

kccqzy 7y ago

I actually have a similar concern. In a previous job I tried to push for internal services communicating over TLS with a custom certificate authority. It was ultimately not adopted as too much complexity, including operationally when something goes wrong the developer should always be able to issue unencrypted requests from an internal network. It seemed to boil down to removing as many hoops as possible when something terrible happens and automated tools fail and manual recovery is needed.

jupp0r 7y ago

I came here to mention this, too. It's been shown from numerous leaks that defense in depth is a much more effective approach than simply relying on network security. You mention that query response data can be sniffed by an attacker in case networks are compromised. The much worse scenario is that the passwords themselves can be intercepted, giving attackers full access.

fulafel 7y ago
Relying on network security is usually a bad idea even if you don't have defense in depth. (Ie it's less bad to rely on some other one thing)

manigandham 7y ago

CockroachDB annoyingly requires manual certificate creation and distribution to each node, and if you don't set those up then you can't use any authorization at all. It's strange and user-hostile to disable auth completely just because you don't use certs. Also some environments like Kubernetes might already have network encryption (provided by a service mesh) and don't need another layer. Certificates serve both encryption and authentication, but encryption can be done with self-signed certs automatically generated by the nodes upon joining the cluster and authentication can be handled in the joining process by using a token or some other bootstrap value that's easier to manage.

knz42 7y ago
> CockroachDB annoyingly requires manual certificate creation and distribution to each node, and if you don't set those up then you can't use any authorization at all. It's strange and user-hostile to disable auth completely just because you don't use certs. This comment is not any more true as of CockroachDB 20.1 (Upcoming in Q2 2020). This version will enable you to set up your own transport-level security with clients using password authentication to crdb.
- deleted 7y ago
  [deleted]

latch 7y ago

All of our internal communication already is secured via wireguard and communication can only flow through the wireguard interface. Does adding cockroach's own encryption on top of this add anything?

thenewnewguy 7y ago
I am unfamiliar with how setups like these usually work, but it could an active attacker MITM your network?
- vsareto 7y ago
  You wouldn't be able to decrypt the wireguard packets unless you had each party's private keys
fulafel 7y ago
What happens when someone compromises your wireguard overlay network? The cert business is not just about encryption, it's also about authentication over an untrusted network.
- tridentlead 7y ago
  Not much if it’s set up in a standard way because each node will only communicate directly with the node it’s expecting too. The communications are secured with the other node’s public key.
  - fulafel 7y ago
    So if you have 10 boxes interconnected in a full mesh, and one of those boxes gets compromised, then it's all fine even if you have been relying on the wireguard vpn being secure?

fulafel 7y ago

Yep, and besides these concerns, there's the difficulty in modeling the trust relationships and dependencies in this kind of "secure internal network" world view. The relationships are complex and you have low likelihood of your mental model matching the reality, because the relationships are not explicitly configured.

jrockway 7y ago

I am a huge fan of mTLS but it is quite difficult to set up and I'm sort of taking a "wait and see" approach, especially with Google pushing ALTS, which sounds pretty sane. My dream is that you can ignore "VPCs" (i.e. trusted networking, firewalls, etc.) and let applications decide whether or not to allow traffic. "You have presented an incorrect TLS certificate, no response for you." The problem I have is that I'm not sure what method you want to use to inject certificates. The current systems seem to be "something will figure it out for you at runtime, and your application will be none the wiser". (This is things like Istio and Linkerd.) I am not sure I'm convinced this adds much security. Anyone that can talk to the apiserver that injects certificates can probably convince the system to give you a certificate for your rogue container. And, I'm not sure that applications have enough visibility into the networking layer to really add any trust. I'd like for each incoming request to have some provenance information attached with dimensions like "this container was built from trusted source code and each commit that it contains was code reviewed", "this container was launched from a manifest written and applied by a trusted engineer from a trusted machine", and of course "this request was authorized by a human user whose data is being accessed". Right now, I feel like you have to hand-wave to get the first two. Sure, maybe Istio thinks the container is legit, perhaps because it asked a container notary (very early stages of support), and so now it can open a TLS connection to anything in the cluster. And, adding a request-scoped JWT is easy enough to provide the "here is the human user that made the request". I just don't think you can hide this from the application with a service mesh; every application needs to be aware of what's going on so it can say no to suspicious requests. And for that, I don't think we're really there yet. We had this at Google and it worked well, but I haven't found anything that works well in the outside world yet. With that in mind, I totally understand why people trust the "well, it's all behind a firewall" model. It's not very good, but it is easy. I hope that in the next year or two we can get to where I want to be, but I'm also not sure that anyone cares. "Good is the enemy of great" and a private network seems to fit people's mental model pretty well. The only thing that automatic mTLS gets you right now is protection against "SSL added and removed here ;-)", and it's only the NSA that's really doing that. If you aren't a target of the NSA, your data is being exfiltrated by employees with access, or by simple application level bugs (XSRF, SQL injection). This is all very rambly and I'm not sure where I'm going with this... but I wish that someone made application-level transport security work and it was trivial to set up.

bionsystem 7y ago

See, I do infrastructure work in companies that often don't even expose their services. They just want a database and a frontend for a very small internal app that will never get out. And yet, a lot of the tools are designed "for the web" and, rightfully so, enforce https, ssl, certs, you name it. Now the issue is that they are a real PITA to implement in some of those networks, where you have to use tons of private registries and repositories, and to get everything working with certificates. Because those companies are not used to issue certs, often times it take them months to do so, as they have a very rigorous process, so everyone wonders why this tiny app never goes to "production" and why we are blocked by such restrictions. Now I'm not even accounting for the fact that you sometimes actually need to go online to install some of those stuff. Recent example was node-sass, dependancy from a composer package, which you would expect to just install from the composer registry ; but no, it has a setup script that goes to nodejs.org or something, which of course you cannot fake because it will have to check certs as well. So again, something designed for the web that'll never work seemlessly elsewhere. (There are workaround of course but I'd love to spend my time elsewhere). To give you an idea, the cloud I'm working on right now is pretty much entirely offline, for both egress and ingress. There is a nexus mirror-proxy to dockerhub and that's it, even this is in the LAN. So really, having to configure cockroachdb certs (or anything else for that matter, like etcd...) is useless, time consuming, frustrating, and counterproductive.

dkersten 7y ago
> very small internal app Is cockroachdb the right DB for this use case, though? A more traditional Postgres/MySQL or evening sqlite (depending on needs) is often plenty good for a small internal app without requiring this extra security > the cloud I’m working on right now is pretty much entirely offline I fee like “cloud” has lost all meaning it may have once had
- bionsystem 7y ago
  > Is cockroachdb the right DB for this use case, though? A more traditional Postgres/MySQL or evening sqlite (depending on needs) is often plenty good for a small internal app without requiring this extra security It's not, but the app I described is just an example. Some use cases involve bigger dataset and resilience. It's not cockroachdb specifically but any of those "web" tools that, used internally in a closed environment, become a PITA to manage. Language tooling, repositories, dbs... Etcd for example is used as an internal storage for tenants (for example to store DNS information) and the team that manages it cannot upgrade right now because they have to figure out first how to implement ssl. > I fee like “cloud” has lost all meaning it may have once had Well, how would you call an on-prem install of OpenStack ? It's their internal cloud. Cloud has not much to do with the web or with the fact that it is publicly exposed, it's a tool to manage infrastructure. And to be perfectly honest, "cloud" isn't a proper term to begin with. Years ago when the word came out one of my professor said they basically renamed "grid" to "cluster" then "cloud". They used to administer hundreds+ of machines and multiples of that of jobs using pssh. They didn't wait for kube, and they didn't need ssl between each and every API and services. Now don't get me wrong, the public cloud provides a great service and tooling coming out now should be web ready and web safe. Just, please, pretty please, give us a --insecure and --offline even if it means having it all over my ansible code, so we can get the job done without having to spend my days working around those tools.

tyingq 7y ago

One part where it makes sense to me is that CRDB often talks about deploying their product in K8S as a good idea. In that scenario, it's very easy to end up in a situation where CRDB communications are encrypted twice, with separate certificate management systems. Not just the client->db connections, but also the node->node ones. Especially with things like Istio becoming more popular.

pc86 7y ago

I roll my eyes so hard at people who say "well our network is secured, so it can insecure on the inside" I'm surprised I don't have a repetitive stress injury by now. It shows a shocking lack of foresight and is honestly insanely unprofessional. This is the exact same kind of opinions that lead to just about every major data breach in recent memory. Secure everything, and assume that everything is compromised. Anything less is downright negligent.

wwarner 7y ago
With all due respect, this is like saying the more locks you have, the more secure your building is. I mean, yes you have to lock some doors to have security, but on the other hand putting a lock on the bathroom door, the light switch and the flush handle isn't increasing security, and could be reducing security. I feel somewhat the opposite -- there should be a very good reason to add security overhead, and without a compelling story the default should be open. This is why i've been so happy reading about wireguard in the kernel recently. This makes it possible to have a very secure, low hassle default that is completely independent of applications.
- bradstewart 7y ago
  > putting a lock on the bathroom door, the light switch and the flush handle isn't increasing security It certainly is if the thing you're securing is the flush handle itself. Like all security questions, understanding exactly what you need/want to secure and the threat models surrounding it is extremely important.
- yjftsjthsd-h 7y ago
  We put valuables in a safe even if the door locks.
  - wwarner 7y ago
    The key work is "valuables". You probably wouldn't put your pants in a safe (though I acknowledge that depends on your roommates). I'm saying that going through the motions doesn't make something secure. How many passwords are in slack channels or github repos? Creating a password barrier, but neglecting a secure mechanism for rotating them and distributing them doesn't buy any security.