6 ms·

Security folks, which would you feel more comfortable with?

Hi all, I work at a SaaS company that needs to securely connect our cloud control plane to customer on-premise infrastructure in order to run orchestration and automation tasks. We’re trying to avoid requiring customers to open inbound firewall rules or stand up full VPNs. We’ve narrowed it down to two models: Agent-based HTTPS/mTLS connector * Customer deploys a small VM/Pod (our agent) inside their environment. * The agent makes an outbound TLS connection (443) to our SaaS, authenticates with mTLS, polls for jobs, and executes them locally. * Simple setup (firewall-friendly, “just outbound HTTPS”), similar to how Datadog agents, GitHub Actions runners, or Terraform Cloud Agents work. WireGuard-based connector * Customer deploys the same kind of connector, but instead of plain HTTPS, it establishes a WireGuard tunnel back to our cloud. * Provides a stable overlay /32 per connector, potentially lower latency, and allows us to send jobs and receive results over a secure tunnel. * Requires outbound UDP (or TCP fallback with something like Tailscale/Netbird). * More networking moving parts, but possibly a more robust transport. We want to balance security posture, customer comfort during security review, and ease of deployment. From your perspective (especially those who review SaaS vendors for security), which approach would give you more confidence, and why? Thanks!

NoahZuniga 1y ago
I would use an existing product, maybe something like cloudflare tunnels.
xyzzy123 1y ago
Hi, I have worked on security teams, been on both sides of supplier assessments and built infrastructure / deployed vendor agents. Both of these options (assuming best practice configuration) would be acceptable to any security team I've worked on based on the fundamentals. But the limitations of corporate environments and processes mean that one is much easier than the other. Practically speaking, supplier assessment forms usually have standard questions for TLS, endpoints, settings / ciphers etc. The paperwork is more complicated for WireGuard because there are no "boxes" for it, you have to write out lengthy explanations into spreadsheet cells / forms. Often the people filling out the forms are the project team who are not networking experts and they have trouble figuring out what to put where. It produces a small feeling of discomfort for people filling out and reviewing the form because it does not look "completely standard". Since WireGuard is a tunnel, an "alert" person might also ask "oh, what protocols are going over that tunnel and to where? How are THEY secured?" which is possibly a whole other set of questions that never seem to come up with TLS. With my deployer hat on - TLS is the happy path. As a deployer, you often don't "own" the network, that's a separate team. These teams (and their equipment) very greatly in their capability, sometimes they are excellent, sometimes they are dysfunctional. There is usually a standard process or form for "TLS egress to XYZ" - but anything that is NOT this will trigger a complicated workflow requiring actual meetings and people looking at architectures and diagrams etc. You will basically find it "impossible" to deploy into some customer environments if you don't go with TLS, that is the bitter reason why everyone uses it despite other things being "better" along certain dimensions. Even mTLS or non-public CAs won't work "out of the box" if there is a middlebox that does inspection by terminating and re-establishing the TLS connection. There is usually a ticket / standard process to request a bypass for this this - but some customers will need hand-holding through that. That (plus the practical difficulties of terminating it in cloud on the SaaS end, which are mostly solved in 2025 fortunately) is a reason even mTLS is not more widely used. Imagine that the person deploying a pod into k8s, the team managing the cluster, the 2 or 3 teams managing the cloud environment, the team doing the supplier assessment and the team managing the network are all different, don't understand each other's "layers" or job functions well, and often fail to communicate with each other. The interface between some teams might mainly be a ticketing system. Making the deployment work at all in these situations is a challenge, and anything "non-standard" is something that needs to be communicated seamlessly across all these layers.
- hellosecpeeps 1y ago
  I really appreciate the reply - everything you mentioned basically confirms my suspicions and gives us some good pointers to look at next. Is my interpretation of your message correct in that, even in the HTTPS/TLS-based agent approach, that mTLS might be hard to achieve in practice? I am trying to guage if the absense of mTLS would be something that would frequently be nit-picked in security reviews. My first-hand experience reviewing vendor agents (Newrelic, Datadog, AWS SSM, Azure Connected Machine etc) tells me probably not as its generally not default-required in those agents, but just looking to crowd-source some other opinions. Another value prop of TLS-based agents appears to be that it allows customers to potentially keep the credentials to systems we are automating inside the agent config, as opposed to us needing to store those credentials in our SaaS platform. Wondering if you have any thoughts on this topic as well. Lastly, are you aware of any open-sourced python-based agents that could serve as a good reference point in terms of best practices? Thanks again!
  - xyzzy123 1y ago
    Thanks. What I've seen is that organisations are focused on process compliance, not technical security. As long as people can tick the boxes in standard supplier assessment forms (which you can do with plain TLS, not even mTLS), you can be successful. The question is, does the product do something that people desperately want? They will find a way to fill out the forms / justify the use of the product if it does. It can actually make the process more difficult if you are very different from your competitors in your security strategy, people have to expend effort to understand why you are different. For mTLS specifically, it would be great as an option / thing that customers can configure if they want to go above and beyond, but may cause difficulties for many customer(s) as a requirement. As you observe, comparable vendors are generally plain TLS with a license / token / secret.

NoahZuniga 1y ago

I would use an existing product, maybe something like cloudflare tunnels.

xyzzy123 1y ago

Hi, I have worked on security teams, been on both sides of supplier assessments and built infrastructure / deployed vendor agents. Both of these options (assuming best practice configuration) would be acceptable to any security team I've worked on based on the fundamentals. But the limitations of corporate environments and processes mean that one is much easier than the other. Practically speaking, supplier assessment forms usually have standard questions for TLS, endpoints, settings / ciphers etc. The paperwork is more complicated for WireGuard because there are no "boxes" for it, you have to write out lengthy explanations into spreadsheet cells / forms. Often the people filling out the forms are the project team who are not networking experts and they have trouble figuring out what to put where. It produces a small feeling of discomfort for people filling out and reviewing the form because it does not look "completely standard". Since WireGuard is a tunnel, an "alert" person might also ask "oh, what protocols are going over that tunnel and to where? How are THEY secured?" which is possibly a whole other set of questions that never seem to come up with TLS. With my deployer hat on - TLS is the happy path. As a deployer, you often don't "own" the network, that's a separate team. These teams (and their equipment) very greatly in their capability, sometimes they are excellent, sometimes they are dysfunctional. There is usually a standard process or form for "TLS egress to XYZ" - but anything that is NOT this will trigger a complicated workflow requiring actual meetings and people looking at architectures and diagrams etc. You will basically find it "impossible" to deploy into some customer environments if you don't go with TLS, that is the bitter reason why everyone uses it despite other things being "better" along certain dimensions. Even mTLS or non-public CAs won't work "out of the box" if there is a middlebox that does inspection by terminating and re-establishing the TLS connection. There is usually a ticket / standard process to request a bypass for this this - but some customers will need hand-holding through that. That (plus the practical difficulties of terminating it in cloud on the SaaS end, which are mostly solved in 2025 fortunately) is a reason even mTLS is not more widely used. Imagine that the person deploying a pod into k8s, the team managing the cluster, the 2 or 3 teams managing the cloud environment, the team doing the supplier assessment and the team managing the network are all different, don't understand each other's "layers" or job functions well, and often fail to communicate with each other. The interface between some teams might mainly be a ticketing system. Making the deployment work at all in these situations is a challenge, and anything "non-standard" is something that needs to be communicated seamlessly across all these layers.

hellosecpeeps 1y ago
I really appreciate the reply - everything you mentioned basically confirms my suspicions and gives us some good pointers to look at next. Is my interpretation of your message correct in that, even in the HTTPS/TLS-based agent approach, that mTLS might be hard to achieve in practice? I am trying to guage if the absense of mTLS would be something that would frequently be nit-picked in security reviews. My first-hand experience reviewing vendor agents (Newrelic, Datadog, AWS SSM, Azure Connected Machine etc) tells me probably not as its generally not default-required in those agents, but just looking to crowd-source some other opinions. Another value prop of TLS-based agents appears to be that it allows customers to potentially keep the credentials to systems we are automating inside the agent config, as opposed to us needing to store those credentials in our SaaS platform. Wondering if you have any thoughts on this topic as well. Lastly, are you aware of any open-sourced python-based agents that could serve as a good reference point in terms of best practices? Thanks again!
- xyzzy123 1y ago
  Thanks. What I've seen is that organisations are focused on process compliance, not technical security. As long as people can tick the boxes in standard supplier assessment forms (which you can do with plain TLS, not even mTLS), you can be successful. The question is, does the product do something that people desperately want? They will find a way to fill out the forms / justify the use of the product if it does. It can actually make the process more difficult if you are very different from your competitors in your security strategy, people have to expend effort to understand why you are different. For mTLS specifically, it would be great as an option / thing that customers can configure if they want to go above and beyond, but may cause difficulties for many customer(s) as a requirement. As you observe, comparable vendors are generally plain TLS with a license / token / secret.