4 ms·

I personally secure my postgres instances by putting them in a silod vpc without internet access. I then limit incoming connections to only be allowed from the

I personally secure my postgres instances by putting them in a silod vpc without internet access. I then limit incoming connections to only be allowed from the separate vpc containing the application server I then use a bastion host when I need to access ssh on the instance. The bastion host remains off and inaccessible except for when I need to perform maintenance. The advantages of this is that there is no "always-open" access to the instance. Not sure why the author does not advocate this.

polskibus 6y ago
How do you monitor node health?
- duckfang 6y ago
  You can put other machines with a highly restrictive set of network rules that allow data to cross inside the network and outside the network in very controlled ways. Email is one such way.
  - capableweb 6y ago
    > Email is one such way. I'm sorry but could you clarify? You mean that you monitor and/or collect data from hosts inside of a network via email somehow?
  - abhishekjha 6y ago
    I am curious. Most of the monitoring stacks include something like telegram/Prometheus to collect metrics and send to influxdb. How would you do this via email?
    - duckfang 6y ago
      Inside the he bastion network/VPC, there can be logging, monitoring, ad/kerb+openldap stuff to do the requisite thing. And your logging and monitoring can send trigger emails when they catch bad things. Your inbox needs to be an automatable inbox, say controlled by bleeder or another flowengine. From there you can build a full messaging dashboard and have it done in something that could pass a very stringent audit.
deleted 6y ago
[deleted]
duckfang 6y ago
Yep, and your answer is the better default answer. Bastion<->app host<->DB Bastion can talk to the net. App host can talk outbound but inbound only accepts bastion and DB. DB can only talk with app host. Obviously, you harden everything appropriately... But with this arrangement, it's very difficult to penetrate this sort of network. Think of it as a network that as a whole is default-deny.
chatmasta 6y ago
> Not sure why the author does not advocate this. Probably because the blog post is an advertisement for their product, which already allows you to implement bastion hosts as you describe. From the bottom: > Databases do not need to be exposed on the public Internet and can safely operate in air-gapped environments using Teleport’s built-in reverse tunnel subsystem.
- greatjack613 6y ago
  Did not catch that. Sneaky self promotion there :) Not sure what there reverse tunnel product is, but a bastion host is super easy to implement, just spin up an ec2 and walla. Curious as to what value they are providing
  - chatmasta 6y ago
    They've been around a while, it actually seems quite cool. Bastion hosts are simple to setup, sure, but Teleport adds a whole bunch of porcelain on top, e.g. integration with SSO, web UI for administration, etc. Haven't used them myself but I wouldn't be against trying it if in the market for something like that.
    - duckfang 6y ago
      I find that porcelain has a very bad time in breaking when you least suspect it. The same is true for server applications that have weird 3rd party dependencies that may go down when you least suspect it.
  - jdmichal 6y ago
    Unsure whether you meant walla, but just in case this is what you meant: It's "voila", from French "et violà". https://www.collinsdictionary.com/dictionary/french-english/et-voil%C3%A0 https://www.collinsdictionary.com/dictionary/french-english/...
zomgwat 6y ago
One alternative to keeping the bastion off is to restrict access to the bastion by IP address. Ideally, the list of IP addresses that need access to the bastion should be relatively small. In cloud environments, it's straightforward to update network firewall rules as IP addresses change. Residential and office IP addresses don't change much so it's not much of a hassle in my experience. That said, it can get annoying if you find your self working on a network that rotates your IP address frequently (e.g., a hotspot).
brightball 6y ago
Agreed. What would be the best way when using something like Heroku?
- fmajid 6y ago
  Something like Nebula (https://github.com/slackhq/nebula https://github.com/slackhq/nebula) or Tailscale, perhaps?
  - brightball 6y ago
    Can you setup Nebula on your Heroku dynos and an RDS instance?
120bits 6y ago
Currently I have a postgres/timescaledb running on EC2 in VPC which has no internet access. I do VPN tunnel to access private local subnet and have security group settings that allows my and 1 more machine access only. I usually have a jump server that I use, but I don't want to keep my ssh keys on the server or leave it behind.
- clan 6y ago
  If the jumpserver is trusted and controlled by you then you should have a look at ssh agent forwarding. Then you avoid leaving keys on the jumphost.
  - ylk 6y ago
    A better alternative would be ProxyJump. See e.g. https://serverfault.com/questions/958222/bastion-server-use-tcp-forwarding-vs-placing-private-key-on-server https://serverfault.com/questions/958222/bastion-server-use-... Edit: To add some details - using ProxyJump you don’t have to expose anything to the jump host and instead just proxy through it.
lovedswain 6y ago
What value is added by using a separate VPC? Equivalent restrictions can more easily be done with security groups, including on the outbound networking
- cj 6y ago
  I would suggest considering segregation by subnets (in addition to security groups), using public / private subnets, where any server in a private subnet (behind a nat gateway) doesn't/can't have a public ip address, and therefore cannot be accessed via the public internet. You could then use a bastion to access servers in the private subnet, or use something like AWS Session Manager which provides command line access via web browser in lieu of a bastion.
  - lovedswain 6y ago
    > What value is added by using a separate VPC? Adding more mechanisms on top is pointless when the effort could be invested in, for example, automated auditing of SGs, which is vastly more potent from a hardening perspective than adding additional layers of technical redundancy that are still exposed to the same flawed human processes. When you reach a team of 10-20 folk on a project, stuff tends to get confusing and/or lazy with elaborate configurations. Security design therefore is about more about managing that outcome through simplicity and process hardening than.. well.. I don't even know what threats a separate VPC protects against
    - cj 6y ago
      > When you reach a team of 10-20 folk on a project, stuff tends to get confusing From the perspective of maintaining security in the least confusing way possible in a team setting, I could see a scenario where you have a VPC that only has private subnets, and no public subnets at all. You could call it “Database VPC” and assuming your team doesn’t reconfigure the VPC to add public subnets, you can be comfortable with your team adding more EC2 servers / databases / whatever in that VPC since they would all be in private subnets inaccessible by the public internet. And then you could have a 2nd VPC with private/public subnets that are less locked down than the database VPC, which might contain your load balancers, application servers, etc. I suppose the benefit would be the logical separation of databases into a VPC without any public subnets. And the only way to gain access to subnets in that VPC would be through VPC peering. (The above assumes you’re using AWS). Although after typing all that, I still think you can accomplish a comparably secure (on a network level) architecture using security groups, or even separate subnets, without separate VPCs. In general I try to avoid multiple VPCs because VPC peering in AWS can get tricky (or impossible if both VPCs have overlapping CIDR blocks).
moltar 6y ago
I use almost the same approach. I also limit access to just one IP, which is a VPN server hosted on a different cloud provider. You can run a bulletproof VPN server easily using something like algo (https://github.com/trailofbits/algo https://github.com/trailofbits/algo)

polskibus 6y ago

How do you monitor node health?

duckfang 6y ago
You can put other machines with a highly restrictive set of network rules that allow data to cross inside the network and outside the network in very controlled ways. Email is one such way.
- capableweb 6y ago
  > Email is one such way. I'm sorry but could you clarify? You mean that you monitor and/or collect data from hosts inside of a network via email somehow?
- abhishekjha 6y ago
  I am curious. Most of the monitoring stacks include something like telegram/Prometheus to collect metrics and send to influxdb. How would you do this via email?
  - duckfang 6y ago
    Inside the he bastion network/VPC, there can be logging, monitoring, ad/kerb+openldap stuff to do the requisite thing. And your logging and monitoring can send trigger emails when they catch bad things. Your inbox needs to be an automatable inbox, say controlled by bleeder or another flowengine. From there you can build a full messaging dashboard and have it done in something that could pass a very stringent audit.

deleted 6y ago

[deleted]

duckfang 6y ago

Yep, and your answer is the better default answer. Bastion<->app host<->DB Bastion can talk to the net. App host can talk outbound but inbound only accepts bastion and DB. DB can only talk with app host. Obviously, you harden everything appropriately... But with this arrangement, it's very difficult to penetrate this sort of network. Think of it as a network that as a whole is default-deny.

chatmasta 6y ago

> Not sure why the author does not advocate this. Probably because the blog post is an advertisement for their product, which already allows you to implement bastion hosts as you describe. From the bottom: > Databases do not need to be exposed on the public Internet and can safely operate in air-gapped environments using Teleport’s built-in reverse tunnel subsystem.

greatjack613 6y ago
Did not catch that. Sneaky self promotion there :) Not sure what there reverse tunnel product is, but a bastion host is super easy to implement, just spin up an ec2 and walla. Curious as to what value they are providing
- chatmasta 6y ago
  They've been around a while, it actually seems quite cool. Bastion hosts are simple to setup, sure, but Teleport adds a whole bunch of porcelain on top, e.g. integration with SSO, web UI for administration, etc. Haven't used them myself but I wouldn't be against trying it if in the market for something like that.
  - duckfang 6y ago
    I find that porcelain has a very bad time in breaking when you least suspect it. The same is true for server applications that have weird 3rd party dependencies that may go down when you least suspect it.
- jdmichal 6y ago
  Unsure whether you meant walla, but just in case this is what you meant: It's "voila", from French "et violà". https://www.collinsdictionary.com/dictionary/french-english/et-voil%C3%A0 https://www.collinsdictionary.com/dictionary/french-english/...

zomgwat 6y ago

One alternative to keeping the bastion off is to restrict access to the bastion by IP address. Ideally, the list of IP addresses that need access to the bastion should be relatively small. In cloud environments, it's straightforward to update network firewall rules as IP addresses change. Residential and office IP addresses don't change much so it's not much of a hassle in my experience. That said, it can get annoying if you find your self working on a network that rotates your IP address frequently (e.g., a hotspot).

brightball 6y ago

Agreed. What would be the best way when using something like Heroku?

fmajid 6y ago
Something like Nebula (https://github.com/slackhq/nebula https://github.com/slackhq/nebula) or Tailscale, perhaps?
- brightball 6y ago
  Can you setup Nebula on your Heroku dynos and an RDS instance?

120bits 6y ago

Currently I have a postgres/timescaledb running on EC2 in VPC which has no internet access. I do VPN tunnel to access private local subnet and have security group settings that allows my and 1 more machine access only. I usually have a jump server that I use, but I don't want to keep my ssh keys on the server or leave it behind.

clan 6y ago
If the jumpserver is trusted and controlled by you then you should have a look at ssh agent forwarding. Then you avoid leaving keys on the jumphost.
- ylk 6y ago
  A better alternative would be ProxyJump. See e.g. https://serverfault.com/questions/958222/bastion-server-use-tcp-forwarding-vs-placing-private-key-on-server https://serverfault.com/questions/958222/bastion-server-use-... Edit: To add some details - using ProxyJump you don’t have to expose anything to the jump host and instead just proxy through it.

lovedswain 6y ago

What value is added by using a separate VPC? Equivalent restrictions can more easily be done with security groups, including on the outbound networking

cj 6y ago
I would suggest considering segregation by subnets (in addition to security groups), using public / private subnets, where any server in a private subnet (behind a nat gateway) doesn't/can't have a public ip address, and therefore cannot be accessed via the public internet. You could then use a bastion to access servers in the private subnet, or use something like AWS Session Manager which provides command line access via web browser in lieu of a bastion.
- lovedswain 6y ago
  > What value is added by using a separate VPC? Adding more mechanisms on top is pointless when the effort could be invested in, for example, automated auditing of SGs, which is vastly more potent from a hardening perspective than adding additional layers of technical redundancy that are still exposed to the same flawed human processes. When you reach a team of 10-20 folk on a project, stuff tends to get confusing and/or lazy with elaborate configurations. Security design therefore is about more about managing that outcome through simplicity and process hardening than.. well.. I don't even know what threats a separate VPC protects against
  - cj 6y ago
    > When you reach a team of 10-20 folk on a project, stuff tends to get confusing From the perspective of maintaining security in the least confusing way possible in a team setting, I could see a scenario where you have a VPC that only has private subnets, and no public subnets at all. You could call it “Database VPC” and assuming your team doesn’t reconfigure the VPC to add public subnets, you can be comfortable with your team adding more EC2 servers / databases / whatever in that VPC since they would all be in private subnets inaccessible by the public internet. And then you could have a 2nd VPC with private/public subnets that are less locked down than the database VPC, which might contain your load balancers, application servers, etc. I suppose the benefit would be the logical separation of databases into a VPC without any public subnets. And the only way to gain access to subnets in that VPC would be through VPC peering. (The above assumes you’re using AWS). Although after typing all that, I still think you can accomplish a comparably secure (on a network level) architecture using security groups, or even separate subnets, without separate VPCs. In general I try to avoid multiple VPCs because VPC peering in AWS can get tricky (or impossible if both VPCs have overlapping CIDR blocks).

moltar 6y ago

I use almost the same approach. I also limit access to just one IP, which is a VPN server hosted on a different cloud provider. You can run a bulletproof VPN server easily using something like algo (https://github.com/trailofbits/algo https://github.com/trailofbits/algo)