There are several benefits to leveraging private sub-networks as part of your deployment infrastructure. We work with AWS, so I’ll speak to that specifically. Servers launched in private sub-networks have no public IP address, so they may be less prone to attack. They also maintain their private IP address even if stopped and restarted (unlike EC2 public IP addresses). Thus you can maintain IP address reliability without needing to use one of your limited elastic IP addresses.

When server resources don’t have a public IP address, communicating via HTTP(S) or SSH becomes more complicated. If using AWS, HTTP(S) traffic can be routed to servers on private subnets by using Route 53 or Elastic Load Balancers (or both). Direct connection via SSH to private sub-network resources, however, is not possible. This presents a problem, because provisioning and configuration of servers is often executed via SSH. The solution is to route SSH traffic through a “bastion” host on an companion public network.

A “bastion” or gateway host is simply a server that sits on the public network and acts as a hub through which you SSH to all of your servers on the companion private network. Since the bastion is on the public network and has a public IP address it is more open to attack; but it can be fortified or hardened (e.g., white-listing IP addresses); it is also never loaded with any critical resources (e.g., your data). You simply use it as a gateway to other critical IT resources. It’s far easier to fortify and maintain one gateway than fortify each and every server in your deployment infrastructure.

Here’s a simplified diagram for the suggested architecture:

You can lock this down by setting specific inbound and outbound network traffic rules on SSH port 22 (e.g. AWS security groups) for all servers. The bastion server should receive inbound traffic on port 22 from the IP of your workstation and should only allow port 22 traffic to the private IP addresses of servers in the private sub-network. Those servers, in turn, should only receive incoming traffic on port 22 from the IP of the bastion server.

Configuring SSH session to pass through a bastion server

If you have all the port settings noted above implemented correctly, the only remaining part of tunneling through a bastion server is properly configuring the SSH session on your local workstation. Create an ssh.cfg file like so:

#######################################
# bastion server on public sub-network
#######################################
Host <bastion-server-publid-ip>
  User <ssh-user-on-bastion-server>
  IdentityFile <path/to/private-key/for/bastion-server>
  PasswordAuthentication no
  ForwardAgent yes
  ServerAliveInterval 60
  TCPKeepAlive yes
  ControlMaster auto
  ControlPath ~/.ssh/%r@%h:%p
  ControlPersist 15m
  ProxyCommand none
 
###################################
# web server on private sub-network
###################################
Host <private-ip-address-for-web-host>
  User <ssh-user-on-web-server>
  IdentityFile <path/to/private-key/for/web-server>
  ServerAliveInterval 60
  TCPKeepAlive yes
  ProxyCommand ssh -q -A -F <path/to/this/ssh.cfg>  <ssh-user-on-bastion-server>@<bastion-server-publid-ip> nc %h %p
  

If you want to SSH the web-server in this example, you simple do something like this:

$ ssh -F <path/to/ssh.cfg> <ssh-user-on-web-server>@<web-server-private-ip> 
  

Given the above command, the ssh.cfg is read for the host associated with the web-server ip. Since we have defined a ProxyCommand with instructions to first start an SSH session with the bastion, that happens first; but after connecting, you will automatically start a second SSH session with the targeted private network server. The beauty of this is that the private key for the actual target server is forwarded to the bastion with the initial SSH command, so there is no need to store that on the bastion host itself.

Bastion hosts are useful if you have a complex IT infrastructure composed on multiple servers which exist on a private network. In addition to typical SSH communication from workstation to target server, they can also be leveraged during server provisioning and other DevOps scripting. I’ll have a follow-up post on using a bastion host with Ansible playbooks.