How to Secure a Kubernetes Cluster with VPC Gen2

6 min readJan 30, 2021

Security is an important step and air gapping your cluster is a crucial first step in securing your cluster, network and containers.

This tutorial explains how to secure a Kubernetes cluster and physically isolate your cluster using a Virtual Private Cloud (VPC). You can secure your cluster by creating a Virtual Private Cloud (VPC) and add rules to a Security Group of an Application Load Balancer (ALB) to allow certain inbound traffic where needed and expose the gateway to manage traffic, for instance by using OAuth, rate limiting and API key access control.

With IBM Cloud Kubernetes Service (IKS) on VPC Generation 2 Computeon IBM Cloud, you can create a Kubernetes cluster on a Virtual Private Cloud (VPC) infrastructure in a matter of minutes.

Steps:

Create a VPC with new subnet
Create a Cluster with VPC Generation 2 Compute
Create a Kubernetes Cluster
Deploy the Guestbook Application,
Update the Security Group
Understanding the Load Balancer
Cleanup

Create a VPC with new subnet

This section uses the browser to create a VPC with new subnet. Navigate to https://cloud.ibm.com/vpc-ext/provision/vpc to provision a new virtual private cloud.

This will setup the following infrastructure for you,

a Virtual Private Cloud, watch this good primer on VPC by Ryan Sumner,
a Subnet,
a Security Group,
an Access Control List (ACL),
a Public Gateway with a Floating IP for Public Access,
an encrypted VPN Gateway or a Direct Link Private Circuits for Private Access,
an elastic Load Balancer for VPC, using an Application Load Balancer (ALB), which includes Sysdig monitoring, Hihg Availability (HA) with a Domain Name Server (DNS), Multi-Zone Region (MZR) support, L4 and L7 support for OSI network and application layer, both public and private load balancing.

Create a Cluster with VPC Generation 2 Compute

Using the UI, create an IBM Cloud Kubernetes Service (IKS), for infrastructure provider choose Generation 2 Compute instead of Classic. For a comparison of infrastructure providers, read Supported Infrastructure Providers, or read an Overview of VPC Networking in IBM Cloud Kubernetes Service (IKS).

Deploy the Guestbook Application

This section uses the IBM Cloud CLI. Log in to the IBM Cloud account, if you use a Single Sign-On (SSO) provider, use the --sso flag instead of the username flag.

ibmcloud login -u $IBMID [--sso]

Target the region where you have created your VPC environment. The resource group flag is optional, if you happen to know it, you can target it explicitly.

ibmcloud target -r $MY_REGION [-g <resource_group>]

Target generation 2 infrastructure.

ibmcloud is target --gen 2

Connect to your cluster to set the current-context,

ibmcloud ks cluster config --cluster $MY_CLUSTER_IDkubectl config current-context

Deploy the guestbook application,

kubectl create namespace $MY_NAMESPACEkubectl create deployment guestbook --image=ibmcom/guestbook:v1 -n $MY_NAMESPACEkubectl expose deployment guestbook --type="LoadBalancer" --port=3000 --target-port=3000 -n $MY_NAMESPACEkubectl get svc -n $MY_NAMESPACENAMESVC_EXTERNAL_IP=$(kubectl get svc guestbook -n $MY_NAMESPACE --output json | jq -r '.status.loadBalancer.ingress[0].hostname')echo $SVC_EXTERNAL_IPSVC_NODEPORT=$(kubectl get svc guestbook -n $MY_NAMESPACE --output json | jq -r '.spec.ports[0].nodePort')echo $SVC_NODEPORTSVC_PORT=$(kubectl get svc guestbook -n $MY_NAMESPACE --output json | jq -r '.spec.ports[0].port')echo $SVC_PORT

Try to send a request to the guestbook application

curl http://$SVC_EXTERNAL_IP:$SVC_PORT

You should now be able to see the HTML response object from the Guestbook application. Open the guestbook URL in a browser to review the web page.

You will see the below error if you are using the IKS 1.18 version

curl http://$SVC_EXTERNAL_IP:$SVC_PORT curl: (52) Empty reply from server

Update the Security Group (Skip this section for 1.19)

To allow any traffic to applications that are deployed on your cluster’s worker nodes, you have to modify the VPC’s default security group by ID.

Update security group and add an inbound rule for NodePort of the service you created when exposing the guestbook deployment.

Or add a security group rule to allow inbound TCP traffic on all Kubernetes ports in the range of 30000–32767.

Try again to reach the guestbook application,

curl http://$SVC_EXTERNAL_IP:$SVC_PORT

You should now be able to see the HTML response object from the Guestbook application. Open the guestbook URL in a browser to review the web page.

Understanding the Load Balancer

Right now, the most important thing is that our tutorial works. Oof. You secured your Kubernetes cluster with a Virtual Private Cloud (VPC) and blocking direct access to your cluster.

To better understand what exactly happened in the end, especially to see why we are using the port of the Service object instead of the NodePortwhich is the port we allowed inbound traffic on, let’s inspect the Service resource of the Guestbook deployment in more detail.

kubectl get svc -n $MY_NAMESPACE --output json

Note:

Service resource lists a property status.loadBalancer.ingress with hostname. We used the hostname to access the Guestbook Service.
spec.ports.port is the port on which the service listens for external network traffic.
externalTrafficPolicy property is set to Cluster, which always routes traffic to all pods running a service with equal distribution.
we used a Service type of LoadBalancer.

Ingress Application Load Balancer (ALB)

Ingress is a Kubernetes service that balances network traffic workloads in your cluster by forwarding public or private requests to your apps by using a unique public or private route. The Ingress application load balancer (ALB) is a layer 7 load balancer which implements the NGINX Ingress controller. A layer 4 LoadBalancer service exposes the ALB so that the ALB can receive external requests that come into your cluster.

Ingress consists of three components:

Ingress resources,
Application load balancers (ALBs),
A load balancer to handle incoming requests across zones. For classic clusters, this component is the Multi-Zone Load Balancer (MZLB) that IBM Cloud Kubernetes Service creates for you. For VPC clusters, this component is the VPC load balancer created in your VPC.

To expose an app by using Ingress, you must create a Kubernetes service for your app and register this service with Ingress by defining an Ingress resource. The Ingress resource is a Kubernetes resource that defines the rules for how to route incoming requests for apps.

The application load balancer (ALB) is an external load balancer that listens for incoming HTTP, HTTPS, or TCP service requests. The ALB then forwards requests to the appropriate app pod according to the rules defined in the Ingress resource. The services that expose your ALBs on public or private IP addresses are formatted as public-abc12 in the kube-system namespace.

The following diagram shows how Ingress directs communication from the internet to an app in a VPC multizone cluster.

source: https://cloud.ibm.com/docs/containers?topic=containers-ingress-about#architecture-vpc

Based on the resolved IP address, the VPC load balancer sends the request to an ALB. The ALB checks if routing rules exist for the application.

Our cluster has 1 worker node. There is a private and a public Ingress ALB installed, the private Ingress ALB is disabled.

ibmcloud ks ingress alb ls -c $MY_CLUSTER_NAME

List the VPC Load Balancers that were created,

ibmcloud is load-balancers

From the Kubernetes docs about External Load Balancer Providers you see that:

When the Service type is set to LoadBalancer, Kubernetes provides functionality equivalent to type equals ClusterIP to pods within the cluster and extends it by programming the (external to Kubernetes) load balancer with entries for the Kubernetes pods. The Kubernetes service controller automates the creation of the external load balancer, health checks (if needed), firewall rules (if needed) and retrieves the external IP allocated by the cloud provider and populates it in the service object.

Cleanup

Delete your Kubernetes cluster,

ibmcloud ks cluster rm --cluster $MY_CLUSTER_NAME

Delete the Gateways, Load Balancers, Network Interfaces, subnet, public gateways, and finally delete the VPC,

ibmcloud is vpn-gateways
ibmcloud is vpn-gateway-delete $vpnid
ibmcloud is load-balancers
ibmcloud is load-balancer-delete $lbid
ibmcloud is instance-network-interfaces $vsi
ibmcloud is instances
ibmcloud is instance-delete $vsiibmcloud is subnets
ibmcloud is subnet-delete $subnetibmcloud is public-gateways
ibmcloud is public-gateway-delete $gatewayibmcloud is vpcs
ibmcloud is vpc-delete $vpc

Cheers! Check out my page for more stories :)