One of the basics of running Kubernetes in a production environment is security—how to ensure container images, pods, specific microservices, and the entire Kubernetes cluster, are all locked down and protected against cyber attacks.
What is Kubernetes Security?
- Cloud (or Corporate Datacenter/Colocation facility)—the fundamental physical infrastructure is the foundation of Kubernetes security. Organizations should uphold best practices, whether the cluster is created by the organization or by a managed service provider, and whether it is hosted in the organization's data center or in the cloud.
- Cluster—securing a Kubernetes cluster involves securing all the applications running in the cluster and the configurable components, mainly the Kubernetes API. Given that most cloud-native applications are built around APIs and microservices, applications are as secure as the most vulnerable element out of all the services that comprise them.
- Container—best practices for container design include starting with a minimal container image (excluding unneeded functions or libraries), avoiding unnecessary privileges to users in the container, and scanning container images for vulnerabilities during build time.
- Code—source code is another attack surface for all Kubernetes environments. A first step to addressing vulnerabilities is to test for code quality and security flaws. In addition, you can use policies such as closing unused ports and encrypting traffic using TLS, to limit the impact of vulnerable code.
Security Essentials for a Kubernetes Developer
Securing a Cluster
If you manage your own cluster, you’ll need to restrict access to it. The first step is to restrict access to kubectl, the command-line tools used to control a Kubernetes cluster. Every request to kubectl must be authenticated and validated, either through token-based authentication or via a public-private key pair.
You can implement more advanced access models using
Keep in mind that Kubernetes RBAC supports Normal roles, which are limited to a specific namespace in the Kubernetes cluster, and Cluster roles that have access to all namespaces. Use Cluster roles sparingly because they can provide broad access to your cluster if compromised by an attacker.
Security Hygiene
Here are a few ways you can practice security hygiene in a Kubernetes cluster:
- System updates—you should regularly update Kubernetes. Because it is an open-source project, security issues and vulnerabilities are discovered quite regularly. Updating Kubernetes is complex, especially in a production cluster. Managed Kubernetes services can automate and significantly ease this process.
- Minimal operating system—it is desirable to use a lightweight image and minimal operating system for your Kubernetes nodes, in order to limit the attack surface.
- Minimal identity and access management (IAM) roles—Kubernetes is often managed in the cloud, and permissions to cloud resources are defined using IAM roles. Ensure that each IAM role only grants permission to the minimal resources needed by the role owners. This is the second layer of permissions, managed at the cloud provider level, while RBAC permissions are managed at the cluster or namespace level.
Securing workloads
- Workload security—ensure you are aware of what container engines are running on your Kubernetes nodes. These could be Docker, CRI-O, Containerd, or others. Each container engine has its own security best practices. In addition, pay close attention to applications running on the Kubernetes cluster and ensure those applications are updated, have all security patches applied, and have strong authentication.
- Declarative configuration—Kubernetes uses declarative configuration, which is defined using YAML files, Helm charts, or various templating technologies. Declarative configuration is very sensitive because it defines how workloads should run and provides credentials for sensitive systems. Ensure you protect secrets across the Kubernetes cluster to ensure configurations never contain plaintext credentials. And carefully protect access to configurations to avoid malicious modification.
Securing Networking
Once a Kubernetes cluster is breached, everything in the network - including services and machines - becomes vulnerable to attack. You prevent a network-wide breach by limiting communication and isolating nodes.
Isolate Kubernetes Nodes
Here are several practices to implement when isolating Kubernetes nodes:
- Configure all Kubernetes nodes on a separate network that does not allow direct access via public networks.
- Avoid any direct connections to the main corporate network.
- Isolate Kubernetes control and data traffic to ensure they cannot flow through the same pipe.
- Always configure nodes with an ingress controller. Set it to allow only connections from the master node on a specified port through a network access control list (ACL).
Monitor Network Traffic to Limit Communications
Here are key tips to help you secure containerized applications:
- Monitoring - set up a Kubernetes network policy that includes allowed traffic and use it to identify anomalous communications.
- Policies - compare allowed traffic to active traffic to identify inactive network policies. This information can help you continuously strengthen your network policy by keeping it up to date, and reducing the attack surface by removing any unnecessary connections.
Incident Response for Kubernetes and the Kubernetes Audit Log
Security teams need logs to identify and investigate attacks on production Kubernetes infrastructure. As part of
Kubernetes Audit Logging is known as a cluster-level capability, which organizations should initiate for production clusters. The audit log records Kubernetes API requests chronologically. Kubernetes audit log entries may assist incident response teams, helping them look into suspicious API requests, generate monitoring alerts for undesirable API calls, and collect statistics.
Audit policies define which events must be logged and what information must appear in the logs. The audit policy object’s structure is defined in the audit.k8s.io API group.
When events are processed, they are compared sequentially against the list of rules defined in the policy.
To support incident response, define audit policies covering critical activities such as:
- Changes to the cluster configuration
- Creation of pods
- Autoscaling events
- Authentication and login events
- Ingress and egress traffic
- Unusual loads which may point to denial of service (DoS)
Conclusion
In this article, I covered a few security essentials for a Kubernetes developer:
- Securing the cluster by locking down kubectl and using authorization methods like RBAC or ABAC.
- Following security hygiene such as regular updates and minimal operating systems for Kubernetes nodes and containers.
- Protecting against known Kubernetes attacks such as dashboard compromise.
- Preventing microservice compromise by establishing pod security policies and protecting secrets.
In addition, I showed how to prepare for an actual security incident against your Kubernetes cluster by ensuring you have logging of events critical to security. I hope this will be useful as you improve your security game when developing containerized applications.