Container adoption is increasing across industries, but as Matt Colman, Senior Cloud Security Architect for IBM Security explains, developing, deploying and managing secure applications has its challenges.

Container and Kubernetes: Security is not an afterthought
Play video

The use of application containers is becoming increasingly popular due to their portability and support of modern development and architecture. Research conducted in 2020 by IBM Market Development & Insights showed 78% of those surveyed experienced improved application quality and faster response to changes in market segment and 76% experienced faster time to market.

What are containers?

For those new to application containers, here is a little background: containers are made possible due to operating system kernel features - specifically, namespaces (kernel level isolation of processes) and cgroups (used to control resource e.g., CPU, memory, network, disk I/O accessed / used by a process / set of processes).

A container is software unit, consisting of application code plus its required binaries and libraries. A container is created from a container image (which is a read-only template of the content just described). A container image is created from a configuration file known as a Dockerfile (or containerfile).

This configuration file describes the necessary base container image to use and then additional commands to run on top to make the container image. The container building tool (popular ones are Docker, Buildah and Kaniko) creates the container image from the configuration file.

Container runtime engines are used to run containers. They abstract the complexities of namespaces and cgroup creation and management from the end user and sit between the running containers and the host system.

Kubernetes, a container orchestration tool, comes into the mix because the next step in the journey is container orchestration - a system that can automate container deployment, scaling, and the management of containerised applications.

Kubernetes interacts with and utilises the container runtime engine to create containers. Crucially, it provides the ability to set scaling requirements, replace failed containers and schedule across a cluster of nodes amongst other functions.

Container technology risk areas

This technology is, however, not without risk. It is important to note that containers are lightweight and portable, due to the absence of requiring a full guest OS; instead, they share the OS of the underlying host. They do not have the isolation of the traditional virtual machine, which is provided by its hypervisor.

Therefore, to take advantage of all the benefits containers can offer (and in a safe manner) security must not be an afterthought. Kubernetes provides a means for securing container orchestration, but it doesn’t just happen: we must plan and act, otherwise shortcuts will be taken to ‘just make it work’ and unpalatable outcomes may be encountered.

Fortunately, there are numerous sources of container security guidance available, NIST Special Publication 800-190 is one of them. It highlights five key risk areas:

  • image,
  • registry,
  • orchestrator,
  • containers,
  • host OS.

Educating development / engineering teams and IT architects designing systems is just as important as securing these risk areas. Our teams and colleagues must appreciate security for container-based systems is available, we must know what to secure, why we must secure it and how we must secure it.

Securing containers and container orchestration involves activities across the software development lifecycle (SDLC), including:

  • planning so that the system and components are secure by design;
  • developing code with best security practices built in;
  • ensuring that the building of artefacts adequately tests the security of configurations and checks for vulnerable components;
  • ensuring deployment controls are in place to prohibit compromised artefacts from reaching production;
  • using runtime monitoring to visualise and track components and configurations to assess against a changing threat landscape and monitor for malicious activity requiring attention.

Prior to creating secure containers, we must ensure the orchestrator and the underlying OS hosting it are secure. Container-optimised OS distributions are available and they are designed for this specific use case. They provide a minimal OS image with only the tools required to run containers included, immutable filesystems and automated, atomic upgrades.

Orchestrator security

The security of the orchestrator has two areas of focus:

  1. It needs to be installed securely (following hardening guides such as CIS Kubernetes Benchmark).
  2. Security configurations, features and functionality need to be implemented.

The former (with a copy of a hardening guide on hand) is relatively self-explanatory. The latter requires knowledge of components of the Kubernetes architecture and the roles they play, along with consideration as to how we wish to utilise them.

Some key areas for consideration are:

  • identity and access management, including human and programmatic accounts and the roles based access control (RBAC) model available;
  • controlling the security and availability of container workloads through admission control, placement and resourcing policies;
  • securing the orchestrator internal network and exposure to outside networks;
  • secrets management;
  • cluster auditing.

Kubernetes provides mechanisms for user authentication (via an external identity provider) and authorisation (via RBAC). Permissions assigned to users and groups must follow the rule of least privilege and several default roles are available.

For you

Be part of something bigger, join the Chartered Institute for IT.

Controlling the security of container workloads is a key element of the container security topic. Kubernetes has the admission control construct to enforce policies over the content of objects sent to the Kubernetes API server. The pod security admission controller is the final opportunity to stop a workload with non-optimal security settings from entering the orchestrator ecosystem.

The pod security admission controller can check ‘security context’ specifications on workloads and enforce that the containers being deployed cannot run with privilege, run as root, have superuser permissions, have access to the underlying host namespaces or network and thus play a massive role in ensuring good security practice. Together with the creation of containers adhering to best security practice, the pod security admission controller can greatly remove risk from running container-based systems.

Orchestrators use internal network overlays for the containers (deployed as units called ‘pods’ in Kubernetes) to communicate with each other. By default, all containers can communicate with each other and so good security practice dictates that we need to limit these communication flows to only those we wish to permit, in order to gain control and prohibit unauthorised communication.

Containers are only exposed for access from outside the cluster if we actively configure them to be so. A key decision point here is whether the application being exposed is secured and needs to be made available for external access.

Kubernetes secrets are stored within the cluster and made available to containers that require them (for example, for application container ‘A’ to connect to database container ‘B’). Access to these secrets must be restricted to the human or programmatic accounts that need them and the secrets need to be encrypted, which requires configuration in Kubernetes to ensure this is so.

The requirement for auditing is simple - if you want to know what happened, when, who did it, etc., then ensure the cluster is configured to create audit logs. More importantly, ensure something is done with them!

Container image security

Container images (the templates for the containers that’ll be run) should be built with security in mind. There are numerous best-practice guides available on the internet. A few key items are:

  • the use of a minimal starting (base) image, referenced by a version tag or even its immutable image digest (a hash of the image contents) can help both reduce the number of components that might have vulnerabilities or need patching, as well as ensuring we know which versions of OS and components we have used;
  • specifying a user within the container image is incredibility important as containers will default to running as root in the absence of a user declaration.

Security should be built into DevSecOps pipelines to scan applications and containers for vulnerabilities and configuration flaws - and, ideally, a container security tool that provides security across the lifecycle should be implemented (I refer to this as an end-to-end container security tool). These end-to-end tools can advise during development and build time through command line clients, enforce admission control at deployment time and, crucially, also provide runtime visibility, monitoring and alerting.

Runtime security

Runtime monitoring gives visibility of the orchestration cluster and the workloads within. The compliance of the cluster configuration against an industry standard should be monitored to ensure a secure posture.

Workload monitoring is vital too. A container runtime security tool can track components in workloads and alert as new vulnerabilities are uncovered, advising on which workloads must be patched. The monitoring can also provide us with visibility of network flows, interactions with the orchestrator API and, significantly, processes within containers.

An unexplained and potentially malicious process that isn’t part of the baseline formed by the security tool for that container can be alerted upon and instigate a proactive response whereby the container is killed and thus, the process terminated. This provides the operations team an incredibly powerful security response and reiterates why security across the entire container lifecycle is so important.


In summary, good container and orchestration security means:

  • planning security practices and the appropriate tools into the application container lifecycle;
  • ensuring the five key risk areas are secured;
  • not missing ‘security enforcement opportunities’, including:
    • code & build: chance to shift-left discovery and remediation;
    • deployment: last chance to stop something;
    • runtime: workload visibility and ongoing security status.