K8s security best practices 2
Forbidden Container Registries
Running a compromised image in a cluster can compromise the cluster. Attackers who get access to a private registry can plant their own compromised images in the registry. The latter can then be pulled by a user. In addition, users often use untrusted images from public registries (such as Docker Hub) that may be malicious. Building images based on untrusted base images can also lead to similar results.Note, this control is configurable. See bellow the details. https://hub.armo.cloud/docs/c-0001
Exec into container
Attackers who have permissions, can run malicious commands in containers in the cluster using exec command (“kubectl exec”). In this method, attackers can use legitimate images, such as an OS image (e.g., Ubuntu) as a backdoor container, and run their malicious code remotely by using “kubectl exec”. https://hub.armo.cloud/docs/c-0002
Resources memory limit and request
This control identifies all Pods for which the memory limit is not set. https://hub.armo.cloud/docs/c-0004
Control plane hardening
The control plane is the core of Kubernetes and gives users the ability to view containers, schedule new Pods, read Secrets, and execute commands in the cluster. Therefore, it should be protected. It is recommended to avoid control plane exposure to the Internet or to an untrusted network. The API server runs on ports 6443 and 8080. We recommend to block them in the firewall. Note also that port 8080, when accessed through the local machine, does not require TLS encryption, and the requests bypass authentication and authorization modules. https://hub.armo.cloud/docs/c-0005
Allowed hostPath
hostPath mount can be used by attackers to get access to the underlying host and thus break from the container to the host. (See “3: Writable hostPath mount” for details). https://hub.armo.cloud/docs/c-0006
Data Destruction
Attackers may attempt to destroy data and resources in the cluster. This includes deleting deployments, configurations, storage, and compute resources. https://hub.armo.cloud/docs/c-0007
Resource policies
CPU and memory resources should have a limit set for every container or a namespace to prevent resource exhaustion. This control identifies all the Pods without resource limit definitions by checking thier yaml definition file as well as their namespace LimitRange objects. It is also recommended to use ResourceQuota object to restrict overall namespace resources, but this is not verified by this control. https://hub.armo.cloud/docs/c-0009
Network policies
We have a MITRE check that fails if there are no policies defined for a specific namespace (cluster internal networking). https://hub.armo.cloud/docs/c-0011
Applications credentials in configuration files
Developers store secrets in the Kubernetes configuration files, such as environment variables in the pod configuration. Such behavior is commonly seen in clusters that are monitored by Azure Security Center. Attackers who have access to those configurations, by querying the API server or by accessing those files on the developer’s endpoint, can steal the stored secrets and use them.Note, this control is configurable. See bellow the details. https://hub.armo.cloud/docs/c-0012
Non-root containers
Container engines allow containers to run applications as a non-root user with non-root group membership. Typically, this non-default setting is configured when the container image is built. . Alternatively, Kubernetes can load containers into a Pod with SecurityContext:runAsUser specifying a non-zero user. While the runAsUser directive effectively forces non-root execution at deployment, NSA and CISA encourage developers to build container applications to execute as a non-root user. Having non-root execution integrated at build time provides better assurance that applications will function correctly without root privileges. https://hub.armo.cloud/docs/c-0013
Access Kubernetes dashboard
The Kubernetes dashboard is a web-based UI that is used for monitoring and managing the Kubernetes cluster. The dashboard allows users to perform actions in the cluster using its service account (Kubernetes-dashboard) with the permissions that are determined by the binding or cluster-binding for this service account. Attackers who gain access to a container in the cluster, can use its network access to the dashboard pod. Consequently, attackers may retrieve information about the various resources in the cluster using the dashboard’s identity. https://hub.armo.cloud/docs/c-0014
List Kubernetes secrets
A Kubernetes secret is an object that lets users store and manage sensitive information, such as passwords and connection strings in the cluster. Secrets can be consumed by reference in the pod configuration. Attackers who have permissions to retrieve the secrets from the API server (by using the pod service account, for example) can access sensitive information that might include credentials to various services. https://hub.armo.cloud/docs/c-0015
Allow privilege escalation
Attackers may gain access to a container and uplift its privilege to enable excessive capabilities. https://hub.armo.cloud/docs/c-0016
Immutable container filesystem
By default, containers are permitted mostly unrestricted execution within their own context. An attacker who has access to a container, can create files and download scripts as he wishes, and modify the underlying application running on the container. https://hub.armo.cloud/docs/c-0017
Configured readiness probe
Readiness probe is intended to ensure that workload is ready to process network traffic. It is highly recommended to define readiness probe for every worker container. This control finds all the PODs where the readiness probe is not configured. https://hub.armo.cloud/docs/c-0018
Bash/cmd inside container
Attackers who have permissions to run a cmd/bash script inside a container can use it to execute malicious code. Note, this control is configurable. See bellow the details. https://hub.armo.cloud/docs/c-0019
Mount service principal
When the cluster is deployed in the cloud, in some cases attackers can leverage their access to a container in the cluster to gain cloud credentials. For example, in AKS each node contains service principal credential. https://hub.armo.cloud/docs/c-0020
Exposed sensitive interfaces
Exposing a sensitive interface to the internet poses a security risk. Some popular frameworks were not intended to be exposed to the internet, and therefore don’t require authentication by default. Thus, exposing them to the internet allows unauthenticated access to a sensitive interface which might enable running code or deploying containers in the cluster by a malicious actor. Examples of such interfaces that were seen exploited include Apache NiFi, Kubeflow, Argo Workflows, Weave Scope, and the Kubernetes dashboard.Note, this control is configurable. See bellow the details. https://hub.armo.cloud/docs/c-0021
Vulnerable application
Running a public-facing vulnerable application in a cluster can enable initial access to the cluster. A container that runs an application that is vulnerable to remote code execution vulnerability (RCE) may be exploited. If service account is mounted to the container (default behavior in Kubernetes), the attacker will be able to send requests to the API server using this service account credentials. https://hub.armo.cloud/docs/c-0024
Application exploit (RCE)
An application that is deployed in the cluster and is vulnerable to a remote code execution vulnerability, or a vulnerability that eventually allows code execution, enables attackers to run code in the cluster. If service account is mounted to the container (default behavior in Kubernetes), the attacker will be able to send requests to the API server using this service account credentials. https://hub.armo.cloud/docs/c-0025
Kubernetes CronJob
Kubernetes Job is a controller that creates one or more pods and ensures that a specified number of them successfully terminate. Kubernetes Job can be used to run containers that perform finite tasks for batch jobs. Kubernetes CronJob is used to schedule Jobs. Attackers may use Kubernetes CronJob for scheduling execution of malicious code that would run as a container in the cluster. https://hub.armo.cloud/docs/c-0026
Dangerous capabilities
Giving dangerous and unnecessary capabilities for a container can increase the impact of a container compromise. https://hub.armo.cloud/docs/c-0028
Ingress and Egress blocked
Network policies control traffic flow between Pods, namespaces, and external IP addresses. By default, no network policies are applied to Pods or namespaces, resulting in unrestricted ingress and egress traffic within the Pod network. Pods become isolated through a network policy that applies to the Pod or the Pod’s namespace. Once a Pod is selected in a network policy, it rejects any connections that are not specifically allowed by any applicable policy object.Administrators should use a default policy selecting all Pods to deny all ingress and egress traffic and ensure any unselected Pods are isolated. Additional policies could then relax these restrictions for permissible connections.(For ARMO runtime needs to add exception) https://hub.armo.cloud/docs/c-0030
Delete Kubernetes events
A Kubernetes event is a Kubernetes object that logs state changes and failures of the resources in the cluster. Example events are a container creation, an image pull, or a pod scheduling on a node. Kubernetes events can be very useful for identifying changes that occur in the cluster. Therefore, attackers may want to delete these events (e.g., by using: “kubectl delete events–all”) in an attempt to avoid detection of their activity in the cluster. https://hub.armo.cloud/docs/c-0031
Access tiller endpoint
Helm is a popular package manager for Kubernetes maintained by CNCF. Tiller is the server-side component of Helm up to version 2. Tiller exposes internal gRPC endpoint in the cluster, listens to port 44134. By default, this endpoint does not require authentication. Attackers may run code on any container that is accessible to the tiller’s service and perform actions in the cluster, using the tiller’s service account, which often has high privileges. https://hub.armo.cloud/docs/c-0033
Automatic mapping of service account
We have it in Armo best (Automatic mapping of service account token). https://hub.armo.cloud/docs/c-0034
Cluster-admin binding
Role-based access control (RBAC) is a key security feature in Kubernetes. RBAC can restrict the allowed actions of the various identities in the cluster. Cluster-admin is a built-in high privileged role in Kubernetes. Attackers who have permissions to create bindings and cluster-bindings in the cluster can create a binding to the cluster-admin ClusterRole or to other high privileges roles. https://hub.armo.cloud/docs/c-0035
Malicious admission controller (validating)
Attackers can use validating webhooks to intercept and discover all the resources in the cluster. This control lists all the validating webhook configurations that must be verified. https://hub.armo.cloud/docs/c-0036
CoreDNS poisoning
CoreDNS is a modular Domain Name System (DNS) server written in Go, hosted by Cloud Native Computing Foundation (CNCF). CoreDNS is the main DNS service that is being used in Kubernetes. The configuration of CoreDNS can be modified by a file named corefile. In Kubernetes, this file is stored in a ConfigMap object, located at the kube-system namespace. If attackers have permissions to modify the ConfigMap, for example by using the container’s service account, they can change the behavior of the cluster’s DNS, poison it, and take the network identity of other services. https://hub.armo.cloud/docs/c-0037
Host PID/IPC privileges
Containers should be isolated from the host machine as much as possible. The hostPID and hostIPC fields in deployment yaml may allow cross-container influence and may expose the host itself to potentially malicious or destructive actions. This control identifies all PODs using hostPID or hostIPC privileges. https://hub.armo.cloud/docs/c-0038
Malicious admission controller (mutating)
Attackers may use mutating webhooks to intercept and modify all the resources in the cluster. This control lists all mutating webhook configurations that must be verified. https://hub.armo.cloud/docs/c-0039
HostNetwork access
Only connect PODs to host network when it is necessary. If not, set the hostNetwork field of the pod spec to false, or completely remove it (false is the default). Whitelist only those PODs that must have access to host network by design. https://hub.armo.cloud/docs/c-0041
SSH server running inside container
SSH server that is running inside a container may be used by attackers. If attackers gain valid credentials to a container, whether by brute force attempts or by other methods (such as phishing), they can use it to get remote access to the container by SSH. https://hub.armo.cloud/docs/c-0042
Container hostPort
Workloads (like pod, deployment, etc) that contain a container with hostport. The problem that arises is that if the scale of your workload is larger than the number of nodes in your Kubernetes cluster, the deployment fails. And any two workloads that specify the same HostPort cannot be deployed to the same node. In addition, if the host where your pods are running becomes unavailable, Kubernetes reschedules the pods to different nodes. Thus, if the IP address for your workload changes, external clients of your application will lose access to the pod. The same thing happens when you restart your pods — Kubernetes reschedules them to a different node if available. https://hub.armo.cloud/docs/c-0044
Writable hostPath mount
hostPath volume mounts a directory or a file from the host to the container. Attackers who have permissions to create a new container in the cluster may create one with a writable hostPath volume and gain persistence on the underlying host. For example, the latter can be achieved by creating a cron job on the host. https://hub.armo.cloud/docs/c-0045
Insecure capabilities
Giving insecure and unnecessary capabilities for a container can increase the impact of a container compromise.Note, this control is configurable. See bellow the details. https://hub.armo.cloud/docs/c-0046
Exposed dashboard
The Kubernetes dashboard is a web-based user interface that enables monitoring and managment of the Kubernetes cluster. By default, the dashboard exposes an internal endpoint (ClusterIP service). If the dashboard is exposed externally, it can allow unauthenticated remote management of the cluster. https://hub.armo.cloud/docs/c-0047
HostPath mount
Mounting host directory to the container can be used by attackers to get access to the underlying host. This control identifies all the PODs using hostPath mount. https://hub.armo.cloud/docs/c-0048
Network mapping
Attackers may try to map the cluster network to get information on the running applications, including scanning for known vulnerabilities. By default, there is no restriction on pods communication in Kubernetes. Therefore, attackers who gain access to a single container, may use it to probe the network. https://hub.armo.cloud/docs/c-0049
Resources CPU limit and request
This control identifies all Pods for which the CPU limit is not set. https://hub.armo.cloud/docs/c-0050
Instance Metadata API
Cloud providers provide instance metadata service for retrieving information about the virtual machine, such as network configuration, disks, and SSH public keys. This service is accessible to the VMs via a non-routable IP address that can be accessed from within the VM only. Attackers who gain access to a container, may query the metadata API service for getting information about the underlying node. For example, in Azure, the following request would retrieve all the metadata information of an instance: http:///metadata/instance?api-version=2019-06-01 https://hub.armo.cloud/docs/c-0052
Access container service account
Service account (SA) represents an application identity in Kubernetes. By default, an SA is mounted to every created pod in the cluster. Using the SA, containers in the pod can send requests to the Kubernetes API server. Attackers who get access to a pod can access the SA token (located in /var/run/secrets/kubernetes.io/serviceaccount/token) and perform actions in the cluster, according to the SA permissions. If RBAC is not enabled, the SA has unlimited permissions in the cluster. If RBAC is enabled, its permissions are determined by the RoleBindings\ClusterRoleBindings that are associated with it. https://hub.armo.cloud/docs/c-0053
Cluster internal networking
Kubernetes networking behavior allows traffic between pods in the cluster as a default behavior. Attackers who gain access to a single container may use it for network reachability to another container in the cluster. https://hub.armo.cloud/docs/c-0054
Linux hardening
In order to reduce the attack surface, it is recommend, when it is possible, to harden your application using security services such as SELinux®, AppArmor®, and seccomp. Starting from Kubernetes version 22, SELinux is enabled by default. https://hub.armo.cloud/docs/c-0055
Configured liveness probe
Liveness probe is intended to ensure that workload remains healthy during its entire execution lifecycle, or otherwise restrat the container. It is highly recommended to define liveness probe for every worker container. This control finds all the PODs where the Liveness probe is not configured. https://hub.armo.cloud/docs/c-0056
Privileged container
A privileged container is a container that has all the capabilities of the host machine, which lifts all the limitations regular containers have. Practically, this means that privileged containers can do almost every action that can be performed directly on the host. Attackers who gain access to a privileged container or have permissions to create a new privileged container (by using the compromised pod’s service account, for example), can get access to the host’s resources. https://hub.armo.cloud/docs/c-0057
CVE-2021-25741 - Using symlink for arbitrary host file system access.
A user may be able to create a container with subPath or subPathExpr volume mounts to access files & directories anywhere on the host filesystem. Following Kubernetes versions are affected: v1.22.0 - v1.22.1, v1.21.0 - v1.21.4, v1.20.0 - v1.20.10, version v1.19.14 and lower. This control checks the vulnerable versions and the actual usage of the subPath feature in all Pods in the cluster. If you want to learn more about the CVE, please refer to the CVE link: https://nvd.nist.gov/vuln/detail/CVE-2021-25741 https://hub.armo.cloud/docs/c-0058
CVE-2021-25742-nginx-ingress-snippet-annotation-vulnerability
Security issue in ingress-nginx where a user that can create or update ingress objects can use the custom snippets feature to obtain all secrets in the cluster (see more at https://github.com/kubernetes/ingress-nginx/issues/7837) https://hub.armo.cloud/docs/c-0059
Namespace without service accounts
It is recommended not to use default service account anywhere in production environment. This control identifies all namespaces without explicit non-default service account. https://hub.armo.cloud/docs/c-0060
Pods in default namespace
It is recommended to avoid running PODs in cluster without explicit namespace assignment. This may lead to wrong capabilities and permissions assignment and potential compromises. This control identifies all the PODs running in the default namespace. https://hub.armo.cloud/docs/c-0061
Sudo in container entrypoint
Adding sudo to a container entry point command may escalate process privileges and allow access to forbiden resources. This control checks all the entry point commands in all containers in the POD to find those that have sudo command. https://hub.armo.cloud/docs/c-0062
Portforwarding privileges
Attackers who have relevant RBAC permissions, can run open a backdoor communication channel directly to the sockets inside target container using exec command “kubectl portforward” command. Using this method, attackers can bypass network security restrictions and communicate directly with software in the containers. https://hub.armo.cloud/docs/c-0063
Image pull policy on latest image tag
Check all images with latest tag (or no tag at all) that have the 'imagePullPolicy' field set to 'Never' or 'IfNotPresent'. If exist, Kubernetes may run older version of the container images (if they happen to be cached on a node) without user knowing about this. https://hub.armo.cloud/docs/c-0064
No impersonation
Impersonation is an explicit RBAC permission to use other roles rather than the one assigned to a user, group or service account. This is sometimes needed for testing purposes. However, it is highly recommended not to use this capability in the production environments for daily operations. This control identifies all subjects whose roles include impersonate verb. https://hub.armo.cloud/docs/c-0065
Secret/ETCD encryption enabled
etcd is a consistent and highly-available key value store used as Kubernetes' backing store for all cluster data. All object data in Kubernetes, like secrets, are stored there. This is the reason why it is important to protect the contents of etcd and use its data encryption feature. https://hub.armo.cloud/docs/c-0066
Audit logs enabled
Audit logging is an important security feature in Kubernetes, it enables the operator to track requests to the cluster. It is important to use it so the operator has a record of events happened in Kubernetes https://hub.armo.cloud/docs/c-0067
PSP enabled
Pod Security Policies enable fine-grained authorization of pod creation and updates and it extends authorization beyond RBAC. It is an important to use PSP to control the creation of sensitive PODs in your cluster. https://hub.armo.cloud/docs/c-0068
Disable anonymous access to Kubelet service
By default, requests to the kubelet's HTTPS endpoint that are not rejected by other configured authentication methods are treated as anonymous requests, and given a username of system:anonymous and a group of system:unauthenticated. https://hub.armo.cloud/docs/c-0069
Enforce Kubelet client TLS authentication
Kubelets are the node level orchestrator in Kubernetes control plane. They are publishing service port 10250 where they accept commands from API server. Operator must make sure that only API server is allowed to submit commands to Kubelet. This is done through client certificate verification, must configure Kubelet with client CA file to use for this purpose. https://hub.armo.cloud/docs/c-0070
Validate Kubelet TLS configuration
Kubelets are the node level orchestrator in Kubernetes control plane. They are publishing service port 10250 where they accept commands from API server which contain sensitive information. The connection between API server and Kubelet must be encrypted and secured. https://hub.armo.cloud/docs/c-0071
Naked PODs
It is not recommended to create PODs without parental Deployment, ReplicaSet, StatefulSet etc.Manual creation if PODs may lead to a configuration drifts and other untracked changes in the system. Such PODs won't be automatically rescheduled by Kubernetes in case of a crash or infrastructure failure. This control identifies every POD that does not have corresponding parental object. https://hub.armo.cloud/docs/c-0073
Containers mounting Docker socket
Mounting Docker socket (Unix socket) enables container to access Docker internals, retrieve sensitive information and execute Docker commands, if Docker runtime is available. This control identifies PODs that attempt to mount Docker socket for accessing Docker runtime. https://hub.armo.cloud/docs/c-0074
Image pull policy on latest tag
While usage of the latest tag is not generally recommended, in some cases this is necessary. If it is, the ImagePullPolicy must be set to Always, otherwise Kubernetes may run an older image with the same name that happens to be present in the node cache. Note that using Always will not cause additional image downloads because Kubernetes will check the image hash of the local local against the registry and only pull the image if this hash has changed, which is exactly what users want when use the latest tag. This control will identify all PODs with latest tag that have ImagePullSecret not set to Always. Note as well that some vendors don't use the word latest in the tag. Some other word may also behave like the latest. For example, Redis uses redis:alpine to signify the latest. Therefore, this control treats any word that does not contain digits as the latest. If no tag is specified, the image is treated as latests too. https://hub.armo.cloud/docs/c-0075
Label usage for resources
It is recommended to set labels that identify semantic attributes of your application or deployment. For example, { app: myapp, tier: frontend, phase: test, deployment: v3 }. These labels can used to asign policies to logical groups of the deployments as well as for presentation and tracking purposes. This control helps you find deployments without any of the expected labels. https://hub.armo.cloud/docs/c-0076
K8s common labels usage
Kubernetes common labels help manage and monitor Kubernetes cluster using different tools such as kubectl, dashboard and others in an interoperable way. Refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/ for more information. This contol helps you find objects that don't have any of these labels defined. https://hub.armo.cloud/docs/c-0077
Images from private registry
The images that are running in the cluster can be stored in a private registry. For pulling those images, the container runtime engine (such as Docker or containerd) needs to have valid credentials to those registries. If the registry is hosted by the cloud provider, in services like Azure Container Registry (ACR) or Amazon Elastic Container Registry (ECR), cloud credentials are used to authenticate to the registry. If attackers get access to the cluster, in some cases they can obtain access to the private registry and pull its images. For example, attackers can use the managed identity token as described in the 'Access managed identity credential' technique. Similarly, in EKS, attackers can use the AmazonEC2ContainerRegistryReadOnly policy that is bound by default to the node's IAM role.Note, this control is configurable. See bellow the details. https://hub.armo.cloud/docs/c-0078
CVE-2022-0185-linux-kernel-container-escape
Linux maintainers disclosed a broadly available Linux kernel vulnerability (CVE-2022-0185) which enables attackers to escape containers and get full control over the node. In order to be able to exploit this vulnerability, the attacker needs to be able to run code on in the container and the container must have CAP_SYS_ADMIN privileges. Linux kernel and all major distro maintainers have released patches. This control alerts on vulnerable kernel versions of Kubernetes nodes. https://hub.armo.cloud/docs/c-0079
CVE-2022-24348-argocddirtraversal
CVE-2022-24348 is a major software supply chain 0-day vulnerability in the popular open source CD platform Argo CD. Exploiting it enables attackers to obtain sensitive information like credentials, secrets, API keys from other applications on the platform. This in turn can lead to privilege escalation, lateral movements and information disclosure. https://hub.armo.cloud/docs/c-0081