Skip to content

K8s security best practices

General

Basics

  • update your kubernetes environment version early and othen
  • don't use the cluster amdin user for your daily work, treat it like root
  • if ou can, use a managed K8s services (AKS, EKS, GKE)
  • Check out the CIS Kubernetes Benchmark documents for more security best practises

RBAC Authorization

  • RBAC is a standard Kubernetes authorization mode, and can easily be used to authorize use of policies.
  • Allows to configure who can access what is the cluster
  • Enabled by default on the latestK8s
  • --authorization-mode=Node,RBAC
  • Has 4 different objects: Role, RoleBinding, ClusterRole and ClusterRoleBinding
  • https://kubernetes.io/docs/reference/access-authn-authz/rbac/
    kubectl api-versions | grep rbac 
    
    kubectl cluster-info dump | grep authorization-mode
    

TLS Everywhere

TLS should be enabled for every component that supports it to prevent traffic sniffing, verify the identity of the server, and (for mutual TLS) verify the identity of the client. more

Kubelet API

Kubelet expose HTTPS endpoints which grant powerful control over the node and containers. By default, Kubelets allow unauthenticated access to this API. Production clusters should enable Kubelet authentication and authorization.

  • https://127.0.0.1:10250/runningpods → list running pods
  • https://127.0.0.1:10250/exec → runs a command in a container
  • k8s-hacking-10250

Audit Logs

  • audit logs are not enabled by default
  • highly recommended enabling them for security and troubleshooting
  • need at least tow things: a log path and a policy file
  • set those up on the kube-apiserver configuration

ETCS

  • main data storage location for your cluster
  • all the cluster objects are saved here
  • use authentication and firewalls to restrict access to etcs
  • encrypt data in etcs (ecryption at rest)

Encryption in transit - default

ps -ef | grep etcd
# --cert-file=/etc/kubernetes/pkietcd/server.crt
# --key-file= /etc/kubernetes/pki/server,key 

Encryption at rest - NOT default

  • By default, it's all stored in plain text
  • Create an EncryptionConfiguration object

Pods

Limit Resources

CPU & Memory

apiVersion: v1
kind: ResourceQuota
metadata:
  name: pods-low
spec:
  hard:
    cpu: "5"
    memory: 10Gi
    pods: "10"

Network Policies

  • by default, all pods can communicate with any other pod in the cluster
  • make sure you create a proper network policy for your cluster
  • does the front-end pod really nned to talk to the DB pod ?
  • what if an attacker can access the pods on the kube-system namespace ?
  • Restrict access to database to only the Pods that require it
  • Specify access via labels
  • If you want to control traffic flow at the IP address or port level (OSI layer 3 or 4), then you might consider using Kubernetes NetworkPolicies for particular applications in your cluster. NetworkPolicies are an application-centric construct which allow you to specify how a pod is allowed to communicate with various network "entities" (we use the word "entity" here to avoid overloading the more common terms such as "endpoints" and "services", which have specific Kubernetes connotations) over the network. NetworkPolicies apply to a connection with a pod on one or both ends, and are not relevant to other connections.
  • https://editor.cilium.io
  • more
    kind: NetworkPolicy
    ...
    spec:
      ingress:
        - form:
          - podSelector:
              matchLabels:
                name: guestbook
    

Run as non-root

RunAsNotRoot - Least Privilege - Only give as much permission/privilege as is absolutely necessary - In the configuration file, the runAsUser field specifies that for any Containers in the Pod, all processes run with user ID 1000. - more

spec:
  securityContext:
    runAsUser: 1000

Read only root filesystem

ReadOnlyRootFilesystem - Requires that containers must run with a read-only root filesystem (i.e. no writable layer) more

spec:
  securityContext:
    readOnlyRootFilesystem: true

Privilege Escalation (no_new_privs)

AllowPrivilegeEscalation - Gates whether or not a user is allowed to set the security context of a container to allowPrivilegeEscalation=true. This defaults to allowed so as to not break setuid binaries. Setting it to false ensures that no child process of a container can gain more privileges than its parent. more

spec:
  securityContext:
    allowPrivilegeEscalation: false

SecComp

  • filters a process's system calls
    apiVersion: v1
    kind: Pod
    metadata:
      name: mypod
      annotation:
        seccomp.security.alpha.kubernetes.io/pod: runtime/defailt
    

AppArmor

  • uses program profiles to restrict the capabilities of individual programs
    apiVersion: v1
    kind: Pod
    metadata:
      name: mypod
      annotation:
        apparmor.security.beta.kubernetes.io/hello: runtime/defailt
    spec:
      containers:
        - name: hello
    

SELinux

  • applies security labels to objects and evaluates all security-relevant interactions via the security policy
    apiVersion: v1
    kind: Pod
    metadata:
      name: mypod
    spec:
    securityContext:
      seLinuxOptions:
        level: "s0:c123,c456"
    

Cluster

Open Policy Agent

In Kubernetes, Admission Controllers enforce semantic validation of objects during create, update, and delete operations. With OPA you can enforce custom policies on Kubernetes objects without recompiling or reconfiguring the Kubernetes API server.

Resources