SCC assignments and permissions in OpenShift

· 1495 words · 8 minute read

SCCs 🔗

There are tons of information out there about SCCs, but in this post we will be focused on how to create and use a custom SCC only.

See the OpenShift official documentation on Managing Security Context Constraints for more details.

Custom SCC 🔗

In the event of requiring a custom SCC, there are a few steps that need to be done to be able to use the SCC properly.

Minimal capabilities 🔗

The best way to create a custom SCC would be to build it based on the most restricted one (hint: its name is restricted) and then start adding capabilities and permissions depending on the application requisites.

In this example we are going to create a custom SCC based on the restricted one but adding permissions to use hostpath:

cat << EOF | oc apply -f -
apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
allowHostDirVolumePlugin: true
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: true
allowPrivilegedContainer: false
allowedCapabilities: null
fsGroup:
  type: MustRunAs
metadata:
  name: mycustomscc
readOnlyRootFilesystem: false
requiredDropCapabilities:
- KILL
- MKNOD
- SETUID
- SETGID
runAsUser:
  type: MustRunAsRange
seLinuxContext:
  type: MustRunAs
supplementalGroups:
  type: RunAsAny
volumes:
- configMap
- downwardAPI
- emptyDir
- persistentVolumeClaim
- projected
- secret
- hostpath
EOF

NOTE: The SCC management procedures are restricted to the cluster-admin user for security reasons.

SCC assignation and permissions 🔗

In order to be able to use a custom SCC, it is required the user or serviceaccount running the pod has access to the custom SCC.

This can be achieved in two different ways:

  • Modifying the SCC to set users/groups.
  • Using RBAC (roles, rolebindings, etc.)

But first, let’s do a regular deployment:

oc login -u nonadmin -p whatever api.example.com
oc new-project myproject
oc create deployment hello-openshift --image=openshift/hello-openshift

Note: As an exercise, you can try to create those objects using just yaml files.

The pod will have the restricted SCC by default:

oc get po -l app=hello-openshift -o jsonpath='{.items[*].metadata.annotations.openshift\.io/scc}'
restricted

Let’s create a serviceaccount:

oc create sa myserviceaccount

Note: As an exercise, you can try to create the service using just a yaml file.

And then, let’s modify the deployment to use that serviceaccount instead:

oc set serviceaccount deployment/hello-openshift myserviceaccount

This will modify the deployment object to set the serviceAccountName parameter (serviceAccount is currently deprecated, see oc explain pod.spec for more information.)

Hint: You can use oc patch or oc edit as well.

oc get deployment hello-openshift -o jsonpath='{.spec.template.spec.serviceAccountName}'
myserviceaccount

Let’s modify the deployment to use a hostPath volume:

oc patch deployment/hello-openshift -p '{"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"hello-openshift"}],"containers":[{"name":"hello-openshift","volumeMounts":[{"mountPath":"/empty","name":"myvolume"}]}],"volumes":[{"hostPath":{"path":"/var/empty","type":"Directory"},"name":"myvolume"}]}}}}' deployment/hello-openshift

Hum… nothing happens… let’s see the events:

oc get events
...
42s         Warning   FailedCreate        replicaset/hello-openshift-855649976d   Error creating: pods "hello-openshift-855649976d-" is forbidden: unable to validate against any security context constraint: [spec.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used]

It seems the pod is not allowed to use hostPath… as expected. This means we need to give that serviceaccount permission to use the custom SCC.

Method A: Modify the SCC definition 🔗

This method involves modifying the SCC to add the serviceaccount to the list of users (or groups), as:

users:
- <user>
- system:serviceaccount:<namespace>:<serviceaccountname>

Assigning users, groups or service accounts directly to an SCC retains cluster-wide scope and require cluster-admin permissions:

oc whoami
system:admin

oc patch scc mycustomscc --type=merge -p '{"users":["system:serviceaccount:myproject:myserviceaccount"]}'

Let’s see now:

oc whoami
nonadmin

oc get po -l app=hello-openshift -o jsonpath='{.items[*].metadata.annotations.openshift\.io/scc}'
mycustomscc

Nice!

Before continuing, let’s clean the SCC:

oc whoami
system:admin

oc patch scc mycustomscc --type=merge -p '{"users":[]}'

And remove the pod:

oc whoami
nonadmin

oc delete $(oc get po -o name -l app=hello-openshift)

Method B: SCC RBAC 🔗

Since OpenShift 3.11, you can specify SCCs as a resource that is handled by RBAC. This allows you to scope access to the SCCs to a certain project or to the entire cluster.

This means that in order for the serviceaccount to use the SCC, you need to create a role (or clusterrole) with the proper permissions and a rolebinding (or clusterrolebinding) for the serviceaccount to that role. It sounds complicated but it is not. Let’s do it with just a single command:

oc whoami
system:admin

oc create role mycustomsccrole --verb=use --resource=scc --resource-name=mycustomscc -n myproject

This will create a role such as:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: mycustomsccrole
  namespace: myproject
rules:
- apiGroups:
  - security.openshift.io
  resourceNames:
  - mycustomscc
  resources:
  - securitycontextconstraints
  verbs:
  - use

Note: You can create the role using the yaml file instead the oc create role command.

And we need to give access to that role to the serviceaccount (meaning, creating a rolebinding):

oc adm policy add-role-to-user mycustomsccrole -z myserviceaccount --role-namespace=myproject

This will create a rolebinding to link the service account to the role:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: mycustomsccrole
  namespace: myproject
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mycustomsccrole
subjects:
- kind: ServiceAccount
  name: myserviceaccount
  namespace: testscc

Note: You can create the rolebinding using the yaml file instead the oc adm policy add-role-to-user command.

To verify:

oc whoami
nonadmin

oc adm policy who-can use scc mycustomscc | grep myserviceaccount
        system:serviceaccount:myproject:myserviceaccount

oc get po -l app=hello-openshift -o jsonpath='{.items[*].metadata.annotations.openshift\.io/scc}'
mycustomscc

Nice!

NOTE: In recent versions of OCP (since https://github.com/openshift/oc/pull/412 was merged), the oc adm policy add-scc-to-user command creates a cluster role:

oc adm policy add-scc-to-user mycustomscc system:serviceaccount:myproject:myserviceaccount
clusterrole.rbac.authorization.k8s.io/system:openshift:scc:mycustomscc added: "myserviceaccount"

oc get clusterrole system:openshift:scc:mycustomscc -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:openshift:scc:mycustomscc
rules:
- apiGroups:
  - security.openshift.io
  resourceNames:
  - mycustomscc
  resources:
  - securitycontextconstraints
  verbs:
  - use

And a clusterrolebinding to link that clusterrole with the service account:

oc get clusterrolebinding system:openshift:scc:mycustomscc -o yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:openshift:scc:mycustomscc
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:openshift:scc:mycustomscc
subjects:
- kind: ServiceAccount
  name: myserviceaccount
  namespace: myproject

Priorities 🔗

When a pod request is created, the admission controller evaluates the request and assign an SCC depending on the permissions the pod requests as well as the permissions the user/sa running this pod is allowed.

The SCCs have a priority field to affect the ordering. This means:

  • Highest priority first, no priority is considered 0
  • If priorities are equal, the SCCs will be sorted from most restrictive to least restrictive
  • If both priorities and restrictions are equal the SCCs will be sorted by name

See [the admission controller code] (https://github.com/openshift/apiserver-library-go/blob/master/pkg/securitycontextconstraints/sccadmission/admission.go) if you are brave enough to understand it :)

The out-of-the-box SCCs included in OpenShift 4 and their priorities are:

oc get scc -o custom-columns=NAME:.metadata.name,PRIORITY:.priority
NAME                          PRIORITY
anyuid                        10
hostaccess                    <nil>
hostmount-anyuid              <nil>
hostnetwork                   <nil>
node-exporter                 <nil>
nonroot                       <nil>
privileged                    <nil>
restricted                    <nil>

Warning about priorities 🔗

Some service accounts included by default in OpenShift are granted cluster-admin permissions, meaning they can perform any action on any resource, including using any SCC, despite of not having being granted it explicitly.

In our case, we only give access to the myserviceaccount service account to use the mycustomscc SCC… but in fact, there are quite a few service accounts allowed to use it:

oc adm policy who-can use scc mycustomscc
...
Users:  admin
        system:admin
        system:serviceaccount:myproject:myserviceaccount
        system:serviceaccount:openshift-apiserver-operator:openshift-apiserver-operator
        system:serviceaccount:openshift-apiserver:openshift-apiserver-sa
        system:serviceaccount:openshift-authentication-operator:authentication-operator
        system:serviceaccount:openshift-authentication:oauth-openshift
        system:serviceaccount:openshift-cluster-node-tuning-operator:cluster-node-tuning-operator
        system:serviceaccount:openshift-cluster-storage-operator:csi-snapshot-controller-operator
        system:serviceaccount:openshift-cluster-version:default
        system:serviceaccount:openshift-config-operator:openshift-config-operator
        system:serviceaccount:openshift-controller-manager-operator:openshift-controller-manager-operator
        system:serviceaccount:openshift-etcd-operator:etcd-operator
        system:serviceaccount:openshift-etcd:installer-sa
        system:serviceaccount:openshift-kube-apiserver-operator:kube-apiserver-operator
        system:serviceaccount:openshift-kube-apiserver:installer-sa
        system:serviceaccount:openshift-kube-apiserver:localhost-recovery-client
        system:serviceaccount:openshift-kube-controller-manager-operator:kube-controller-manager-operator
        system:serviceaccount:openshift-kube-controller-manager:installer-sa
        system:serviceaccount:openshift-kube-controller-manager:localhost-recovery-client
        system:serviceaccount:openshift-kube-scheduler-operator:openshift-kube-scheduler-operator
        system:serviceaccount:openshift-kube-scheduler:installer-sa
        system:serviceaccount:openshift-kube-scheduler:localhost-recovery-client
        system:serviceaccount:openshift-kube-storage-version-migrator-operator:kube-storage-version-migrator-operator
        system:serviceaccount:openshift-kube-storage-version-migrator:kube-storage-version-migrator-sa
        system:serviceaccount:openshift-machine-config-operator:default
        system:serviceaccount:openshift-network-operator:default
        system:serviceaccount:openshift-operator-lifecycle-manager:olm-operator-serviceaccount
        system:serviceaccount:openshift-service-ca-operator:service-ca-operator
        system:serviceaccount:openshift-service-catalog-removed:openshift-service-catalog-apiserver-remover
        system:serviceaccount:openshift-service-catalog-removed:openshift-service-catalog-controller-manager-remover
Groups: system:cluster-admins
        system:masters

This means any of those serviceaccounts can effectively use the custom SCC and if the priority is higher than the one intended to be used, things can go wrong. For more information see this KCS article

Let’s see it in action. Modify the mycustomscc to have higher priority than the default ones (any number higher than 10 would be enough… let’s choose 99):

oc patch scc mycustomscc --type merge -p '{"priority":99}'

oc get scc mycustomscc -o jsonpath='{.priority}'
99

Now, let’s force the admission controller to ‘recalculate’ the SCCs for one of the serviceaccounts that can use any SCC. In this example, we will choose one of the oauth pods:

oc get po -n openshift-authentication
NAME                               READY   STATUS             RESTARTS   AGE
oauth-openshift-5c4466d8f6-djg89   1/1     Running            0          3d4h
oauth-openshift-5c4466d8f6-hrrkf   1/1     Running            0          3d4h

oc get po -n openshift-authentication -o jsonpath='{range .items[*]}{.metadata.annotations.openshift\.io/scc}{"\t"}{.metadata.name}{"\t"}{.metadata.namespace}{"\n"}{end}'
anyuid	oauth-openshift-5c4466d8f6-djg89	openshift-authentication
anyuid	oauth-openshift-5c4466d8f6-hrrkf	openshift-authentication

oc get deploy -n openshift-authentication oauth-openshift -o jsonpath='{.spec.template.spec.serviceAccountName}'
oauth-openshift

oc adm policy who-can use scc mycustomscc | grep oauth-openshift
        system:serviceaccount:openshift-authentication:oauth-openshift

They are using the anyuid SCC, but they can use any… and based on the order, mycustomscc will win:

oc get scc -o custom-columns=NAME:.metadata.name,PRIORITY:.priority
NAME                          PRIORITY
anyuid                        10
hostaccess                    <nil>
hostmount-anyuid              <nil>
hostnetwork                   <nil>
mycustomscc                   99
node-exporter                 <nil>
nonroot                       <nil>
privileged                    <nil>
restricted                    <nil>

Let’s see:

oc delete po oauth-openshift-5c4466d8f6-hrrkf

oc get po
NAME                               READY   STATUS             RESTARTS   AGE
oauth-openshift-5c4466d8f6-djg89   1/1     Running            0          3d4h
oauth-openshift-5c4466d8f6-wq7cv   0/1     CrashLoopBackOff   6          7m45s

oc get po -n openshift-authentication -o jsonpath='{range .items[*]}{.metadata.annotations.openshift\.io/scc}{"\t"}{.metadata.name}{"\t"}{.metadata.namespace}{"\n"}{end}'
anyuid	oauth-openshift-5c4466d8f6-djg89	openshift-authentication
mycustomscc	oauth-openshift-5c4466d8f6-wq7cv	openshift-authentication

Ooops, we just broke our cluster… the admission controller choose the mycustomscc SCC instead the anyuid one and now the pod is not able to run. Let’s fix it by lowering the priority of our custom SCC:

oc patch scc mycustomscc --type merge -p '{"priority":9}'
oc delete po oauth-openshift-5c4466d8f6-wq7cv
oc get po
NAME                               READY   STATUS    RESTARTS   AGE
oauth-openshift-5c4466d8f6-djg89   1/1     Running   0          3d4h
oauth-openshift-5c4466d8f6-tbphl   1/1     Running   0          46s

oc get po -n openshift-authentication -o jsonpath='{range .items[*]}{.metadata.annotations.openshift\.io/scc}{"\t"}{.metadata.name}{"\t"}{.metadata.namespace}{"\n"}{end}'
anyuid	oauth-openshift-5c4466d8f6-djg89	openshift-authentication
anyuid	oauth-openshift-5c4466d8f6-tbphl	openshift-authentication

PHEW!

openshift scc permissions rbac