The problem

During the process of upgrade of my K8s course I decided to add more limitations to what student is allowed to do in terms of accident or intentional cloud resorce over-usage. From my perspective we have these concerns that should not be overlooked to avoid unexpectedly high costs in cloud billing:

• Nodepool size(number of nodes/cpu)
• Block storage usage(size and number of PersistentVolumes)
• Network Load Balancer(service type=LoadBalancer)

Limiting nodepool size

The first thing that comes into mind is to use built-in RBAC. So, initially I create new ServiceAccount with token and give Kubeconfig file that uses this token. That means no admin account is provided for student. But at the same time I launch one cluster for one user, no multi-tenant approaches to save money sacrificing the studying process.

I give permissions to read all resources in cluster, but write operations are possible only in default namespace, excluding ResourceQuota(see below). In OVH cloud k8s node pool size can be controlled via CRD pre-installed in cluster called nodepool. Since RBAC prevents service account to change anything outside default namespace the nodepool size modification is also unavailable.

Limiting block storage usage

ResourceQuota is k8s resource that enforces limit of number of PVC and overall volume size in defined namespace. I create it during the initialization process with values of 5 claims and 5 GiB per default namespace. The alteration of this resource is forbidden by RBAC

Service type LoadBalancer limitation

This is a bit more tricky since RBAC does not provide such level of access granularity. The obvious way to solve this is to create ValidatingWebhookConfiguration with corresponding webhook server written using Golang. The code itself is pretty easy. In order to avoid calling to the k8s API I found it simplier to check by the name of the ingress, and deny in case it is different than ingress-nginx-controller. Exactly this name of the service is created in one of the course lessons. It guarantees that only one service with this name can be created.