In my continuous learning about Kubernetes, my interest wandered a bit and moved towards the extensions that it offers.
Got more curiosity in things like operators, custom resources, definitions, webhooks, and other goods that a Kubernetes cluster can be customized with.
After reading and researching for a bit, I’ve decided to try out and build my own validating webhook.
It’s excellent as a pet project, with lots of things to consider until everything fits together and starts working.
Now, what is a webhook or an admission controller for that matter?
An admission controller is a piece of code that intercepts requests to the Kubernetes API server before the persistence of the object, but after the request is authenticated and authorized.
In Kubernetes, there are two types of admission controllers, called the ValidatingAdmissionWebhook and MutatingAdmissionWebhook.
As each of their names implies; one just validates the requests, and the other modifies it if it isn’t up to spec.
Now I won’t be going into details about them. If you want to read more here is a link to the official documentation with a much more extensive explanation.
The great thing with the webhooks is that you can write your own, in whatever language you like, along with custom logic, be it for validation or mutation of requests.
I’ve decided to start with the validating webhook using Python and the Flask framework.
Writing only the webhook logic is one thing but setting it up and getting it to work is another.
To bootstrap the whole thing, in my case it required:
- Python and the Flask framework
- Docker
- SSL cert
- And a bunch of Kubernetes files(service, deployment, secret…)
All that for one simple webhook?!
Although it seems a bit tedious, I promise it’s quite a fun and rewarding project.
The project files are also available on my GitHub repository here.
Let’s start from the beginning.
Python code
With this basic functionality, I wrote the webhook code in less than fifty lines of Python code. Using Flask makes this super easy.
There are a lot of examples on the internet that you can take inspiration for writing it.
It’ll eventually boil down to two things.
You need to:
- Analyze the request and validate or mutate it against a set rule
- Send back an HTTP response to the Kubernetes admission controller
Everything else you add to the code is a plus.
This validating webhook will check for a label if it’s present on Deployment creation.
If it isn’t the request is blocked and an error message is displayed.
I’ve added the option to set the required label from an environment variable and also included logging.
Initially, I wanted to go with Django but considered it would be an overkill for this, so I chose to do it with Flask.
It came up really clean and simple:
from flask import Flask, request, jsonify
from os import environ
import logging
webhook = Flask(__name__)
webhook.config['LABEL'] = environ.get('LABEL')
webhook.logger.setLevel(logging.INFO)
if "LABEL" not in environ:
webhook.logger.error("Required environment variable for label isn't set. Exiting...")
exit(1)
@webhook.route('/validate', methods=['POST'])
def validating_webhook():
request_info = request.get_json()
uid = request_info["request"].get("uid")
if request_info["request"]["object"]["metadata"]["labels"].get(webhook.config['LABEL']):
webhook.logger.info(f'Object {request_info["request"]["object"]["kind"]}/{request_info["request"]["object"]["metadata"]["name"]} contains the required \"{webhook.config["LABEL"]}\" label. Allowing the request.')
return admission_response(True, uid, f"{webhook.config['LABEL']} label exists.")
else:
webhook.logger.error(f'Object {request_info["request"]["object"]["kind"]}/{request_info["request"]["object"]["metadata"]["name"]} doesn\'t have the required \"{webhook.config["LABEL"]}\" label. Request rejected!')
return admission_response(False, uid, f"The label \"{webhook.config['LABEL']}\" isn't set!")
def admission_response(allowed, uid, message):
return jsonify({"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"response":
{"allowed": allowed,
"uid": uid,
"status": {"message": message}
}
})
if __name__ == '__main__':
webhook.run(host='0.0.0.0', port=5000)
The first thing is setting the environment variable and logging.
The initial if condition checks that the label environment variable isn’t empty, which is required for the webhook to work.
From there, on POST request the code checks if the set label is present in the Deployment’s metadata field. And if the label isn’t present the admission controller will reject the Deployment.
I added the logger function so it will print out a log message depending on the status of the request.
The second function is only to respond to the admission controller. With an HTTP response if allowing the request is true or false.
The Kubernetes docs, also have an example of how the response should look like.
As stated in the documentation, the UID and the allowed fields are mandatory to be present in the AdmissionReview response.
Now one crucial thing… You must supply SSL encryption between the traffic from the webhook and the admission controller!
How you choose to supply the cert and key is up to you.
You can package them in the docker image or inject them separately using Kubernetes secrets. I did the latter.
Docker
The requirements.txt with the libraries can be generated, or found on the repository here.
The Dockerfile is self-explanatory:
FROM ubuntu:20.10 RUN apt-get update -y && apt-get install -y python3-pip python-dev WORKDIR /app COPY requirements.txt /app/requirements.txt RUN pip3 install -r /app/requirements.txt COPY validate.py /app COPY wsgi.py /app CMD gunicorn --certfile=/certs/webhook.crt --keyfile=/certs/webhook.key --bind 0.0.0.0:443 wsgi:webhook
For the image, well, you can choose any base image you want.
The other things are to install Python, Flask, Gunicorn, the libraries, and to copy over the code.
The important bit is the certificate and key.
Gunicorn will look in the /certs path, so they must be available there when mounted on the pod.
Note: If you aren’t familiar, Gunicorn is the application server that will forward the requests to your webhook app via the WSGI protocol.
Deployment files
Deploying the webhook requires four pieces for the puzzle to be complete.
First is the webhook deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: validation-webhook
labels:
app: validate
spec:
replicas: 1
selector:
matchLabels:
app: validate
template:
metadata:
labels:
app: validate
spec:
containers:
- name: webhook
image: kmitevski/webhook:gunicorn
ports:
- containerPort: 443
env:
- name: LABEL
value: development
volumeMounts:
- name: certs-volume
readOnly: true
mountPath: "/certs"
imagePullPolicy: Always
volumes:
- name: certs-volume
secret:
secretName: admission-tls
The service on which the webhook will be available:
apiVersion: v1
kind: Service
metadata:
name: validate
spec:
selector:
app: validate
ports:
- port: 443
The secret manifest containing the certificate and key:
apiVersion: v1 kind: Secret metadata: name: admission-tls type: Opaque data: webhook.crt: YOUR ENCODED BASE64 CERT webhook.key: YOUR ENCODED BASE64 KEY
Finally the Validating Webhook configuration file:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: validating-webhook
webhooks:
- name: validate.default.svc
failurePolicy: Fail
sideEffects: None
admissionReviewVersions: ["v1","v1beta1"]
rules:
- apiGroups: ["apps", ""]
resources:
- "deployments"
apiVersions:
- "*"
operations:
- CREATE
clientConfig:
service:
name: validate
namespace: default
path: /validate/
caBundle: YOUR ENCODED BASE64 CERT
Whew… Nearly there.
Compiling it all together
To get it all together and for the webhook to work, there is somewhat an order in which you should deploy things.
I started with generating the certificate, which you will need to supply to Gunicorn, and the webhook configuration.
I had a lot of trial and error to do until the whole thing fit in place and started to work.
Let’s just say that generating a certificate a hundred times until it worked wasn’t quite fun.
Generating certificate
To generate the cert, the openssl tool is used:
openssl req -x509 -sha256 -newkey rsa:2048 -keyout webhook.key -out webhook.crt -days 1024 -nodes -addext "subjectAltName = DNS.1:validate.default.svc"
Now here is the catch, SAN – subjectAltName that has a DNS record must be included with the cert!
You must match that DNS to your service name and namespace. The notation is service_name.namespace.svc.
Failing to do so may leave you with a bitter feel and desire to just rm -rf * the whole thing.
I was close to this multiple times!
Everything will run, but when you decide to test the webhook this little annoying message will pop up:
open ssl x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0
To have it in mind, you will need to reference the name of the service in multiple places.
Make sure it stays the same, otherwise changing it later will require changes in the certificate and the webhook config.
Creating the Kubernetes secret
The Kubernetes Secret will need to include the certificate and key, encoded in base64 format.
This is easy to do on Linux with:
cat webhook.key | base64 | tr -d '\n' #LS0tLS1CRUdJTiBQUklWQVRFI.... cat webhook.crt | base64 | tr -d '\n' #LS0tLS1CRUdJTiBDRVJUSUZJQ0FUR
Copy the webhook.crt output and paste it to the secret and the webhook config manifests.
Copy the webhook.key as well, although the key only needs to be included in the secret.
Just be careful when copying so you don’t include your username from the prompt :).
The mount point in the Pod spec is set on /certs.
You can change it, but make sure to create a new image with the change done in the Gunicorn cert and key paths.
Creating the Image
The folder and file structure are like this:
tree . . |-- Dockerfile |-- kubernetes-manifests | |-- label.yaml | |-- webhook-config.yaml | |-- webhook-deploy.yaml | |-- webhook-secret.yaml | `-- webhook-service.yaml |-- requirements.txt |-- validate.py |-- webhook.crt |-- webhook.key `-- wsgi.py
I built the Docker image from the root folder. Where the Dockerfile is located, along with the Python files and certificates.
The wsgi.py is just a helper file for Gunicorn to run the webhook app.
If you intend to use your own image, you’ll need to build, tag, and push it to a repository using the following commands as examples:
docker build -t webhook:gunicorn -f Dockerfile . docker tag webhook:gunicorn kmitevski/webhook docker push kmitevski/webhook:gunicorn
Bootstrapping the app
- Create the certificate
- Create the Docker image
- Apply the Secret manifest to the cluster
- Apply the Deployment, Service, and the Webhook configuration manifest
- Test the webhook
Testing
For testing, you can try and create a simple Deployment with an Nginx image.
This is easily tested using the imperative approach:
$ kubectl create deploy nginx --image=nginx error: failed to create deployment: admission webhook "validate.default.svc" denied the request: The label "development" isn't set!
And if you check in the webhook pod logs:
ERROR in validate: Object Deployment/nginx doesn't have the required "development" label. Request rejected!
Now try and deploy the label.yaml manifest.
That one contains the development label, and with it, creating the Deployment will be allowed.
$ kubectl apply -f label.yaml deployment.apps/nginx created
INFO in validate: Object Deployment/nginx contains the required "development" label. Allowing the request.
Success!! The validating webhook works!
Next up on the list is building a mutating webhook, I’ll see how that one works out :).