Testing a Java-based Kubernetes Admission Webhook Locally

9 min read —
— ● kubernetes ● AdmissionWebhook ●

A Kubernetes Admission webhook can be an effective way to enforce and apply custom system policies. Admission webhooks are callbacks which allow admission requests (creating or updating a kubernetes resource) to first be forwarded to a custom external service. In the external service, administrators can enforce constraints (Validating Admission Webhook) or apply changes (Mutating Admission Webhook) to their Kubernetes resources.

The Platform Core team at theScore exposes some of our internal tooling as Kubernetes custom resources. We use Validating Admission Webhooks to enforce that any resource and schema changes in a service's custom resource files are valid. This allows users to catch configuration issues quickly and avoid their instances transitioning into a bad state.

However, one issue with Kubernetes Webhooks is that is can be tedious to test them in a local dev environment as they are called directly from the Kubernetes API Server and to properly test, we would like to mimic this behaviour. This outlines how the Platform Team's setup to test a Validating Webhook.

Prerequisites

Before being able to test a webhook, the service that is invoked from the Kubernetes API server needs to be implemented. For more information on implementing webhooks, see the kubernetes documentation here

For the sake of this blog, it is assumed you have a Admission Webhook Service implementation that can be called via the command line and accepts keystore and trust store paths as arguments. This is helpful as kubernetes requires webhooks to have a valid TLS certificate to be successfully called.

On the Platform Core Team at theScore, our webhook can be invoked using these commands:

# Build the webhook with maven
mvn package -DskipTests


datadex webhook --port=<port num> --keystore-path=<path to keystore> --keystore-pass=<keystore password> --truststore-path=<truststore path> --truststore-pass=<truststore password>

Testing Process

Webhook Resource Definition

As part of the development of your Admission Webhook, you should have created a CRD file for the custom service to be called by the API Server. For local testing, there are some particular configurations which are needed that will be pointed out here.

Below is an example webhook definition that we use on the Platform Team at theScore. As seen by the webhooks.rules section, this webhook gets invoked whenever a Datadex Instance (a Platform Team custom resource) get created or updated. For local testing, you should pay attention to the webhooks.clientConfig section: * url is set to https://host.docker.internal:443/validate * This defines the location of the locally running webhook. We will be using docker-desktop to create a local kubernetes cluster and the https://host.docker.internal allows docker to access services running on the local machine outside the docker container. In this example, calls to this url will hit the server running on localhost at port 443. * caBundle defines the CAs (Certificate Authorities) used to verify the TLS connection of the webhook. See the "Generating Local TLS Certificate Files" section below for more details

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: "datadex-operator.thescore.com"
webhooks:
  - name: datadex-operator.thescore.com
    rules:
      - apiGroups: ["datadex.thescore.com"]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["datadexinstances"]
        scope: "Namespaced"
    clientConfig:
      url: "https://host.docker.internal:443/validate"
      caBundle: <This field will be populated later in this blog>
    admissionReviewVersions: ["v1"]
    sideEffects: None
    timeoutSeconds: 30

Generating Local TLS Certificate Files

As mentioned above, webhook services must have a valid TLS certificate. The cert can be obtained in several ways but the easiest for local testing is to generate a self-signed certificate. They can be requested at any time by any developer, and never expire making testing easy. However, this is only for local testing. For other live instances and especially production, be sure to obtain a certificate from an approved Certificate Authority.

To generate a self-signed cert, openssl can be used. The following bash script can be used to generate all the necessary data. It is originally from Slack's open source code and modified to this specific use case.

#!/bin/bash

openssl genrsa -out ca.key 2048

openssl req -new -x509 -sha256 -days 3650 -key ca.key \
  -subj "/C=CA/CN=datadex-webhook"\
  -out ca.crt

openssl req -newkey rsa:2048 -nodes -sha256 -keyout server.key \
  -subj "/C=CA/CN=datadex-webhook" \
  -out server.csr

openssl x509 -req \
  -extfile <(printf "subjectAltName=DNS:host.docker.internal") \
  -sha256 \
  -days 3650 \
  -in server.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out server.crt

cat server.crt ca.crt > server.bundle.crt

echo
echo ">> Generating kube secrets..."
kubectl create secret tls validating-webhook-tls \
  --cert=server.bundle.crt \
  --key=server.key \
  --dry-run=client -o yaml \
  > webhook-secret.yaml

formatted_crt=$(cat ca.crt | base64 | fold)
# yq is a command line yaml processor. See https://github.com/mikefarah/yq for info and install instructions
yq -y -i -e ".webhooks[0].clientConfig.caBundle = \"$formatted_crt\"" webhook-configuration.yaml

echo
echo ">> Generating Keystore and truststore files"
openssl pkcs12 -export -inkey server.key -in server.crt -out key.pkcs12 -password pass:123456
keytool -importkeystore -noprompt -srckeystore key.pkcs12 -srcstoretype pkcs12 -destkeystore key.jks -storepass 123456 -srcstorepass 123456

csplit -f crt- -n 2 server.bundle.crt '/-----BEGIN CERTIFICATE-----/'
for file in crt-*; do
  keytool -delete -noprompt -keystore trust.jks -trustcacerts -storepass 123456 -alias "service-$file" 
  keytool -import -noprompt -keystore trust.jks -file "$file" -storepass 123456 -alias "service-$file"
  rm "$file"
done

rm ca.crt ca.key ca.srl server.crt server.csr server.key server.bundle.crt key.pkcs12

This script does a couple of things 1. Outputs two files; key.jks and trust.jks. These are stores for the self-signed server key and TLS certificate which will later be provided to the webhook service as arguments 2. Creates a YAML file (webhook-secret.yaml) which defines a Kubernetes TLS secret resource 3. Populates the caBundle field of the local Validating Webhook Kubernetes CRD (webhook-configuration.yaml). This defines the local CA which allows the self-signed cert to be verified.

Local Kubernetes Environment Setup

Now we must set up and configure our local kubernetes cluster to enable it to call our Custom Webhook.

Docker-Desktop Setup

For this example we will use docker-desktop. It provides out of the box kubernetes so to get a local kubernetes cluster running, all we need to do is download docker-desktop and enable Kubernetes. See docker's documentation for more information on this.

Kubernetes Setup

Once docker is all setup, next we need to install kubectl if not done already: * MacOS: brew install kubectl * See documentation for other installation method

Webhook Configuration

The final step is to configure the kubernetes cluster to actually call our webhook whenever an update/create resource request is made. 1. Switch to the docker-desktop k8s context * kubectl config use-context docker-desktop 2. [Optional] You can create a custom namespace for testing if desired or just use the default namespace. 3. Configure the cluster to call the local webhook: * Setup TLS secret: kubectl apply -f webhook-secret.yaml * Apply the webhook configuration: kubectl apply -f webhook-configuration.yaml

Invoking Webhook Endpoints

At this point all the setup and configuration is complete and the webhook is ready to be tested: 1. Start a local instance of the webhook (substitute the command with the command to start up your webhook) * datadex webhook --port=443 --keystore-path=key.jks --keystore-pass=123456 --truststore-path=trust.jks --truststore-pass=123456 2. In the docker-desktop cluster environment you previously setup, create or update the resource that the webhook is listening to: * kubectl apply -f <yaml file>

That's it! At this point, the Kubernetes API Server in docker-desktop will have called your local webhook instance and the behaviour of your webhook in an actual live kubernetes cluster can be tested. You can actively develop and debug with instant feedback.

Alternate Testing Options

If you do not want to test your webhook being called from the actual Kubernetes API Server, an alternative is to use cURL as a quick and easy way to manually call the endpoints of your Webhook:

  1. Generate the TLS certificates. See the "Generating Local TLS Certificate Files" section above
  2. Run the webhook service from the command line:
     mvn package -DskipTests
     datadex webhook --port=443 --keystore-path=key.jks --keystore-pass=123456 --truststore-path=trust.jks --truststore-pass=123456
  1. Use cURL to hit webhook endpoint:
     # Pinging a GET endpoint with no Payload (eg: health endpoint)
     #
     curl -k https://localhost:8443/health

     # Hitting an endpoint expecting data:
     #  
     # The --data option is the payload the the k8s API server will provide when it calls the webhook
     # See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#webhook-request-and-response for details
     #
     curl --data "@<Request JSON File Path>" -k https://localhost:443/<endpoint>