Run Kubernetes in production - it'll be easy, they said

How to replace the default fake “ACME” certificate for Kubernetes/AKS?

This post was most recently updated on March 30th, 2023.

4 min read.

A while back I got a request to change the default certificate for an AKS cluster. A customer didn’t like the “Kubernetes Fake Certificate” that’s shown when no other matching certificates can be found, and they wanted it changed.

So yours truly started googling, and as a result, this article explains how to do just that!

Background

The customer didn’t invent this requirement out of thin air. The certificate does expose one important piece of information – that it’s being served from a Kubernetes instance. This has some cybersecurity implications, as it makes the recon work of a potential attacker a tiny bit easier.

See, if someone knows the hostname or domain of their potential victim, they can easily figure out the IP address serving that address, and navigate to it – even just by using their browser. And then Kubernetes ingress controllers will happily serve your https request with an invalid, self-signed, fake certificate that says “kubernetes” in cat-sized letters.

Well – it’s not a big problem, but it’s a problem nonetheless. So the customer wanted to get rid of it.

Solution

The solution is to configure the ingress controllers for your Kubernetes cluster to serve a different default SSL/TLS certificate. This requires a bit of scripting, but shouldn’t be too hard for us!

As a prerequisite, you’ll need the az module for PowerShell, kubectl (included in the module), and an Azure Subscription with AKS cluster(s) available. The same guide probably partially applies to any Kubernetes cluster, but AKS is the flavour I use.

Time needed: 20 minutes

How to change the default SSL/TLS certificate for your AKS ingress controllers?

  1. Generate a new certificate

    This can be whatever you like. It’s going to be recognized as an invalid certificate nonetheless.

    Verify it’s in PEM format – i.e., you can open it up in notepad and the first line you see is something like this:
    —–BEGIN CERTIFICATE—– 

    This guide supposes your certificate is called acme.cer and the matching private key is in a file called acme.key, and both are PEM-formatted (base64 with standard headers).

  2. Figure out your k8s details

    You’ll need a namespace for your secret, and you’ll need to identify your ingress controllers and the external IP address.

    Namespace can be any valid one, but pay attention to the IP address!
    How to find out your external IP address in AKS?

  3. Upload your certificate

    You’ll need to create a new secret for your certificate contents and upload the certificate into it using kubectl.

    kubectl is available in Azure CLI (or you can get it separately if you really want to). The commands are going to look somewhat like the below:

    az account set --subscription your-guid-here
    az aks get-credentials --resource-group Your-Group --name f15feb1
    kubectl create secret tls 'custom-fake-cert' --cert='./acme.cer' --key='./acme.key'


    You should get something like this back:
    secret/custom-fake-cert created

    You can verify the deployment was successful by checking the Configuration > Secrets view in AKS and see if your secret is there:
    Our new secret for the custom certificate in AKS

  4. Find your ingress controllers

    Now we’ll want to navigate to Workloads and find our ingress controllers. You’ll hopefully recognize them by name.

  5. Modify the ingress controller deployment

    Now, open up the deployment of your ingress controller(s) and click “YAML”. You’ll want to add a new argument to the startup of your controller.

    You’ll want to add this:
    '--default-ssl-certificate=your-namespace/your-secret-name'

    So in our case, that’ll of course be:
     '--default-ssl-certificate=default/custom-fake-cert'

    That’ll come in the “args” section for your controllers – the whole thing looks somewhat like the excerpt below:
     containers:
            - name: controller
              image: >-
                registry.k8s.io/ingress-nginx/controller:v1.5.1@sha256:...
              args:
                - /nginx-ingress-controller
                - '--publish-service=$(POD_NAMESPACE)/ingress-nginx-controller'
                - '--election-id=ingress-nginx-leader'
                - '--controller-class=k8s.io/ingress-nginx'
                - '--ingress-class=nginx'
                - '--configmap=$(POD_NAMESPACE)/ingress-nginx-controller'
                - '--validating-webhook=:8443'
                - '--validating-webhook-certificate=/usr/local/certificates/cert'
                - '--validating-webhook-key=/usr/local/certificates/key'
                - '--default-ssl-certificate=default/custom-fake-cert'
              ports:
                - name: http
                  containerPort: 80
                  protocol: TCP
                - name: https
                  containerPort: 443
                  protocol: TCP
                - name: webhook
                  containerPort: 8443
                  protocol: TCP

  6. Recreate your pods

    Theoretically, this should happen automatically. But you can make sure it’s quick by removing the pods yourself (supposing this isn’t a live production environment with a lot of users trying to access it right now!)

  7. Verify your changes have been applied

    You can open the YAML for your pods, and see if the argument is there.

    If it isn’t, just do the contra-Einsteinian thing, and try again until it works. Eventually, it should.

    It just isn’t particle physics, is it? Oh well.

  8. Test!

    Now you should be able to navigate to your desired IP Address and give the certificate another closer look.

    Our new fake certificate throwing the expected validation error in browser. Beautiful!
    Our new fake certificate throwing the expected validation error in the browser. Beautiful!


And that should be it! The change should stick until you redeploy – so if you want to make it permanent, better incorporate it into your deployment pipeline!

FAQ

What is the Kubernetes ACME certificate, anyway?

ACME is not just A Company Making Everything, it also stands for stands for Automatic Certificate Management Environment. The default SSL/TLS certificate that AKS (and Kubernetes in general) uses for ingresses is an untrusted one, and if you access it via the IP of the node hosting your pods, it won’t be valid anyway. It’s not a very useful certificate and shouldn’t be used for any serious workloads.

How can I make sure Kubernetes picks up the changes to a deployment?

If you want to force pods to pick up any updates to the deployment in AKS, you need to update the deployment.kubernetes.io/revision label for your deployment.

Why this isn’t always required to update your pods definition is beyond me. But it works to force it.

References

mm
5 1 vote
Article Rating
Subscribe
Notify of
guest

0 Comments
Inline Feedbacks
View all comments