This post was most recently updated on March 30th, 2023.
4 min read.A while back I got a request to change the default certificate for an AKS cluster. A customer didn’t like the “Kubernetes Fake Certificate” that’s shown when no other matching certificates can be found, and they wanted it changed.
So yours truly started googling, and as a result, this article explains how to do just that!
Background
The customer didn’t invent this requirement out of thin air. The certificate does expose one important piece of information – that it’s being served from a Kubernetes instance. This has some cybersecurity implications, as it makes the recon work of a potential attacker a tiny bit easier.
See, if someone knows the hostname or domain of their potential victim, they can easily figure out the IP address serving that address, and navigate to it – even just by using their browser. And then Kubernetes ingress controllers will happily serve your https request with an invalid, self-signed, fake certificate that says “kubernetes” in cat-sized letters.
Well – it’s not a big problem, but it’s a problem nonetheless. So the customer wanted to get rid of it.
Solution
The solution is to configure the ingress controllers for your Kubernetes cluster to serve a different default SSL/TLS certificate. This requires a bit of scripting, but shouldn’t be too hard for us!
As a prerequisite, you’ll need the az module for PowerShell, kubectl (included in the module), and an Azure Subscription with AKS cluster(s) available. The same guide probably partially applies to any Kubernetes cluster, but AKS is the flavour I use.
Time needed: 20 minutes
How to change the default SSL/TLS certificate for your AKS ingress controllers?
- Generate a new certificate
This can be whatever you like. It’s going to be recognized as an invalid certificate nonetheless.
Verify it’s in PEM format – i.e., you can open it up in notepad and the first line you see is something like this:
—–BEGIN CERTIFICATE—–
This guide supposes your certificate is called acme.cer and the matching private key is in a file called acme.key, and both are PEM-formatted (base64 with standard headers). - Figure out your k8s details
You’ll need a namespace for your secret, and you’ll need to identify your ingress controllers and the external IP address.
Namespace can be any valid one, but pay attention to the IP address! - Upload your certificate
You’ll need to create a new secret for your certificate contents and upload the certificate into it using kubectl.
kubectl is available in Azure CLI (or you can get it separately if you really want to). The commands are going to look somewhat like the below:az account set --subscription your-guid-here
az aks get-credentials --resource-group Your-Group --name f15feb1
kubectl create secret tls 'custom-fake-cert' --cert='./acme.cer' --key='./acme.key'
You should get something like this back:secret/custom-fake-cert created
You can verify the deployment was successful by checking the Configuration > Secrets view in AKS and see if your secret is there: - Find your ingress controllers
Now we’ll want to navigate to Workloads and find our ingress controllers. You’ll hopefully recognize them by name.
- Modify the ingress controller deployment
Now, open up the deployment of your ingress controller(s) and click “YAML”. You’ll want to add a new argument to the startup of your controller.
You’ll want to add this:'--default-ssl-certificate=your-namespace/your-secret-name'
So in our case, that’ll of course be:'--default-ssl-certificate=default/custom-fake-cert'
That’ll come in the “args” section for your controllers – the whole thing looks somewhat like the excerpt below:containers:
- name: controller
image: >-
registry.k8s.io/ingress-nginx/controller:v1.5.1@sha256:...
args:
- /nginx-ingress-controller
- '--publish-service=$(POD_NAMESPACE)/ingress-nginx-controller'
- '--election-id=ingress-nginx-leader'
- '--controller-class=k8s.io/ingress-nginx'
- '--ingress-class=nginx'
- '--configmap=$(POD_NAMESPACE)/ingress-nginx-controller'
- '--validating-webhook=:8443'
- '--validating-webhook-certificate=/usr/local/certificates/cert'
- '--validating-webhook-key=/usr/local/certificates/key'
- '--default-ssl-certificate=default/custom-fake-cert'
ports:
- name: http
containerPort: 80
protocol: TCP
- name: https
containerPort: 443
protocol: TCP
- name: webhook
containerPort: 8443
protocol: TCP - Recreate your pods
Theoretically, this should happen automatically. But you can make sure it’s quick by removing the pods yourself (supposing this isn’t a live production environment with a lot of users trying to access it right now!)
- Verify your changes have been applied
You can open the YAML for your pods, and see if the argument is there.
If it isn’t, just do the contra-Einsteinian thing, and try again until it works. Eventually, it should.
It just isn’t particle physics, is it? Oh well. - Test!
Now you should be able to navigate to your desired IP Address and give the certificate another closer look.
Our new fake certificate throwing the expected validation error in the browser. Beautiful!
And that should be it! The change should stick until you redeploy – so if you want to make it permanent, better incorporate it into your deployment pipeline!
FAQ
ACME is not just A Company Making Everything, it also stands for stands for Automatic Certificate Management Environment. The default SSL/TLS certificate that AKS (and Kubernetes in general) uses for ingresses is an untrusted one, and if you access it via the IP of the node hosting your pods, it won’t be valid anyway. It’s not a very useful certificate and shouldn’t be used for any serious workloads.
If you want to force pods to pick up any updates to the deployment in AKS, you need to update the deployment.kubernetes.io/revision label for your deployment.
Why this isn’t always required to update your pods definition is beyond me. But it works to force it.
References
- https://kubernetes.io/docs/tasks/configmap-secret/managing-secret-using-kubectl/
- How to verify your private key matches the certificate
- “Destination Path Too Long” when copying files in File Explorer? Easy workaround(s)! - August 27, 2024
- Where does NetSpot (wifi heatmapping tool) store its project files? - August 20, 2024
- How to fix Bitlocker failing to activate? - August 13, 2024