Cert Manager With Http01 Challeges

· 6 min read

Any traffic traversing Internet should be encrypted, as, even without we knowing it, it usually contains sensitive information, even the browser’s user agent can be sensitive, for instance, if an unpatched vulnerability affects that version. Until few years ago, SSL Certificates were very expensive, but the Electronic Frontier Foundation with the Let’s Encrypt project made them free and universally accessible.

I’ve been using Azure Kubernetes Service for a while thanks to a VS Subscription provided by one of my customers, the UNICC, but was using certificates issued by a personal CA thanks to EasyRSA as mentioned on my previous post. But the cert-manager team made easy to perform automatic requests for Let’s Encrypt certificates from kubernetes clusters, so I decided to give it a try.

I read several posts, some of them outdated, and others overcomplicated, most of my test was based on the Deploy cert-manager on Azure Kubernetes Service (AKS) and use Let’s Encrypt to sign a certificate for an HTTPS website tutorial on the official cert-manager documentation, but I wanted a simpler solution, so I changed the challenge to use the http01 instead of the dns01 so no modifications on the dns are required each time a challenge has to be solved. Also I previously add a wildcard (*) record on my dns zone pointing all *.aks.garciaamaya.com entries to my cluster’s load balancer address.

Deploying the controller and the CRDs

The first step was to deploy the CRDs and the controllers for cert-manager, I used the official helm chart as suggested on the tutorial:

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade cert-manager jetstack/cert-manager \
    --install \
    --create-namespace \
    --wait \
    --namespace cert-manager \
    --set installCRDs=true

It created several resources:

$ kubectl -n cert-manager get all
NAME                                          READY   STATUS    RESTARTS   AGE
pod/cert-manager-7ccffc4d98-b5m6l             1/1     Running   0          3h9m
pod/cert-manager-cainjector-7dbbc4f4f-r5jvd   1/1     Running   0          3h9m
pod/cert-manager-webhook-66d49bbf6f-nlpwj     1/1     Running   0          3h9m

NAME                           TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
service/cert-manager           ClusterIP   10.0.31.58    <none>        9402/TCP   3h9m
service/cert-manager-webhook   ClusterIP   10.0.34.192   <none>        443/TCP    3h9m

NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cert-manager              1/1     1            1           3h9m
deployment.apps/cert-manager-cainjector   1/1     1            1           3h9m
deployment.apps/cert-manager-webhook      1/1     1            1           3h9m

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/cert-manager-7ccffc4d98             1         1         1       3h9m
replicaset.apps/cert-manager-cainjector-7dbbc4f4f   1         1         1       3h9m
replicaset.apps/cert-manager-webhook-66d49bbf6f     1         1         1       3h9m

And some CRDs:

$ kubectl api-resources --api-group=cert-manager.io
NAME                  SHORTNAMES   APIVERSION           NAMESPACED   KIND
certificaterequests   cr,crs       cert-manager.io/v1   true         CertificateRequest
certificates          cert,certs   cert-manager.io/v1   true         Certificate
clusterissuers                     cert-manager.io/v1   false        ClusterIssuer
issuers                            cert-manager.io/v1   true         Issuer

Creating the issuers

For creating the certificate issuer, I followed the HTTP01 configuration page on cert-manager’s documentation, but, as this was a testing cluster, and was meant to be used only by me, I deployed the issuer cluster-wide to reuse it on every namespace.

First I created the letsencrypt-staging.yaml file with the following contents:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: $EMAIL_ADDRESS
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - http01:
        ingress:
          class: nginx

And applied it using the following command (use your own email):

EMAIL_ADDRESS=myemail@domain.tld envsubst < letsencrypt-staging.yaml | kubectl apply -f -

and got:

clusterissuer.cert-manager.io/letsencrypt-staging created

Then I created the production issuer, also cluster-wide, using the file letsencrypt-production.yaml:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    email: $EMAIL_ADDRESS
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-production
    solvers:
    - http01:
        ingress:
          class: nginx

And applied it as before:

EMAIL_ADDRESS=myemail@domain.tld envsubst < letsencrypt-production.yaml | kubectl apply -f -

and also got:

clusterissuer.cert-manager.io/letsencrypt-production created

Testing the certificate issuance

For testing the solution, I followed the Securing NGINX-ingress tutorial, also from the cert-manager documentation. I had issues and messages about being unable to find the the letsencrypt-staging issuer, but then I found that in the Issuers section of the tutorial, explains that the annotation cert-manager.io/issuer must be changed to cert-manager.io/cluster-issuer if using ClusterIssuers.

I put all the needed resources in a yaml file called kuard.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: kuard

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kuard
  namespace: kuard
spec:
  selector:
    matchLabels:
      app: kuard
  replicas: 1
  template:
    metadata:
      labels:
        app: kuard
    spec:
      containers:
      - image: gcr.io/kuar-demo/kuard-amd64:1
        imagePullPolicy: Always
        name: kuard
        ports:
        - containerPort: 8080

---
apiVersion: v1
kind: Service
metadata:
  name: kuard
  namespace: kuard
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    app: kuard

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kuard
  namespace: kuard
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-staging"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - kuard.$BASE_DOMAIN
    secretName: kuard.tls
  rules:
  - host: kuard.$BASE_DOMAIN
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kuard
            port:
              number: 80

And applied it

BASE_DOMAIN=aks.garciaamaya.com envsubst < kuard.yaml |kubectl apply -f -

and got the confirmation messages for all resources:

namespace/kuard created
deployment.apps/kuard created
service/kuard created
ingress.networking.k8s.io/kuard created

I tested it using curl and got the data of the Fake Certificate as expected.

curl -kv https://kuard.aks.garciaamaya.com

*   Trying 20.71.64.0:443...
* Connected to kuard.aks.garciaamaya.com (20.71.64.0) port 443 (#0)
... REDACTED ...
* Server certificate:
*  subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  start date: Apr 24 16:50:36 2023 GMT
*  expire date: Apr 23 16:50:36 2024 GMT
*  issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.

Then I edit the ingress resource to use the letsencrypt-production certificate authority changing the cert-manager.io/issuer annotation value to letsencrypt-production, and checked the status with:

kubectl -n kuard describe ingress kuard

Events:
  Type     Reason     Age               From                                       Message
  ----     ------     ----              ----                                       -------
  Normal   Generated  20m               cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "kuard.tls-wh82c"
  Normal   Requested  20m               cert-manager-certificates-request-manager  Created new CertificateRequest resource "kuard.tls-2vffn"
  Normal   Requested  18m               cert-manager-certificates-request-manager  Created new CertificateRequest resource "kuard.tls-qw6dc"
  Normal   Issuing    5s (x2 over 20m)  cert-manager-certificates-trigger          Issuing certificate as Secret does not exist
  Normal   Generated  4s                cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "kuard.tls-zxvft"
  Normal   Requested  4s                cert-manager-certificates-request-manager  Created new CertificateRequest resource "kuard.tls-9d4vk"

In the Events section I was able to check the status of the request. Once updated, I tested the certificate using openssl:

echo|openssl s_client -connect kuard.aks.garciaamaya.com:443 -servername kuard.aks.garciaamaya.com 2>/dev/null \
  |openssl x509 -noout -subject -issuer

subject=CN = kuard.aks.garciaamaya.com
issuer=C = US, O = Let's Encrypt, CN = R3

Now the page is using a certificate issued by a trusted CA and got my lock in the browser. kuar screenshot

Conclusion

Installing cert-manager for kubernetes was easier than I originally thought, almost all the work is done by the helm chart, at least for a basic deployment, working with several issuers including self-signed certificates or even as a subCA from an Enterprise CA gives it power and flexibility, and, of course, automating the request of certificates and their renewals, saves a lot of time for DevOps Engineers like me.

References