Skip to content

Configuring Knative cert-manager integration

Knative Serving relies on a bridging component to use cert-manager for automated certificate provisioning. If you intend to use that feature, you need to enable the Knative cert-manager integration.

Prerequisites

The following must be installed on your Knative cluster:

Warning

Make sure you have installed cert-manager. Otherwise, the Serving controller will not start up correctly.

Issuer configuration

The Knative cert-manager integration defines three references to cert-manager issuers to configure different CAs for the three Knative Serving encryption features:

  • issuerRef: issuer for external-domain certificates used for ingress.
  • clusterLocalIssuerRef: issuer for cluster-local-domain certificates used for ingress.
  • systemInternalIssuerRef: issuer for certificates for system-internal-tls certificates used by Knative internal components.

The following example uses a self-signed ClusterIssuer and the Knative cert-manager integration references that ClusterIssuer for all three configurations. As this should not be used in production (and does not support rotating the CA without downtime), you should think about which CA should be used for each use case and how trust will be distributed to the clients calling the encrypted services. For the Knative system components, Knative provides a way to specify a bundle of CAs that should be trusted (more on this below).

There is no general answer on how to structure this, here an example on how it could look like:

Feature Certificate Authority Trusted via
external-domain-tls Let's encrypt Browser clients have the Let's encrypt chain already, all the root CAs will be added in company-wide Docker base image by DevOps team.
cluster-local-domain-tls CA provided by cluster operator The CA is managed by the DevOps team and will be added in company-wide Docker base image.
system-internal-tls Self-signed Cert-Manager ClusterIssuer The CA will be populated by cert-manager. DevOps team will use trust-manager to distribute the CA to Knative system components.

Issuer selection

In general, you can refer to the cert-manager documentation. There are examples available for:

Important

Please note, that not all issuer types work for each Knative feature.

cluster-local-domain-tls needs to be able to sign certificates for cluster-local domains like myapp.<namespace>, myapp.<namespace>.svc and myapp.<namespace>.svc.cluster.local. The CA is usually outside the cluster, so verification via ACME protocol (DNS01/HTTP01) is impossible. You can use an issuer that allows the creation of these certificates (e.g., a CA issuer).

system-internal-tls needs to be able to sign specific SANs that Knative validates for. The defined set of SANs is:

  • kn-routing
  • kn-user-<namespace> ( is each namespace where Knative Services are/will be created)
  • data-plane.knative.dev

As this is also not possible via ACME protocol (DNS01/HTTP01), you need to configure an issuer that allows creating the these certificates (e.g. CA issuer).

Configuring issuers

Warning

The self-signed cluster issuer should not be used in production, please see Issuer configuration above for more information.

  1. Create and apply the following self-signed ClusterIssuer to your cluster:

    # this issuer is used by cert-manager to sign all certificates
    apiVersion: cert-manager.io/v1
    kind: ClusterIssuer
    metadata:
      name: cluster-selfsigned-issuer
    spec:
      selfSigned: {}
    ---
    apiVersion: cert-manager.io/v1
    kind: ClusterIssuer # this issuer is specifically for Knative, it will use the CA stored in the secret created by the Certificate below
    metadata:
      name: knative-selfsigned-issuer
    spec:
      ca:
        secretName: knative-selfsigned-ca
    ---
    apiVersion: cert-manager.io/v1
    kind: Certificate # this creates a CA certificate, signed by cluster-selfsigned-issuer and stored in the secret knative-selfsigned-ca
    metadata:
      name: knative-selfsigned-ca
      namespace: cert-manager #  If you want to use it as a ClusterIssuer the secret must be in the cert-manager namespace.
    spec:
      secretName: knative-selfsigned-ca
      commonName: knative.dev
      usages:
        - server auth
      isCA: true
      issuerRef:
        kind: ClusterIssuer
        name: cluster-selfsigned-issuer
    
  2. Ensure that the ClusterIssuer is ready:

    kubectl get clusterissuer cluster-selfsigned-issuer -o yaml
    kubectl get clusterissuer knative-selfsigned-issuer -o yaml
    
    Result: The Status.Conditions should include Ready=True.

  3. Then reference the ClusterIssuer in the config-certmanager ConfigMap:

    kubectl edit configmap config-certmanager -n knative-serving
    

    Add the fields within the data section:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: config-certmanager
      namespace: knative-serving
      labels:
        networking.knative.dev/certificate-provider: cert-manager
    data:
      issuerRef: |
        kind: ClusterIssuer
        name: knative-selfsigned-issuer
      clusterLocalIssuerRef: |
        kind: ClusterIssuer
        name: knative-selfsigned-issuer
      systemInternalIssuerRef: |
        kind: ClusterIssuer
        name: knative-selfsigned-issuer
    

    Ensure that the file was updated successfully:

    kubectl get configmap config-certmanager -n knative-serving -o yaml
    

Managing trust and rotation without downtime

As pointed out above, each client that calls a Knative Service using HTTPS needs to trust the CA and/or intermediate chain. If you take a look at the encryption overview, you can see that there are multiple places where trust needs to be distributed:

  • Cluster external client (Browser and/or other application): this is considered out of scope of Knative.
  • Cluster internal client (e.g. Knative or Vanilla K8s workload): see below.
  • Knative system components (e.g. Activator, Queue-Proxy, Ingress-Controller): see below.

Trusting for cluster internal clients (e.g. Knative or Vanilla K8s workload)

As Knative does not control all workload and the settings are highly dependent on your runtime and/or language, this is out of scope of Knative. But here a few points to consider, as there are several ways on how to provide CAs to your application:

  • Adding the CA bundle to a Container image on build-time (be aware that this complicates CA rotation, you basically need to rebuild every application)
  • Mounting a CA bundle to the filesystem (e.g. from a Secret or ConfigMap)
  • Reading it from environment variable
  • Accessing it from a Secret/ConfigMap via K8s API

If reloading certificates without downtime is important for your client, the workload must either watch changes on the K8s resource (Secret/ConfigMap) or watch the filesystem. If the workload is watching the filesystem, it is important to note that using ionotify to catch changing Secrets/ConfigMaps is not very reliable on K8s. Tests have shown that it is more reliable to regularly poll and check the certificate on the filesystem for changes.

Here are a few examples for golang:

Trusting for Knative system components

Knative system components can be configured to trust one or many CA bundles from ConfigMaps. The cluster operator needs to ensure, to configure them accordingly to avoid any downtimes during a rotation. Knative components look for a ConfigMap in the namespace where the component runs, e.g:

  • knative-serving
  • istio-system (when using net-istio)
  • kourier-system (when using net-kourier)
  • Each namespace where a Knative Service runs

Knative looks for a ConfigMap with the label networking.knative.dev/trust-bundle: "true" and will read all data keys (regardless of the name). One key can contain one or multiple CAs/Intermediates. If they are valid, they will be added to the trust store of the Knative components.

Here is an example of how ConfigMap could look like:

apiVersion: v1
data:
  cacerts.pem: |
    -----BEGIN CERTIFICATE-----
    MIIDDTCCAfWgAwIBAgIQMQuip05h7NLQq2TB+j9ZmTANBgkqhkiG9w0BAQsFADAW
    MRQwEgYDVQQDEwtrbmF0aXZlLmRldjAeFw0yMzExMjIwOTAwNDhaFw0yNDAyMjAw
    OTAwNDhaMBYxFDASBgNVBAMTC2tuYXRpdmUuZGV2MIIBIjANBgkqhkiG9w0BAQEF
    AAOCAQ8AMIIBCgKCAQEA3clC3CV7sy0TpUKNuTku6QmP9z8JUCbLCPCLACCUc1zG
    FEokqOva6TakgvAntXLkB3TEsbdCJlNm6qFbbko6DBfX6rEggqZs40x3/T+KH66u
    4PvMT3fzEtaMJDK/KQOBIvVHrKmPkvccUYK/qWY7rgBjVjjLVSJrCn4dKaEZ2JNr
    Fd0KNnaaW/dP9/FvviLqVJvHnTMHH5qyRRr1kUGTrc8njRKwpHcnUdauiDoWRKxo
    Zlyy+MhQfdbbyapX984WsDjCvrDXzkdGgbRNAf+erl6yUm6pHpQhyFFo/zndx6Uq
    QXA7jYvM2M3qCnXmaFowidoLDsDyhwoxD7WT8zur/QIDAQABo1cwVTAOBgNVHQ8B
    Af8EBAMCAgQwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDwYDVR0TAQH/BAUwAwEB/zAd
    BgNVHQ4EFgQU7p4VuECNOcnrP9ulOjc4J37Q2VUwDQYJKoZIhvcNAQELBQADggEB
    AAv26Vnk+ptQrppouF7yHV8fZbfnehpm07HIZkmnXO2vAP+MZJDNrHjy8JAVzXjt
    +OlzqAL0cRQLsUptB0btoJuw23eq8RXgJo05OLOPQ2iGNbAATQh2kLwBWd/CMg+V
    KJ4EIEpF4dmwOohsNR6xa/JoArIYH0D7gh2CwjrdGZr/tq1eMSL+uZcuX5OiE44A
    2oXF9/jsqerOcH7QUMejSnB8N7X0LmUvH4jAesQgr7jo1JTOBs7GF6wb+U76NzFa
    8ms2iAWhoplQ+EHR52wffWb0k6trXspq4O6v/J+nq9Ky3vC36so+G1ZFkMhCdTVJ
    ZmrBsSMWeT2l07qeei2UFRU=
    -----END CERTIFICATE-----
kind: ConfigMap
metadata:
  labels:
    networking.knative.dev/trust-bundle: "true"
  name: knative-bundle
  namespace: knative-serving

Using trust-manager to distribute the bundle

As it can be a cumbersome task to distribute the CA bundle to all the namespaces, you can use trust-manager to automatically distribute the CA bundles. Please refer to their documentation for more information on how to do this.

Trust during rotation

During a rotation of a CA and/or Intermediate certificates your clients will need to trust the old and the new CA/chain until the rotation is done. Using the trust approach from above, you can do a full chain rotation without downtime:

  1. Make sure your existing setup is up and running.
  2. Make sure all Knative Services have the relevant certificates and are not expired.
  3. Make sure your CA (and full chain) is not expired.
  4. Add the existing and the new CA (and the full chain) to the trust bundle (either manually or via trust-manager).
  5. Reconfigure your cert-manager ClusterIssuers or Issuers to use the new CA.
  6. Wait until all certificates are expired and are renewed by cert-manager.
  7. All certificates are now signed by the new CA.
  8. Add some grace period to make sure all components did pick up all the changes.
  9. Remove the old CA from the trust bundle.

We use analytics and cookies to understand site traffic. Information about your use of our site is shared with Google for that purpose. Learn more.

× OK