Security Guard

Security-Guard

Note

This integration is current Alpha.

Security-Guard provides visibility into the security status of deployed Knative Services, by monitoring the behaviors of user containers and events. Security-Guard also supports optional blocking of events and termination of user container instances, all based on behavior.

Security-Guard profile and criteria

Security-Guard creates a profile of each user container behavior and of each event behavior. The behaviors are then compared to a pre-defined criteria. If the profile does not meet the criteria, Security-Guard can log alerts, block misbehaving events, or stop misbehaving Service instances, depending on user configurations.

The criteria that a profile is compared to is composed of a set of micro-rules. These rules describe expected behaviors for events and user containers, including expected responses. You can choose to set micro-rules manually, or use Security-Guard's machine learning feature to automate the creation of micro-rules.

Guardians

A per-Service set of micro-rules is stored in the Kubernetes system as a Guardian object. Under Knative, Security-Guard store Guardians using the guardians.guard.security.knative.dev CRDs.

To list all CRD Guardians use:

kubectl get guardians.guard.security.knative.dev

Example Output:

NAME            AGE
helloworld-go   10h

Using Security-Guard

Security-Guard offers situational awareness by writing its alerts to the Service queue proxy log. You may observe the queue-proxy to see alerts.

Security alerts appear in the queue proxy log file and start with the string SECURITY ALERT!. The default setup of Security-Guard is to to learn any new pattern after reporting it. By default, Security-Guard will never block events and will never stop Service instances.

When a new Service is deployed and is actively serving requests, it typically takes about 30 min for Security-Guard to learn the patterns of the Service requests and responses and build corresponding micro-rules. After the initial learning period, Security-Guard updates the micro-rules in the Service Guardian, following which, it sends alerts only when a change in behavior is detected.

Note that in the default setup, Security-Guard continues to learn any new behavior and therefore avoids reporting alerts repeatedly when the new behavior reoccurs. Correct security procedures should include reviewing any new behavior detected by Security-Guard.

Security-Guard can also be configured to operate in other modes of operation, such as:

• Move from auto learning to manual micro-rules management after the initial learning period
• Block requests/responses when they do not conform to the micro-rules

For more information or for troubleshooting help, see the #knative-security channel in Knative Slack.

Security-Guard Use Cases

Security-Guard support four different stages in the life of a knative service from a security standpoint.

• Zero-Day
• Vulnerable
• Exploitable
• Misused

We next detail each stage and how Security-Guard is used to manage the security of the service in that stage.

Zero-Day

Under normal conditions, the Knative user who owns the service is not aware of any known vulnerabilities in the service. Yet, it is reasonable to assume that the service has weaknesses.

Security-Guard offers Knative users the ability to detect/block patterns sent as part of incoming events that may be used to exploit unknown, zero-day, service vulnerabilities.

Vulnerable

Once a CVE that describes a vulnerability in the service is published, the Knative user who owns the service is required to start a process to eliminate the vulnerability by introducing a new revision of the service. This process of removing a known vulnerability may take many weeks to accomplish.

Security-Guard enables Knative users to set micro-rules to detect/block incoming events that include patterns that may be used as part of some future exploit targeting the discovered vulnerability. In this way, users are able to continue offering services, although the service has a known vulnerability.

Exploitable

When a known exploit is found effective in compromising a service, the Knative user who owns the Service needs a way to filter incoming events that contain the specific exploit. This is normally the case during a successful attack, where a working exploit is able to compromise the user-container.

Security-Guard enables Knative users a way to set micro-rules to detect/block incoming events that include specific exploits while allowing other events to be served.

Misused

When an offender has established an attack pattern that is able to take over a service instance, by first exploiting one or more vulnerabilities and then starting to misuse the service instance, stopping the service instance requires the offender to repeat the attack pattern. At any given time, some service instances may be compromised and misused while others behave as designed.

Security-Guard enables Knative users a way to detect/remove misused Service instances while allowing other instances to continue serve events.

Additional resources

See Readme files in the Security-Guard Github Repository.

Installing Security-Guard

Here we show how to install Security-Guard in Knative. Security-Guard is an enhancement to knative-Serving and needs to be installed after the Knative-Serving is successfully installed.

Using Security-Guard requires that your cluster will use an enhanced queue-proxy image.

In addition, Security-Guard includes automation for auto-learning a per service Guardian. Auto-learning requires you to deploy a guard-service on your kubernetes cluster. guard-service should be installed in in the knative-serving namespace.

In production you would typically also wish to enable TLS and Token support to protect the queue-proxy communication with the guard-service as described below.

Install steps

To start this tutorial, after installing Knative Serving, run the following procedure to replace your queue-proxy image and deploy a guard-service.

Install from sourceInstall from released images and yamlsInstall using the Knative Operator

1. Clone the Security-Guard repository using git clone git@github.com:knative-extensions/security-guard.git

2. Do cd security-guard

3. Run ko apply -Rf ./config

Use released images to update your system to enable Security-Guard:

1. Set the feature named queueproxy.mount-podinfo to allowed in the config-features ConfigMap.

   An easy way to do that is using:

   kubectl apply -f https://raw.githubusercontent.com/knative-extensions/security-guard/release-0.4/config/deploy/config-features.yaml
   

2. Set the deployment parameter queue-sidecar-image to gcr.io/knative-releases/knative.dev/security-guard/cmd/queue in the config-deployment ConfigMap.

   An easy way to do that is using:

   kubectl apply -f https://github.com/knative-extensions/security-guard/releases/download/v0.4.0/queue-proxy.yaml
   

3. Add the necessary Security-Guard resources to your cluster using:

   kubectl apply -f https://raw.githubusercontent.com/knative-extensions/security-guard/release-0.4/config/resources/gateAccount.yaml
   kubectl apply -f https://raw.githubusercontent.com/knative-extensions/security-guard/release-0.4/config/resources/serviceAccount.yaml
   kubectl apply -f https://raw.githubusercontent.com/knative-extensions/security-guard/release-0.4/config/resources/guardiansCrd.yaml
   

4. Deploy guard-service on your system to enable automated learning of micro-rules.

   An easy way to do that is using:

   kubectl apply -f https://github.com/knative-extensions/security-guard/releases/download/v0.4.0/guard-service.yaml
   

Note

The example below shows a case where kourier ingress is used, make the necessary changes when installing with istio or contour.

Example script to install Security-Guard and Serving with Kourier using the Knative Operator.

kubectl apply --filename - <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: knative-serving
---
apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
spec:
  security:
    securityGuard:
      enabled: true
  ingress:
    kourier:
      enabled: true
  config:
    network:
      ingress.class: "kourier.ingress.networking.knative.dev"
EOF

kubectl apply -f https://raw.githubusercontent.com/knative-extensions/security-guard/release-0.4/config/resources/gateAccount.yaml

Per Namespace Setup

In order to deploy guard protected services in a namespace, provide guard-gate with the necessary permissions on each namespace used:

kubectl apply -f https://raw.githubusercontent.com/knative-extensions/security-guard/release-0.4/config/resources/gateAccount.yaml

Additional Production Configuration

It is recommended to secure the communication between queue-proxy with the guard-service using one of the following methods:

Manual changesUsing scriptsUsing Knative Operator

1. Add GUARD_SERVICE_TLS=true to the environment of guard-service to enable TLS and server side authentication using a Knative issued certificate. The guard-service will be using the keys in the knative-serving-certs secret of the knative-serving namespace.

2. Add GUARD_SERVICE_AUTH=true to the environment of guard-service to enable client side authentication using tokens

3. Set the queue-sidecar-rootca parameter of the config-deployment configmap in the knative-serving namespace to the public key defined under ca-cert.pem key in the knative-serving-certs secret of the knative-serving namespace. This will inform queue-proxy to use TLS and approve the guard-service certificates.

4. Set queue-sidecar-token-audiences = "guard-service" at the config-deployment configmap in the knative-serving namespace. This will produce a a token with audience guard-service for every queue-proxy instance.

Use the following script to set TLS and Tokens support in guard-service:

echo "Add TLS and Tokens to guard-service"
kubectl patch deployment guard-service -n knative-serving -p '{"spec":{"template":{"spec":{"containers":[{"name":"guard-service","env":[{"name": "GUARD_SERVICE_TLS", "value": "true"}, {"name": "GUARD_SERVICE_AUTH", "value": "true"}]}]}}}}'

Use the following script to set TLS and Tokens support in guard-gates:

echo "Copy the certificate to a temporary file"
ROOTCA="$(mktemp)"
FILENAME=`basename $ROOTCA`
kubectl get secret -n knative-serving knative-serving-certs -o json| jq -r '.data."ca-cert.pem"' | base64 -d >  $ROOTCA

echo "Get the certificate in a configmap friendly form"
CERT=`kubectl create cm config-deployment --from-file $ROOTCA -o json --dry-run=client |jq .data.\"$FILENAME\"`

echo "Add TLS and Tokens to config-deployment configmap"
kubectl patch cm config-deployment -n knative-serving -p '{"data":{"queue-sidecar-token-audiences": "guard-service", "queue-sidecar-rootca": '"$CERT"'}}'

echo "cleanup"
rm  $ROOTCA

Use the following script to read the TLS and Token settings of both guard-service and guard-gates:

echo "Results:"
kubectl get cm config-deployment -n knative-serving -o json|jq '.data'
kubectl get deployment guard-service -n knative-serving -o json|jq .spec.template.spec.containers[0].env

Use the following script to unset TLS and Tokens support in guard-service:

echo "Remove TLS and Tokens from  guard-service deployment"
kubectl patch deployment guard-service -n knative-serving -p '{"spec":{"template":{"spec":{"containers":[{"name":"guard-service","env":[{"name": "GUARD_SERVICE_TLS", "value": "false"}, {"name": "GUARD_SERVICE_AUTH", "value": "false"}]}]}}}}'

Use the following script to unset TLS and Tokens support in guard-gates:

echo "Remove TLS and Tokens from config-deployment configmap"
kubectl patch cm config-deployment -n knative-serving -p '{"data":{"queue-sidecar-token-audiences": "", "queue-sidecar-rootca": ""}}'

Note

The example below shows a case where kourier ingress is used, make the necessary changes when installing with istio or contour.

Example script to install Security-Guard with TLS and Serving with Kourier using the Knative Operator.

kubectl apply --filename - <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: knative-serving
---
apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
EOF

echo "Waiting for secret to be created (CTRL-C to exit)"
PEM=""
while [[ -z $PEM ]]
do
  echo -n "."
  sleep 1
  DOC=`kubectl get secret -n knative-serving knative-serving-certs -o json 2> /dev/null`
  PEM=`echo $DOC | jq -r '.data."ca-cert.pem"'`
done
echo " Secret found!"

echo "Copy the certificate to file"
ROOTCA="$(mktemp)"
FILENAME=`basename $ROOTCA`
echo $PEM | base64 -d >  $ROOTCA

echo "Create a temporary config-deployment configmap with the certificate"
CERT=`kubectl create cm config-deployment --from-file $ROOTCA -o json --dry-run=client |jq .data.\"$FILENAME\"`

echo "cleanup"
rm $ROOTCA

kubectl apply --filename - <<EOF
apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
spec:
  deployments:
  - name: guard-service
    env:
    - container: guard-service
      envVars:
      - name: GUARD_SERVICE_TLS
        value: "true"
      - name: GUARD_SERVICE_AUTH
        value: "true"
  security:
    securityGuard:
      enabled: true
  ingress:
    kourier:
      enabled: true
  config:
    network:
      ingress.class: "kourier.ingress.networking.knative.dev"
    deployment:
      queue-sidecar-rootca: ${CERT}
      queue-sidecar-token-audiences: guard-service
EOF

Security-Guard monitoring quickstart

This tutorial shows how you can use Security-Guard to protect a deployed Knative Service.

Creating and deploying a service

Tip

The following commands create a helloworld-go sample Service while activating and configuring the Security-Guard extension for this Service. You can modify these commands, including changing the Security-Guard configuration for your service using either the kn CLI or changing the service yaml based on this example.

Create a sample securedService:

Apply YAMLkn services CLIkn func CLI

1. Create a YAML file using the following example:

   apiVersion: serving.knative.dev/v1
   kind: Service
   metadata:
     name: helloworld-go
     namespace: default
   spec:
     template:
       metadata:
           annotations:
             features.knative.dev/queueproxy-podinfo: enabled
             qpoption.knative.dev/guard-activate: enable
       spec:
         containers:
           - image: ghcr.io/knative/helloworld-go:latest
             env:
               - name: TARGET
                 value: "Secured World"
   

2. Apply the YAML file by running the command:

   kubectl apply -f <filename>.yaml
   

   Where <filename> is the name of the file you created in the previous step.

Creating a service using CLI

kn service create helloworld-go \
    --image ghcr.io/knative/helloworld-go:latest \
    --env "TARGET=Secured World" \
    --annotation features.knative.dev/queueproxy-podinfo=enabled \
    --annotation qpoption.knative.dev/guard-activate=enable

Creating a function using CLI.

Add the following deploy.annotations to your func.yaml file located in your project dir"

deploy:
  annotations:
    features.knative.dev/queueproxy-podinfo: enabled
    qpoption.knative.dev/guard-activate: enable

Deploy as you would deploy any other function

kn func deploy

After the Service has been created, Guard starts monitoring the Service Pods and all Events sent to the Service.

Cleanup

To remove the deployed service use:

Apply YAMLkn CLI

Delete using the YAML file used to create the service by running the command:

kubectl delete -f <filename>.yaml

Where <filename> is the name of the file you created in the previous step.

kn service delete helloworld-go

To remove the Guardian of the deployed service use:

```bash
kubectl delete guardians.guard.security.knative.dev helloworld-go
```

Security-Guard example alerts

1. Send an event with unexpected query string, for example:

   curl "http://helloworld-go.default.52.118.14.2.sslip.io?a=3"
   

   This returns an output similar to the following:

   Hello Secured World!
   

2. Check alerts:

   kubectl logs deployment/helloworld-go-00001-deployment queue-proxy|grep "SECURITY ALERT!"
   

   This returns an output similar to the following:

   ...SECURITY ALERT! HttpRequest -> [QueryString:[KeyVal:[Key a is not known,],],]
   

3. Send an event with unexpected long url, for example:

   curl "http://helloworld-go.default.52.118.14.2.sslip.io/AAAAAAAAAAAAAAAA"
   

   This returns an output similar to the following:

   Hello Secured World!
   

4. Check alerts:

   kubectl logs deployment/helloworld-go-00001-deployment queue-proxy|grep "SECURITY ALERT!"
   

   This returns an output similar to the following:

   ...SECURITY ALERT! HttpRequest -> [Url:[Segments:[Counter out of Range: 1,],Val:[Letters:[Counter out of Range: 16,],Sequences:[Counter out of Range: 1,],],],].