GCP Cloud Pub/Sub source

This sample shows how to configure the GCP PubSub event source. This event source is most useful as a bridge from other GCP services, such as Cloud Storage, IoT Core and Cloud Scheduler.

Prerequisites

  1. Create a Google Cloud project and install the gcloud CLI and run gcloud auth login. This sample will use a mix of gcloud and kubectl commands. The rest of the sample assumes that you’ve set the $PROJECT_ID environment variable to your Google Cloud project id, and also set your project ID as default using gcloud config set project $PROJECT_ID.

  2. Setup Knative Serving

  3. Setup Knative Eventing. In addition, install the GCP PubSub event source from release-gcppubsub.yaml:

   kubectl apply --filename https://github.com/knative/eventing-sources/releases/download/v0.5.0/gcppubsub.yaml
  1. Enable the Cloud Pub/Sub API on your project:
   gcloud services enable pubsub.googleapis.com
  1. Create a GCP Service Account. This sample creates one service account for both registration and receiving messages, but you can also create a separate service account for receiving messages if you want additional privilege separation.

    1. Create a new service account named knative-source with the following command: shell gcloud iam service-accounts create knative-source
    2. Give that Service Account the Pub/Sub Editor role on your GCP project: shell gcloud projects add-iam-policy-binding $PROJECT_ID \ --member=serviceAccount:knative-source@$PROJECT_ID.iam.gserviceaccount.com \ --role roles/pubsub.editor
    3. Download a new JSON private key for that Service Account. Be sure not to check this key into source control! shell gcloud iam service-accounts keys create knative-source.json \ --iam-account=knative-source@$PROJECT_ID.iam.gserviceaccount.com
    4. Create two secrets on the kubernetes cluster with the downloaded key:
      # Note that the first secret may already have been created when installing
      # Knative Eventing. The following command will overwrite it. If you don't
      # want to overwrite it, then skip this command.
      kubectl --namespace knative-sources create secret generic gcppubsub-source-key --from-file=key.json=knative-source.json --dry-run --output yaml | kubectl apply --filename -
    
      # The second secret should not already exist, so just try to create it.
      kubectl --namespace default create secret generic google-cloud-key --from-file=key.json=knative-source.json

    gcppubsub-source-key and key.json are pre-configured values in the controller-manager StatefulSet which manages your Eventing sources.

    google-cloud-key and key.json are pre-configured values in gcp-pubsub-source.yaml.

Deployment

  1. Create the default Broker in your namespace. These instructions assume the namespace default, feel free to change to any other namespace you would like to use instead:
  kubectl label namespace default knative-eventing-injection=enabled
  1. Create a GCP PubSub Topic. If you change its name (testing), you also need to update the topic in the gcp-pubsub-source.yaml file:
   gcloud pubsub topics create testing
  1. Replace the MY_GCP_PROJECT placeholder in gcp-pubsub-source.yaml and apply it.

If you’re in the samples directory, you can replace MY_GCP_PROJECT and apply in one command:

   sed "s/MY_GCP_PROJECT/$PROJECT_ID/g" gcp-pubsub-source.yaml | \
       kubectl apply --filename -

If you are replacing MY_GCP_PROJECT manually, then make sure you apply the resulting YAML:

   kubectl apply --filename gcp-pubsub-source.yaml
  1. Create a function and create a Trigger that will send all events from the Broker to the function:
   kubectl apply --filename trigger.yaml

Publish

Publish messages to your GCP PubSub Topic:

gcloud pubsub topics publish testing --message="Hello world"

Verify

We will verify that the published message was sent into the Knative eventing system by looking at what is downstream of the GcpPubSubSource. If you deployed the Subscriber, then continue using this section. If not, then you will need to look downstream yourself.

  1. We need to wait for the downstream pods to get started and receive our event, wait 60 seconds.

    • You can check the status of the downstream pods with:
     kubectl get pods --selector serving.knative.dev/service=event-display

    You should see at least one.

  2. Inspect the logs of the subscriber:

   kubectl logs --selector serving.knative.dev/service=event-display -c user-container

You should see log lines similar to:

{
  "ID": "284375451531353",
  "Data": "SGVsbG8sIHdvcmxk",
  "Attributes": null,
  "PublishTime": "2018-10-31T00:00:00.00Z"
}

The log message is a dump of the message sent by GCP PubSub. In particular, if you base-64 decode the Data field, you should see the sent message:

echo "SGVsbG8sIHdvcmxk" | base64 --decode

Results in: “Hello world”

For more information about the format of the message, see the PubsubMessage documentation.