How To Configure PubSub Notifications In Cloud Storage
Hello Everyone
Welcome to CloudAffaire and this is Debjeet.
In the last blog post, we have discussed Object Lifecycle Management in Cloud Storage.
https://cloudaffaire.com/how-to-create-object-lifecycle-management-in-cloud-storage/
In this blog post, we will discuss how to configure PubSub Notification in Cloud Storage.
PubSub Notifications for Cloud Storage:
PubSub Notifications sends information about changes to objects in your buckets to PubSub, where the information is added to a PubSub topic of your choice in the form of messages. For example, you can track objects that are created and deleted in your bucket. Each notification contains information describing both the event that triggered it and the object that changed.
You can send notifications to any PubSub topic in any project for which you have sufficient permissions. Once received by the PubSub topic, the resulting message can be sent to any number of subscribers to the topic.
Notification configurations:
A notification configuration is a rule you attach to a bucket that specifies:
- The topic in PubSub that receives notifications.
- The events that trigger a notification to be sent.
- The information contained within notifications.
You can attach multiple notification configurations to a bucket. A bucket can have up to 100 total notification configurations and up to 10 notification configurations set to trigger for a specific event.
Each notification configuration is identified by an integer. This integer is returned:
- When you create the notification configuration.
- When you list the notification configurations attached to a bucket.
- In the notificationConfig attribute of each notification triggered by the notification configuration.
Creating and deleting notification configurations increment a bucket’s metageneration number.
Event types:
The following is a list of event types currently supported by Cloud Storage:
- OBJECT_FINALIZE: Sent when a new object (or a new generation of an existing object) is successfully created in the bucket. This includes copying or rewriting an existing object.
- OBJECT_METADATA_UPDATE: Sent when the metadata of an existing object changes.
- OBJECT_DELETE: Sent when an object has been permanently deleted. This includes objects that are overwritten or are deleted as part of the bucket’s lifecycle configuration.
- OBJECT_ARCHIVE: Only sent when a bucket has enabled object versioning. This event indicates that the live version of an object has become a noncurrent version
Notification format:
Notifications sent to the PubSub topic consist of two parts:
- Attributes: A set of key:value pairs describing the event.
- Payload: A string that contains the metadata of the changed object.
Delivery guarantees:
When you add a notification configuration, Cloud Storage may take up to 30 seconds to begin sending notifications associated with it. Once started, Cloud Storage guarantees at-least-once delivery to PubSub. PubSub also offers at-least-once delivery to the recipient, which means that you could receive multiple messages, with multiple IDs, that represent the same Cloud Storage event.
Notifications are not guaranteed to be published in the order PubSub receives them. If you plan to modify the Cloud Storage object based on a notification, it is recommended that you use the object’s generation and metageneration numbers as preconditions on your update request. If a notification consistently cannot be delivered to a PubSub topic, Cloud Storage may delete the notification after 7 days. Delivery failure can occur when the PubSub topic no longer exists, when Cloud Storage no longer has permission to publish to the topic, or when the project that owns the topic exceeds its publishing quota.
How To Configure PubSub Notifications In Cloud Storage:
Step 1: Enable PubSub API for your project.
|
1 2 3 4 5 6 7 8 9 10 11 |
########################################### ## PubSub Notifications In Cloud Storage ## ########################################### ## Prerequisite: gsutil/gcloud installed and configured ## https://cloudaffaire.com/gcloud-installation-and-configuration/ ## Check if pubsub api is enabled for your project gcloud services list --enabled | grep pubsub* ## Enable pubsub api for your project gcloud services enable pubsub.googleapis.com |
Step 2: Create a Topic and Subscription in your PubSub.
|
1 2 3 4 5 |
## Create a pubsub topic gcloud pubsub topics create mytopic ## Create a subscription to your pubsub topic gcloud pubsub subscriptions create --topic mytopic mysubscription |
Step 3: Create a bucket in Cloud Storage.
|
1 2 |
## Create a bucket in cloud storage (bucket name needs to be globally unique) gsutil mb -c standard -l asia-south1 -p cloudaffaire gs://cloudaffaire_bucket_psn |
Step 4: Create a PubSub Notification Configuration for your bucket.
|
1 2 3 4 5 |
## Create a pubsub notification configuration gsutil notification create -t mytopic -f json gs://cloudaffaire_bucket_psn ## List current notification configuration for the bucket gsutil notification list gs://cloudaffaire_bucket_psn |
Step 5: PubSub Notifications for Cloud Storage.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
## Upload an object into your bucket ## Create a file for upload echo "this is my object version one" > cloudaffaire_object.txt ## Upload an object into the bucket gsutil cp cloudaffaire_object.txt gs://cloudaffaire_bucket_psn ## Get the pubsub notification gcloud pubsub subscriptions pull --auto-ack mysubscription ## eventType=OBJECT_FINALIZE ## Edit the metadata associated with an object gsutil setmeta -h "content-type:json" gs://cloudaffaire_bucket_psn/cloudaffaire_object.txt ## Get the pubsub notification gcloud pubsub subscriptions pull --auto-ack mysubscription ## eventType=OBJECT_METADATA_UPDATE ## Delete the object gsutil rm gs://cloudaffaire_bucket_psn/cloudaffaire_object.txt ## Get the pubsub notification gcloud pubsub subscriptions pull --auto-ack mysubscription ## eventType=OBJECT_DELETE |
Step 6: Cleanup.
|
1 2 3 4 5 6 7 8 9 10 11 |
## Delete the notification configuration gsutil notification delete projects/_/buckets/cloudaffaire_bucket_psn/notificationConfigs/1 ## Delete the pubsub subscription gcloud pubsub subscriptions delete mysubscription ## Delete the pubsub topic gcloud pubsub topics delete mytopic ## Delete the cloud storage bucket gsutil rb gs://cloudaffaire_bucket_psn |
Hope you have enjoyed this article. In the next blog post, we will discuss object metadata in cloud storage.
All the public cloud providers are changing the console user interface rapidly and due to this some of the screenshots used in our previous AWS blogs are no longer relevant. Hence, we have decided that from now onwards most of the demo will be done programmatically. Let us know your feedback on this in the comment section.
To get more details on cloud storage, please refer below GCP documentation.
https://cloud.google.com/storage/docs/
https://cloud.google.com/pubsub/docs/admin#pubsub-create-topic-cli
https://cloud.google.com/storage/docs/reporting-changes