How To Create Object Lifecycle Management In Cloud Storage

Hello Everyone

Welcome to CloudAffaire and this is Debjeet.

In the last blog post, we have discussed object versioning in Cloud Storage.

https://cloudaffaire.com/how-to-enable-object-versioning-in-cloud-storage/

In this blog post, we will discuss how to create Object Lifecycle Management in cloud storage.

Object Lifecycle Management:

You can assign a lifecycle management configuration to a bucket. The configuration contains a set of rules which apply to current and future objects in the bucket. When an object meets the criteria of one of the rules, Cloud Storage automatically performs a specified action on the object.

Lifecycle configuration:

Each lifecycle management configuration contains a set of rules. When defining a rule, you can specify any set of conditions for any action. If you specify multiple conditions in a rule, an object has to match all of the conditions for the action to be taken. If you specify multiple rules that contain the same action, the action is taken when an object matches the condition(s) in any of the rules. Each rule should contain only one action.

If a single object is subject to multiple actions, Cloud Storage performs only one of the actions, and the object will be re-evaluated before any additional actions are taken. A Delete action takes precedence over a SetStorageClass action. If multiple SetStorageClass actions are specified, the action switching to the storage class with the lowest at-rest storage pricing is chosen.

Lifecycle actions:

The following actions are supported for a lifecycle rule:

Delete: Delete objects based on some rule like age of the object.
SetStorageClass: Change the storage class of objects.

Note: For buckets in a region, the new storage class cannot be Multi-Regional Storage. For buckets in a multi-region or dual-region, the new storage class cannot be Regional Storage.

Important: Cloud Storage performs an action asynchronously, so there can be a lag between when the conditions are satisfied and when the action is taken.

Lifecycle conditions:

The following conditions are supported for a lifecycle rule:

Age: This condition is satisfied when an object reaches the specified age (in days). Age is measured from the object’s creation time.
CreatedBefore: This condition is satisfied when an object is created before midnight of the specified date in UTC.
IsLive: This condition is typically only used in conjunction with object versioning. When set to false, this condition is satisfied for any noncurrent version of an object.
MatchesStorageClass: This condition is satisfied when an object in the bucket is stored as the specified storage class.
NumberOfNewerVersions: This condition is typically only used in conjunction with object versioning. If the value of this condition is set to N, an object version satisfies the condition when there are at least N versions (including the live version) newer than it.

Note: All conditions are optional, but at least one condition is required.

Object Lifecycle behavior:

Cloud Storage regularly inspects all the objects in a bucket for which Object Lifecycle Management is configured and performs all actions applicable according to the bucket’s rules.
Updates to your lifecycle configuration may take up to 24 hours to go into effect.
An object lifecycle Delete action will not take effect on an object while the object either has an object hold placed on it or a retention policy that it has not yet fulfilled.
An object lifecycle SetStorageClass action is not affected by the existence of object holds or retention policies.
Object Lifecycle Management does not rewrite an object when changing its storage class.
If a Delete action is specified for a bucket with the Age condition (and no NumberOfNewerVersions condition), then some objects may be tagged with expiration time metadata.

Object Lifecycle Monitoring Options:

To track the lifecycle management actions that Cloud Storage takes, use one of the following options:

Use Cloud Storage Access Logs. This feature logs both the action and who performed the action. A value of GCS Lifecycle Management in the cs_user_agent field of the log entry indicates the action was taken by Cloud Storage in accordance with a lifecycle configuration.
Enable Pub/Sub Notifications for Cloud Storage for your bucket. This feature sends notifications to a Pub/Sub topic of your choice when specified actions occur. Note that this feature does not record who performed the actions.

How To Create Object Lifecycle Management Demo:

Step 1: Create a bucket and upload an object to your bucket.

##################################################
## Object Lifecycle Management In Cloud Storage ##
##################################################
## Prerequisite: gsutil installed and configured
## https://cloudaffaire.com/gcloud-installation-and-configuration/

## Create a bucket in cloud storage (bucket name needs to be globally unique)
gsutil mb -c standard -l asia-south1 -p cloudaffaire gs://cloudaffaire-bucket-olm

## Upload an object into your bucket
## Create a file for upload
echo "this is my object one" > cloudaffaire_object.txt 

## Upload an object into the bucket
gsutil cp cloudaffaire_object.txt gs://cloudaffaire-bucket-olm

##################################################

## Object Lifecycle Management In Cloud Storage ##

##################################################

## Prerequisite: gsutil installed and configured

## https://cloudaffaire.com/gcloud-installation-and-configuration/

## Create a bucket in cloud storage (bucket name needs to be globally unique)

gsutil mb -c standard -l asia-south1 -p cloudaffaire gs://cloudaffaire-bucket-olm

## Upload an object into your bucket

## Create a file for upload

echo "this is my object one" > cloudaffaire_object.txt

## Upload an object into the bucket

gsutil cp cloudaffaire_object.txt gs://cloudaffaire-bucket-olm

Step 2: Enable Lifecycle Management for your bucket.

## Check current object lifecycle management configuration
gsutil lifecycle get gs://cloudaffaire-bucket-olm

## gs://cloudaffaire-bucket-olm/ has no lifecycle configuration.

## Create a lifecycle configuration
## Delete object after one day
vi lifecycle_delete_configuration.json
-----------------------
{
"lifecycle": {
  "rule": [
  {
    "action": {"type": "Delete"},
    "condition": {
      "age": 1,
      "isLive": true
    }
  }]
}
}
-----------------------
:wq

## Enable lifecycle management for the bucket
gsutil lifecycle set lifecycle_delete_configuration.json \
gs://cloudaffaire-bucket-olm

## Check current object lifecycle management configuration
gsutil lifecycle get gs://cloudaffaire-bucket-olm

## Check current object lifecycle management configuration

gsutil lifecycle get gs://cloudaffaire-bucket-olm

## gs://cloudaffaire-bucket-olm/ has no lifecycle configuration.

## Create a lifecycle configuration

## Delete object after one day

vi lifecycle_delete_configuration.json

-----------------------

{

"lifecycle": {

"rule": [

{

"action": {"type": "Delete"},

"condition": {

"age": 1,

"isLive": true

}

}]

}

-----------------------

:wq

## Enable lifecycle management for the bucket

gsutil lifecycle set lifecycle_delete_configuration.json \

gs://cloudaffaire-bucket-olm

## Check current object lifecycle management configuration

gsutil lifecycle get gs://cloudaffaire-bucket-olm

Step 3: Update Lifecycle Management for your bucket.

## Update the lifecycle configuration
## Change object storage class from standard to nearline after one day
vi lifecycle_change_storageclass_configuration.json
-----------------------
{
"lifecycle": {
  "rule": [
  {
    "action": {
      "type": "SetStorageClass",
      "storageClass": "NEARLINE"
    },
    "condition": {
      "age": 1,
      "matchesStorageClass": ["STANDARD"]
    }
  }
]}
}
-------------------------
:wq

## Update the lifecycle management for the bucket
gsutil lifecycle set lifecycle_change_storageclass_configuration.json \
gs://cloudaffaire-bucket-olm

## Check current object lifecycle management configuration
gsutil lifecycle get gs://cloudaffaire-bucket-olm

## Update the lifecycle configuration

## Change object storage class from standard to nearline after one day

vi lifecycle_change_storageclass_configuration.json

-----------------------

{

"lifecycle": {

"rule": [

{

"action": {

"type": "SetStorageClass",

"storageClass": "NEARLINE"

"condition": {

"age": 1,

"matchesStorageClass": ["STANDARD"]

}

]}

}

-------------------------

:wq

## Update the lifecycle management for the bucket

gsutil lifecycle set lifecycle_change_storageclass_configuration.json \

gs://cloudaffaire-bucket-olm

## Check current object lifecycle management configuration

gsutil lifecycle get gs://cloudaffaire-bucket-olm

Step 4: Disable Lifecycle Management for your bucket.

## Disable lifecycle management for the bucket
vi lifecycle_disable_configuration.json
-------------------------
{}
-------------------------
:wq

## Disable lifecycle management for the bucket
gsutil lifecycle set lifecycle_disable_configuration.json \
gs://cloudaffaire-bucket-olm

## Check current object lifecycle management configuration
gsutil lifecycle get gs://cloudaffaire-bucket-olm

## Disable lifecycle management for the bucket

vi lifecycle_disable_configuration.json

-------------------------

{}

-------------------------

:wq

## Disable lifecycle management for the bucket

gsutil lifecycle set lifecycle_disable_configuration.json \

gs://cloudaffaire-bucket-olm

## Check current object lifecycle management configuration

gsutil lifecycle get gs://cloudaffaire-bucket-olm

Step 5: Cleanup.

## Delete the object
gsutil rm gs://cloudaffaire-bucket-olm/cloudaffaire_object.txt

## Delete the bucket
gsutil rb gs://cloudaffaire-bucket-olm

## Delete the object

gsutil rm gs://cloudaffaire-bucket-olm/cloudaffaire_object.txt

## Delete the bucket

gsutil rb gs://cloudaffaire-bucket-olm

Hope you have enjoyed this article. In the next blog post, we will discuss Pub/Sub Notifications in Cloud Storage.

All the public cloud providers are changing the console user interface rapidly and due to this some of the screenshots used in our previous AWS blogs are no longer relevant. Hence, we have decided that from now onwards most of the demo will be done programmatically. Let us know your feedback on this in the comment section.

To get more details on cloud storage, please refer below GCP documentation.

https://cloud.google.com/storage/docs/

https://cloud.google.com/storage/docs/lifecycle

https://cloud.google.com/storage/docs/gsutil/commands/lifecycle

https://cloud.google.com/storage/docs/json_api/v1/buckets#resource-representations

How To Create Object Lifecycle Management In Cloud Storage

Object Lifecycle Management:

Lifecycle configuration:

Lifecycle actions:

Lifecycle conditions:

Object Lifecycle behavior:

Object Lifecycle Monitoring Options:

How To Create Object Lifecycle Management Demo:

Step 1: Create a bucket and upload an object to your bucket.

Step 2: Enable Lifecycle Management for your bucket.

Step 3: Update Lifecycle Management for your bucket.

Step 4: Disable Lifecycle Management for your bucket.

Step 5: Cleanup.

Leave a Reply Cancel reply