You are currently viewing How To Create Retention Policies And Bucket Lock In Cloud Storage

How To Create Retention Policies And Bucket Lock In Cloud Storage

How To Create Retention Policies And Bucket Lock In Cloud Storage

Hello Everyone

Welcome to CloudAffaire and this is Debjeet.

In the last blog post, we have discussed Object Metadata in Cloud Storage.

In this blog post, we will discuss how to create Retention Policies and Bucket Lock in Cloud Storage.

Retention Policies And Bucket Lock In Cloud Storage:

Bucket Lock feature allows you to configure a data retention policy for a Cloud Storage bucket that governs how long objects in the bucket must be retained. The feature also allows you to lock the data retention policy, permanently preventing the policy from being reduced or removed.

This feature can provide immutable storage on Cloud Storage. Bucket Lock can help with regulatory and compliance requirements, such as those associated with FINRA, SEC, and CFTC. Bucket Lock may also help you address certain health care industry retention regulations.

Retention policies:

You can include a retention policy when creating a new bucket, or you can add a retention policy to an existing bucket. Placing a retention policy on a bucket ensures that all current and future objects in the bucket cannot be deleted or overwritten until they reach the age you define in the retention policy.

To help track when individual objects are eligible for deletion, objects in a bucket with a retention policy each have retention expiration time metadata. This piece of metadata shows the date and time when an object fulfills the retention period.

Retention Policy Key Features:

  • You can add a retention policy to a bucket to specify a retention period.
  • If a bucket does not have a retention policy, you can delete or overwrite objects in the bucket at any time.
  • If a bucket has a retention policy, objects in the bucket can only be deleted or overwritten once their age is greater than the retention period.
  • A retention policy retroactively applies to existing objects in the bucket as well as new objects added to the bucket.
  • You can lock a retention policy to permanently set it on the bucket.
  • Once you lock a retention policy, you cannot remove it or reduce the retention period it has.
  • You cannot delete a bucket with a locked retention policy unless every object in the bucket has met the retention period.
  • You can increase the retention period of a locked retention policy.
  • Changing a retention policy is considered a single Class A operation, regardless of the number of objects affected.
  • An object’s editable metadata is not subject to the retention policy and can be modified even when the object itself cannot be.
  • A retention policy contains an effective time, the time after which all objects in the bucket are guaranteed to be in compliance with the retention period.
  • To see the earliest date when a given object is eligible for deletion in a bucket with a retention policy, view the retention expiration date portion of the object’s metadata.
  • Retention policies and Object Versioning are mutually exclusive features in Cloud Storage: for a given bucket, only one of these can be enabled at a time. Any versioned objects remaining in a bucket when you apply a retention policy are also protected by the retention policy.
  • You can use Object Lifecycle Management to automatically delete objects in a bucket, including in a bucket with a locked policy. A lifecycle rule won’t delete an object until after the object fulfills the retention policy.
  • You should not perform parallel composite uploads if your bucket has a retention policy, because the component pieces cannot be deleted until each has met the bucket’s minimum retention period.
  • You use constraints in organization policies to require that retention policies with specific retention periods be included as part of creating a new bucket or as part of adding/updating the a retention policy on an existing bucket.

Important: Locking a retention policy is an irreversible action and once locked, you cannot revert until the retention period is over.

Retention periods:

Retention periods are measured in seconds; however, some tools, like the Google Cloud Console and gsutil allow you to set and view retention periods with other units of time for convenience. The following conversions apply in such cases:

  • A day is considered to be 86,400 seconds.
  • A month is considered to be 31 days, which is 2,678,400 seconds.
  • A year is considered to be 365.25 days, which is 31,557,600 seconds.
  • You can set a maximum retention period of 3,155,760,000 seconds (100 years).

For gsutil, when specifying a retention period, you use a [NUMBER][UNIT] format, where [UNIT] can be s, m, d, or y to signify seconds, minutes, days, or years, respectively. Only one unit of time can be used in a command. For example, you can use 900s or 15m, but you cannot use 15m30s.

Retention policy locks:

When you lock a retention policy on a bucket, you prevent the policy from ever being removed or the retention period from ever being reduced (although you can still increase the retention period). If you try to remove or reduce the policy duration of a locked bucket, you get a 400 BadRequestException error. Once a retention policy is locked, you cannot delete the bucket until every object in the bucket has met the retention period.

Object holds:

Object holds are metadata flags that you place on individual objects. When an object has a hold placed on it, it cannot be deleted. Cloud Storage offers the following types of holds:

  • Event-based holds: Event-based holds can be used in conjunction with a retention policy to control retention based on the occurrence of some event, such as holding loan documents for a certain period after loan was paid.
  • Temporary holds: Temporary holds can be used for regulatory or legal investigation purposes, such as holding trading documents for legal investigation.

An object can have one, both, or neither hold placed on it. Both types of holds behave the same if the object is in a bucket that doesn’t have a retention policy. If the object is in a bucket that has a retention policy, they have different effects on the object when the hold is released:

  • An event-based hold resets the object’s time in the bucket for the purposes of the retention period.
  • A temporary hold does not affect the object’s time in the bucket for the purposes of the retention period.

How To Create Retention Policies And Bucket Lock In Cloud Storage:

Step 1: Create one bucket and upload an object.

Step 2: Create a retention policy for your bucket.

Step 3: Remove the retention policy from your bucket.

Step 4: Place a lock to your bucket.

Step 5: Place an event-based hold to your object.

Step 6: Place a temporary hold to your object.

Hope you have enjoyed this article. In the next blog post, we will discuss Object Transcoding in cloud storage.

All the public cloud providers are changing the console user interface rapidly and due to this some of the screenshots used in our previous AWS blogs are no longer relevant. Hence, we have decided that from now onwards most of the demo will be done programmatically. Let us know your feedback on this in the comment section.

To get more details on cloud storage, please refer below GCP documentation.


Leave a Reply