What Is Object Transcoding In Cloud Storage

Hello Everyone

Welcome to CloudAffaire and this is Debjeet.

In the last blog post, we have discussed Retention Policies and Bucket Lock in Cloud Storage.

https://cloudaffaire.com/how-to-create-retention-policies-and-bucket-lock-in-cloud-storage/

In this blog post, we will discuss Object Transcoding in Cloud Storage.

What Is Object Transcoding In Cloud Storage?

Transcoding, in Cloud Storage, is the automatic changing of a file’s compression before it’s served to a requester. When transcoding results in a file becoming gzip-compressed, it can be considered compressive, whereas when the result is a file that is no longer gzip-compressed, it can be considered decompressive. Cloud Storage supports the decompressive form of transcoding.

Note: Decompressive transcoding invalidates integrity checking on affected objects. This is because the stored hash represents the compressed data, while the served data has compression removed. If requesters of your data rely on integrity checking, you should not use decompressive transcoding.

Decompressive transcoding:

If files are stored as gzip-compressed objects on Cloud Storage, they can be automatically decompressed before being sent to a requester, resulting in a file that is identity encoded (that is, not compressed). This allows for reduced storage costs for the object within Cloud Storage, but gives the requester the file itself, without any compression.

In order to be eligible for decompressive transcoding, an object must meet two criteria:

The file is gzip-compressed when stored in Cloud Storage.
The associated metadata includes Content-Encoding: gzip.

When an object meets these two criteria, it undergoes decompressive transcoding when requested, in which case it is also served without a Content-Encoding header. If you want an object that meets both criteria to be served in its compressed state (for example, to reduce egress cost or time), there are two ways to prevent decompressive transcoding from occurring:

If the request for the object includes an Accept-Encoding: gzip header, the object is served as-is in that specific request, along with a Content-Encoding: gzip response header.
If the Cache-Control metadata field for the object is set to no-transform, the object is served as a compressed object in all subsequent requests, regardless of any Accept-Encoding request headers.

Content-Type vs. Content-Encoding:

There are several behaviors that you should be aware of concerning how Content-Type and Content-Encoding relate to transcoding. Both are metadata stored along with an object.

Content-Type: should be included in all uploads and indicates the type of object being uploaded. For example, Content-Type text/plain indicates that the uploaded object is a plain-text file.
Content-Encoding: is optional and can, if desired, be included in the upload of files that are compressed. For example, Content-Encoding gzip indicates that the uploaded object is gzip-compressed.

Object Transcoding In Cloud Storage:

#########################################
## Object Transcoding In Cloud Storage ##
#########################################
## Prerequisite: gsutil/gcloud installed and configured
## https://cloudaffaire.com/gcloud-installation-and-configuration/

## Create a bucket in cloud storage (bucket name needs to be globally unique)
gsutil mb -c standard -l asia-south1 -p cloudaffaire gs://cloudaffaire_bucket_ot

## Create two files with same size (1 mb size)
dd if=/dev/zero of=cloudaffaire_object_one.txt bs=1048576  count=1 &&
dd if=/dev/zero of=cloudaffaire_object_two.txt bs=1048576  count=1

## Check the file size in your disk
ls -lh cloudaffaire_object*

## Both file consumes exactly 1mb of disk space

## Upload an object into your bucket without trasncoding
gsutil cp cloudaffaire_object_one.txt gs://cloudaffaire_bucket_ot

## Upload an object into your bucket using trasncoding
gsutil cp -z txt cloudaffaire_object_two.txt gs://cloudaffaire_bucket_ot

## Observe the upload speed is much faster for the object with transcoding than 
## the object without transcoding

## Get object metadata for object without transcoding
gsutil stat gs://cloudaffaire_bucket_ot/cloudaffaire_object_one.txt

##    Content-Length:         1048576
##    Content-Type:           text/plain

## Get object metadata for object with transcoding
gsutil stat gs://cloudaffaire_bucket_ot/cloudaffaire_object_two.txt

##    Content-Encoding:       gzip
##    Content-Length:         1061
##    Content-Type:           text/plain

## Observe the content lenght is much less for the object with transcoding

## Delete the objects from your bucket
gsutil rm gs://cloudaffaire_bucket_ot/cloudaffaire_object_one.txt &&
gsutil rm gs://cloudaffaire_bucket_ot/cloudaffaire_object_two.txt

## Delete your bucket
gsutil rb gs://cloudaffaire_bucket_ot

#########################################

## Object Transcoding In Cloud Storage ##

#########################################

## Prerequisite: gsutil/gcloud installed and configured

## https://cloudaffaire.com/gcloud-installation-and-configuration/

## Create a bucket in cloud storage (bucket name needs to be globally unique)

gsutil mb -c standard -l asia-south1 -p cloudaffaire gs://cloudaffaire_bucket_ot

## Create two files with same size (1 mb size)

dd if=/dev/zero of=cloudaffaire_object_one.txt bs=1048576 count=1 &&

dd if=/dev/zero of=cloudaffaire_object_two.txt bs=1048576 count=1

## Check the file size in your disk

ls -lh cloudaffaire_object*

## Both file consumes exactly 1mb of disk space

## Upload an object into your bucket without trasncoding

gsutil cp cloudaffaire_object_one.txt gs://cloudaffaire_bucket_ot

## Upload an object into your bucket using trasncoding

gsutil cp -z txt cloudaffaire_object_two.txt gs://cloudaffaire_bucket_ot

## Observe the upload speed is much faster for the object with transcoding than

## the object without transcoding

## Get object metadata for object without transcoding

gsutil stat gs://cloudaffaire_bucket_ot/cloudaffaire_object_one.txt

## Content-Length: 1048576

## Content-Type: text/plain

## Get object metadata for object with transcoding

gsutil stat gs://cloudaffaire_bucket_ot/cloudaffaire_object_two.txt

## Content-Encoding: gzip

## Content-Length: 1061

## Content-Type: text/plain

## Observe the content lenght is much less for the object with transcoding

## Delete the objects from your bucket

gsutil rm gs://cloudaffaire_bucket_ot/cloudaffaire_object_one.txt &&

gsutil rm gs://cloudaffaire_bucket_ot/cloudaffaire_object_two.txt

## Delete your bucket

gsutil rb gs://cloudaffaire_bucket_ot

Hope you have enjoyed this article. In the next blog post, we will discuss Composite Objects in cloud storage.

All the public cloud providers are changing the console user interface rapidly and due to this some of the screenshots used in our previous AWS blogs are no longer relevant. Hence, we have decided that from now onwards most of the demo will be done programmatically. Let us know your feedback on this in the comment section.

To get more details on cloud storage, please refer below GCP documentation.

https://cloud.google.com/storage/docs/

https://cloud.google.com/storage/docs/transcoding

https://cloud.google.com/storage/docs/viewing-editing-metadata

https://cloud.google.com/storage/docs/gsutil/commands/cp#options