Cloud Storage in GCP
Hello Everyone
Welcome to CloudAffaire and this is Debjeet.
In the last blog post, we have learned how to create a compute engine using gcloud.
https://cloudaffaire.com/how-to-create-a-compute-engine-instance-using-gcloud/
In this blog post, we will discuss cloud storage in GCP.
Cloud Storage in GCP:
Cloud Storage is a service for storing your objects in Google Cloud. An object is an immutable piece of data consisting of a file of any format. You store objects in containers called buckets. All buckets are associated with a project, and you can group your projects under an organization.
After you create a project, you can create Cloud Storage buckets, upload objects to your buckets, and download objects from your buckets. You can also grant permissions to make your data accessible to members you specify, or – for certain use cases such as hosting a website
Key Components of cloud storage:
Projects:
All data in Cloud Storage belongs inside a project. A project consists of a set of users, a set of APIs, and billing, authentication, and monitoring settings for those APIs. You can have one project or multiple projects.
Buckets:
Buckets are the basic containers that hold your data. Everything that you store in Cloud Storage must be contained in a bucket. You can use buckets to organize your data and control access to your data, but unlike directories and folders, you cannot nest buckets.
When you create a bucket, you specify a globally-unique name, a geographic location where the bucket and its contents are stored, and a default storage class. The default storage class you choose applies to objects added to the bucket that don’t have a storage class specified explicitly.
After you create a bucket, you can still change its default storage class, to any class supported in the bucket’s location; however, you can only change the bucket name and location by deleting and re-creating the bucket.
Objects:
Objects are the individual pieces of data that you store in Cloud Storage. There is no limit on the number of objects that you can create in a bucket.
Objects have two components: object data and object metadata. Object data is typically a file that you want to store in Cloud Storage. Object metadata is a collection of name-value pairs that describe various object qualities.
Key features of cloud storage:
Object metadata:
Objects stored in Cloud Storage have metadata associated with them. Metadata identifies properties of the object, as well as specifies how the object should be handled when it’s accessed. Metadata exists as key:value pairs. For example, the storage class of an object is represented by the metadata entry storageClass:STANDARD. storageClass is the key for the metadata, and all objects have such a key associated with them. STANDARD specifies the value this specific object has, and the value varies from object to object.
Pub/Sub Notifications:
Pub/Sub Notifications sends information about changes to objects in your buckets to Pub/Sub, where the information is added to a Pub/Sub topic of your choice in the form of messages. For example, you can track objects that are created and deleted in your bucket. Each notification contains information describing both the event that triggered it and the object that changed.
Object Versioning:
To support the retrieval of objects that are deleted or overwritten, Cloud Storage offers the Object Versioning feature. Enable Object Versioning to protect your Cloud Storage data from being overwritten or accidentally deleted. Enabling Object Versioning increases storage costs.
Object Lifecycle Management:
You can assign a lifecycle management configuration to a bucket. The configuration contains a set of rules which apply to current and future objects in the bucket. When an object meets the criteria of one of the rules, Cloud Storage automatically performs a specified action on the object.
Object transcoding:
Transcoding, in Cloud Storage, is the automatic changing of a file’s compression before it’s served to a requester. When transcoding results in a file becoming gzip-compressed, it can be considered compressive, whereas when the result is a file that is no longer gzip-compressed, it can be considered decompressive. Cloud Storage supports the decompressive form of transcoding.
Composite objects:
Composite objects are created from existing objects without transferring additional object data. Composite objects are useful for making appends to an object, as well as for building an object using components that you’ve uploaded in parallel.
Cross-origin resource sharing (CORS):
Cross Origin Resource Sharing (CORS) is a mechanism for allowing interactions between resources from different origins, something that is normally prohibited in order to prevent malicious behavior. You set CORS configuration on a bucket by specifying information, such as HTTP methods and originating domains, that identify the types of requests it will accept.
Cloud Audit Logs:
Cloud Audit Logs maintains three audit logs for each Google Cloud project, folder, and organization: Admin Activity, Data Access, and System Event. Google Cloud services write audit log entries to these logs to help you answer the questions of “who did what, where, and when?” within your Google Cloud resources.
Retention policies using Bucket Lock:
The Bucket Lock feature, which allows you to configure a data retention policy for a Cloud Storage bucket that governs how long objects in the bucket must be retained. The feature also allows you to lock the data retention policy, permanently preventing the policy from being reduced or removed.
Requester Pays:
Whenever a user accesses a Cloud Storage resource such as a bucket or object, there are charges associated with making and executing the request. Normally, the project owner of the resource is billed for these access charges; however, if the requester provides a billing project with their request, the requester’s project is billed instead.
Encryption:
Cloud Storage always encrypts your data on the server-side, before it is written to disk, at no additional charge. Besides this standard behavior, there are additional ways to encrypt your data when using Cloud Storage. You can also create and manage your own encryption keys for server-side encryption using Customer-supplied encryption keys or Customer-managed encryption keys, which act as an additional encryption layer on top of the standard Cloud Storage encryption.
Access control:
You can control who has access to your Cloud Storage buckets and objects as well as what level of access they have using Cloud Identity and Access Management (Cloud IAM) permissions, Access Control Lists (ACLs), Signed URLs (query string authentication), Signed Policy Documents, Firebase Security Rules.
Storage classes:
The storage class you set for an object affects the object’s availability and pricing model. When you create a bucket, you can specify a default storage class for the bucket. When you add objects to the bucket, they inherit this storage class unless explicitly set otherwise. There are three types of storage classes available in GCP:
- Standard Storage: Standard Storage is best for data that is frequently accessed (“hot” data) and/or stored for only brief periods of time.
- Nearline Storage: Nearline Storage is a low-cost, highly durable storage service for storing infrequently accessed data.
- Coldline Storage: Coldline Storage is a very-low-cost, highly durable storage service for data archiving, online backup, and disaster recovery.
Bucket locations:
You specify a location for storing your object data when you create a bucket. There are different location types:
- region: A region is a specific geographic place, such as London.
- dual-region: A dual-region is a specific pair of regions, such as Finland and the Netherlands.
- multi-region: A multi-region is a large geographic area, such as the United States, that contains two or more geographic places.
Hope you have enjoyed this article. In the next blog post, we will create our 1st bucket using gsutil.
To get more details on cloud storage, please refer below GCP documentation.
https://cloud.google.com/storage/docs/