How To Manage Encryption In Cloud Storage
Hello Everyone
Welcome to CloudAffaire and this is Debjeet.
In the last blog post, we have discussed Access Logs in Cloud Storage.
https://cloudaffaire.com/how-to-enable-and-view-access-logs-in-cloud-storage/
In this blog post, we will discuss encryption in Cloud Storage.
Encryption In Cloud Storage:
Cloud Storage always encrypts your data on the server-side, before it is written to disk, at no additional charge. Besides this standard behavior, there are additional ways to encrypt your data when using Cloud Storage.
- Server-side encryption: encryption that occurs after Cloud Storage receives your data, but before the data is written to disk and stored.
- Customer-supplied encryption keys: You can create and manage your own encryption keys for server-side encryption, which act as an additional encryption layer on top of the standard Cloud Storage encryption.
- Customer-managed encryption keys: You can generate and manage your encryption keys using Cloud Key Management Service, which act as an additional encryption layer on top of the standard Cloud Storage encryption.
- Client-side encryption: encryption that occurs before data is sent to Cloud Storage. Such data arrives at Cloud Storage already encrypted but also undergoes server-side encryption.
Google-managed encryption keys:
Cloud Storage manages server-side encryption keys on your behalf using the same hardened key management systems that we use for our own encrypted data, including strict key access controls and auditing. Cloud Storage encrypts user data at rest using AES-256. There is no setup or configuration required, no need to modify the way you access the service, and no visible performance impact. Data is automatically and transparently decrypted when read by an authorized user.
Customer-supplied encryption keys:
As an additional layer on top of Google-managed encryption keys, you can choose to provide your own AES-256 key, encoded in standard Base64. This key is known as a customer-supplied encryption key. If you provide a customer-supplied encryption key, Cloud Storage does not permanently store your key on Google’s servers or otherwise manage your key. Instead, you provide your key for each Cloud Storage operation, and your key is purged from Google’s servers after the operation is complete. Cloud Storage stores only a cryptographic hash of the key so that future requests can be validated against the hash. Your key cannot be recovered from this hash, and the hash cannot be used to decrypt your data.
When you apply a customer-supplied encryption key to an object, Cloud Storage uses the key when encrypting:
- The object’s data.
- The object’s CRC32C checksum.
- The object’s MD5 hash.
Cloud Storage uses standard server-side keys to encrypt the remaining metadata for the object, including the object’s name. This allows you to read and update general metadata, as well as list and delete objects, without needing the customer-supplied encryption key.
For example, if an object is encrypted with a customer-supplied encryption key, the key must be used to perform operations on the object such as downloading or moving it. If you attempt to read the object’s metadata without providing the key, you receive metadata such as the object name and Content-Type, but not the object’s CRC32C checksum or MD5 hash. If you do supply your key with the request for the object metadata, the object’s CRC32C checksum and MD5 hash are included with the metadata.
Restrictions:
- Cloud Storage Transfer Service, Cloud Dataflow, and Cloud Dataproc do not currently support objects encrypted with customer-supplied encryption keys.
- You cannot use the Google Cloud Console to download objects that are encrypted with a customer-supplied encryption key. Similarly, when you use the Google Cloud Console to upload an object, you cannot encrypt the object with a customer-supplied encryption key. You can perform these actions with customer-managed encryption keys.
- To use customer-supplied encryption keys with gsutil, you must have gsutil 4.18 or later.
- You can only set customer-supplied encryption keys on individual objects. You cannot set a default customer-supplied encryption key for a bucket.
- If you are performing a compose operation on objects encrypted by customer-supplied encryption keys, the component objects must be encrypted by the same key, and you need to provide the key with the compose request. The resulting composite object is encrypted by the same key.
Customer-managed encryption keys:
As an additional layer on top of Google-managed encryption keys, you can choose to use keys generated by Cloud Key Management Service. Such keys are known as customer-managed encryption keys. If you use a customer-managed encryption key, your encryption keys are stored within Cloud KMS. The project that holds your encryption keys can then be independent from the project that contains your buckets, thus allowing for better separation of duties.
When you apply a customer-managed encryption key to an object, Cloud Storage uses the key when encrypting:
- The object’s data.
- The object’s CRC32C checksum.
- The object’s MD5 hash.
Cloud Storage uses standard server-side keys to encrypt the remaining metadata for the object, including the object’s name. Thus, if you have sufficient permission, you can perform actions such as reading most metadata, listing objects, and deleting objects even after you’ve disabled or destroyed the associated customer-managed encryption key.
Restrictions
The following restrictions apply when using customer-managed encryption keys:
- Cloud SQL exports to Cloud Storage and Dataflow do not currently support objects encrypted with customer-managed encryption keys.
- You cannot use the JSON API Copy Object method when the source object is encrypted with a customer-managed encryption key or when the destination object would become encrypted by a customer-managed encryption key. Use the Rewrite Object method instead.
- You cannot use the JSON API Compose Object method when one or more of the source objects are encrypted with a customer-managed encryption key.
- You cannot encrypt an object with a customer-managed encryption key by updating the object’s metadata. Include the key as part of a rewrite of the object instead.
- You must create the Cloud KMS key in the same location as the data you intend to encrypt. For example, if your bucket is located in US-EAST1, any Cloud KMS key encrypting objects in that bucket must also be created in US-EAST1.
- You cannot specify a customer-managed encryption key as part of a Storage Transfer Service transfer, and any such keys on source objects are not applied to the transferred objects. Set a default customer-managed key on your bucket prior to performing the transfer.
Note: Because most metadata can be read regardless of the encryption method, do not include sensitive information in your object and bucket metadata or names.
How To Manage Encryption In Cloud Storage
Encryption using customer-supplied encryption key.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 |
################################# ## Encryption In Cloud Storage ## ################################# ## Prerequisite: gsutil/gcloud installed and configured ## https://cloudaffaire.com/gcloud-installation-and-configuration/ ##-------------------------------------------------- ## Encryption with customer-supplied encryption keys ##-------------------------------------------------- ## Generate an encryption key ## Prerequisite: Python python -c 'import base64; import os; \ print(base64.encodestring(os.urandom(32)))' ## You will get a random encrption key (see below) ## [YOUR_ENCRYPTION_KEY_ONE] = jD+BO6blF8hqx53eCmUeMgwrkIw7STXSCWZ9l+L3wQw= ## Set encryption and decryption key's in your environment variable ## Replace the value from above output YOUR_ENCRYPTION_KEY_ONE=jD+BO6blF8hqx53eCmUeMgwrkIw7STXSCWZ9l+L3wQw= && YOUR_DECRYPTION_KEY_ONE=jD+BO6blF8hqx53eCmUeMgwrkIw7STXSCWZ9l+L3wQw= ## Create a bucket in cloud storage (bucket name needs to be globally unique) gsutil mb -c standard -l asia-south1 -p cloudaffaire gs://cloudaffaire_bucket_encrpt ## Create a file echo 'my encrypted content' > myobject.txt ## Upload an object to your bucket with customer-supplied encryption key gsutil -o "GSUtil:encryption_key=$YOUR_ENCRYPTION_KEY_ONE" \ cp myobject.txt gs://cloudaffaire_bucket_encrpt ## Get the metadata for your object gsutil stat gs://cloudaffaire_bucket_encrpt/myobject.txt # Hash (crc32c): encrypted # Hash (md5): encrypted # Encryption algorithm: AES256 # Encryption key SHA256: CzNZD3z+pHHic0lHzWmdJO7IajIwWlUOambvn1dpncs= ## Try to get the content of the object without decryption key gsutil cat gs://cloudaffaire_bucket_encrpt/myobject.txt ## Error, Missing decryption key ## Get the content of the object using decryption key gsutil -o "GSUtil:encryption_key=$YOUR_DECRYPTION_KEY_ONE" \ cat gs://cloudaffaire_bucket_encrpt/myobject.txt ## my encrypted content ## Delete all objects and your bucket gsutil rm -r gs://cloudaffaire_bucket_encrpt |
Encryption using customer-managed encryption key.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 |
##------------------------------------------------- ## Encryption with customer-managed encryption keys ##------------------------------------------------- ## Check if kms api is enabled for your project gcloud services list | grep kms ## Enable kms api for your project gcloud services enable cloudkms.googleapis.com ## Create a kms key ring gcloud kms keyrings create mykeyring \ --location asia-south1 ## Create a kms symetric key gcloud kms keys create mykey \ --location asia-south1 \ --keyring mykeyring \ --purpose encryption ## Create a bucket in cloud storage (bucket name needs to be globally unique) gsutil mb -c standard -l asia-south1 -p cloudaffaire gs://cloudaffaire_bucket_encrpt ## Grant permission to cloud storage service account to use the kms key gsutil kms authorize \ -p cloudaffaire \ -k projects/cloudaffaire/locations/asia-south1/keyRings/mykeyring/cryptoKeys/mykey ## Encrypt the bucket using kms key gsutil kms encryption \ -k projects/cloudaffaire/locations/asia-south1/keyRings/mykeyring/cryptoKeys/mykey \ gs://cloudaffaire_bucket_encrpt ## Upload an object into your bucket gsutil cp myobject.txt gs://cloudaffaire_bucket_encrpt ## Get details for your bucket encryption gsutil kms encryption gs://cloudaffaire_bucket_encrpt ## Get the metadata for your object gsutil stat gs://cloudaffaire_bucket_encrpt/myobject.txt ## KMS key: projects/cloudaffaire/locations/asia-south1/keyRings/mykeyring/cryptoKeys/mykey/cryptoKeyVersions/1 ## Get the content of the object using decryption key gsutil cat gs://cloudaffaire_bucket_encrpt/myobject.txt ## my encrypted content ## No need to provide decryption key, only cloud storage service account needs access to kms ## Delete all objects and your bucket gsutil rm -r gs://cloudaffaire_bucket_encrpt ## Destroy the kms key gcloud kms keys versions destroy 1 \ --location=asia-south1 \ --keyring=mykeyring \ --key=mykey |
Hope you have enjoyed this article. In the next blog post, we will discuss Access Control in cloud storage.
All the public cloud providers are changing the console user interface rapidly and due to this some of the screenshots used in our previous AWS blogs are no longer relevant. Hence, we have decided that from now onwards most of the demo will be done programmatically. Let us know your feedback on this in the comment section.
To get more details on cloud storage, please refer below GCP documentation.
https://cloud.google.com/storage/docs/
https://cloud.google.com/storage/docs/encryption/
https://cloud.google.com/storage/docs/encryption/default-keys
https://cloud.google.com/storage/docs/encryption/customer-managed-keys
https://cloud.google.com/storage/docs/encryption/client-side-keys
https://cloud.google.com/storage/docs/encryption/using-customer-supplied-keys
https://cloud.google.com/kms/docs/creating-keys