How To Enable And View Access Logs In Cloud Storage

Hello Everyone

Welcome to CloudAffaire and this is Debjeet.

In the last blog post, we have discussed Composite Objects in Cloud Storage.

https://cloudaffaire.com/how-to-create-a-composite-object-in-cloud-storage/

In this blog post, we will discuss Access Logs in Cloud Storage.

Audit Logs:

Cloud Audit Logs maintains three audit logs for each Google Cloud project, folder, and organization: Admin Activity, Data Access, and System Event. Google Cloud services write audit log entries to these logs to help you answer the questions of “who did what, where, and when?” within your Google Cloud resources.

Admin Activity audit logs:

Admin Activity audit logs contain log entries for API calls or other administrative actions that modify the configuration or metadata of resources. For example, these logs record when users create VM instances or change Cloud Identity and Access Management permissions. Admin Activity audit logs are always written; you can’t configure or disable them. There is no charge for your Admin Activity audit logs.

Data Access audit logs:

Data Access audit logs contain API calls that read the configuration or metadata of resources, as well as user-driven API calls that create, modify, or read user-provided resource data. Data Access audit logs do not record the data-access operations on resources that are publicly shared (available to All Users or All Authenticated Users) or that can be accessed without logging into Google Cloud. Data Access audit logs are disabled by default because they can be quite large; they must be explicitly enabled to be written. Enabling the logs might result in your project being charged for the additional logs usage.

System Event audit logs:

System Event audit logs contain log entries for Google Cloud administrative actions that modify the configuration of resources. System Event audit logs are generated by Google systems; they are not driven by direct user action. System Event audit logs are always written; you can’t configure or disable them. There is no charge for your System Event audit logs.

Audit Logs In Cloud Storage:

In Cloud Storage, you can enable Data Access Logs which keep entries for operations that modify objects or read a project, bucket, or object. There are several sub-types of data access logs:

ADMIN_READ: Entries for operations that read the configuration or metadata of a project, bucket, or object.
DATA_READ: Entries for operations that read an object.
DATA_WRITE: Entries for operations that create or modify an object.

Note: Admin Activity logs are enabled by default and keep entries for operations that modify the configuration or metadata of a project, bucket, or object.

Log settings:

Logs pertaining to Cloud Storage operations are generated by the service storage.googleapis.com.
Admin Activity logs are recorded by default. These logs do not count towards your log ingestion quota.
Data Access logs pertaining to Cloud Storage operations are not recorded by default.

Restrictions:

Cloud Audit Logs does not track access to public objects.
Cloud Audit Logs does not track changes made by the Object Lifecycle Management feature.
You cannot use authenticated browser downloads to access objects when Cloud Audit Logs Data Access logs are enabled on the bucket containing the objects.

How To Enable And View Access Logs In Cloud Storage:

##################################
## Access Logs In Cloud Storage ##
##################################
## Prerequisite: gsutil/gcloud installed and configured
## https://cloudaffaire.com/gcloud-installation-and-configuration/

## Create a bucket in cloud storage (bucket name needs to be globally unique)
gsutil mb -c standard -l asia-south1 -p cloudaffaire gs://cloudaffaire_bucket_al

## Create a log bucket to store access logs (bucket name needs to be globally unique)
gsutil mb -c standard -l asia-south1 -p cloudaffaire gs://cloudaffaire_log_bucket_al

## Allow Cloud Storage WRITE permission to the log bucket
gsutil acl ch -g cloud-storage-analytics@google.com:W gs://cloudaffaire_log_bucket_al

## Enable logging for your bucket
gsutil logging set on -b gs://cloudaffaire_log_bucket_al -o mylogs gs://cloudaffaire_bucket_al

## Get the logging for your bucket
gsutil logging get gs://cloudaffaire_bucket_al

## {"logBucket": "cloudaffaire_log_bucket_al", "logObjectPrefix": "mylogs"}

## Create a file and upload to cloud storage
echo "hello from cloudaffaire" > cloudaffaire_objecte.txt &&
gsutil cp cloudaffaire_objecte.txt gs://cloudaffaire_bucket_al 

## Get the logs from your logging bucket
gsutil ls gs://cloudaffaire_log_bucket_al

## gs://cloudaffaire_log_bucket_al/mylogs_usage_2020_01_14_14_00_00_06155500ed398c4da1_v0

## View the access logs for your bucket
## Replace the object name (mylogs_usage_2020_01_14_14_00_00_06155500ed398c4da1_v0) from above output
gsutil cat gs://cloudaffaire_log_bucket_al/mylogs_usage_2020_01_14_14_00_00_06155500ed398c4da1_v0
## Note: Usage logs are generated hourly when there is activity to report in the monitored bucket.
## Usage logs are typically created 15 minutes after the end of the hour. If your are not able to
## View any logs in your log bucket, then wait for some time for the log to get generated

## Disable logging for your bucket
gsutil logging set off gs://cloudaffaire_bucket_al

## Delete all access logs and your logging bucket
gsutil rm -r gs://cloudaffaire_log_bucket_al

## Delete all objects and your bucket
gsutil rm -r gs://cloudaffaire_bucket_al

##################################

## Access Logs In Cloud Storage ##

##################################

## Prerequisite: gsutil/gcloud installed and configured

## https://cloudaffaire.com/gcloud-installation-and-configuration/

## Create a bucket in cloud storage (bucket name needs to be globally unique)

gsutil mb -c standard -l asia-south1 -p cloudaffaire gs://cloudaffaire_bucket_al

## Create a log bucket to store access logs (bucket name needs to be globally unique)

gsutil mb -c standard -l asia-south1 -p cloudaffaire gs://cloudaffaire_log_bucket_al

## Allow Cloud Storage WRITE permission to the log bucket

gsutil acl ch -g cloud-storage-analytics@google.com:W gs://cloudaffaire_log_bucket_al

## Enable logging for your bucket

gsutil logging set on -b gs://cloudaffaire_log_bucket_al -o mylogs gs://cloudaffaire_bucket_al

## Get the logging for your bucket

gsutil logging get gs://cloudaffaire_bucket_al

## {"logBucket": "cloudaffaire_log_bucket_al", "logObjectPrefix": "mylogs"}

## Create a file and upload to cloud storage

echo "hello from cloudaffaire" > cloudaffaire_objecte.txt &&

gsutil cp cloudaffaire_objecte.txt gs://cloudaffaire_bucket_al

## Get the logs from your logging bucket

gsutil ls gs://cloudaffaire_log_bucket_al

## gs://cloudaffaire_log_bucket_al/mylogs_usage_2020_01_14_14_00_00_06155500ed398c4da1_v0

## View the access logs for your bucket

## Replace the object name (mylogs_usage_2020_01_14_14_00_00_06155500ed398c4da1_v0) from above output

gsutil cat gs://cloudaffaire_log_bucket_al/mylogs_usage_2020_01_14_14_00_00_06155500ed398c4da1_v0

## Note: Usage logs are generated hourly when there is activity to report in the monitored bucket.

## Usage logs are typically created 15 minutes after the end of the hour. If your are not able to

## View any logs in your log bucket, then wait for some time for the log to get generated

## Disable logging for your bucket

gsutil logging set off gs://cloudaffaire_bucket_al

## Delete all access logs and your logging bucket

gsutil rm -r gs://cloudaffaire_log_bucket_al

## Delete all objects and your bucket

gsutil rm -r gs://cloudaffaire_bucket_al

Hope you have enjoyed this article. In the next blog post, we will discuss Encryption in cloud storage.

All the public cloud providers are changing the console user interface rapidly and due to this some of the screenshots used in our previous AWS blogs are no longer relevant. Hence, we have decided that from now onwards most of the demo will be done programmatically. Let us know your feedback on this in the comment section.

To get more details on cloud storage, please refer below GCP documentation.

https://cloud.google.com/storage/docs/

https://cloud.google.com/storage/docs/audit-logs

https://cloud.google.com/storage/docs/access-logs

https://cloud.google.com/logging/docs/audit/

https://cloud.google.com/logging/docs/audit/configure-data-access

https://cloud.google.com/logging/docs/audit/configure-data-access#config-api