A Deep Dive into Google Cloud Storage: What it is, why, how, and when to use it.

Introduction

What is Google Cloud Storage?

Google Cloud Storage (GCS) is a type of object storage offered by Google. But then, what is object storage you may ask. Object storage is a type of storage service that stores data in the form of objects or blobs. These objects are usually uniquely identifiable by an addressable URL. It is also one of the 4 main categories of storage services provided by most public cloud providers. Other types of storage services include File Storage, Block Storage, and Caches.

Object storage treats stored objects as atomic in nature and is stored in a logical construct called “Buckets”, but more on this later.

Another parallel service offering that directly competes with Google Cloud Storage will be Amazon Simple Storage Service(S3).

To understand object storage better, please read my previous post on types of storage services.

Note: GCS is used as an abbreviation for Google Cloud Storage in later sections of this post

What we will cover in this post

• Business use cases for GCS
• What are Buckets? and how to use them
• Classes of storage in GCS
• Access Control Policies and Permissions
• GCS Durability
• Keys and Encryption

Business use cases for GCS

GCS is perfect for use in any of, but not limited to the following use cases:

• Large and continually growing unstructured datasets.
• Data backups and archiving.
• When finely grained access controls are required at the data layer.
• Hosting simple web pages and applications.

Large and continually growing unstructured datasets.

Google cloud storage is perfect for storing large unstructured datasets. Unstructured in this sense could mean any data type requiring a basic and atomic read/write capability. GCS has an infinite scale and can accommodate whatever size of data you produce.
You can migrate petabytes of data to google cloud storage using the Google Transfer Appliance.

Data backups and archiving

GCS has special storage classes for storing backups and archived data at a very minimal cost and in an efficient way. These storage classes are Nearline and Coldline storage. Use nearline storage if you intend to access this data at least once a month and cold line storage when access is estimated to be not more than once a year. While this data is still accessible whenever you need it, these storage classes are not designed for frequent access and you might be charged more for this.

When finely grained access controls are required at the data layer

With GCS, managing fine-grained access to data becomes a breeze. Access controls can be defined at the bucket and object levels. If you’ve ever used IAM to manage permissions on a GCP resource, this same functionality can be used to managed access to objects and their containing buckets across your organization and the general public. read more about IAM

Hosting simple web pages and applications

You may not have heard of this one, but you can actually host simple web pages on google cloud storage by simply uploading the contents of your static website to a cloud storage bucket, setting permissions, and enabling the DNS service. You can set this up with a custom domain and viola!
GCS is RESTful by design and so all contents of your buckets can be served directly without any issues. In a future post, we will explore this in greater detail.

What are Buckets? and types of Buckets

In this section, we will explore buckets in GCP. what they are, how to create and manage them. As a personal preference, I will be using the gsutil tool to manage my buckets. gsutil is a google cloud command-line tool for managing your storage resources on google cloud.

Google cloud shell comes with gsutil and a suite of other CLI components preinstalled, so we will use that going forward.

so, what are Buckets?
Buckets are logical containers for your objects. They exist in a global namespace. This means no two buckets can have the same name on GCS. Bucket names must be unique and follow a DNS-like naming convention since, in fact, these names will be used in the RESTful URLs GCS provides you to access the contents of our bucket.

What you will need before proceeding:

• A Google cloud account. Creating a GCP account
• Billing enabled on your account. How to enable billing?
• A Google cloud shell interface or gsutil installed on your machine.

I will also recommend creating a new GCP project for this and deleting it afterward to avoid additional costs.

Now to the exciting parts, let's dive in!

Creating a bucketgsutil mb gs://BUCKET_NAME

BUCKET_NAME: A globally unique name you will like to give your bucket. It is recommended to use DNS-like names for buckets. You may get a 409 error if you use an already existing bucket name.
The gsutil helpcommand allows us to explore gsutil specific commands in greater detail. Some popular flags to use with the mb command include:

• -p: Used to specify the project to associate your new bucket
• -c: Used to specify the default storage class of your bucket. (we will explore storage classes in a later post).
• -l: Used to specify the location of your bucket. For example, US-EAST1

Useful defaults
Without specifying any of the above options, GCS creates reasonable defaults for us based on the selected project and location. To view these set defaults for our earlier created bucket, we can use the command.gsutil ls -L -b gs://BUCKET_NAME

Adding and viewing contents of a bucket

In this section, we explore the following:

• Setting environment variables
• Uploading an object (photo) to CloudShell
• Copying our object to GCS
• Viewing contents of our bucket to validate upload

You can download the cat photo from here

export BUCKET_NAME=BUCKET_NAME
gsutil cp ~/cat-photo.jpeg $BUCKET_NAME
gsutil ls $BUCKET_NAME

you can also view the number of items in your bucket with the command.gsutil ls -l $BUCKET_NAME

This should produce a result such as:93026  2021-07-18T04:23:22Z  gs://my-gsutil-demo-bucket-1/cat-photo.jpeg
TOTAL: 1 objects, 93026 bytes (90.85 KiB)

In the next post, we will explore classes of GCS storage, access controls, and permissions for the bucket and objects we created above.
Till next time, always remember gsutil help is your friend.