Using Google Cloud Storage with Go

Most of Google’s client libraries have been automatically generated. This is understandable, given that Google maintains a very large number of APIs and wish to provide libraries in a variety of languages. This results in client libraries for all or most of Google’s APIs in most mainstream languages, but it comes with a cost. These libraries are not always the easiest to get started with. The Go library for Google Cloud Storage, or GCS, is not among the more confusing Google client libraries, but sometimes it’s easier to read a tutorial than to read client documentation, so this will hopefully be of help to somebody.

If you do want to read through Google’s client docs, head over to https://pkg.go.dev/cloud.google.com/go/storage

Boilerplate

My Github user is j18e, so I’m going to create a new module by entering go mod init github.com/j18e/go-gcs-demo . Now we have a file called go.mod which will help keep track of imports.

Creating a client

To start off, let’s create a GCS client. We’re creating main.go and putting in the following:

package main
import (
    "context"
    "log"

    "cloud.google.com/go/storage"
)
const dirType = `inode/directory`
func main() {
    ctx := context.Background()
    cli, err := storage.NewClient(ctx)
    if err != nil {
        log.Fatal(err)
    }
    defer cli.Close()
}

So far, so good. We’re creating a client from Google Cloud’s Storage lib. What is not shown here is how we’re authenticating. What I’m doing when I run this is creating a service account in Google Cloud which has editor permissions on my GCS bucket, downloading a JSON service account key, and running export GOOGLE_APPLICATION_CREDENTIALS=/path/to/sa-json in the terminal from which I run my code. The storage library will automatically read the key file, or error if it doesn’t exist.

Listing bucket contents

Now let’s see if we can get our little app to do a simple ls in the bucket.

bkt := cli.Bucket("j18e-gcs-bucket")
it := bkt.Objects(ctx, nil)
for {
    attrs, err := it.Next()
    if err == iterator.Done {
        break
    }
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(attrs.Name, attrs.ContentType)
}

Before this will work, we’ll need to add"google.golang.org/api/iterator"to our imports. Given a bucket called j18e-gcs-bucket , we’re iterating through each item found in the bucket, printing both the item name and the content type. The content type will indicate to use if it’s a directory or regular file, for example. There are other attributes we could be printing, such as storage class, but we won’t be exploring those today.

Creating a directory

Folders aren’t strictly required in Google Cloud Storage. The standard cloud bucket paradigm is that the folder names eg some/path/are part of the entire file name. So some/path/file is the actual file name, not simply file . Having said all that, it is possible to create a directory as a discrete object. Let’s add that to our app:

contentType := `inode/directory`
obj := bkt.Object("data/myfolder")
writer := obj.NewWriter(ctx)
writer.ObjectAttrs.ContentType = contentType
if _, err := writer.Write(nil); err != nil {
    log.Fatal(err)
}
if err := writer.Close(); err != nil {
    log.Fatalf("closing writer: %v", err)
}

The method for creating a directory is the same as that for creating a regular file, you just refrain from adding any content and set the content type to inode/directory . Note that checking for an error on the Close method is necessary if we want to catch all failures. This is because the Write method executes asynchronously, so errors that occur after Write is called may not be returned by the Write method, however once Close returns we should have all possible errors reported.

Creating a file

Now let’s create a file in our new directory. Note that creating the directory beforehand was not strictly necessary, we just did that to demonstrate that it’s possible.

fileWriter := bkt.Object("data/myfolder/myfile").NewWriter(ctx)
msg := `this is a test`
if _, err := fileWriter.Write([]byte(msg)); err != nil {
    log.Fatal(err)
}
if err := fileWriter.Close(); err != nil {
    log.Fatalf("closing writer: %v", err)
}

Now we’re actually writing the file with some content as well as not setting the content type to inode/directory , so we’ll end up with a regular text file.

Putting it all together

package main
import (
    "context"
    "fmt"
    "log"
"cloud.google.com/go/storage"
    "google.golang.org/api/iterator"
)
func main() {
    ctx := context.Background()
    cli, err := storage.NewClient(ctx)
    if err != nil {
        log.Fatal(err)
    }
    defer cli.Close()
bkt := cli.Bucket("j18e-gcs-bucket")
it := bkt.Objects(ctx, nil)
    for {
        attrs, err := it.Next()
        if err == iterator.Done {
            break
        }
        if err != nil {
            log.Fatal(err)
        }
        fmt.Println(attrs.Name, attrs.ContentType)
    }
contentType := `inode/directory`
    obj := bkt.Object("data/myfolder")
    writer := obj.NewWriter(ctx)
    writer.ObjectAttrs.ContentType = contentType
    if _, err := writer.Write(nil); err != nil {
        log.Fatal(err)
    }
    if err := writer.Close(); err != nil {
        log.Fatalf("closing writer: %v", err)
    }
fileWriter := bkt.Object("data/myfolder/myfile").NewWriter(ctx)
    msg := `this is a test`
    if _, err := fileWriter.Write([]byte(msg)); err != nil {
        log.Fatal(err)
    }
    if err := fileWriter.Close(); err != nil {
        log.Fatalf("closing writer: %v", err)
    }
}

Hopefully this has been helpful to you in getting started with the Go client for Google Cloud Storage.