Google Cloud Storage Authentication

Most of the operations you perform in Cloud Storage must be authenticated. The only exceptions are operations on objects that allow anonymous access. Objects are anonymously accessible if the allUsers group has READ permission. The allUsers group includes anyone on the Internet.

OAuth 2.0

Authentication

Google Cloud Storage uses OAuth 2.0 for API authentication and authorization. Authentication is the process of determining the identity of a client.The details of authentication vary depending on how you are accessing Google Cloud Storage, but fall into two general types:

• A server-centric flow allows an application to directly hold the credentials of a service account to complete authentication. Use this flow if your application works with its own data rather than user data. Google Cloud Platform projects have default service accounts you can use, or you can create new ones.

• A user-centric flow allows an application to obtain credentials from an end user. The user signs in to complete authentication. Use this flow if your application needs to access user data. See the User account credentials section later in this page for scenarios where a user-centric flow is appropriate.

Keep in mind that you can use both types of authentication together in an application. For more background information about authentication, see the Google Cloud Platform Auth Guide.

Scopes

Authorization is the process of determining what permissions an authenticated identity has on a set of specified resources. OAuth uses scopes to determine if an authenticated identity is authorized. Applications use a credential (obtained from a user-centric or server-centric authentication flow) together with one or more scopes to request an access token from a Google authorization server to access protected resources. For example, application A with an access token with read-only scope can only read, while application B with an access token with read-write scope can read and modify data. Neither application can read or modify access control lists on objects and buckets; only an application with full-control scope can do so.

gsutil authentication

With gsutil installed from the Cloud SDK, you can authenticate with user or service account credentials.

gcloud beta auth application-default login

gcloud auth uses the cloud-platform scope when getting an access token.

If you installed gsutil independent of the Cloud SDK, then see the gsutil install page for information about how to authenticate with a user or service account.

Client library authentication

Client libraries can use Application Default Credentials to easily authenticate with Google APIs and send requests to those APIs. With Application Default Credentials, you can test your application locally and deploy it without changing the underlying code. For more information, including code samples, see Google Cloud Platform Auth Guide.

Application Default Credentials are part of the client libraries you can use to access Google Cloud Storage. Default credentials identify your application with either a user credential or a default service account. When working with a library, it's typical that you'll choose credentials based on the type of environment where you run your code.

Initialize your environment

Local Development Environment

To initialize your local development environment, you can use the Google Cloud SDK to authenticate with user credentials. On Linux/Mac OS X, use the script:

curl https://sdk.cloud.google.com | bash

Once installed, you can use your own user account credentials as a proxy to test code calling APIs from your local machine by running:

gcloud beta auth application-default login

This command requests access tokens with the cloud-platform scope.

Google Cloud Platform

If you're running your application on Google App Engine or Google Compute Engine, the environment already provides a service account's authentication information, so no further setup is required. For Compute Engine, the service account scope depends on how you created the instance. See Setting the scope of service account access for instances. For App Engine, the cloud-platform scope is used.

Other Environments

For production environments outside the Google Cloud Platform, you provide a service account's authentication information through an environment variable and the scope is set in code. For more information, see How the Application Default Credentials work. Creating a service account is discussed later in this page.

Examples of using Application Default Credentials

using System;
// Imports the Google Cloud client library
using Google.Storage.V1;

public class QuickstartSample
{
    public static void Main()
    {
        // Your Google Cloud Platform project ID
        string projectId = "YOUR_PROJECT_ID";

        // Instantiates a client
        StorageClient storageClient = StorageClient.Create();

        // The name for the new bucket
        string bucketName = "my-new-bucket";

        // Creates the new bucket
        storageClient.CreateBucket(projectId, bucketName);

        Console.WriteLine($"Bucket {bucket.Name} created.");
    }
}

// Sample storage_quickstart creates a Google Cloud Storage bucket.
package main

import (
	"fmt"
	"golang.org/x/net/context"
	"log"

	// Imports the Google Cloud Storage client package
	"cloud.google.com/go/storage"
)

func main() {
	ctx := context.Background()

	// Your Google Cloud Platform project ID
	projectID := "YOUR_PROJECT_ID"

	// Creates a client
	client, err := storage.NewClient(ctx)
	if err != nil {
		log.Fatalf("Failed to create client: %v", err)
	}

	// The name for the new bucket
	bucketName := "my-new-bucket"

	// Prepares a new bucket
	bucket := client.Bucket(bucketName)

	// Creates the new bucket
	if err := bucket.Create(ctx, projectID, nil); err != nil {
		log.Fatalf("Failed to create bucket: %v", err)
	}

	fmt.Printf("Bucket %v created.", bucketName)
}

// Imports the Google Cloud client library
import com.google.cloud.storage.Bucket;
import com.google.cloud.storage.BucketInfo;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;

public class QuickstartSample {
  public static void main(String... args) throws Exception {
    // Instantiates a client
    Storage storage = StorageOptions.getDefaultInstance().getService();

    // The name for the new bucket
    String bucketName = args[0];  // "my-new-bucket";

    // Creates the new bucket
    Bucket bucket = storage.create(BucketInfo.of(bucketName));

    System.out.printf("Bucket %s created.%n", bucket.getName());
  }
}

// Imports the Google Cloud client library
const Storage = require('@google-cloud/storage');

// Your Google Cloud Platform project ID
const projectId = 'YOUR_PROJECT_ID';

// Instantiates a client
const storageClient = Storage({
  projectId: projectId
});

// The name for the new bucket
const bucketName = 'my-new-bucket';

// Creates the new bucket
storageClient.createBucket(bucketName, (err, bucket) => {
  if (err) {
    console.error(err);
    return;
  }

  console.log(`Bucket ${bucket.name} created.`);
});

# Includes the autoloader for libraries installed with composer
require __DIR__ . '/vendor/autoload.php';

# Imports the Google Cloud client library
use Google\Cloud\Storage\StorageClient;

# Your Google Cloud Platform project ID
$projectId = 'YOUR_PROJECT_ID';

# Instantiates a client
$storage = new StorageClient([
    'projectId' => $projectId
]);

# The name for the new bucket
$bucketName = 'my-new-bucket';

# Creates the new bucket
$bucket = $storage->createBucket($bucketName);

echo 'Bucket ' . $bucket->name() . ' created.';

# Imports the Google Cloud client library
from google.cloud import storage

# Instantiates a client
storage_client = storage.Client()

# The name for the new bucket
bucket_name = 'my-new-bucket'

# Creates the new bucket
bucket = storage_client.create_bucket(bucket_name)

print('Bucket {} created.'.format(bucket.name))

# Imports the Google Cloud client library
require "google/cloud/storage"

# Your Google Cloud Platform project ID
project_id = "YOUR_PROJECT_ID"

# Instantiates a client
storage = Google::Cloud::Storage.new project: project_id

# The name for the new bucket
bucket_name = "my-new-bucket"

# Creates the new bucket
bucket = storage.create_bucket bucket_name

puts "Bucket #{bucket.name} was created."

API authentication

To make requests using OAuth 2.0 to either the Google Cloud Storage XML API or JSON API, include your application's access token in the Authorization header in every request that requires authentication.

Authorization: Bearer <oauth2_token>

The following is an example of a request that lists objects in a bucket.

GET /storage/v1/b/example-bucket/o HTTP/1.1
Host: www.googleapis.com
Authorization: Bearer ya29.AHES6ZRVmB7fkLtd1XTmq6mo0S1wqZZi3-Lh_s-6Uw7p8vtgSwg

GET / HTTP/1.1
Host: example-bucket.storage.googleapis.com
Authorization: Bearer ya29.AHES6ZRVmB7fkLtd1XTmq6mo0S1wqZZi3-Lh_s-6Uw7p8vtgSwg

Due to the complexity of managing and refreshing access tokens and the security risk when dealing directly with cryptographic applications, we strongly encourage you to use a verified client library.

If you looking for developer keys to use with the XML API for interoperable access with Amazon S3, see Managing developer keys for a simple migration.

Cookie-based authentication

Cloud Storage lets you provide browser-based authenticated downloads to users who have Google accounts but do not necessarily have Cloud Storage accounts. To do this, apply Google account-based Access Control Lists (ACLs) to the object, then provide users with a URL that is scoped to the object. The URL for browser-based authenticated downloads is:

https://storage.cloud.google.com/bucket/object

For example, to let Jane download an object named /europe/france/paris.jpg in a bucket named example-travel-maps, you grant READ permission to [email protected] for the object /europe/france/paris.jpg. Then you provide Jane with the following URL:

https://storage.cloud.google.com/example-travel-maps/europe/france/paris.jpg

When Jane clicks the URL in her browser, she's automatically prompted to sign in to her Google account (if she's not already logged in). After she is authenticated, and her browser has acquired a cookie with an encapsulated identity token, she is redirected to the object in the Cloud Storage repository. Cloud Storage then verifies that Jane is allowed to read the object, and the object is downloaded to Jane's computer.

The following figure shows how the authentication process for browser-based authenticated downloads works.

Service account credentials

Service accounts are special accounts that represent software rather than people. They are the most common way applications authenticate with Google Cloud Storage. Every project has service accounts associated with it, which may be used for different authentication scenarios, as well as to enable advanced features such as Signed URLs and browser uploads using POST.

When you use a service account to authenticate your application, you do not need a user to authenticate to get an access token. Instead, you obtain a private key from the Google Cloud Platform Console, which you then use to send a signed request for an access token. You can then use the access token like you normally would. For more information see the Google Cloud Platform Auth Guide.

Generating a service account credential

You can create a private key in the Cloud Platform Console by creating an OAuth Client ID for a service account. You can get your private key in JSON and PKCS12 format:

• JSON keys are required if you are using Application Default Credentials in a production environment outside of Google Cloud Platform. JSON keys cannot be converted to other formats.
• PKCS12 is supported by many different programming languages and libraries. If needed, you can convert the key into other formats using OpenSSL (see Converting the private key to other formats). However, PKCS12 keys cannot be converted to JSON format.

To generate a private key in JSON or PKCS12 format:

1. Open the list of credentials in the Google Cloud Platform Console.

   Open the list of credentials

2. Click Create credentials.
3. Select Service account key.

   A Create service account key window opens.

4. Click the drop-down box below Service account, then click New service account.
5. Enter a name for the service account in Name.
6. Use the default Service account ID or generate a different one.
7. Select the Key type: JSON or P12.
8. Click Create.

   A Service account created window is displayed and the private key for the Key type you selected is downloaded automatically. If you selected a P12 key, the private key's password ("notasecret") is displayed.

9. Click Close.

For more information about security and identity in projects, see the Google Cloud Platform Console Help.

Service credential details

The OAuth client ID of a service account uniquely identifies the account being used for the authentication. After you create a Client ID for a service account (see Generating a service account credential), you will see the following information:

• Client ID
• Email address
• Certificate fingerprints

Notice that there are two identifiers of the service account, the Client ID and the Email address. You must use the Email address for RSA signatures, for example, for Signed URLs (Query String Authentication).

Converting the service account credential to other formats

After you create a Client ID for a service account (see Generating a service account credential) and download a private key in PKCS12 format, you can convert the key to other formats if necessary by using OpenSSL. For example, to convert the key to PKCS1 (PEM):

# Convert the key from pkcs12 to pkcs1 (PEM).
cat /path/to/xxxx-privatekey.p12 | openssl pkcs12 -nodes -nocerts -passin pass:notasecret | openssl rsa > /path/to/secret.pem

You can then use openssl to generate a signature digest for your data:

# Generate a signature digest using the PEM key.
openssl dgst -sign path/to/key.pem -sha256 path/to/data_to_sign | base64 -w0

You can also use openssl to convert the PEM file format to DER format. Some libraries (for example, PyCrypto) require that keys be in DER format.

# Convert the key from pkcs1 (PEM) to DER.
openssl rsa -in path/to/key.pem -inform PEM -out path/to/key.der -outform DER

JSON private keys cannot be converted to other formats.

User account credentials

Use user account credentials for authentication when your application requires access to data on a user's behalf; otherwise, use service account credentials. Here are examples of scenarios where user account credentials can be used:

• Web server applications
• Installed and desktop applications
• Client-side JavaScript
• Applications on limited-input devices

For more information on these scenarios, see OAuth scenarios.

If you are designing an application to support multiple authentication options for end users, then use Google Identity Toolkit, which supports password authentication as well as federated sign in with identity providers like Google, Facebook, Yahoo, and Microsoft.

When an application is granted an access token in a user-centric auth flow by an end-user, that access token will only have the permissions available to the user who grants the token. For example, if [email protected] has read-only access to example-bucket, an application which Jane has granted read-write access to will be unable to write to example-bucket on her behalf.