Google Cloud Storage (GCS)

Overview

Google Cloud Storage (GCS) is a scalable, fully managed, and highly reliable object storage service offered by Google Cloud, designed to store and access data from anywhere in the world. It provides a secure and cost-effective way to store data, including in common data storage formats such as CSV and parquet. Rill supports natively connecting to GCS using the provided Google Cloud Storage URI of your bucket to retrieve and read files.

Rill Developer (Local credentials)

When using Rill Developer on your local machine (i.e. rill start), Rill uses the credentials configured in your local environment using the Google Cloud CLI (gcloud). Follow these steps to configure it:

prerequisites

In order to use the Google Cloud CLI, you will need to install the Google Cloud CLI. If you are unsure if this has been done, you can run the following command from the command line and see if it returns your authenticated user.

gcloud auth list

If an error or no users are returned, please follow Google's documentation on setting up your command line before continuing. Make sure to run gcloud init after installation as described in the tutorial.

1. Install the Google Cloud CLI.
2. Initiate the Google Cloud CLI by running gcloud init.
3. Set up your user by running gcloud auth application-default login.

Service Accounts

If you are using a service account, you will need to run the following command:

gcloud auth activate-service-account --key-file=`path_to_json_key_file`

You have now configured Google Cloud access from your local environment. Rill will detect and use your credentials next time you try to ingest a source.

info

As an alternative, to ensure that you are running rill with a specifc service account, you can provide the key in the rill start command. This is useful for when you have multiple profiles or may receive limited access to a bucket.

GOOGLE_APPLICATION_CREDENTIALS=<path_to_json_key_file> rill start

Did you know?

If this project has already been deployed to Rill Cloud and credentials have been set for this source, you can use rill env pull to pull these cloud credentials locally (into your local .env file). Please note that this may override any credentials that you have set locally for this source.

Rill Cloud deployment

When deploying a project to Rill Cloud, Rill requires a JSON key file to be explicitly provided for a Google Cloud service account with appropriate read access / permissions to the buckets used in your project.

When you first deploy a project using rill deploy, you will be prompted to provide credentials for the remote sources in your project that require authentication.

If you subsequently add sources that require new credentials (or if you input the wrong credentials during the initial deploy), you can update the credentials used by Rill Cloud by running:

rill env configure

info

Note that you must cd into the Git repository that your project was deployed from before running rill env configure or set the --project flag in the command: rill env configure --project 'project_name'.

Did you know?

If you've configured credentials locally already (in your <RILL_PROJECT_DIRECTORY>/.env file), you can use rill env push to push these credentials to your Rill Cloud project. This will allow other users to retrieve / reuse the same credentials automatically by running rill env pull.

Appendix

How to create a service account using the Google Cloud Console

Here is a step-by-step guide on how to create a Google Cloud service account with read-only access to GCS:

1. Navigate to the Service Accounts page under "IAM & Admin" in the Google Cloud Console.

2. Click the "Create Service Account" button at the top of the page.

3. In the "Create Service Account" window, enter a name for the service account, then click "Create and continue".

4. In the "Role" field, search for and select the "Storage Object Viewer" role. Click "Continue", then click "Done".

   • This grants the service account access to data in all buckets. To only grant access to data in a specific bucket, leave the "Role" field blank, click "Done", then follow the steps described in Add a principal to a bucket-level policy.

5. On the "Service Accounts" page, locate the service account you just created and click on the three dots on the right-hand side. Select "Manage Keys" from the dropdown menu.

6. On the "Keys" page, click the "Add key" button and select "Create new key".

7. Choose the "JSON" key type and click "Create".

8. Download and save the JSON key file to a secure location on your computer.

How to create a service account using the gcloud CLI

1. Open a terminal window and follow the steps on Install the Google Cloud CLI if you haven't already done so.

2. You will need your Google Cloud project ID to complete this tutorial. Run the following command to show it:

   gcloud config get project

3. Replace [PROJECT_ID] with your project ID in the following command, and run it to create a new service account (optionally also replace rill-service-account with a name of your choice):

   gcloud iam service-accounts create rill-service-account --project [PROJECT_ID]

4. Grant the service account access to data in Google Cloud Storage:

   • To grant access to data in all buckets, replace [PROJECT_ID] with your project ID in the following command, and run it:

     gcloud projects add-iam-policy-binding [PROJECT_ID] \
         --member="serviceAccount:rill-service-account@[PROJECT_ID].iam.gserviceaccount.com" \
         --role="roles/storage.objectViewer"
   • To only grant access to data in a specific bucket, replace [BUCKET_NAME] and [PROJECT_ID] with your details in the following command, and run it:

     gcloud storage buckets add-iam-policy-binding gs://[BUCKET_NAME] \
         --member="serviceAccount:rill-service-account@[PROJECT_ID].iam.gserviceaccount.com" \
         --role="roles/storage.objectViewer"

5. Replace [PROJECT_ID] with your project ID in the following command, and run it to create a key file for the service account:

   gcloud iam service-accounts keys create rill-service-account.json \
       --iam-account rill-service-account@[PROJECT_ID].iam.gserviceaccount.com

6. You have now created a JSON key file named rill-service-account.json in your current working directory.