Skip to main content

Google Cloud Storage (GCS)

Overview​

Google Cloud Storage (GCS) is a scalable, fully managed, and highly reliable object storage service offered by Google Cloud, designed to store and access data from anywhere in the world. It provides a secure and cost-effective way to store data, including common data storage formats such as CSV and Parquet. Rill supports natively connecting to GCS using the provided Google Cloud Storage URI of your bucket to retrieve and read files.


Rill Developer (Local credentials)​

When using Rill Developer on your local machine (i.e., rill start), Rill will use either the credentials configured in your local environment using the Google Cloud CLI (gcloud) or an explicitly defined connector YAML.

ollow these steps to configure your local environmental credentials:

Prerequisites

To use the Google Cloud CLI, you will need to install the Google Cloud CLI. If you are unsure if this has been done, you can run the following command from the command line to see if it returns your authenticated user.

gcloud auth list

If an error or no users are returned, please follow Google's documentation on setting up your command line before continuing. Make sure to run gcloud init after installation as described in the tutorial.

  1. Install the Google Cloud CLI.
  2. Initiate the Google Cloud CLI by running gcloud init.
  3. Set up your user by running gcloud auth application-default login.
Service Accounts

If you are using a service account, you will need to run the following command:

gcloud auth activate-service-account --key-file=path_to_json_key_file

You have now configured Google Cloud access from your local environment. Rill will detect and use your credentials the next time you try to ingest a source.

Did you know?

If this project has already been deployed to Rill Cloud and credentials have been set for this source, you can use rill env pull to pull these cloud credentials locally (into your local .env file). Please note that this may override any credentials you have set locally for this source.

Separating Dev and Prod Environments​

When ingesting data locally, consider setting parameters in your connector file to limit how much data is retrieved, since costs can scale with the data source. This also helps other developers clone the project and iterate quickly by reducing ingestion time.

For more details, see our Dev/Prod setup docs.

Rill Cloud Deployment​

When deploying a project to Rill Cloud, Rill requires a JSON key file to be explicitly provided for a Google Cloud service account with appropriate read access/permissions to the buckets used in your project.

When you first deploy a project using rill deploy, you will be prompted to provide credentials for the remote sources in your project that require authentication.

If you subsequently add sources that require new credentials (or if you enter the wrong credentials during the initial deploy), you can update the credentials used by Rill Cloud by running:

rill env configure
Did you know?

If you've already configured credentials locally (in your <RILL_PROJECT_DIRECTORY>/.env file), you can use rill env push to push these credentials to your Rill Cloud project. This will allow other users to retrieve and reuse the same credentials automatically by running rill env pull.

Appendix​

How to create a service account using the Google Cloud Console​

Here is a step-by-step guide on how to create a Google Cloud service account with read-only access to GCS:

  1. Navigate to the Service Accounts page under "IAM & Admin" in the Google Cloud Console.
  2. Click the "Create Service Account" button at the top of the page.
  3. In the "Create Service Account" window, enter a name for the service account, then click "Create and continue".
  4. In the "Role" field, search for and select the "Storage Object Viewer" role. Click "Continue", then click "Done".
    • This grants the service account access to data in all buckets. To only grant access to data in a specific bucket, leave the "Role" field blank, click "Done", then follow the steps described in Add a principal to a bucket-level policy.
  5. On the "Service Accounts" page, locate the service account you just created and click on the three dots on the right-hand side. Select "Manage Keys" from the dropdown menu.
  6. On the "Keys" page, click the "Add key" button and select "Create new key".
  7. Choose the "JSON" key type and click "Create".
  8. Download and save the JSON key file to a secure location on your computer.

How to create a service account using the gcloud CLI​

  1. Open a terminal window and follow the steps on Install the Google Cloud CLI if you haven't already done so.
  2. You will need your Google Cloud project ID to complete this tutorial. Run the following command to show it: 
    gcloud config get project
  3. Replace [PROJECT_ID] with your project ID in the following command, and run it to create a new service account (optionally also replace rill-service-account with a name of your choice): 
    gcloud iam service-accounts create rill-service-account --project [PROJECT_ID]
  4. Grant the service account access to data in Google Cloud Storage:
    • To grant access to data in all buckets, replace [PROJECT_ID] with your project ID in the following command, and run it: 
      gcloud projects add-iam-policy-binding [PROJECT_ID] \
          --member="serviceAccount:rill-service-account@[PROJECT_ID].iam.gserviceaccount.com" \
          --role="roles/storage.objectViewer"
    • To only grant access to data in a specific bucket, replace [BUCKET_NAME] and [PROJECT_ID] with your details in the following command, and run it: 
      gcloud storage buckets add-iam-policy-binding gs://[BUCKET_NAME] \
          --member="serviceAccount:rill-service-account@[PROJECT_ID].iam.gserviceaccount.com" \
          --role="roles/storage.objectViewer"
  5. Replace [PROJECT_ID] with your project ID in the following command, and run it to create a key file for the service account: 
    gcloud iam service-accounts keys create rill-service-account.json \
        --iam-account rill-service-account@[PROJECT_ID].iam.gserviceaccount.com
  6. You have now created a JSON key file named rill-service-account.json in your current working directory.
info

As an alternative, to ensure that you are running Rill with a specific service account, you can provide the key in the rill start command. This is useful when you have multiple profiles or may receive limited access to a bucket.

GOOGLE_APPLICATION_CREDENTIALS=<path_to_json_key_file> rill start