Skip to main content

Azure Blob Storage

Overview​

Azure Blob Storage (ABS) is a scalable, fully managed, and highly reliable object storage solution offered by Microsoft Azure, designed to store and access data from anywhere in the world. It provides a secure and cost-effective way to store data, including common storage formats such as CSV and Parquet. Rill supports connecting to and reading from Azure Blob Storage using the following Resource URI syntax:

azure://<account>.blob.core.windows.net/<container>/path/file.csv

Authentication Methods​

To connect to Azure Blob Storage, you can choose from five authentication options:

  1. Storage Account Key (recommended for cloud deployment)
  2. Connection String (alternative for cloud deployment)
  3. Shared Access Signature (SAS) Token (most secure, fine-grained control)
  4. Public (for publicly accessible containers - no authentication required)
  5. Azure CLI authentication (local development only - not recommended for production)

When you add data from Azure Blob Storage through the Rill UI, you'll see three authentication options:

  • Storage Account Key, Connection String, or SAS Token: The process follows two steps:

    1. Configure Authentication - Set up your Azure connector with credentials
    2. Configure Data Model - Define which container and objects to ingest The UI will automatically create both the connector file and model file for you.

  • Public: For publicly accessible containers, you skip the connector creation step and go directly to:

    1. Configure Data Model - Define which container and objects to ingest The UI will only create the model file (no connector file is needed).
Manual Configuration Only

Azure CLI authentication is only available through manual configuration. See Method 5: Azure CLI Authentication for setup instructions.

Storage Account Key credentials provide reliable authentication for Azure Blob Storage. This method works for both local development and Rill Cloud deployments.

Using the UI​

  1. Click Add Data in your Rill project
  2. Select Azure Blob Storage as the data model type
  3. In the authentication step:
    • Choose Storage Account Key
    • Enter your Storage Account name
    • Enter your Storage Account Key
  4. In the data model configuration step, enter your SQL query
  5. Click Create to finalize

After the model YAML is generated, you can add additional model settings directly to the file.

Manual Configuration​

If you prefer to configure manually:

Step 1: Create connector configuration

Create connectors/azure.yaml:

type: connector
driver: azure

azure_storage_account: rilltest
azure_storage_key: "{{ .env.AZURE_STORAGE_KEY }}"

Step 2: Add credentials to .env

AZURE_STORAGE_KEY=your_storage_account_key

Follow the Azure Documentation to retrieve your storage account keys. Then, create your first model.

Method 2: Connection String​

Connection String provides an alternative authentication method for Azure Blob Storage.

Using the UI​

  1. Click Add Data in your Rill project
  2. Select Azure Blob Storage as the data model type
  3. In the authentication step:
    • Choose Connection String
    • Enter your Connection String
  4. In the data model configuration step, enter your SQL query
  5. Click Create to finalize

After the model YAML is generated, you can add additional model settings directly to the file.

Manual Configuration​

Step 1: Create connector configuration

Create connectors/azure_conn.yaml:

type: connector
driver: azure

azure_storage_connection_string: "{{ .env.AZURE_STORAGE_CONNECTION_STRING }}"

Step 2: Add credentials to .env

AZURE_STORAGE_CONNECTION_STRING=your_connection_string

Follow the Azure Documentation to retrieve your connection string. Then, create your first model.

Method 3: Shared Access Signature (SAS) Token​

SAS tokens provide fine-grained access control with specific permissions and expiration times for secure access to your storage resources.

Using the UI​

  1. Click Add Data in your Rill project
  2. Select Azure Blob Storage as the data model type
  3. In the authentication step:
    • Choose SAS Token
    • Enter your Storage Account name
    • Enter your SAS Token
  4. In the data model configuration step, enter your SQL query
  5. Click Create to finalize

After the model YAML is generated, you can add additional model settings directly to the file.

Manual Configuration​

Step 1: Create connector configuration

Create connectors/azure.yaml:

type: connector
driver: azure

azure_storage_account: rilltest
azure_storage_sas_token: "{{ .env.AZURE_STORAGE_SAS_TOKEN }}"

Step 2: Add credentials to .env

AZURE_STORAGE_SAS_TOKEN=your_sas_token

Follow the Azure Documentation to create your Azure SAS token. Then, create your first model.

Method 4: Public Containers​

For publicly accessible Azure Blob Storage containers, you don't need to create a connector. Simply use the Azure URI directly in your model configuration.

Using the UI​

  1. Click Add Data in your Rill project
  2. Select Azure Blob Storage as the data model type
  3. In the authentication step:
    • Choose Public
    • The UI will skip connector creation and proceed directly to data model configuration
  4. In the data model configuration step, enter your SQL query
  5. Click Create to finalize

After the model YAML is generated, you can add additional model settings directly to the file.

Manual Configuration​

No connector configuration is needed for public containers. Proceed to Create Your First Model below and omit the create_secrets_from_connectors line.

Method 5: Azure CLI Authentication (Local Development Only)​

For local development, you can use credentials from the Azure CLI. This method is not suitable for production or Rill Cloud deployments. This method is only available through manual configuration, and you don't need to create a connector file.

Setup​

  1. Install the Azure CLI if not already installed
  2. Authenticate with your Azure account: 
    az login
  3. Verify your authentication status: 
    az account show
  4. Create your model file (no connector needed)

Manual Configuration​

No connector is needed when using local credentials. Proceed to Create Your First Model below and omit the create_secrets_from_connectors line.

Rill will automatically detect and use your local Azure CLI credentials when no connector is specified.

warning

This method only works for local development. Deploying to Rill Cloud with this configuration will fail because the cloud environment doesn't have access to your local credentials. Always use Storage Account Key, Connection String, or SAS tokens for production deployments.

Path Patterns​

You can use wildcards to read multiple files:

-- Single file
SELECT * FROM read_parquet('azure://account.blob.core.windows.net/container/data/file.parquet')

-- All files in a directory
SELECT * FROM read_parquet('azure://account.blob.core.windows.net/container/data/*.parquet')

-- All files in nested directories
SELECT * FROM read_parquet('azure://account.blob.core.windows.net/container/data/**/*.parquet')

-- Files matching a pattern
SELECT * FROM read_parquet('azure://account.blob.core.windows.net/container/data/2024-*.parquet')

Create Your First Model​

After setting up your authentication above, create a model to define what data to pull.

Create models/azure_data.yaml:

type: model
connector: duckdb
create_secrets_from_connectors: azure

dev:
  sql: SELECT * FROM read_parquet('azure://rilltest.blob.core.windows.net/my-container/path/to/data/*.parquet')

sql: SELECT * FROM read_parquet('azure://rilltest.blob.core.windows.net/my-container/**/*.parquet')

  • Replace azure in create_secrets_from_connectors with the name of the connector you created.
  • If you're using a public container or local credentials (no connector), omit the create_secrets_from_connectors line.

After creating the model, you can add additional model settings directly to the file.

Deploy to Rill Cloud​

When deploying a project to Rill Cloud, Rill requires you to explicitly provide an Azure Blob Storage connection string, Azure Storage Key, or Azure Storage SAS token with access to Azure Blob Storage used in your project. Please refer to our connector YAML reference docs for more information.

If you subsequently add sources that require new credentials (or if you simply entered the wrong credentials during the initial deploy), you can update the credentials by pushing the Deploy button to update your project or by running the following command in the CLI:

rill env push
Did you know?

If you've already configured credentials locally (in your <RILL_PROJECT_DIRECTORY>/.env file), you can use rill env push to push these credentials to your Rill Cloud project. This will allow other users to retrieve and reuse the same credentials automatically by running rill env pull.