Azure Blob Storage
Overview
Azure Blob Storage (ABS) is a scalable, fully managed, and highly reliable object storage solution offered by Microsoft Azure, designed to store and access data from anywhere in the world. It provides a secure and cost-effective way to store data, including common storage formats such as CSV and Parquet. Rill supports connecting to and reading from Azure Blob Storage using the following Resource URI syntax:
azure://<account>.blob.core.windows.net/<container>/path/file.csv
Authentication Methods
To connect to Azure Blob Storage, you can choose from four authentication options:
- Storage Account Key (recommended for cloud deployment)
- Connection String (alternative for cloud deployment)
- Shared Access Signature (SAS) Token (most secure, fine-grained control)
- Public (for publicly accessible containers - no authentication required)
- Azure CLI authentication (local development only - not recommended for production)
Choose the method that best fits your setup. For production deployments to Rill Cloud, use Storage Account Key, Connection String, or SAS tokens. Public containers don't require authentication and skip connector creation. Azure CLI authentication only works for local development and will cause deployment failures.
Using the Add Data UI
When you add an Azure Blob Storage data model through the Rill UI, you'll see four authentication options:
-
Storage Account Key, Connection String, or SAS Token: The process follows two steps:
- Configure Authentication - Set up your Azure connector with credentials
- Configure Data Model - Define which container and objects to ingest The UI will automatically create both the connector file and model file for you.
-
Public: For publicly accessible containers, you skip the connector creation step and go directly to:
- Configure Data Model - Define which container and objects to ingest The UI will only create the model file (no connector file is needed).
Azure CLI authentication is only available through manual configuration. See Method 5: Azure CLI Authentication for setup instructions.
Method 1: Storage Account Key (Recommended)
Storage Account Key credentials provide reliable authentication for Azure Blob Storage. This method works for both local development and Rill Cloud deployments.
Using the UI
- Click Add Data in your Rill project
- Select Azure Blob Storage as the data model type
- In the authentication step:
- Choose Storage Account Key
- Enter your Storage Account name
- Enter your Storage Account Key
- Name your connector (e.g.,
my_azure)
- In the data model configuration step:
- Enter your container name and object path
- Configure other model settings as needed
- Click Create to finalize
The UI will automatically create both the connector file and model file for you.
Manual Configuration
If you prefer to configure manually, create two files:
Step 1: Create connector configuration
Create connectors/my_azure.yaml:
type: connector
driver: azure
azure_storage_account: rilltest
azure_storage_key: "{{ .env.connector.azure.azure_storage_key }}"
Step 2: Create model configuration
Create models/my_azure_data.yaml:
type: model
connector: duckdb
create_secrets_from_connectors: my_azure
sql: SELECT * FROM read_parquet('azure://rilltest.blob.core.windows.net/my-container/path/to/data/*.parquet')
refresh:
cron: "0 */6 * * *"
Step 3: Add credentials to .env
connector.azure.azure_storage_key=your_storage_account_key
Follow the Azure Documentation to retrieve your storage account keys.
Method 2: Connection String
Connection String provides an alternative authentication method for Azure Blob Storage.
Using the UI
- Click Add Data in your Rill project
- Select Azure Blob Storage as the data model type
- In the authentication step:
- Choose Connection String
- Enter your Connection String
- Name your connector (e.g.,
my_azure_conn)
- In the data model configuration step:
- Enter your container name and object path
- Configure other model settings as needed
- Click Create to finalize
Manual Configuration
Step 1: Create connector configuration
Create connectors/my_azure_conn.yaml:
type: connector
driver: azure
azure_storage_connection_string: "{{ .env.connector.azure.azure_storage_connection_string }}"
Step 2: Create model configuration
Create models/my_azure_data.yaml:
type: model
connector: duckdb
create_secrets_from_connectors: my_azure_conn
sql: SELECT * FROM read_parquet('azure://rilltest.blob.core.windows.net/my-container/path/to/data/*.parquet')
refresh:
cron: "0 */6 * * *"
Step 3: Add credentials to .env
connector.azure.azure_storage_connection_string=your_connection_string
Follow the Azure Documentation to retrieve your connection string.
Method 3: Shared Access Signature (SAS) Token
SAS tokens provide fine-grained access control with specific permissions and expiration times for secure access to your storage resources.
Using the UI
- Click Add Data in your Rill project
- Select Azure Blob Storage as the data model type
- In the authentication step:
- Choose SAS Token
- Enter your Storage Account name
- Enter your SAS Token
- Name your connector (e.g.,
my_azure_sas)
- In the data model configuration step:
- Enter your container name and object path
- Configure other model settings as needed
- Click Create to finalize
Manual Configuration
Step 1: Create connector configuration
Create connectors/my_azure_sas.yaml:
type: connector
driver: azure
azure_storage_account: rilltest
azure_storage_sas_token: "{{ .env.connector.azure.azure_storage_sas_token }}"
Step 2: Create model configuration
Create models/my_azure_data.yaml:
type: model
connector: duckdb
create_secrets_from_connectors: my_azure_sas
sql: SELECT * FROM read_parquet('azure://rilltest.blob.core.windows.net/my-container/path/to/data/*.parquet')
refresh:
cron: "0 */6 * * *"
Step 3: Add credentials to .env
connector.azure.azure_storage_sas_token=your_sas_token
Follow the Azure Documentation to create your Azure SAS token.
Method 4: Public Containers
For publicly accessible Azure Blob Storage containers, you don't need to create a connector. Simply use the Azure URI directly in your model configuration.
Using the UI
- Click Add Data in your Rill project
- Select Azure Blob Storage as the data model type
- In the authentication step:
- Choose Public
- The UI will skip connector creation and proceed directly to data model configuration
- In the data model configuration step:
- Enter your container name and object path
- Configure other model settings as needed
- Click Create to finalize
The UI will only create the model file (no connector file is created).
Manual Configuration
For public containers, you only need to create a model file. No connector configuration is required.
Create models/my_azure_data.yaml:
type: model
connector: duckdb
sql: SELECT * FROM read_parquet('azure://publicaccount.blob.core.windows.net/my-public-container/path/to/data/*.parquet')
refresh:
cron: "0 */6 * * *"
Method 5: Azure CLI Authentication (Local Development Only)
For local development, you can use credentials from the Azure CLI. This method is not suitable for production or Rill Cloud deployments. This method is only available through manual configuration, and you don't need to create a connector file.
Setup
- Install the Azure CLI if not already installed
- Authenticate with your Azure account:
az login - Verify your authentication status:
az account show - Create your model file (no connector needed)
Model Configuration
Create models/my_azure_data.yaml:
type: model
connector: duckdb
sql: SELECT * FROM read_parquet('azure://rilltest.blob.core.windows.net/my-container/path/to/data/*.parquet')
refresh:
cron: "0 */6 * * *"
Rill will automatically detect and use your local Azure CLI credentials when no connector is specified.
This method only works for local development. Deploying to Rill Cloud with this configuration will fail because the cloud environment doesn't have access to your local credentials. Always use Storage Account Key, Connection String, or SAS tokens for production deployments.
Using Azure Blob Storage Data in Models
Once your connector is configured (or for public containers, no connector needed), you can reference Azure Blob Storage paths in your model SQL queries using DuckDB's Azure functions.
Basic Example
With a connector (authenticated):
type: model
connector: duckdb
sql: SELECT * FROM read_parquet('azure://rilltest.blob.core.windows.net/my-container/data/*.parquet')
refresh:
cron: "0 */6 * * *"
Public container (no connector needed):
type: model
connector: duckdb
sql: SELECT * FROM read_parquet('azure://publicaccount.blob.core.windows.net/my-public-container/data/*.parquet')
refresh:
cron: "0 */6 * * *"
Path Patterns
You can use wildcards to read multiple files:
-- Single file
SELECT * FROM read_parquet('azure://account.blob.core.windows.net/container/data/file.parquet')
-- All files in a directory
SELECT * FROM read_parquet('azure://account.blob.core.windows.net/container/data/*.parquet')
-- All files in nested directories
SELECT * FROM read_parquet('azure://account.blob.core.windows.net/container/data/**/*.parquet')
-- Files matching a pattern
SELECT * FROM read_parquet('azure://account.blob.core.windows.net/container/data/2024-*.parquet')
Deploy to Rill Cloud
When deploying a project to Rill Cloud, Rill requires you to explicitly provide either an Azure Blob Storage connection string, Azure Storage Key, or Azure Storage SAS token for the containers used in your project. Please refer to our connector YAML reference docs for more information.
If you subsequently add sources that require new credentials (or if you simply entered the wrong credentials during the initial deploy), you can update the credentials by pushing the Deploy button to update your project or by running the following command in the CLI:
rill env push