Skip to main content

Connector YAML

When you add olap_connector to your rill.yaml file, you will need to set up a <connector_name>.yaml file in the 'connectors' directory. This file requires the following parameters,type and driver (see below for more parameter options). Rill will automatically test the connectivity to the OLAP engine upon saving the file. This can be viewed in the connectors tab in the UI.

Did you know?

Starting from Rill 0.46, you can directly create OLAP engines from the UI! Select + Add -> Data -> Connect an OLAP engine

Properties​

type​

[string] - Refers to the resource type and must be connector (required)

Common Properties​

name​

[string] - Name is usually inferred from the filename, but can be specified manually.

refs​

[array of string] - List of resource references

dev​

[object] - Overrides any properties in development environment.

prod​

[object] - Overrides any properties in production environment.

One of Properties Options​

athena​

Configuration properties specific to the athena

driver​

[string] - Refers to the driver type and must be driver athena (required)

aws_access_key_id​

[string] - AWS Access Key ID used for authentication. Required when using static credentials directly or as base credentials for assuming a role.

aws_secret_access_key​

[string] - AWS Secret Access Key paired with the Access Key ID. Required when using static credentials directly or as base credentials for assuming a role.

aws_access_token​

[string] - AWS session token used with temporary credentials. Required only if the Access Key and Secret Key are part of a temporary session credentials.

role_arn​

[string] - ARN of the IAM role to assume. When specified, the SDK uses the base credentials to call STS AssumeRole and obtain temporary credentials scoped to this role.

role_session_name​

[string] - Session name to associate with the STS AssumeRole session. Used only if 'role_arn' is specified. Useful for identifying and auditing the session.

external_id​

[string] - External ID required by some roles when assuming them, typically for cross-account access. Used only if 'role_arn' is specified and the role's trust policy requires it.

workgroup​

[string] - Athena workgroup to use for query execution. Defaults to 'primary' if not specified.

output_location​

[string] - S3 URI where Athena query results should be stored (e.g., s3://your-bucket/athena/results/). Optional if the selected workgroup has a default result configuration.

aws_region​

[string] - AWS region where Athena and the result S3 bucket are located (e.g., us-east-1). Defaults to 'us-east-1' if not specified.

allow_host_access​

[boolean] - Allow the Athena client to access host environment configurations such as environment variables or local AWS credential files. Defaults to true, enabling use of credentials and settings from the host environment unless explicitly disabled.

azure​

Configuration properties specific to the azure

driver​

[string] - Refers to the driver type and must be driver azure (required)

azure_storage_account​

[string] - Azure storage account name

azure_storage_key​

[string] - Azure storage access key

azure_storage_sas_token​

[string] - Optional azure SAS token for authentication

azure_storage_connection_string​

[string] - Optional azure connection string for storage account

azure_storage_bucket​

[string] - Name of the Azure Blob Storage container (equivalent to an S3 bucket) (required)

allow_host_access​

[boolean] - Allow access to host environment configuration

bigquery​

Configuration properties specific to the bigquery

driver​

[string] - Refers to the driver type and must be driver bigquery (required)

google_application_credentials​

[string] - Raw contents of the Google Cloud service account key (in JSON format) used for authentication.

project_id​

[string] - ID of the Google Cloud project to use for BigQuery operations. This can be omitted only if the project ID is included in the service account key.

allow_host_access​

[boolean] - Enable the BigQuery client to use credentials from the host environment when no service account JSON is provided. This includes Application Default Credentials from environment variables, local credential files, or the Google Compute Engine metadata server. Defaults to true, allowing seamless authentication in GCP environments.

clickhouse​

Configuration properties specific to the clickhouse

driver​

[string] - Refers to the driver type and must be driver clickhouse (required)

managed​

[boolean] - true means Rill will provision the connector using the default provisioner. false disables automatic provisioning.

mode​

[string] - read - Controls the operation mode for the ClickHouse connection. Defaults to 'read' for safe operation with external databases. Set to 'readwrite' to enable model creation and table mutations. Note: When 'managed: true', this is automatically set to 'readwrite'.

dsn​

[string] - DSN(Data Source Name) for the ClickHouse connection

username​

[string] - Username for authentication

password​

[string] - Password for authentication

host​

[string] - Host where the ClickHouse instance is running

port​

[integer] - Port where the ClickHouse instance is accessible

database​

[string] - Name of the ClickHouse database within the cluster

ssl​

[boolean] - Indicates whether a secured SSL connection is required

cluster​

[string] - Cluster name, required for running distributed queries

log_queries​

[boolean] - Controls whether to log raw SQL queries

settings_override​

[string] - override the default settings used in queries. example readonly = 1, session_timezone = 'UTC'

embed_port​

[integer] - Port to run ClickHouse locally (0 for random port)

can_scale_to_zero​

[boolean] - Indicates if the database can scale to zero

max_open_conns​

[integer] - Maximum number of open connections to the database

max_idle_conns​

[integer] - Maximum number of idle connections in the pool

dial_timeout​

[string] - Timeout for dialing the ClickHouse server

conn_max_lifetime​

[string] - Maximum time a connection may be reused

read_timeout​

[string] - Maximum time for a connection to read data

druid​

Configuration properties specific to the druid

driver​

[string] - Refers to the driver type and must be driver druid (required)

dsn​

[string] - Data Source Name (DSN) for connecting to Druid (required)

username​

[string] - Username for authenticating with Druid

password​

[string] - Password for authenticating with Druid

host​

[string] - Hostname of the Druid coordinator or broker

port​

[integer] - Port number of the Druid service

ssl​

[boolean] - Enable SSL for secure connection

log_queries​

[boolean] - Log raw SQL queries sent to Druid

max_open_conns​

[integer] - Maximum number of open database connections (0 = default, -1 = unlimited)

skip_version_check​

[boolean] - Skip checking Druid version compatibility

duckdb​

Configuration properties specific to the duckdb

driver​

[string] - Refers to the driver type and must be driver duckdb (required)

pool_size​

[integer] - Number of concurrent connections and queries allowed

allow_host_access​

[boolean] - Whether access to the local environment and file system is allowed

cpu​

[integer] - Number of CPU cores available to the database

memory_limit_gb​

[integer] - Amount of memory in GB available to the database

read_write_ratio​

[number] - Ratio of resources allocated to the read database; used to divide CPU and memory

init_sql​

[string] - is executed during database initialization.

conn_init_sql​

[string] - is executed when a new connection is initialized.

secrets​

[string] - Comma-separated list of other connector names to create temporary secrets for in DuckDB before executing a model.

log_queries​

[boolean] - Whether to log raw SQL queries executed through OLAP

gcs​

Configuration properties specific to the gcs

driver​

[string] - Refers to the driver type and must be driver gcs (required)

google_application_credentials​

[string] - Google Cloud credentials JSON string

bucket​

[string] - Name of gcs bucket (required)

allow_host_access​

[boolean] - Allow access to host environment configuration

key_id​

[string] - Optional S3-compatible Key ID when used in compatibility mode

secret​

[string] - Optional S3-compatible Secret when used in compatibility mode

https​

Configuration properties specific to the https

driver​

[string] - Refers to the driver type and must be driver https (required)

path​

[string] - The full HTTPS URI to fetch data from (required)

headers​

[object] - HTTP headers to include in the request

local_file​

Configuration properties specific to the local_file

driver​

[string] - Refers to the driver type and must be driver local_file (required)

dsn​

[string] - Data Source Name (DSN) indicating the file path or location of the local file (required)

allow_host_access​

[boolean] - Flag to indicate if access to host-level file paths is permitted

motherduck​

Configuration properties specific to the motherduck

driver​

[string] - Refers to the driver type and must be driver motherduck (required)

dsn​

[string] - Data Source Name (DSN) specifying the MotherDuck connection endpoint (required)

token​

[string] - Authentication token for accessing MotherDuck (secret) (required)

mysql​

Configuration properties specific to the mysql

driver​

[string] - Refers to the driver type and must be driver mysql (required)

dsn​

[string] - DSN(Data Source Name) for the mysql connection (required)

pinot​

Configuration properties specific to the pinot

driver​

[string] - Refers to the driver type and must be driver pinot (required)

dsn​

[string] - DSN(Data Source Name) for the Pinot connection (required)

username​

[string] - Username for authenticating with Pinot

password​

[string] - Password for authenticating with Pinot

broker_host​

[string] - Hostname of the Pinot broker (required)

broker_port​

[integer] - Port number for the Pinot broker

controller_host​

[string] - Hostname of the Pinot controller (required)

controller_port​

[integer] - Port number for the Pinot controller

ssl​

[boolean] - Enable SSL connection to Pinot

log_queries​

[boolean] - Log raw SQL queries executed through Pinot

max_open_conns​

[integer] - Maximum number of open connections to the Pinot database

postgres​

Configuration properties specific to the postgres

driver​

[string] - Refers to the driver type and must be driver postgres (required)

dsn​

[string] - DSN(Data Source Name) for the postgres connection (required)

redshift​

Configuration properties specific to the redshift

driver​

[string] - Refers to the driver type and must be driver redshift (required)

aws_access_key_id​

[string] - AWS Access Key ID used for authenticating with Redshift. (required)

aws_secret_access_key​

[string] - AWS Secret Access Key used for authenticating with Redshift. (required)

aws_access_token​

[string] - AWS Session Token for temporary credentials (optional).

region​

[string] - AWS region where the Redshift cluster or workgroup is hosted (e.g., 'us-east-1').

database​

[string] - Name of the Redshift database to query. (required)

workgroup​

[string] - Workgroup name for Redshift Serverless, in case of provisioned Redshift clusters use 'cluster_identifier'.

cluster_identifier​

[string] - Cluster identifier for provisioned Redshift clusters, in case of Redshift Serverless use 'workgroup' .

s3​

Configuration properties specific to the s3

driver​

[string] - Refers to the driver type and must be driver s3 (required)

aws_access_key_id​

[string] - AWS Access Key ID used for authentication

aws_secret_access_key​

[string] - AWS Secret Access Key used for authentication

aws_access_token​

[string] - Optional AWS session token for temporary credentials

bucket​

[string] - Name of s3 bucket (required)

endpoint​

[string] - Optional custom endpoint URL for S3-compatible storage

region​

[string] - AWS region of the S3 bucket

allow_host_access​

[boolean] - Allow access to host environment configuration

retain_files​

[boolean] - Whether to retain intermediate files after processing

salesforce​

Configuration properties specific to the salesforce

driver​

[string] - Refers to the driver type and must be driver salesforce (required)

username​

[string] - Salesforce account username (required)

password​

[string] - Salesforce account password (secret)

key​

[string] - Authentication key for Salesforce (secret)

endpoint​

[string] - Salesforce API endpoint URL (required)

client_id​

[string] - Client ID used for Salesforce OAuth authentication

slack​

Configuration properties specific to the slack

driver​

[string] - Refers to the driver type and must be driver slack (required)

bot_token​

[string] - Bot token used for authenticating Slack API requests (required)

snowflake​

Configuration properties specific to the snowflake

driver​

[string] - Refers to the driver type and must be driver snowflake (required)

dsn​

[string] - DSN (Data Source Name) for the Snowflake connection (required)

parallel_fetch_limit​

[integer] - Maximum number of concurrent fetches during query execution

sqlite​

Configuration properties specific to the sqlite

driver​

[string] - Refers to the driver type and must be driver sqlite (required)

dsn​

[string] - DSN(Data Source Name) for the sqlite connection (required)