## Contact --- #### Contact us --- sidebar_class_name: hidden title: Contact us hide_table_of_contents: true --- Rill has a globally distributed team and we'd love to hear from you—whether you're a customer, prospect, or simply a Rill enthusiast. There are several ways to get in touch with the Rill team and engage with the community, which are described below. ## GitHub Rill is part of the open source community, and you can find the source code of our Rill Developer project on GitHub [here](https://github.com/rilldata/rill). We welcome contributions, but if you simply want to report a bug, file an issue, or request a feature, you can do so on [this page](https://github.com/rilldata/rill/issues). ## Discord Curious to see if someone else has encountered an issue before, looking for help getting started, interested in tapping into the collective knowledge of others, or simply want to engage with other members of the Rill community? Rill's community can be found on Discord, and we'd love for you to be a part of it! Please click [here](https://discord.gg/2ubRfjC7Rh) to access our dedicated Discord channel. ## Contacting Support If you'd like to get in contact with the Rill team directly and/or contact Rill Support, there are a few different options available. ### Slack channel If you're an active prospect or customer, you will have access to a dedicated Slack channel with the Rill team. This ensures we can provide you with individualized responses to your questions or issues, with the fastest response times from our distributed team. :::info Interested in evaluating Rill Cloud? Please let us know and we'd be happy to set up a Slack channel for your team! ::: ### Email For individuals who prefer a more traditional medium, you can open a support ticket by reaching out to support@rilldata.com. ### In-app chat For logged-in users, Rill provides the ability to ask a question or report an issue to the Rill Support team directly from within the application via an embedded in-app chat widget. Select your user icon in the top right corner and choose "Contact Rill support"—a chat widget will appear on your screen! ![Rill Developer Chat](/img/contact/rill-developer-chat.png) --- ## Developers --- ### Build --- #### Connectors --- ##### Data Source --- ####### Data Sources --- title: Data Sources sidebar_label: Data Sources sidebar_position: 0 toc_max_heading_level: 3 className: connect-connect --- By default, Rill will use a managed embedded analytics engine (**DuckDB** or **ClickHouse**) to support data ingestion. Whether you're working with cloud data warehouses, databases, file storage, or streaming data sources, Rill provides seamless connectivity and data ingestion capabilities. Once you have connected to your data source you can create [downstream models](/developers/build/models), [metrics views](/developers/build/metrics-view) and [visualize your data](/developers/build/dashboards). :::tip using clickhouse? Don't forget to [create a managed ClickHouse server](/developers/build/connectors/olap/clickhouse#rill-managed-clickhouse) before getting started! ```yaml type: connector driver: clickhouse managed: true ``` ::: import ConnectorIcon from '@site/src/components/ConnectorIcon'; In order to connect and browse through your data, you'll need to create a connector file. Browse through the options below for our supported connectors. Each connector is designed to handle the specific authentication and configuration requirements of your data source. :::warning OLAP Engine Limitations Rill supports connecting your data to both [DuckDB](/developers/build/connectors/olap/duckdb) and [ClickHouse](/developers/build/connectors/olap/clickhouse). However, there are still some features in development for managed ClickHouse. For more information see our [managed ClickHouse docs](/developers/build/connectors/olap/clickhouse#rill-managed-clickhouse). If you've still got questions, [contact our team](/contact) for more information and scheduled feature releases! ::: ## Data Warehouses ### Athena ### BigQuery ### Databricks ### Redshift ### Snowflake
} header="Athena" content="Connect to Amazon Athena for serverless querying of data stored in S3 using standard SQL." link="/developers/build/connectors/data-source/athena" linkLabel="Learn more" referenceLink="athena" /> } header="BigQuery" content="Connect to Google BigQuery for analytics and data warehousing with service account authentication." link="/developers/build/connectors/data-source/bigquery" linkLabel="Learn more" referenceLink="bigquery" /> } header="Databricks" content="Connect to Databricks SQL warehouses with personal access token authentication." link="/developers/build/connectors/data-source/databricks" linkLabel="Learn more" referenceLink="databricks" /> } header="Redshift" content="Connect to Amazon Redshift data warehouse with AWS credentials and support for both provisioned and serverless clusters." link="/developers/build/connectors/data-source/redshift" linkLabel="Learn more" referenceLink="redshift" /> } header="Snowflake" content="Connect to Snowflake data warehouse with support for individual credentials and JWT authentication." link="/developers/build/connectors/data-source/snowflake" linkLabel="Learn more" referenceLink="snowflake" />
## Databases ### MySQL ### PostgreSQL ### SQLite ### Supabase
} header="MySQL" content="Connect to MySQL databases with support for various authentication methods and SSL connections." link="/developers/build/connectors/data-source/mysql" linkLabel="Learn more" referenceLink="mysql" /> } header="PostgreSQL" content="Connect to PostgreSQL databases with support for SSL connections and various authentication methods." link="/developers/build/connectors/data-source/postgres" linkLabel="Learn more" referenceLink="postgresql" /> } header="SQLite" content="Connect to SQLite databases for lightweight, file-based data storage and querying." link="/developers/build/connectors/data-source/sqlite" linkLabel="Learn more" /> } header="Supabase" content="Connect to Supabase's managed PostgreSQL databases with SSL support and standard connection methods." link="/developers/build/connectors/data-source/supabase" linkLabel="Learn more" referenceLink="supabase" />
## Object Storage ### Amazon S3 ### Google Cloud Storage ### Microsoft Azure Blob Storage
} header="Amazon S3" content="Connect to Amazon S3 buckets to read data files including CSV, JSON, Parquet, and compressed formats." link="/developers/build/connectors/data-source/s3" linkLabel="Learn more" referenceLink="s3" /> } header="Google Cloud Storage" content="Google Cloud Storage for scalable object storage and data lakes." link="/developers/build/connectors/data-source/gcs" linkLabel="Learn more" referenceLink="gcs" /> } header="Azure" content="Connect to Microsoft Azure Blob Storage to read data files with support for various formats." link="/developers/build/connectors/data-source/azure" linkLabel="Learn more" referenceLink="azure" />
## Table Formats ### Delta Lake ### Apache Iceberg
} header="Delta Lake" content="Read Delta tables directly from object storage through compatible query engines." link="/developers/build/connectors/data-source/delta" linkLabel="Learn more" /> } header="Apache Iceberg" content="Read Iceberg tables directly from object storage through compatible query engines." link="/developers/build/connectors/data-source/iceberg" linkLabel="Learn more" />
## Other Data Connectors ### External DuckDB ### Google Sheets ### HTTPS ### Local File ### Salesforce
} header="DuckDB" content="Attach your local DuckDB database to Rill's embedded database." link="/developers/build/connectors/data-source/duckdb" linkLabel="Learn more" referenceLink="external-duckdb" /> } header="Google Sheets" content="Connect to Google Sheets to read data from spreadsheets." link="/developers/build/connectors/data-source/googlesheets" linkLabel="Learn more" /> https://

} header="HTTPS" content="Download data from HTTP/HTTPS URLs with support for various authentication methods." link="/developers/build/connectors/data-source/https" linkLabel="Learn more" referenceLink="https" /> } header="Local File" content="Read data from local files including CSV, JSON, Parquet, and compressed formats." link="/developers/build/connectors/data-source/local-file" linkLabel="Learn more" /> } header="Salesforce" content="Connect to Salesforce to extract data from objects and queries using the Salesforce API." link="/developers/build/connectors/data-source/salesforce" linkLabel="Learn more" />
## Externally Hosted Services If you have a firewall in front of your externally hosted service, you will need to whitelist the IP addresses below. This will allow you to connect to/from your service once your project is deployed to Rill Cloud. ``` 35.196.245.100 34.74.117.37 35.196.153.31 34.75.22.143 34.148.167.51 35.237.60.193 ``` ## Managed OLAP Engine Caveats When deciding on which managed OLAP engine to use with Rill, you'll need to decide based on the following factors: - **Size of data**: Consider the volume and growth rate of your datasets - **Familiarity with respective OLAP engine features**: Assess your team's expertise with each engine's capabilities - **Integration complexity**: Consider how well each engine integrates with your existing data infrastructure In the case of **sub 100GB of data**, we recommend keeping the default engine, DuckDB, in order to minimize the integration complexity. The reason for this is that [DuckDB has built-in functions](https://duckdb.org/docs/stable/data/data_sources) to support the connectors listed on this page. On the other hand, if you need to analyze **100s of GB of data**, we would recommend using Managed ClickHouse. This will add some complexity ([staging tables](/developers/build/models/staging-models)), but will in turn provide better dashboard performance. If data leans either way, a good deciding factor for which OLAP engine to use is your **familiarity with their SQL syntax**. Whether you're [creating models](/developers/build/models/models-101#intermediate-processing) or using [arithmetic functions](/developers/build/metrics-view) in the metrics view, you'll need to utilize the engine's built-in functions. :::note Supported Connectors If there's a connector that you're interested in or you're looking for the list of currently supported ClickHouse connectors, [contact us](/contact)! ::: --- ####### Delta Lake --- title: Delta Lake description: Read Delta Lake tables from object storage sidebar_label: Delta Lake sidebar_position: 12 --- ## Overview [Delta Lake](https://delta.io/) is an open-source storage framework that brings ACID transactions to data lakes. Rill supports reading Delta tables directly from object storage through compatible query engine integrations. Today, this is powered by DuckDB's [Delta extension](https://duckdb.org/docs/stable/core_extensions/delta). :::note Direct file access only Rill reads Delta tables by scanning the table's transaction log and data files directly from object storage. Catalog-based access (e.g., through Unity Catalog) is not currently supported. ::: ## Storage Backends Delta tables can be read from any of the following storage backends: | Backend | URI format | Authentication | |---|---|---| | Amazon S3 | `s3://bucket/path/to/table` | Requires an [S3 connector](/developers/build/connectors/data-source/s3) | | Azure Blob Storage | `azure://container/path/to/table` | Requires an [Azure connector](/developers/build/connectors/data-source/azure) | | Local filesystem | `/path/to/table` | No authentication needed | :::info GCS not yet supported Google Cloud Storage is not currently supported for Delta tables. GCS support depends on the upstream DuckDB Delta extension adding it. ::: For cloud storage backends, you must first configure the corresponding storage connector with valid credentials. Rill uses these credentials to authenticate when reading the Delta table files. ## Using the UI 1. Click **Add Data** in your Rill project 2. Select **Delta Lake** as the data source type 3. Choose your storage backend (S3, Azure, or Local) 4. Enter the path to your Delta table directory 5. Enter a model name and click **Create** For cloud storage backends, the UI will prompt you to set up the corresponding storage connector if one doesn't already exist. ## Manual Configuration Create a model that uses DuckDB's `delta_scan()` function to read the table. ### Reading from S3 Create `models/delta_data.yaml`: ```yaml type: model connector: duckdb create_secrets_from_connectors: s3 materialize: true sql: | SELECT * FROM delta_scan('s3://my-bucket/path/to/delta_table') ``` ### Reading from Azure ```yaml type: model connector: duckdb create_secrets_from_connectors: azure materialize: true sql: | SELECT * FROM delta_scan('azure://my-container/path/to/delta_table') ``` ### Reading from local filesystem ```yaml type: model connector: duckdb materialize: true sql: | SELECT * FROM delta_scan('/path/to/delta_table') ``` ## Deploy to Rill Cloud Since Delta tables are read through DuckDB using your existing storage connector credentials, deploying to Rill Cloud follows the same process as the underlying storage connector: - **S3**: Follow the [S3 deployment guide](/developers/build/connectors/data-source/s3#deploy-to-rill-cloud) - **Azure**: Follow the [Azure deployment guide](/developers/build/connectors/data-source/azure#deploy-to-rill-cloud) Ensure your storage connector credentials are configured in your Rill Cloud project before deploying. ## Limitations - **Direct file access only**: Rill reads Delta transaction logs and data files directly from storage. Catalog integrations (e.g., Unity Catalog) are not supported. - **DuckDB engine**: Delta support is currently provided through DuckDB's Delta extension. Additional engine support is planned. - **No GCS support**: Google Cloud Storage is not yet supported by the Delta extension. - **Read-only**: Rill reads from Delta tables but does not write to them. - **Experimental**: The DuckDB Delta extension is currently marked as experimental. --- ####### External DuckDB --- title: External DuckDB description: Connect to external DuckDB databases and ingest data into Rill sidebar_label: External DuckDB sidebar_position: 11 --- While not recommended for production use, Rill allows you to `attach` external DuckDB databases to ingest data from them as a data source. This approach has several caveats and limitations during deployment and is primarily intended for local testing scenarios. :::warning Local Development Only There are several limitations with deployment to Rill Cloud, so we do not recommend this method for production environments. Key limitations include: - Moving your DuckDB file into the `data/` folder within your project directory - A size limitation of 100MB when deploying to Rill Cloud [Contact us](/contact) if you have questions or encounter issues with these limitations. ::: ## Attaching an External DuckDB In the default `connectors/duckdb.yaml` file, you can use the `init_sql` parameter to execute SQL statements during database initialization, such as attaching an external database to Rill's embedded DuckDB. For more details on the YAML configuration, see the [DuckDB reference page](/reference/project-files/connectors#duckdb). ```yaml type: connector driver: duckdb managed: true init_sql: ATTACH '/path/to/your/duckdb.db' AS external_duckdb; INSTALL httpfs; LOAD httpfs; ``` ## Importing Data to Your External DuckDB After establishing a connection, you can import data through the connector UI. This process will write data from your attached database to [Rill's embedded DuckDB.](/developers/build/connectors/olap/duckdb#rill-managed-duckdb) ```yaml # Model YAML # Reference documentation: https://docs.rilldata.com/reference/project-files/models type: model materialize: true connector: duckdb sql: SELECT * from external_duckdb.local_table ``` --- ####### Google Sheets --- title: Google Sheets description: Connect to data in Google Sheets sidebar_label: Google Sheets sidebar_position: 20 --- ### Google Sheets Rill can read from any HTTP(S) URL endpoint that produces a valid data file in a supported format. For example, to bring in data from [Google Sheets](https://www.google.com/sheets/about/) as a CSV file directly into Rill as a source ([leveraging the direct download link syntax](https://www.highviewapps.com/blog/how-to-create-a-csv-or-excel-direct-download-link-in-google-sheets/)), you can create a `source_name.yaml` file in the `sources` directory of your Rill project with the following content. :::note Prerequisite The Google Sheet must be a public sheet with `Anyone with the link` enabled. ::: ```yaml type: model connector: "duckdb" sql: "select * from read_csv_auto('https://docs.google.com/spreadsheets/d//export?format=csv&gid=', normalize_names=True)" ``` :::note Updating the URL Make sure to replace `SPREADSHEET_ID` and `SHEET_ID` with the ID of your spreadsheet and tab, respectively (which you can obtain from the URL when Google Sheets is open). ::: ![Googlesheets](/img/build/connectors/data-sources/googlesheets.png) :::tip `gsheets` DuckDB Community Extension In cases where setting the Google Sheet to `Anyone with the link` is not allowed, DuckDB has an extension that allows you to share the sheet to a Google service account. However, we have not yet implemented the DuckDB community extension in our deployment of DuckDB. Please contact us if you are interested in using this feature. ::: --- ####### Apache Iceberg --- title: Apache Iceberg description: Read Iceberg tables from object storage sidebar_label: Apache Iceberg sidebar_position: 27 --- ## Overview [Apache Iceberg](https://iceberg.apache.org/) is an open table format for large analytic datasets. Rill supports reading Iceberg tables directly from object storage through compatible query engine integrations. Today, this is powered by DuckDB's native [Iceberg extension](https://duckdb.org/docs/extensions/iceberg/overview.html). :::note Direct file access only Rill reads Iceberg tables by scanning the table's metadata and data files directly from object storage. Catalog-based access (e.g., through a Hive Metastore, AWS Glue, or REST catalog) is not currently supported. ::: ## Storage Backends Iceberg tables can be read from any of the following storage backends: | Backend | URI format | Authentication | |---|---|---| | Amazon S3 | `s3://bucket/path/to/table` | Requires an [S3 connector](/developers/build/connectors/data-source/s3) | | Google Cloud Storage | `gs://bucket/path/to/table` | Requires a [GCS connector](/developers/build/connectors/data-source/gcs) with HMAC keys | | Azure Blob Storage | `azure://container/path/to/table` | Requires an [Azure connector](/developers/build/connectors/data-source/azure) | | Local filesystem | `/path/to/table` | No authentication needed | For cloud storage backends, you must first configure the corresponding storage connector with valid credentials. Rill uses these credentials to authenticate when reading the Iceberg table files. ## Using the UI 1. Click **Add Data** in your Rill project 2. Select **Apache Iceberg** as the data source type 3. Choose your storage backend (S3, GCS, Azure, or Local) 4. Enter the path to your Iceberg table directory 5. Optionally configure advanced parameters (allow moved paths, snapshot version) 6. Enter a model name and click **Create** For cloud storage backends, the UI will prompt you to set up the corresponding storage connector if one doesn't already exist. ## Manual Configuration Create a model that uses DuckDB's `iceberg_scan()` function to read the table. ### Reading from S3 Create `models/iceberg_data.yaml`: ```yaml type: model connector: duckdb create_secrets_from_connectors: s3 materialize: true sql: | SELECT * FROM iceberg_scan('s3://my-bucket/path/to/iceberg_table') ``` ### Reading from GCS :::info HMAC keys required DuckDB's `iceberg_scan()` authenticates to GCS using HMAC keys, not JSON service account credentials. When configuring your [GCS connector](/developers/build/connectors/data-source/gcs), use the `key_id` and `secret` (HMAC) properties instead of `google_application_credentials`. ::: ```yaml type: model connector: duckdb create_secrets_from_connectors: gcs materialize: true sql: | SELECT * FROM iceberg_scan('gs://my-bucket/path/to/iceberg_table') ``` ### Reading from Azure ```yaml type: model connector: duckdb create_secrets_from_connectors: azure materialize: true sql: | SELECT * FROM iceberg_scan('azure://my-container/path/to/iceberg_table') ``` ### Reading from local filesystem ```yaml type: model connector: duckdb materialize: true sql: | SELECT * FROM iceberg_scan('/path/to/iceberg_table') ``` ## Optional Parameters The `iceberg_scan()` function accepts additional parameters: | Parameter | Type | Description | |---|---|---| | `allow_moved_paths` | boolean | Allow reading tables where data files have been moved from their original location. Defaults to `true` in the UI. | | `version` | string | Read a specific Iceberg snapshot version instead of the latest. | Example with optional parameters: ```sql SELECT * FROM iceberg_scan('s3://my-bucket/path/to/iceberg_table', allow_moved_paths = true, version = '2') ``` ## Deploy to Rill Cloud Since Iceberg tables are read through DuckDB using your existing storage connector credentials, deploying to Rill Cloud follows the same process as the underlying storage connector: - **S3**: Follow the [S3 deployment guide](/developers/build/connectors/data-source/s3#deploy-to-rill-cloud) - **GCS**: Follow the [GCS deployment guide](/developers/build/connectors/data-source/gcs#deploy-to-rill-cloud) - **Azure**: Follow the [Azure deployment guide](/developers/build/connectors/data-source/azure#deploy-to-rill-cloud) Ensure your storage connector credentials are configured in your Rill Cloud project before deploying. ## Limitations - **Direct file access only**: Rill reads Iceberg metadata and data files directly from storage. Catalog integrations (Hive Metastore, AWS Glue, REST catalog) are not supported. - **DuckDB engine**: Iceberg support is currently provided through DuckDB's Iceberg extension. Additional engine support (e.g., ClickHouse) is planned. - **GCS requires HMAC keys**: DuckDB's `iceberg_scan()` only supports HMAC authentication for GCS, not JSON service account credentials. - **Read-only**: Rill reads from Iceberg tables but does not write to them. --- ####### Kafka --- title: Kafka description: Connect Kafka to OLAP Engines sidebar_label: Kafka sidebar_position: 30 --- :::warning For Customers with Druid/ClickHouse Engines Only Note: The setup instructions below are for customers using Rill's hosted OLAP solution. ::: ## Setup Instructions Follow the instructions below to grant Rill access to your Apache Kafka Cluster and the data on a given topic within the cluster. Providing access to a cloud-provided service, such as Confluent Cloud, is easier because all of the connection and security is already taken care of for you. If you are using a self-managed cluster, ensure security and encryption are configured accordingly. The following scenarios are covered. Apache Kafka is designed to work within almost any enterprise environment, allowing for a variety of unique configurations. Work with your engineering team and Rill Data to ensure your configuration and setup are performant and secure. * Confluent Cloud * Private Kafka Cluster * VPC Kafka Peering * Public Kafka Cluster * AWS Kinesis ### Confluent Cloud Confluent Cloud is inherently secure and accessible. Communication with Confluent Cloud can be set up in a few minutes. The main consideration is to determine the level of accessibility for the credentials you will be using from the Rill platform to access the Confluent Cloud Cluster. The concepts here apply to other Apache Kafka SaaS offerings, even though the means to access them might be different. Please reach out if you have any questions about integrating with your Apache Kafka SaaS provider. ### Credentials The first step is to create a security key you can use from Rill Data to access your Kafka cluster. Select a granular access key to reduce exposure. You can use an existing key or an actual account when creating the client API access. #### Create Key and Secret Select "+ Add Key" from the "Cloud API Keys" menu option and create a non-admin account. Create a service account unique to accessing data from Rill Data and be sure to download and secure the client key and secret for use from Rill Data Druid Ingestion. It is recommended to create a unique service account so your access control can be uniquely established for access from Rill Data into your Confluent Cloud Kafka Cluster. If you fail to download or lose the key/secret, a new key/secret will need to be generated. #### Create Cluster API Access Rill Data manages the Kafka topic offset internally, so the Apache Kafka connection only needs topic read access. The best way to provide Rill access to your cluster is through granular access with the above service account. Ideally, using a topic-name prefix is preferred, as it minimizes the number of ACL rules you need to create and manage for the API access key/secret. #### Data Ingestion The key aspects for Confluent Cloud ingestion are establishing the bootstrap server, customer properties for access, and the topic to consume from. - Bootstrap servers: *Cluster's Broker and Port* - Topic: *The Kafka Topic* - Consumer properties: *At minimum, the properties needed to access the cluster with key/secret* - Where should the data be sampled from? *Start of stream* or *End of stream* *See **ioConfig** example shown from a complete ingestion specification for pulling data from Kafka. Insert the **key** and **secret** into the JAAS config.* ```json "ioConfig": { "type": "kafka", "consumerProperties": { "bootstrap.servers": "{{ CLUSTER_HOSTNAME }}:9092", "security.protocol": "SASL_SSL", "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username='{{ CLIENT_KEY }}' password='{{ CLIENT_SECRET }}';", "sasl.mechanism": "PLAIN" }, "topic": "rilldata-sourcedata", "inputFormat": { "type": "json" }, "useEarliestOffset": true } ``` ### Private Kafka Cluster When connecting to a private Apache Kafka Cluster, accessibility and security are the most significant configuration areas. Rill Data connects directly to the Kafka Cluster as a Kafka client and will access your cluster as any other consumer client. * For Apache Kafka to be highly performant, the client API communicates directly to the active Kafka broker for a given partition. * Configure your cluster so the brokers are each individually accessible from the Rill services. * Apache Kafka has the server property **advertised.listeners** to ensure that the client has the correct information to communicate with an individual broker. When configuring your Kafka Cluster, ensure it can be accessed from Rill by establishing a VPC. An example setup is shown via AWS Private Link below. AWS Private Link allows exposure of the Kafka brokers over a Network Load Balancer using VPC Endpoints. The network packets always remain within the AWS Network. #### AWS Private Link Kafka **EXTERNAL** advertised listeners should match the DNS name configured. Network Load Balancer will expose all the brokers with different ports. Each broker can have an incremented port for **EXTERNAL** access, e.g., 9092, 9093, and 9094. ```shell listener.security.protocol.map=EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT advertised.listeners=PLAINTEXT://:9092,EXTERNAL://${PRIVATE_DNS_NAME}:${INCREMENTED_PORT} ## Eg: # Broker #1 # advertised.listeners=PLAINTEXT://:9092,EXTERNAL://private-kafka.rilldata.com:19092 # Broker #2 # advertised.listeners=PLAINTEXT://:9092,EXTERNAL://private-kafka.rilldata.com:19093 # Broker #3 # advertised.listeners=PLAINTEXT://:9092,EXTERNAL://private-kafka.rilldata.com:19094 ``` #### AWS Private Link using CloudFormation 1. Open AWS CloudFormation to create a new Stack. 2. Use the Amazon S3 URL: https://s3.amazonaws.com/cf-templates.rilldata.com/rilldata-private-link.yaml 3. Specify stack details: * **Stack Name**: `rilldata-privatelink` * **AccountId**: Rill Data AWS Account ID. * **NlbArn**: ARN of the Network Load Balancer (Internal) through which we can share the internal Endpoints 4. Click Next, then Next again, acknowledge the capabilities, and create the stack. 5. You can check the events and it should create the resources for you. 6. Share the outputs with Rill Data. We would be using the following CloudFormation Template. ```yaml title="YAML" AWSTemplateFormatVersion: 2010-09-09 Metadata: License: Apache-2.0 Description: 'AWS CloudFormation Template for creating a Private Link for a given Network Load Balancer' Parameters: NlbArn: Type: String Description: ARN of the Network Load Balancer Default: arn:aws:elasticloadbalancing:us-east-1:248432388601:loadbalancer/net/kafka-broker/de46ce872b289b14 AccountId: Type: String Description: ID of the account to share the private link with. Default: 417306524257 Resources: EndpointService: Type: AWS::EC2::VPCEndpointService Properties: AcceptanceRequired: True NetworkLoadBalancerArns: - !Ref NlbArn EndpointServicePermissions: Type: AWS::EC2::VPCEndpointServicePermissions Properties: AllowedPrincipals: - !Join - '' - - 'arn:aws:iam::' - !Ref AccountId - ':root' ServiceId: !Ref EndpointService Outputs: PrivateLinkServiceId: Value: !Ref EndpointService Description: Service ID of the Private Link ``` ### VPC Peering * Since the cluster is private, setting up SSL may not be needed -- as it would depend on your corporate policies. * Authentication: create a user to be used for consuming * Authorization: allow the user to consume from a specific topic * Do VPC network peering * Make sure every broker’s advertised endpoint is accessible from a consumer. * Set up the consumer by storing the user’s credentials into the consumer’s configuration file. ### Public Kafka Cluster By default, there is no encryption, authentication, or ACLs (access control list) configured. Any client can communicate to Kafka brokers via the PLAINTEXT port. :::danger Kafka Security It is critical that access via this port is restricted to trusted clients only. Network segmentation and/or authorization ACLs can be used to restrict access to trusted IPs in such cases. ::: If neither is used, the cluster is wide open and can be accessed by anyone. * **Enable SSL encryption**: SSL uses private-key/certificate pairs which are used during the SSL handshake process. * **Authentication & Authorization**: Create a user to be used for consuming and allow the user to consume from a specific topic. * Make sure every broker’s advertised endpoint is accessible from a consumer. * Set up the consumer by adding the broker’s certificate into a trust store and storing the user’s credentials into the consumer’s configuration file. ### AWS Kinesis We can provide access to the Kinesis stream through an IAM Role which will be assumed by the Rill Data AWS Account to gain access. :::info Rill Data AWS Account arn:aws:iam::248432388601:root ::: #### Using CloudFormation Console 1. Open AWS CloudFormation to create a new Stack. 2. Use the Amazon S3 URL: `https://s3.amazonaws.com/cf-templates.rilldata.com/rilldata-kinesis-access.yaml` 3. Specify Stack Details: * **Stack Name**: `rilldata-kinesis-access` * **KinesisARN**: Name of the bucket we want to provide access to. 4. Click Next, then Next again, acknowledge the capabilities, and create the stack. 5. You can check the events and it should create the resources for you. 6. Share the outputs with Rill Data. ##### CloudFormation Template Reference We would be using the following CloudFormation Template. ```yaml AWSTemplateFormatVersion: '2010-09-09' Metadata: License: Apache-2.0 Description: 'AWS CloudFormation Template for providing Rill Data Access to Kinesis. It creates a Role that can be assumed by the Rill Data AWS Account. The Role has an IAM policy associated with it.' Parameters: KinesisARN: Type: String Description: Kinesis Stream ARN NamePrefix: Type: String Description: Name prefix for the IAM Policy and IAM Role. Default: rilldata Resources: KinesisRole: Type: AWS::IAM::Role Properties: Description: 'Rill Data Access to the Kinesis. Managed by: CloudFormation' AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: AWS: - 'arn:aws:iam::248432388601:root' Action: - 'sts:AssumeRole' Policies: - PolicyName: !Join - '' - - !Ref NamePrefix - 'KinesisAccessPolicy' PolicyDocument: Statement: - Effect: Allow Action: - "kinesis:Describe*" - "kinesis:List*" - "kinesis:Get*" Resource: - !Ref KinesisARN RoleName: !Join - '-' - - !Ref NamePrefix - kinesis-access Tags: - Key: Accessor Value: RillData - Key: ManagedBy Value: CloudFormation Outputs: RoleName: Value: !GetAtt [KinesisRole, Arn] Description: Kinesis Access Role Arn, to be shared with Rill Data ``` ## References ### SSL Encryption SSL uses private-key/certificate pairs which are used during the SSL handshake process. * Each broker needs its own private-key/certificate pair, and the client uses the certificate to authenticate the broker. * Each logical client needs a private-key/certificate pair if client authentication is enabled, and the broker uses the certificate to authenticate the client. ```shell # Configure the password, truststore, and keystore # Since this stores passwords directly in the broker configuration file, # it is important to restrict access to these files via file system permissions. ssl.truststore.location=/var/ssl/private/kafka.server.truststore.jks ssl.truststore.password=test1234 ssl.keystore.location=/var/ssl/private/kafka.server.keystore.jks ssl.keystore.password=test1234 ssl.key.password=test1234 # Enable SSL for inter-broker communication, add the following to the broker properties file (it defaults to PLAINTEXT): security.inter.broker.protocol=SSL ``` ### Authorization Kafka supports client authentication via SASL. SASL authentication can be enabled concurrently with SSL encryption. The supported SASL mechanisms are: * GSSAPI (Kerberos) * OAUTHBEARER * SCRAM * PLAIN * Delegation Tokens #### JAAS configurations Kafka uses the Java Authentication and Authorization Service (JAAS) for SASL configuration. Provide JAAS configurations for all SASL authentication mechanisms. Brokers can configure JAAS by passing a static JAAS configuration file into the JVM using the java.security.auth.login.config property at runtime. ```shell export KAFKA_OPTS="-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf" bin/kafka-server-start etc/kafka/server.properties ``` Brokers can also configure JAAS using the broker configuration property sasl.jaas.config. Prefix the property name with the listener prefix, including the SASL mechanism, i.e., listener.name.```listenerName```.```saslMechanism``````.sasl.jaas.config. Specify one login module in the config value. To configure multiple mechanisms on a listener, you must provide a separate config for each mechanism using the listener and mechanism prefix. ```shell listener.name.sasl_ssl.scram-sha-256.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \ username="admin" \ password="admin-secret"; listener.name.sasl_ssl.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="admin" \ password="admin-secret" \ user_admin="admin-secret" \ user_alice="alice-secret"; ``` This is the preferred method of configuring JAAS for brokers. ### Authorization using ACLs Kafka ships with a pluggable, out-of-the-box Authorizer implementation that uses ZooKeeper to store all the ACLs. It is important to set ACLs because otherwise access to resources is limited to super users when an Authorizer is configured. The default behavior is that if a resource has no associated ACLs, then no one is allowed to access the resource, except super users. #### Broker Configuration ##### Authorizer To enable ACLs, you must configure an Authorizer. Kafka provides a simple authorizer implementation, and to use it, you can add the following to server.properties: ``` authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer ``` ##### Super Users By default, if no resource patterns match a specific resource, then the resource has no associated ACLs, and therefore no one other than super users are allowed to access the resource. If you want to change that behavior, you can include the following in server.properties: ``` allow.everyone.if.no.acl.found=true super.users=User:Bob;User:Alice ``` --- ####### Local File --- title: Local File description: Connect to your local data files sidebar_label: Local File sidebar_position: 35 --- Import data from files stored on your local machine into your Rill project. ## Overview The Local File connector allows you to import data from CSV, JSON, Parquet, and other supported file formats directly from your local filesystem. This is perfect for working with local datasets, exports, or files you've downloaded. ## Adding a Local File Source ### Option 1: Using the Rill UI 1. In the left navigation pane, click the **"+"** button next to **Sources** 2. Select **"Local File"** from the connector options 3. Navigate to and select your file, or drag and drop it directly onto the Rill interface ### Option 2: Using Code Create a YAML configuration file in your project's `sources` directory: ```yaml # Model YAML # Reference documentation: https://docs.rilldata.com/reference/project-files/models type: model materialize: true connector: duckdb sql: "select * from read_csv('data/your_file_name.csv', auto_detect=true, ignore_errors=1, header=true) ``` **Important:** When using relative paths, they should be relative to your Rill project root (where `rill.yaml` is located), not the `sources` directory. ## Importing Multiple Files Use glob patterns to import data from multiple files at once: ```yaml type: source connector: local_file path: /path/to/data/*.csv ``` **Examples:** - `data/*.csv` - All CSV files in the data directory - `exports/2024-*.parquet` - All Parquet files from 2024 - `logs/**/*.json` - All JSON files in logs and subdirectories For detailed glob pattern syntax, refer to the [DuckDB multiple files documentation](https://duckdb.org/docs/stable/data/multiple_files/overview.html). ## Supported File Formats The Local File connector supports various file formats including: - CSV - JSON - Parquet :::warning File Size Limits When ingesting the Data into Rill, you'll notice a new `/data` folder path with a copy of the CSV file. This is designed so that when you publish to Rill Cloud, the file will also be included. Note that there is a 100MB limit to each unique file. Files over 100MB will not be deployed with your project. ::: --- ####### Salesforce --- title: Salesforce description: Connect to data in a Salesforce org using the Bulk API sidebar_label: Salesforce sidebar_position: 65 --- ## Overview [Salesforce](https://www.salesforce.com/) is a leading cloud-based Customer Relationship Management (CRM) platform designed to help businesses connect with and understand their customers better. It offers a comprehensive suite of applications focused on sales, customer service, marketing automation, analytics, and application development. Salesforce enables organizations of all sizes to build stronger relationships with their customers through personalized experiences, streamlined communication, and predictive insights. Rill can ingest data from Salesforce as a source by utilizing the Bulk API, which requires a Salesforce username and password (and, in some cases, a token, depending on the org configuration) to authenticate against a Salesforce org. ## Local credentials When using Rill Developer on your local machine, you will need to provide your credentials via a connector file. We would recommend not using plain text to create your file and instead use the `.env` file. For more details on your connector, see [connector YAML](/reference/project-files/connectors) for more details. :::tip Updating the project environmental variable If you've already deployed to Rill Cloud, you can either [push/pull the credential]( /guide/administration/project-settings/variables-and-credentials#pushing-and-pulling-credentials-to--from-rill-cloud-via-the-cli) from the CLI with: ``` rill env push rill env pull ``` ::: Alternatively, you can include the credentials directly in the underlying source YAML by adding the `username` and `password` parameters. For example, your source YAML may contain the following properties (these can also be configured through the UI during source creation): ```yaml type: "model" connector: "salesforce" endpoint: "login.salesforce.com" username: "user@example.com" password: "MyPasswordMyToken" soql: "SELECT Id, Name, CreatedDate FROM Opportunity" sobject: "Opportunity" ``` :::tip Did you know? If this project has already been deployed to Rill Cloud and credentials have been set for this source, you can use `rill env pull` to [pull these cloud credentials](/developers/build/connectors/credentials/#rill-env-pull) locally (into your local `.env` file). Please note that this may override any credentials you have set locally for this source. ::: ## Deploy to Rill Cloud When deploying a project to Rill Cloud, Rill requires you to explicitly provide Salesforce credentials used in your project. Please refer to our [connector YAML reference docs](/reference/project-files/connectors) for more information. If you subsequently add sources that require new credentials (or if you simply entered the wrong credentials during the initial deploy), you can update the credentials by pushing the `Deploy` button to update your project or by running the following command in the CLI: ``` rill env push ``` :::note Leave the `key` and `client_id` fields blank unless you are using JWT (described in the next section [below](#jwt)). ::: ### JWT Authentication using JWT instead of a password is also supported. Set `client_id` to the **Client Id** (also known as the _Consumer Key_) of the Connected App to use, and set `key` to contain the PEM-formatted private key to use for signing. --- ####### SQLite --- title: SQLite description: Connect to data in SQLite sidebar_label: SQLite sidebar_position: 80 --- ## Overview [SQLite](https://www.sqlite.org/about.html) is a lightweight, self-contained SQL database engine renowned for its reliability, speed, and full-featured, serverless architecture. SQLite is primarily known as an in-process database and is widely used in embedded systems, mobile applications, and various small-to-medium-sized applications due to its simplicity, zero-configuration, and single-file database format. SQLite supports standard SQL syntax and includes features such as transactions and atomic commit and rollback, making it a practical choice for applications requiring a compact, efficient data management system. Rill supports connecting to and reading from a SQLite database as a source through the [DuckDB SQLite extension](https://duckdb.org/docs/extensions/sqlite.html). ## Connect to SQLite SQLite databases are read through DuckDB's [SQLite extension](https://duckdb.org/docs/extensions/sqlite.html) using the `sqlite_scan()` function. No separate connector is needed. Create a model file (e.g., `models/my_sqlite_data.yaml`): ```yaml type: model connector: duckdb materialize: true sql: | SELECT * FROM sqlite_scan('data/mydatabase.db', 'my_table') ``` :::tip If you plan to deploy the project to Rill Cloud, place the SQLite database file in a `data` folder in your Rill project directory and use the relative path (e.g., `data/mydatabase.db`). ::: ## Deploy to Rill Cloud Once a project with a SQLite source has been deployed using `rill deploy`, Rill Cloud will need to be able to access and retrieve the underlying database file. In most cases, this means that the corresponding SQLite database file should be included within a directory in your Git repository, which will allow you to specify a relative path in your source definition (from the project root). :::warning When Using An External SQLite Database If the SQLite database file is external to your Rill project directory, you will still be able to use the fully qualified path to read this SQLite database _locally_ using Rill Developer. However, when deployed to Rill Cloud, this source will throw an **error**. ::: --- ##### Olap --- ####### Google BigQuery --- title: Google BigQuery description: Power Rill dashboards using BigQuery sidebar_label: Google BigQuery sidebar_position: 13 --- [Google BigQuery](https://cloud.google.com/bigquery/docs) is a fully managed, serverless data warehouse that enables scalable and cost-effective analysis of large datasets. Rill supports connecting to BigQuery as a live connector, allowing you to build metrics views and dashboards directly on top of existing BigQuery tables — no data movement required. :::tip BigQuery as a Live Connector vs. Data Source Connector Rill supports BigQuery in two distinct modes: - **OLAP (Live Connector)** — Rill queries BigQuery directly at dashboard load time. No data is ingested into Rill. Use this when your data is already modeled and optimized in BigQuery and you want Rill as a visual layer on top. Set `olap_connector: bigquery` in `rill.yaml`. - **Data Source Connector** — Rill extracts data from BigQuery and ingests it into its embedded engine (DuckDB or ClickHouse). Use this when you want Rill to manage the data pipeline, apply transformations via SQL models, or combine BigQuery data with other sources. See the [BigQuery data source docs](/developers/build/connectors/data-source/bigquery). In general, use the live connector if your BigQuery tables are already production-ready and large. Use data source ingestion if you need to transform, join, or enrich the data before building dashboards. ::: ## Connect to BigQuery After selecting "Add Data", select BigQuery and fill in your connection parameters. This will automatically create the `bigquery.yaml` file in your `connectors` directory and populate the `.env` file with your credentials. For more information on supported parameters, see our [BigQuery connector YAML reference docs](/reference/project-files/connectors#bigquery). ```yaml type: connector driver: bigquery project_id: "my-gcp-project" google_application_credentials: "{{ .env.GOOGLE_APPLICATION_CREDENTIALS }}" ``` :::tip Creating a Service Account For production use, authenticate with a Google Cloud service account JSON key. The service account needs the `roles/bigquery.dataViewer`, `roles/bigquery.readSessionUser`, and `roles/bigquery.jobUser` roles. See [the appendix in the BigQuery data source docs](/developers/build/connectors/data-source/bigquery#how-to-create-a-service-account-using-the-google-cloud-console) for a step-by-step guide. ::: ### Application Default Credentials (Local Development) For local development, you can omit `google_application_credentials` and Rill will use your local Google Cloud CLI credentials: ```bash gcloud auth application-default login ``` ```yaml type: connector driver: bigquery project_id: "my-gcp-project" ``` :::warning Not suitable for Rill Cloud Application Default Credentials only work locally. When deploying to Rill Cloud, always provide a service account JSON via `google_application_credentials`. ::: ### Controlling Query Costs BigQuery bills based on bytes scanned. On the on-demand pricing model, we recommend setting `max_bytes_billed` on the connector to cap the amount scanned per dashboard query. Queries that exceed the limit fail with an error instead of running. ```yaml type: connector driver: bigquery project_id: "my-gcp-project" google_application_credentials: "{{ .env.GOOGLE_APPLICATION_CREDENTIALS }}" max_bytes_billed: 100000000000 # 100 GB ``` `max_bytes_billed` only applies to live connector dashboard queries. It is not enforced when ingesting data via the BigQuery data source connector. The default value of `0` disables the limit. ## Build a Metrics View on BigQuery Once connected, set BigQuery as the OLAP connector in your `rill.yaml`: ```yaml olap_connector: bigquery ``` Then create a metrics view that references a BigQuery table: ```yaml type: metrics_view connector: bigquery database: my-gcp-project # BigQuery project (optional; defaults to the connector's project_id) database_schema: my_dataset # BigQuery dataset model: my_table timeseries: created_at dimensions: - column: region - column: category measures: - name: total_revenue expression: SUM(revenue) ``` In BigQuery terminology, `database` maps to the **project**, `database_schema` maps to the **dataset**, and `model` maps to the **table**. Measure expressions must use [BigQuery SQL](https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax) syntax. :::note Rill supports metrics views directly on BigQuery as a live connector. Incremental models and partitioned ingestion are not supported in live connector mode. ::: To reduce bytes scanned on dashboards with repeat traffic, see [Caching query results](/developers/build/metrics-view/underlying-model#caching-query-results) on live connectors. Caching pairs well with [`max_bytes_billed`](#controlling-query-costs) — use both to cap per-query cost and reduce overall query volume. --- ####### ClickHouse --- title: ClickHouse description: Power Rill dashboards using ClickHouse sidebar_label: ClickHouse sidebar_position: 0 --- import LoomVideo from '@site/src/components/LoomVideo'; // Adjust the path as needed [ClickHouse](https://clickhouse.com/docs/en/intro) is an open-source, column-oriented OLAP database management system known for its ability to perform real-time analytical queries on large-scale datasets. Its architecture is optimized for high performance, leveraging columnar storage and advanced compression techniques to speed up data reads and significantly reduce storage costs. ClickHouse's efficiency in query execution, scalability, and ability to handle even petabytes of data make it an excellent choice for real-time analytic use cases.
Rill supports connecting to an existing ClickHouse cluster via a "live connector" and using it as an OLAP engine built against [external tables](/developers/build/connectors/olap#external-olap-tables) to power Rill dashboards. This is particularly useful when working with extremely large datasets (hundreds of GBs or even TB+ in size). :::note Supported Versions Rill supports connecting to ClickHouse v22.7 or newer versions. ::: ## Connect to ClickHouse When using ClickHouse for local development, you can connect via connection parameters or by using the DSN. Both local instances of ClickHouse and ClickHouse Cloud are supported. After selecting "Add Data", select ClickHouse and fill in your connection parameters. This will automatically create the `clickhouse.yaml` file in your `connectors` directory and populate the `.env` file with `CLICKHOUSE_PASSWORD` or `CLICKHOUSE_DSN` depending on which you select in the UI. For more information on supported parameters, see our [ClickHouse connector YAML reference docs](/reference/project-files/connectors#clickhouse). ```yaml type: connector driver: clickhouse host: port: username: password: "{{ .env.CLICKHOUSE_PASSWORD }}" ssl: true # required for ClickHouse Cloud ``` After creating the connector, you can edit the `.env` file manually in the project directory, or the connectors/clickhouse.yaml file. :::tip Getting DSN errors in dashboards after setting `.env`? If you are facing issues related to DSN connection errors in your dashboards even after setting the connection string via the project's `.env` file, try restarting Rill using the `rill start --reset` command. ::: ### Connection String (DSN) Rill is able to connect to ClickHouse using the [ClickHouse Go Driver](https://clickhouse.com/docs/en/integrations/go). An appropriate connection string (DSN) will need to be set through the `CLICKHOUSE_DSN` property in Rill. ```bash CLICKHOUSE_DSN="clickhouse://:?username=&password=" ``` Once the file is created, it will be added directly to the `.env` file in the project directory. To make changes to this connector, modify `CLICKHOUSE_DSN`. ```yaml type: connector driver: clickhouse dsn: "{{ .env.CLICKHOUSE_DSN }}" ``` :::info Check your port In most situations, the default port is 9440 for TLS and 9000 when not using TLS. However, it is worth double-checking the port that your ClickHouse instance is configured to use when setting up your connection string. ::: :::note DSN properties For more information about available DSN properties and setting an appropriate connection string, please refer to ClickHouse's [documentation](https://github.com/ClickHouse/clickhouse-go?tab=readme-ov-file#dsn). ::: ## Connect to ClickHouse Cloud If you are connecting to an existing [ClickHouse Cloud](https://clickhouse.com/cloud) instance, you can retrieve connection details about your instance by clicking on the `Connect` tab from within the admin settings navigation page. This will provide relevant information, such as the hostname, port, and username being used for your instance that you can then use to construct your DSN. ![ClickHouse Cloud](/img/build/connectors/olap-engines/clickhouse/clickhouse-cloud.png) Using the information in the ClickHouse UI, populate the parameters of your connection. ### Connection String (DSN) Because ClickHouse Cloud requires a secure connection over [https](https://github.com/ClickHouse/clickhouse-go?tab=readme-ov-file#http-support-experimental), you will need to pass in `secure=true` and `skip_verify=true` as additional URL parameters as part of your https URL (for your DSN). ```yaml https://:?username=&password=&secure=true&skip_verify=true ``` :::info Need help connecting to ClickHouse? If you would like to connect Rill to an existing ClickHouse instance, please don't hesitate to [contact us](/contact). We'd love to help! ::: ## Rill Managed ClickHouse By setting `managed: true` in your ClickHouse connector, you will enable an embedded ClickHouse server to spin up with Rill. This will allow you to import data directly into this ClickHouse server without having to worry about managing an external database. ```yaml type: connector driver: clickhouse managed: true ``` Data ingestion features are not yet available in the UI, please refer to our [model documentation](/reference/project-files/models) on how to ingest data into ClickHouse. For a guide, see [ingesting data directly into ClickHouse](/developers/tutorials/rill-clickhouse/r_ch_ingest). ## Read Only Connector You can configure your ClickHouse connector to operate in read-only mode by adding a mode parameter. This provides an additional security layer by ensuring your connector can only read data and cannot perform write operations. ```yaml mode: read # readwrite ``` ## Configuring Rill Cloud When deploying a ClickHouse-backed project to Rill Cloud, you have the following options to pass the appropriate connection string to Rill Cloud: 1. If you have followed the UI to create your ClickHouse connector, the password or DSN should already exist in the .env file. During the deployment process, this `.env` file is automatically pushed with the deployment. 2. If `CLICKHOUSE_DSN` has already been set in your project `.env`, you can push and update these variables directly in your cloud deployment by using the `rill env push` command. :::warning Local ClickHouse Server If you are developing on a locally running ClickHouse server, this will not be deployed with your project. You will either need to use ClickHouse Cloud or Managed ClickHouse. ::: ## Setting the Default OLAP Connection Creating a connection to an OLAP engine will automatically add the `olap_connector` property in your project's [rill.yaml](/reference/project-files/rill-yaml) and change the default OLAP engine to ClickHouse. Once this is changed, you'll notice that some of the UI features are removed as we currently do not support modeling and direct source ingestion in ClickHouse. However, this is behind a feature flag, [contact us](/contact) for more information! ```yaml olap_connector: clickhouse ``` :::info Interested in using multiple OLAP engines in the same project? Please see our [Using Multiple OLAP Engines](/developers/build/connectors/olap/multiple-olap) page. ::: ## Reading from Multiple Schemas Rill supports reading from multiple schemas in ClickHouse from within the same project in Rill Developer, and all accessible tables (given the permission set of the underlying user) should automatically be listed in the lower left-hand tab, which can then be used to [create dashboards](/developers/build/dashboards). ## Caching Query Results To reduce load on dashboards with repeat traffic, see [Caching query results](/developers/build/metrics-view/underlying-model#caching-query-results). ## Additional Notes - At the moment, we do not officially support modeling with ClickHouse; however, this is available via a feature flag. If this is something you're interested in, please [contact us](/contact). - For dashboards powered by ClickHouse, [measure definitions](/developers/build/metrics-view/#measures) are required to follow standard [ClickHouse SQL](https://clickhouse.com/docs/en/sql-reference) syntax. - Because string columns in ClickHouse can theoretically contain [arbitrary binary data](https://github.com/ClickHouse/ClickHouse/issues/2976#issuecomment-416694860), if your column contains invalid UTF-8 characters, you may want to first cast the column by applying the `toValidUTF8` function ([see ClickHouse documentation](https://clickhouse.com/docs/en/sql-reference/functions/string-functions#tovalidutf8)) before reading the table into Rill to avoid any downstream issues. --- ####### Databricks --- title: Databricks description: Power Rill dashboards using Databricks sidebar_label: Databricks sidebar_position: 02 --- [Databricks](https://docs.databricks.com) is a unified data and AI platform built on top of Apache Spark and the lakehouse architecture, with Unity Catalog for governance and Databricks SQL for analytics. Rill supports connecting to Databricks as a live connector, allowing you to build metrics views and dashboards directly on top of existing Databricks tables — no data movement required. :::tip Databricks as a Live Connector vs. Data Source Connector Rill supports Databricks in two distinct modes: - **OLAP (Live Connector)** — Rill queries Databricks directly at dashboard load time. No data is ingested into Rill. Use this when your data is already modeled and optimized in Databricks and you want Rill as a visual layer on top. Set `olap_connector: databricks` in `rill.yaml`. - **Data Source Connector** — Rill extracts data from Databricks and ingests it into its embedded engine (DuckDB). Use this when you want Rill to manage the data pipeline, apply transformations via SQL models, or combine Databricks data with other sources. See the [Databricks data source docs](/developers/build/connectors/data-source/databricks). In general, use the live connector if your Databricks tables are already production-ready and large. Use data source ingestion if you need to transform, join, or enrich the data before building dashboards. ::: ## Connect to Databricks After selecting "Add Data", select Databricks and fill in your connection parameters. This will automatically create the `databricks.yaml` file in your `connectors` directory and populate the `.env` file with your credentials. For more information on supported parameters, see our [Databricks connector YAML reference docs](/reference/project-files/connectors#databricks). ```yaml type: connector driver: databricks host: "dbc-xxxxxxxx-xxxx.cloud.databricks.com" http_path: "/sql/1.0/warehouses/xxxxxxxxxxxxxxxx" token: "{{ .env.DATABRICKS_TOKEN }}" catalog: "main" # optional schema: "default" # optional ``` :::tip Finding your connection parameters In the Databricks workspace, navigate to **SQL Warehouses**, select the warehouse you want to use, and open the **Connection details** tab. The **Server hostname** is your `host`, the **HTTP path** is your `http_path`, and you can generate a personal access token under **User Settings → Developer → Access tokens**. ::: ### Connection String (DSN) For advanced configuration, you can specify a single DSN instead of the individual fields above. The DSN cannot be combined with `host`, `http_path`, `token`, `catalog`, or `schema`. ```yaml type: connector driver: databricks dsn: "{{ .env.DATABRICKS_DSN }}" ``` ```bash # .env DATABRICKS_DSN=token:dapi...@dbc-xxxxxxxx-xxxx.cloud.databricks.com:443/sql/1.0/warehouses/xxxxxxxxxxxxxxxx?catalog=main&schema=default ``` See the [Databricks SQL Go driver](https://github.com/databricks/databricks-sql-go) documentation for the full list of supported DSN parameters. ## Build a Metrics View on Databricks Once connected, set Databricks as the OLAP connector in your `rill.yaml`: ```yaml olap_connector: databricks ``` Then create a metrics view that references a Databricks table: ```yaml type: metrics_view connector: databricks database: main # Unity Catalog (optional; defaults to the connector's catalog) database_schema: my_schema # Schema within the catalog model: my_table timeseries: created_at dimensions: - column: region - column: category measures: - name: total_revenue expression: SUM(revenue) ``` In Databricks terminology, `database` maps to the **catalog**, `database_schema` maps to the **schema**, and `model` maps to the **table**. Measure expressions must use [Databricks SQL](https://docs.databricks.com/aws/en/sql/language-manual/) syntax. :::note Rill supports metrics views directly on Databricks as a live connector. Incremental models and partitioned ingestion are not supported in live connector mode. ::: To reduce SQL warehouse spend on dashboards with repeat traffic, see [Caching query results](/developers/build/metrics-view/underlying-model#caching-query-results) on live connectors. --- ####### Druid --- title: Druid description: Power Rill dashboards using Druid sidebar_label: Druid sidebar_position: 05 --- [Apache Druid](https://druid.apache.org/docs/latest/design/) is an open-source, high-performance OLAP engine designed for real-time analytics on large datasets. It excels in analytical workloads due to its columnar storage format, which enables fast data aggregation, querying, and filtering. Druid is particularly well-suited for use cases that require interactive exploration of large-scale data, real-time data ingestion, and fast query responses, making it a popular choice for applications in business intelligence, user behavior analytics, and financial analysis. Rill supports connecting to an existing Druid cluster via a "live connector" and using it as an OLAP engine built against [external tables](/developers/build/connectors/olap#external-olap-tables) to power Rill dashboards. This is particularly useful when working with extremely large datasets (hundreds of GBs or even TB+ in size). ## Configuring Rill Developer with Druid When using Rill for local development, there are a few options to configure Rill to enable Druid as an OLAP engine: 1. Connect to an OLAP engine via Add Data. This will automatically create the `druid.yaml` file in your `connectors` directory and populate the `.env` file with `DRUID_PASSWORD` or `DRUID_DSN` depending on which you select in the UI. For more information on supported parameters, see our [Druid connector YAML reference docs](/reference/project-files/connectors#druid). ```yaml type: connector driver: druid host: port: username: password: "{{ .env.DRUID_PASSWORD }}" ssl: true # or dsn: "{{ .env.DRUID_DSN }}" ``` 2. You can manually set `DRUID_DSN` in your project's `.env` file or try pulling existing credentials locally using `rill env pull` if the project has already been deployed to Rill Cloud. :::tip Getting DSN errors in dashboards after setting `.env`? If you are facing issues related to DSN connection errors in your dashboards even after setting the connection string via the project's `.env` file, try restarting Rill using the `rill start --reset` command. ::: ## Connection String (DSN)
Rill connects to Druid using the [HTTP API](https://druid.apache.org/docs/latest/api-reference/sql-api) and requires a connection string of the following format: `http://:@:/druid/v2/sql`. If `user` or `password` contain special characters, they should be URL encoded (i.e., `p@ssword` -> `p%40ssword`). This should be set in the `DRUID_DSN` property in Rill. As an example, this typically looks like: ```bash DRUID_DSN="https://user:password@localhost:8888/druid/v2/sql" ``` :::info Need help connecting to Druid? If you would like to connect Rill to an existing Druid instance, please don't hesitate to [contact us](/contact). We'd love to help! ::: ## Setting the Default OLAP Connection When connecting to Druid via the UI, the default OLAP connection will be automatically added to your rill.yaml. This will change the way the UI behaves, such as adding new data sources, as this is not supported with a Druid-backed Rill project. ```yaml olap_connector: druid ``` :::note For more information about available properties in `rill.yaml`, see our [project YAML](/reference/project-files/rill-yaml) documentation. ::: :::info Interested in using multiple OLAP engines in the same project? Please see our [Using Multiple OLAP Engines](/developers/build/connectors/olap/multiple-olap) page. ::: ## Configuring Rill Cloud When deploying a Druid-backed project to Rill Cloud, you have the following options to pass the appropriate connection string to Rill Cloud: 1. If you have followed the UI to create your Druid connector, the password or DSN should already exist in the .env file. During the deployment process, this `.env` file is automatically pushed with the deployment. 2. If `DRUID_DSN` has already been set in your project `.env`, you can push and update these variables directly in your cloud deployment by using the `rill env push` command. ## Supported Versions Rill supports connecting to Druid v28.0 or newer versions. ## Caching Query Results To reduce broker load on dashboards with repeat traffic, see [Caching query results](/developers/build/metrics-view/underlying-model#caching-query-results) on live connectors. ## Additional Notes - At the moment, we do not support modeling with Druid. If this is something you're interested in, please [contact us](/contact). - For dashboards powered by Druid, [measure definitions](/developers/build/metrics-view/#measures) are required to follow standard [Druid SQL](https://druid.apache.org/docs/latest/querying/sql/) syntax. --- ####### DuckDB --- title: DuckDB description: Power Rill dashboards using DuckDB (default) sidebar_label: DuckDB sidebar_position: 10 --- [DuckDB](https://duckdb.org/why_duckdb.html) is an in-memory, columnar SQL database designed for analytical (OLAP) workloads, offering high-speed data processing and analysis. Its columnar storage model and vectorized query execution make it highly efficient for OLAP tasks, enabling fast aggregation, filtering, and joins on large datasets. ## Rill Managed DuckDB By default, Rill includes DuckDB as an embedded OLAP engine that ingests data from [data sources](/developers/build/connectors) and powers your dashboards. When you start a new project, you'll see a `connectors/duckdb.yaml` file alongside other project files. No additional configuration is needed to use DuckDB with Rill Developer or Rill Cloud. ```yaml type: connector driver: duckdb managed: true ``` :::tip Performance Considerations DuckDB is an excellent analytical engine but can face performance challenges as data size grows significantly. As a general guideline, we recommend keeping your data size in DuckDB **under 50GB** along with other [performance recommendations](/developers/tutorials/performance). For larger datasets, Rill still provides excellent performance but may require additional backend optimizations. [Contact us](/contact) if you need assistance with large-scale deployments. ::: ## Live Connect to External DuckDB Rill also supports connecting to external DuckDB database files as a "live connector". This allows you to leverage existing DuckDB databases within Rill to create metrics views and dashboards. :::warning Local Development Only This setup is designed for local development and testing only. It will not deploy to Rill Cloud under most circumstances because: - Rill Cloud can only access files within your project directory - If your DuckDB file is outside the project folder, it cannot be bundled for deployment - Files larger than 100MB will fail to deploy due to upload size limits For production deployments, consider using our [external DuckDB data source](/developers/build/connectors/data-source/duckdb) to ingest your data instead. ::: ### Configuration Using the UI, select the DuckDB icon under the OLAP section to add a new DuckDB connector. Any existing connectors with data models will need to be refreshed to ingest the data into your external DuckDB. ```yaml type: connector driver: duckdb path: '/path/to/main.db' ``` ### Setting the Default OLAP Connection Creating a connection to MotherDuck will automatically add the `olap_connector` property in your project's [rill.yaml](/reference/project-files/rill-yaml) and change the default OLAP engine to `duckdb`. ```yaml olap_connector: duckdb ``` ## Using DuckDB Extensions DuckDB supports a wide variety of extensions that can enhance its functionality. To use extensions with Rill's embedded DuckDB, configure them in your connector: ```yaml # connectors/duckdb.yaml type: connector driver: duckdb init_sql: | INSTALL httpfs; LOAD httpfs; INSTALL spatial; LOAD spatial; ``` ### Popular Extensions For a complete list of available extensions, see the [DuckDB Extensions documentation](https://duckdb.org/docs/extensions/overview). ## Multiple OLAP Engines While not recommended, Rill supports using multiple OLAP engines in a single project. For more information, see [Using Multiple OLAP Engines](/developers/build/connectors/olap/multiple-olap). ## Additional Notes - For dashboards powered by DuckDB, [measure definitions](/developers/build/metrics-view/#measures) are required to follow standard [DuckDB SQL](https://duckdb.org/docs/sql/introduction) syntax. - There is a known issue around creating a DuckDB source via the UI; you will need to create the YAML file manually. --- ####### DuckLake --- title: DuckLake description: Power Rill dashboards using DuckLake sidebar_label: DuckLake sidebar_position: 12 --- [DuckLake](https://ducklake.select/) is an open lakehouse format built on DuckDB. A DuckLake keeps table data as Parquet files in object storage (local, S3, GCS, or Azure) while the catalog (schemas, snapshots, statistics) lives in a separate database — DuckDB, SQLite, PostgreSQL, or MySQL. Rill connects to DuckLake through the DuckDB driver and uses it as a live OLAP engine, so no data is ingested into Rill and all queries are pushed down to DuckDB against your lake. :::note DuckLake uses the DuckDB Driver DuckLake connectors use `driver: duckdb` under the hood. The difference from a standard DuckDB connector is the `attach` clause, which points DuckDB at your DuckLake catalog and data path. ::: ## Configuring Rill Developer with DuckLake Create the connector via **Add Data → DuckLake** in the UI. Rill will generate a `connectors/ducklake.yaml` file and set DuckLake as the default OLAP engine in `rill.yaml`. ```yaml type: connector driver: duckdb attach: "'ducklake:metadata.ducklake' (DATA_PATH 'data/')" ``` The `attach` clause is passed directly to DuckDB's `ATTACH` statement. It must begin with the `ducklake:` metadata backend and should include a `DATA_PATH` pointing at the directory (local or object storage) that holds your Parquet files. ### Supported Metadata Backends DuckLake can store its catalog in any of the following: - **DuckDB** — `'ducklake:metadata.ducklake'` - **SQLite** — `'ducklake:sqlite:metadata.sqlite'` - **PostgreSQL** — `'ducklake:postgres:dbname=ducklake host=...'` - **MySQL** — `'ducklake:mysql:host=... user=... database=ducklake'` For cloud-hosted catalogs, credentials for the metadata database can be injected via environment variables, e.g. `host={{ .env.DUCKLAKE_PG_HOST }}`. ### Supported Data Paths The `DATA_PATH` in your `attach` clause can point at either local storage or any object store supported by DuckDB: - Local filesystem — `DATA_PATH 'data/'` - S3 — `DATA_PATH 's3://my-bucket/ducklake/'` - GCS — `DATA_PATH 'gs://my-bucket/ducklake/'` - Azure Blob Storage — `DATA_PATH 'azure://my-container/ducklake/'` See the [DuckLake docs](https://ducklake.select/docs/stable/duckdb/usage/connecting) for the full ATTACH syntax. ### Setting the Default OLAP Connection Creating a DuckLake connector automatically sets `olap_connector` in your project's [rill.yaml](/reference/project-files/rill-yaml) to the new connector. ```yaml olap_connector: ducklake ``` ## Advanced Options The `attach` clause is passed through to DuckDB and accepts the full set of DuckLake ATTACH options — see the [DuckLake ATTACH reference](https://ducklake.select/docs/stable/duckdb/usage/connecting) for the complete list. Example with multiple options: ```yaml type: connector driver: duckdb attach: "'ducklake:metadata.ducklake' (DATA_PATH 's3://my-bucket/ducklake/', OVERRIDE_DATA_PATH true, SNAPSHOT_VERSION '42')" ``` ## Trying DuckLake Without Your Own Data If you want to see DuckLake in action before pointing Rill at your own catalog, DuckDB hosts a public `lineitem` table from TPC-H (scale factor 3) as a read-only DuckLake. Load it in the [DuckDB browser visualizer](https://duckdb.org/visualizer/#resource_path=https%3A%2F%2Fblobs.duckdb.org%2Fdatalake%2Ftpch-sf3.ducklake&resource_type=ducklake&table_name=lineitem) to confirm the catalog is reachable, then point Rill at the same resource: ```yaml type: connector driver: duckdb attach: "'ducklake:https://blobs.duckdb.org/datalake/tpch-sf3.ducklake'" ``` ## Configuring Rill Cloud When deploying a DuckLake-backed project to Rill Cloud: 1. Any secrets referenced in the `attach` clause (e.g. S3 credentials, Postgres passwords) should be set via `{{ .env.KEY_NAME }}` in your YAML and managed with the project `.env` file. 2. Use `rill env push` to sync local environment variables to your cloud deployment. 3. The `DATA_PATH` must be reachable from Rill Cloud — local filesystem paths will not deploy. ## Additional Notes - DuckLake uses the same SQL dialect as DuckDB, so all standard DuckDB functions are available. [Measure definitions](/developers/build/metrics-view/#measures) should follow standard [DuckDB SQL](https://duckdb.org/docs/sql/introduction) syntax. - Rill opens DuckLake in read-only mode by default. To allow Rill to create or modify tables in the lake, enable write mode in the connector advanced options. - Combine DuckLake with [multiple OLAP engines](/developers/build/connectors/olap/multiple-olap) to power different dashboards from different catalogs in the same project. :::info Need help connecting to DuckLake? If you would like to connect Rill to DuckLake or need assistance with setup, please don't hesitate to [contact us](/contact). We'd love to help! ::: --- ####### MotherDuck --- title: MotherDuck description: Power Rill dashboards using MotherDuck sidebar_label: MotherDuck sidebar_position: 15 --- [MotherDuck](https://motherduck.com/) is a cloud-native DuckDB service that provides scalable analytics and data processing capabilities. Built on the same core engine as DuckDB, MotherDuck offers the familiar SQL interface and performance characteristics while adding cloud-native features like serverless compute, automatic scaling, and collaborative data sharing. ![Rill Developer](/img/build/connectors/olap-engines/motherduck/rill-developer.png) Rill supports connecting to MotherDuck and using it as an OLAP engine to power dashboards. This is particularly useful when you want the performance and SQL compatibility of DuckDB with the scalability and collaboration features of a cloud service. :::note Supported Versions Rill supports connecting to MotherDuck using the latest DuckDB-compatible drivers and protocols. ::: ## Getting Your MotherDuck Access Token To connect to MotherDuck, you'll need a access token from your MotherDuck account: 1. Log in to your [MotherDuck account](https://motherduck.com/) 2. Navigate to the **Settings** section 3. Go to **Access Tokens** 4. Create a new access token or copy an existing one ![Service Token](/img/build/connectors/olap-engines/motherduck/service-token.png) :::warning Keep Your Token Secure Your MotherDuck access token provides access to your data. Keep it secure and never commit it directly to version control. Always use environment variables or secure credential management. ::: ## Configuring Rill Developer with MotherDuck Connect to your OLAP engine via Add Data. This will automatically create the motherduck.yaml file in your connectors folder and populate the .env file with `MOTHERDUCK_TOKEN`. For more information on supported parameters, see our [MotherDuck connector YAML reference docs](/reference/project-files/connectors#motherduck). ```yaml type: connector driver: duckdb token: "{{ .env.MOTHERDUCK_TOKEN }}" path: "md:my_database" schema_name: "my_schema" ``` ### Setting the Default OLAP Connection Creating a connection to MotherDuck will automatically add the `olap_connector` property in your project's [rill.yaml](/reference/project-files/rill-yaml) and change the default OLAP engine to `motherduck`. ```yaml olap_connector: motherduck ``` :::info Interested in using multiple OLAP engines in the same project? Please see our [Using Multiple OLAP Engines](/developers/build/connectors/olap/multiple-olap) page. ::: ## Configuring Rill Cloud When deploying a MotherDuck-backed project to Rill Cloud, you have the following options to pass the appropriate access token: 1. If you have followed the UI to create your MotherDuck connector, the token should already exist in the `.env` file. During the deployment process, this `.env` file is automatically pushed with the deployment. 2. If `motherduck_token` has already been set in your project `.env`, you can push and update these variables directly in your cloud deployment by using the `rill env push` command. ## Caching Query Results To increase performance on dashboards with repeat traffic, see [Caching query results](/developers/build/metrics-view/underlying-model#caching-query-results) on live connectors. ## Additional Notes - MotherDuck uses the same SQL syntax as DuckDB, so all standard DuckDB functions and features are available - For dashboards powered by MotherDuck, [measure definitions](/developers/build/metrics-view/#measures) should follow standard [DuckDB SQL](https://duckdb.org/docs/sql/introduction) syntax :::info Need help connecting to MotherDuck? If you would like to connect Rill to MotherDuck or need assistance with setup, please don't hesitate to [contact us](/contact). We'd love to help! ::: --- ####### Using Multiple OLAP Engines --- title: Using Multiple OLAP Engines description: Using multiple OLAP Engines to power dashboards in the same project sidebar_label: Using Multiple OLAP Engines sidebar_position: 50 --- If you have access to another OLAP engine (such as [ClickHouse](/developers/build/connectors/olap/clickhouse) or [Druid](/developers/build/connectors/olap/druid), you have the option to either: - Create dedicated projects that are powered by one specific OLAP engine (default) - Use different OLAP engines _in the same project_ to power separate dashboards On this page, we will walk through how to configure the latter. ### Why Multiple OLAP Engines? There could be reasons why you wish to configure multiple OLAP engines within the same project: - You have data sources that differ greatly in size but which you want to use within the same project. As a rule of thumb, DuckDB handles datasets _up to 50GB quite well_ and is performant. For much larger datasets, you may want a more enterprise-grade OLAP engine powering specific dashboards. - You have existing datasets/tables from other OLAP stores that you wish to use in Rill, which may already be optimized, and which you do not want to separately ingest into Rill. Instead, you would like to create dashboards off these tables directly and have the OLAP engine power them. :::info Don't see an OLAP engine? If there's an OLAP engine you're interested in that isn't available, please don't hesitate to [contact us](/contact). We'd love to hear from you and learn more! ::: ## Enabling Multiple OLAP Engines To configure multiple OLAP engines, you'll want to leave the default OLAP engine as [DuckDB](/developers/build/connectors/olap/duckdb) in your project and configure dashboards that are powered by other OLAP engines individually (more on this below). ### Setting up your OLAP Engine connection string (DSN) Before getting started, you'll need to first configure the appropriate connection string for each OLAP engine that you plan to use in Rill. Besides the built-in DuckDB OLAP engine, each OLAP engine should have its own `connector..dsn` variable that needs to be configured. **For Rill Developer:** - You can set these variables in your project's `.env` file or try pulling existing credentials locally using `rill env pull` if the project has already been deployed to Rill Cloud. :::tip Getting DSN errors in dashboards after setting `.env`? There might be instances where you've configured the project's `.env` file with the appropriate connection DSN strings but dashboards are still throwing errors. In these situations, try restarting Rill using the `rill start --reset` command. ::: **For Rill Cloud:** - Add the required `connector..dsn` parameters to your project's `.env`, and you can push these updated variables to your deployed project directly using `rill env push`. ### Configuring DuckDB as the default OLAP engine Not much needs to be done here as _DuckDB is the inherent default OLAP engine_ that is used by Rill. However, in case a different `olap_connector` is set in the project's `rill.yaml` file, this property should either be removed and/or set back to `duckdb`. ```yaml olap_connector: duckdb ``` :::note rill.yaml For more information about available configurations for `rill.yaml`, please see our [Project YAML](/reference/project-files/rill-yaml) reference documentation. ::: ### Setting the OLAP Engine in the metrics view YAML For each metrics view that is using a separate OLAP engine (other than the default), you'll want to set the `connector` and `table` properties in the underlying [metrics view YAML](/reference/project-files/metrics-views) configuration to the OLAP engine and corresponding [external table](/developers/build/connectors/olap#external-olap-tables) that exists in your OLAP store, respectively. ```yaml type: metrics_view title: connector: table: ... ``` --- ####### "Bring Your Own OLAP Engine (Live Connector)" --- title: "Bring Your Own OLAP Engine (Live Connector)" description: Configure the OLAP engine used by Rill sidebar_label: "OLAP Engines" sidebar_position: 0 toc_max_heading_level: 3 className: connect-connect --- import ConnectorIcon from '@site/src/components/ConnectorIcon'; Rill supports connecting directly to your own OLAP engine via a "live connector". In this mode, no data is ingested into Rill, and all compute is pushed down to the OLAP engine. Use this mode if you've already handled all of your modeling upstream and want to use Rill as your visual application layer. :::tip Models on Live Connectors Rill also offers the ability to ingest and create tables directly from a [data source](/developers/build/connectors/data-source) to your OLAP engine via the live connector, however you'll need to consider a few topics. - **Use a test database** to avoid accidentally overwriting production data - **Incremental processing and related queries are not supported** - **Feature availability may vary** between different OLAP engines ::: In order to connect Rill to your OLAP engine: 1. Create the connector via the UI. 2. [Create the YAML](/reference/project-files/connectors#olap-engines) and set the [default OLAP engine](/reference/project-files/rill-yaml#configuring-the-default-olap-engine) via the rill.yaml file. :::note `olap_connector` in rill.yaml When setting the OLAP Engine via the UI, the `olap_connector` key will automatically update the rill.yaml. ::: ## OLAP Engines Rill supports the use of several different OLAP engines to power your dashboards, including: ### ClickHouse ### Databricks ### Druid ### DuckDB ### DuckLake ### Google BigQuery ### MotherDuck ### Pinot ### Snowflake ### StarRocks
} content="High-performance columnar database for real-time analytics and data warehousing." link="/developers/build/connectors/olap/clickhouse" linkLabel="Learn more" referenceLink="clickhouse" /> } content="Lakehouse platform with native support for metrics views as a live connector." link="/developers/build/connectors/olap/databricks" linkLabel="Learn more" referenceLink="databricks" /> } content="Real-time analytics database designed for high-performance OLAP queries." link="/developers/build/connectors/olap/druid" linkLabel="Learn more" referenceLink="druid" /> } content="Add extra parameters to Rill's embedded DuckDB or connect your own." link="/developers/build/connectors/olap/duckdb" linkLabel="Learn more" referenceLink="duckdb" /> } content="Open lakehouse format built on DuckDB — catalog in SQL, data in Parquet." link="/developers/build/connectors/olap/ducklake" linkLabel="Learn more" referenceLink="ducklake" /> } content="Serverless cloud data warehouse with native support for metrics views as a live connector." link="/developers/build/connectors/olap/bigquery" linkLabel="Learn more" referenceLink="bigquery" /> } content="Cloud-native DuckDB service for scalable analytics and data processing." link="/developers/build/connectors/olap/motherduck" linkLabel="Learn more" referenceLink="motherduck" /> } content="Distributed OLAP datastore for real-time analytics and business intelligence." link="/developers/build/connectors/olap/pinot" linkLabel="Learn more" referenceLink="pinot" /> } content="Cloud data warehouse with native support for metrics views as a live connector." link="/developers/build/connectors/olap/snowflake" linkLabel="Learn more" referenceLink="snowflake" /> } content="Distributed OLAP datastore for real-time analytics and business intelligence." link="/developers/build/connectors/olap/starrocks" linkLabel="Learn more" referenceLink="starrocks" />
:::note Additional OLAP Engines Rill is continually evaluating additional OLAP engines that can be added. For a full list of OLAP engines that we support, refer to our [OLAP Engines](/developers/build/connectors/olap) page. If you don't see an OLAP engine that you'd like to use, please don't hesitate to [reach out](/contact)! ::: ## Multiple OLAP Engines in a Single Project Rill supports the use of multiple OLAP engines in a single project with some limitations. For more detailed information, see our reference on [multiple OLAP engines](/developers/build/connectors/olap/multiple-olap). The basic use cases for multiple engines in a single project are: 1. Using Rill on top of already created and optimized tables from different OLAP sources. 2. Separating data based on size, as performance on different engines differs based on the size of the data. ## Externally Hosted Services If you have a firewall in front of your externally hosted service, you will need to whitelist the IP addresses below. This will allow you to connect to/from your service once your project is deployed to Rill Cloud. ``` 35.196.245.100 34.74.117.37 35.196.153.31 34.75.22.143 34.148.167.51 35.237.60.193 ``` ## What is OLAP? OLAP (or Online Analytical Processing) is a computational approach designed to enable rapid, multidimensional analysis of large volumes of data. With OLAP, data is typically organized into cubes instead of traditional two-dimensional tables, which can facilitate complex queries and data analysis in a way that is significantly more efficient and user-friendly for analytical tasks. In particular, OLAP databases can be especially well suited for BI use cases that require deep, multidimensional analysis or real-time / user-facing analytics and applications. Additionally, many modern OLAP databases are optimized to ingest large volumes of data, execute low-latency queries with high throughput, and process billions of rows quickly with an emphasis on speed and efficiency in data retrieval. Unlike traditional relational databases or data warehouses that are optimized for transaction processing (with a focus on CRUD operations), OLAP databases are designed for query speed and complex analysis. Rather than storing data in a row-oriented manner, optimizing for transactional efficiency and operational queries, most OLAP databases are columnar and use pre-aggregated multidimensional cubes to speed up analytical queries. This allows a broad range of ad hoc queries and analysis to be performed without needing predefined schemas that are tailored to specific queries, and it's this flexibility that enables the highly interactive slice-and-dice exploration of data that powers Rill dashboards. This paradigm allows OLAP to be particularly well-suited for organizations and teams that want to dive deep into and understand their data to support decision-making processes, where speed and flexibility in the actual data analysis are important. :::info Want to see OLAP in action? Check [here](https://www.rilldata.com/case-studies) to see examples of use cases that can be powered by OLAP. ::: ### External OLAP tables Rill supports creating and powering dashboards using existing tables from alternative [OLAP engines](/developers/build/connectors/olap) that have been configured in a particular project. These tables are not managed by Rill—hence, external—but allow users to bring in separate tables or datasets that might already exist in another preferred OLAP database of choice. This prevents the need to unnecessarily ingest this data into Rill, especially if the table is already optimized for use by this other OLAP engine, and allows Rill to connect to the data directly (and submit analytical queries). ![External Olap Db](/img/build/connectors/external-tables/external-olap-db.png) ## Performance Tips ### Data Lifecycle Management One common way to decrease overall data size and improve query performance (by scanning less data) is to roll up your data to higher time grains historically. Typically, this means taking hourly data and rolling up to daily data when the additional level of granularity is no longer necessary for business needs. Databases like Apache Druid have these lifecycle tools built in, or reach out to Rill with questions. A couple of considerations when rolling data from lower to higher time grains: - Daily data loses time zone querying as everything is rolled up to a single time zone (usually UTC) - Consider hashed compaction when going from hourly to daily to reduce data size even further - Watch out for rolling up metrics. Some metrics should be summed—but others (like a bid floor or campaign budget) should stay unique and be rolled up as a max ### Dimension Stripping Dimension stripping is another tool to reduce data size by removing high cardinality fields that are not required for analysis. While this can be done upfront in the dataset, another practice would be to drop these fields at certain intervals when they no longer add business value. Most frequently, we see a couple of decision points where these fields are dropped: - After a day to first week, dropping user level details no longer needed for monitoring - After a week to multiple weeks, dropping "double click" level details that aren't needed for reporting (e.g., the minor release number on an Operating System field) - After a month to months, dropping fields no longer interesting for analysis ### Sampling & Datasketches There are times when you may look at sampling data feeds to trade data accuracy for lower costs and faster query speeds. Sampling involves sending only a percentage of your data, then extrapolating the values to get an estimate. Rill does not recommend sampling your primary KPIs, any records that require a join, or are tied to revenue. This filtered data should be decided in random fashion to not skew or bias the results. Please note, tracking uniques is not recommended if you choose to sample. If looking to track uniques, but with smaller datasets and significantly improved performance, you can load unique values (IP addresses, user IDs, URLs, etc.) with [datasketches](https://datasketches.apache.org). There are multiple types of datasketches supported depending on your engine. At a high level, datasketches use algorithms to approximate unique values. Common use cases for datasketches include count distincts (campaign reach, unique visitors) and quantiles (time spent, frequency). Check out the [Apache Datasketches](https://datasketches.apache.org/docs/Architecture/MajorSketchFamilies.html) site for more details on methodology and use cases. ### Lookups While joins can kill the performance of [OLAP engines](/developers/build/connectors/olap), lookups (key-value pairs) are common to reduce data size and improve query speeds. Lookups can be done during ingestion time (a static lookup to enrich the source data) or at query time (dynamic lookups). **Static Lookups** Static lookups are lookups that are ingested at processing time. When a record is being processed, if a match is found between the record and lookup's key, the lookup's corresponding value at that moment in time is extracted and carbon-copied into Druid for the records it processed. Static lookups are best suited for: - Dimensions with values that require a historical record for how they have changed over time - Values that are never expected to change (leverage Dynamic Lookups if the values are expected to change) - Extremely large lookups (hundreds of thousands of records or >50MB lookup file) to improve query performance Customers typically store lookup values in S3 or GCS, and the lookup file is then updated by customers as needed and consumed by ETL logic. **Dynamic Lookups** Since static lookups transform and store the data permanently, any changes to the mapping would require reprocessing the entire dataset to ensure consistency. To address the case when values in a lookup are expected to change with time, we developed dynamic lookups. Dynamic lookups, also known as Query Time Lookups, are lookups that are retrieved at query time, as opposed to being used at ingestion time. Benefits of dynamic lookups include: - Historical continuity for dimensions that change frequently without reprocessing the entire dataset - Time savings, because there is no dataset reprocessing required to complete the update - Dynamic lookups are kept separate from the dataset. Thus, any human errors introduced in the lookup do not impact the underlying dataset - Ability for users to create new dimension tables from metadata associated with a dimension table. For example, account ownership can change during the course of a quarter. In such cases, a dynamic lookup can be updated on the fly to reflect the most current changes --- ####### Pinot --- title: Pinot description: Power Rill dashboards using Pinot sidebar_label: Pinot sidebar_position: 20 --- [Apache Pinot](https://docs.pinot.apache.org/) is a real-time distributed OLAP datastore purpose-built for low-latency, high-throughput analytics, and is perfect for user-facing analytical workloads. Rill supports connecting to an existing Pinot cluster via a "live connector" and using it as an OLAP engine built against [external tables](/developers/build/connectors/olap#external-olap-tables) to power Rill dashboards. This is particularly useful when working with extremely large datasets (hundreds of GBs or even TB+ in size). ## Configuring Rill Developer with Pinot When using Rill for local development, there are a few options to configure Rill to enable Pinot as an OLAP engine: 1. Connect to an OLAP engine via Add Data. This will automatically create the `pinot.yaml` file in your `connectors` directory and populate the `.env` file with `PINOT_PASSWORD` or `PINOT_DSN` depending on which you select in the UI. For more information on supported parameters, see our [Pinot connector YAML reference docs](/reference/project-files/connectors#pinot). ```yaml type: connector driver: pinot dsn: "{{ .env.PINOT_DSN }}" ``` 1. You can set `PINOT_DSN` in your project's `.env` file or try pulling existing credentials locally using `rill env pull` if the project has already been deployed to Rill Cloud. :::tip Getting DSN errors in dashboards after setting `.env`? If you are facing issues related to DSN connection errors in your dashboards even after setting the connection string via the project's `.env` file, try restarting Rill using the `rill start --reset` command. ::: ## Connection String (DSN) Rill connects to Pinot using the [Pinot Golang Client](https://docs.pinot.apache.org/users/clients/golang) and requires a connection string of the following format: `http://:@:?controller=:`. If `user` or `password` contain special characters, they should be URL encoded (i.e., `p@ssword` -> `p%40ssword`). This should be set in the `PINOT_DSN` property in Rill. As an example, this typically looks like: ```bash PINOT_DSN="http(s)://username:password@localhost:8000?controller=localhost:9000" ``` :::info Need help connecting to Pinot? If you would like to connect Rill to an existing Pinot instance, please don't hesitate to [contact us](/contact). We'd love to help! ::: ## Setting the Default OLAP Connection You'll also need to update the `olap_connector` property in your project's `rill.yaml` to change the default OLAP engine to Pinot: ```yaml olap_connector: pinot ``` :::info Interested in using multiple OLAP engines in the same project? Please see our [Using Multiple OLAP Engines](/developers/build/connectors/olap/multiple-olap) page. ::: ## Configuring Rill Cloud When deploying a Pinot-backed project to Rill Cloud, you have the following options to pass the appropriate connection string to Rill Cloud: 1. If you have followed the UI to create your Pinot connector, the password or DSN should already exist in the .env file. During the deployment process, this `.env` file is automatically pushed with the deployment. 2. If `PINOT_DSN` has already been set in your project `.env`, you can push and update these variables directly in your cloud deployment by using the `rill env push` command. ## Support for Multi-Valued Dimensions Multi-valued dimensions need to be defined in the dashboard YAML as expressions using the `arrayToMv` function. For example, if `RandomAirports` is a multi-valued column in a Pinot table, then the dimension definition will look like: ```yaml - display_name: RandomAirports expression: arrayToMv(RandomAirports) name: RandomAirports description: "Random Airports" ``` Refer to the [Dashboard YAML](/reference/project-files/explore-dashboards) reference page for all dimension properties in detail. :::note Pinot does not support the unnest function, so don't set the `unnest` property to true in the dimension definition of the dashboard YAML. ::: ## Caching Query Results To reduce broker load on dashboards with repeat traffic, see [Caching query results](/developers/build/metrics-view/underlying-model#caching-query-results) on live connectors. ## Additional Notes - At the moment, we do not support modeling with Pinot. If this is something you're interested in, please [contact us](/contact). - For dashboards powered by Pinot, [measure definitions](/developers/build/metrics-view/#measures) are required to follow [Pinot SQL](https://docs.pinot.apache.org/users/user-guide-query/querying-pinot) syntax. --- ####### Snowflake --- title: Snowflake description: Power Rill dashboards using Snowflake sidebar_label: Snowflake sidebar_position: 24 --- [Snowflake](https://docs.snowflake.com) is a cloud data platform known for its scalability, ease of use, and separation of storage and compute. Rill supports connecting to Snowflake as a live connector, allowing you to build metrics views and dashboards directly on top of existing Snowflake tables — no data movement required. :::tip Snowflake as a Live Connector vs. Data Source Connector Rill supports Snowflake in two distinct modes: - **OLAP (Live Connector)** — Rill queries Snowflake directly at dashboard load time. No data is ingested into Rill. Use this when your data is already modeled and optimized in Snowflake and you want Rill as a visual layer on top. Set `olap_connector: snowflake` in `rill.yaml`. - **Data Source Connector** — Rill extracts data from Snowflake and ingests it into its embedded engine (DuckDB or ClickHouse). Use this when you want Rill to manage the data pipeline, apply transformations via SQL models, or combine Snowflake data with other sources. See the [Snowflake data source docs](/developers/build/connectors/data-source/snowflake). In general, use the live connector if your Snowflake tables are already production-ready and large. Use data source ingestion if you need to transform, join, or enrich the data before building dashboards. ::: ## Connect to Snowflake After selecting "Add Data", select Snowflake and fill in your connection parameters. This will automatically create the `snowflake.yaml` file in your `connectors` directory and populate the `.env` file with your credentials. For more information on supported parameters, see our [Snowflake connector YAML reference docs](/reference/project-files/connectors#snowflake). ```yaml type: connector driver: snowflake dsn: "{{ .env.SNOWFLAKE_DSN }}" ``` :::tip Finding your account identifier Your account identifier appears in your Snowflake URL — it's everything before `.snowflakecomputing.com`. For example, if your URL is `https://xy12345.us-east-1.snowflakecomputing.com`, your account identifier is `xy12345.us-east-1`. ::: ### Key-Pair Authentication (Recommended) For production use, Snowflake recommends key-pair authentication over passwords. Generate an unencrypted PKCS#8 private key and base64-encode it: ```bash # Generate a 2048-bit unencrypted PKCS#8 private key openssl genrsa 2048 | openssl pkcs8 -topk8 -nocrypt -out rsa_key.p8 # Base64 URL-encode for use in Rill base64 -w 0 rsa_key.p8 ``` Then set the encoded key in your connector: ```yaml type: connector driver: snowflake account: "" user: "" private_key: "{{ .env.SNOWFLAKE_PRIVATE_KEY }}" ``` See [Snowflake's key-pair authentication docs](https://docs.snowflake.com/en/guide/key-pair-auth) for full setup instructions. ## Build a Metrics View on Snowflake Once connected, set Snowflake as the OLAP connector in your `rill.yaml`: ```yaml olap_connector: snowflake ``` Then create a metrics view that references a Snowflake table: ```yaml type: metrics_view connector: snowflake database: MY_DATABASE database_schema: MY_SCHEMA model: MY_TABLE timeseries: created_at dimensions: - column: region - column: category measures: - name: total_revenue expression: SUM(revenue) ``` :::note Rill supports metrics views directly on Snowflake as a live connector. Incremental models and partitioned ingestion are not supported in live connector mode. ::: To reduce warehouse spend on dashboards with repeat traffic, see [Caching query results](/developers/build/metrics-view/underlying-model#caching-query-results) on live connectors. --- ####### StarRocks --- title: StarRocks description: Power Rill dashboards using StarRocks sidebar_label: StarRocks sidebar_position: 25 --- [StarRocks](https://www.starrocks.io/) is an open-source, high-performance analytical database designed for real-time, multi-dimensional analytics on large-scale data. It supports both primary key and aggregate data models, making it suitable for a variety of analytical workloads including real-time dashboards, ad-hoc queries, and complex analytical tasks. :::note Supported Versions Rill supports connecting to StarRocks 4.0 or newer versions. ::: :::info Rill supports connecting to an existing StarRocks cluster via a read-only OLAP connector and using it to power Rill dashboards with [external tables](/developers/build/connectors/olap#external-olap-tables). ::: ## Connect to StarRocks When using StarRocks for local development, you can connect via connection parameters or by using a DSN. After selecting "Add Data", select StarRocks and fill in your connection parameters. This will automatically create the `starrocks.yaml` file in your `connectors` directory and populate the `.env` file with `STARROCKS_PASSWORD`. ### Connection Parameters ```yaml type: connector driver: starrocks host: port: 9030 username: password: "{{ .env.STARROCKS_PASSWORD }}" catalog: default_catalog database: ssl: false ``` ### Connection String (DSN) Rill can also connect to StarRocks using a DSN connection string. StarRocks uses MySQL protocol, so the connection string must follow the MySQL DSN format: ```yaml type: connector driver: starrocks dsn: "{{ .env.STARROCKS_DSN }}" ``` #### Using default_catalog For `default_catalog`, you can specify database directly in the DSN path (MySQL-style): ``` user:password@tcp(host:9030)/my_database?parseTime=true ``` #### Using external catalogs with DSN For external catalogs (Iceberg, Hive, etc.), set `catalog` and `database` as separate properties (do not include database in DSN): ```yaml type: connector driver: starrocks dsn: "user:password@tcp(host:9030)/?parseTime=true" catalog: iceberg_catalog database: my_database ``` If `catalog` is not specified, it defaults to `default_catalog`. :::warning DSN Format Only MySQL-style DSN format is supported. The `starrocks://` URL scheme is **not** supported. When using DSN, do not set `host`, `port`, `username`, `password` separately — these must be included in the DSN string. ::: ## Configuration Properties | Property | Description | Default | | ------------- | --------------------------------------------------------------------- | -------------------- | | `host` | StarRocks FE (Frontend) server hostname | Required (if no DSN) | | `port` | MySQL protocol port of StarRocks FE | `9030` | | `username` | Username for authentication | `root` | | `password` | Password for authentication | - | | `catalog` | StarRocks catalog name (for external catalogs like Iceberg, Hive) | `default_catalog` | | `database` | StarRocks database name | - | | `ssl` | Enable SSL/TLS encryption | `false` | | `dsn` | MySQL-format connection string (alternative to individual parameters) | - | | `log_queries` | Enable logging of all SQL queries (useful for debugging) | `false` | ## External Catalogs StarRocks supports external catalogs for querying data in Hive, Iceberg, Delta Lake, and other external data sources. To use an external catalog: 1. Set the `catalog` property to your external catalog name (e.g., `iceberg_catalog`) 2. Set the `database` property to the database within that catalog ```yaml type: connector driver: starrocks host: starrocks-fe.example.com port: 9030 username: analyst password: "{{ .env.STARROCKS_PASSWORD }}" catalog: iceberg_catalog database: my_database ``` ## Naming Mapping StarRocks uses a three-level hierarchy: Catalog > Database > Table. In Rill's API: | Rill Parameter | StarRocks Concept | Example | | ---------------- | ----------------- | ------------------------------------ | | `database` | Catalog | `default_catalog`, `iceberg_catalog` | | `databaseSchema` | Database | `my_database` | | `table` | Table | `my_table` | ## Creating Metrics Views When creating metrics views against StarRocks tables, use the `table` property with `database_schema` to reference your data: ```yaml type: metrics_view display_name: My Dashboard table: my_table database_schema: my_database timeseries: timestamp dimensions: - name: category column: category measures: - name: total_count expression: COUNT(*) ``` ## Troubleshooting ### Connection Issues If you encounter connection issues: 1. Verify the FE node hostname and port (default: 9030) 2. Check that your user has appropriate permissions 3. Ensure network connectivity to the StarRocks FE node 4. For SSL connections, verify SSL is enabled on the StarRocks server ### Timezone Handling All timestamp values are returned in UTC. The driver parses DATETIME values from StarRocks as UTC time. ## Known Limitations - **Read-only connector**: StarRocks is a read-only OLAP connector. Model creation and execution is not supported. :::info Need help connecting to StarRocks? If you would like to connect Rill to an existing StarRocks instance, please don't hesitate to [contact us](/contact). We'd love to help! ::: --- ##### Services --- ####### Claude --- title: Claude description: Use Claude for AI features with your own API key sidebar_label: Claude sidebar_position: 10 --- [Claude](https://www.anthropic.com/claude) is Anthropic's AI assistant, designed to be helpful, harmless, and honest. Rill supports connecting to Claude using your own API key and configuration parameters to enable AI-powered conversations and data analysis features. ## API Key Rill will use your configured Claude connector if available, or fall back to its built-in LLM service if no custom configuration is provided. Once configured, your API key will be used for AI features in both Rill Developer and Rill Cloud, including the AI Agent builder. You can configure your API key in your project's `.env` file and reference the [credentials in a connector YAML](/reference/project-files/connectors#claude). ### Claude API Key To configure Claude access, you'll need to obtain an API key and configure it in your project. 1. **Obtain your Claude API key** from the [Anthropic Console](https://console.anthropic.com/settings/keys). 2. **Create the connector YAML:** Create `connectors/claude.yaml` in your project: ```yaml type: connector driver: claude api_key: "{{ .env.CLAUDE_API_KEY }}" ``` 3. **Set up environment variable:** If configuring manually, ensure your project's `.env` file contains the key before starting Rill: ```env claude_api_key=sk-ant-... ``` 4. **Configure Claude as the default AI connector:** Add the following to your `rill.yaml` to use Claude as the AI provider for your project: ```yaml ai_connector: claude ``` For details on managing credentials across environments, see [Configure Local Credentials](/developers/build/connectors/credentials). ## Configuration Options For additional configuration options (model, temperature, token limits, etc.), see the [Claude connector reference](/reference/project-files/connectors#claude). ## Deploy to Rill Cloud Rill requires you to explicitly provide a Claude API key to use the Claude connector. See the [connector reference](/reference/project-files/connectors#claude) for details. For details on pushing and pulling credentials between environments, see [Configure Local Credentials](/developers/build/connectors/credentials#rill-env-push). --- ####### Gemini --- title: Gemini description: Use Gemini for AI features with your own API key sidebar_label: Gemini sidebar_position: 20 --- [Gemini](https://ai.google.dev/) is Google's family of AI models, designed for a wide range of reasoning, coding, and multimodal tasks. Rill supports connecting to Gemini using your own API key and configuration parameters to enable AI-powered conversations and data analysis features. ## API Key Rill will use your configured Gemini connector if available, or fall back to its built-in LLM service if no custom configuration is provided. Once configured, your API key will be used for AI features in both Rill Developer and Rill Cloud, including the AI Agent builder. You can configure your API key in your project's `.env` file and reference the [credentials in a connector YAML](/reference/project-files/connectors#gemini). ### Gemini API Key To configure Gemini access, you'll need to obtain an API key and configure it in your project. 1. **Obtain your Gemini API key** from [Google AI Studio](https://aistudio.google.com/apikey). 2. **Create the connector YAML:** Create `connectors/gemini.yaml` in your project: ```yaml type: connector driver: gemini api_key: "{{ .env.gemini_api_key }}" ``` 3. **Set up environment variable:** If configuring manually, ensure your project's `.env` file contains the key before starting Rill: ```env gemini_api_key=AI... ``` 4. **Configure Gemini as the default AI connector:** Add the following to your `rill.yaml` to use Gemini as the AI provider for your project: ```yaml ai_connector: gemini ``` For details on managing credentials across environments, see [Configure Local Credentials](/developers/build/connectors/credentials). ## Configuration Options For additional configuration options (model, temperature, token limits, etc.), see the [Gemini connector reference](/reference/project-files/connectors#gemini). ## Deploy to Rill Cloud Rill requires you to explicitly provide a Gemini API key to use the Gemini connector. See the [connector reference](/reference/project-files/connectors#gemini) for details. For details on pushing and pulling credentials between environments, see [Configure Local Credentials](/developers/build/connectors/credentials#rill-env-push). --- ####### "Service Integrations" --- title: "Service Integrations" description: Connect Rill to external services like OpenAI and Slack sidebar_position: 7 --- import ConnectorIcon from '@site/src/components/ConnectorIcon'; Service Integrations extend Rill's capabilities by connecting to third-party services. Unlike data connectors that import data into Rill, these integrations enable features like AI-powered analytics and notifications. ## AI Integrations ### Claude ### Gemini ### OpenAI
} header="AI" content="Create and define a Claude Connector with your own API key." link="/developers/build/connectors/services/claude" linkLabel="Learn more" referenceLink="claude" /> } header="Gemini" content="Create and define a Gemini Connector with your own API key." link="/developers/build/connectors/services/gemini" linkLabel="Learn more" referenceLink="gemini" /> } header="AI" content="Create and define an OpenAI Connector with your own API key." link="/developers/build/connectors/services/openai" linkLabel="Learn more" referenceLink="openai" />
## Notifications ### Slack
} header="Slack" content="Connect to Slack to send alerts and messages from Rill." link="/developers/build/connectors/services/slack" linkLabel="Learn more" referenceLink="slack" />
--- ####### OpenAI --- title: OpenAI description: Connect to OpenAI services for AI-powered features sidebar_label: OpenAI sidebar_position: 30 --- [OpenAI](https://openai.com/) provides powerful AI services including language models, embeddings, and other machine learning capabilities. Rill supports connecting to OpenAI services using your own API key and configuration parameters to enable AI-powered conversations and data analysis features. ## API Key Rill will use your configured OpenAI connector if available, or fall back to its built-in LLM service if no custom configuration is provided. Once configured, your API key will be used for AI features in both Rill Developer and Rill Cloud, including the AI Agent builder. You can configure your API key in your project's `.env` file and reference the [credentials in a connector YAML](/reference/project-files/connectors#openai). ### OpenAI API Key To configure OpenAI access, you'll need to obtain an API key and configure it in your project. 1. **Obtain your OpenAI API key** from the [OpenAI Platform](https://platform.openai.com/api-keys). 2. **Create the connector YAML:** Create `connectors/openai.yaml` in your project: ```yaml type: connector driver: openai api_key: "{{ .env.OPENAI_API_KEY }}" ``` 3. **Set up environment variable:** If configuring manually, ensure your project's `.env` file contains the key before starting Rill: ```env OPENAI_API_KEY=sk-... ``` 4. **Configure OpenAI as the default AI connector:** Add the following to your `rill.yaml` to use OpenAI as the AI provider for your project: ```yaml ai_connector: openai ``` For details on managing credentials across environments, see [Configure Local Credentials](/developers/build/connectors/credentials). ## Configuration Options For additional configuration options (model, base URL, API type, etc.), see the [OpenAI connector reference](/reference/project-files/connectors#openai). ## Deploy to Rill Cloud Rill requires you to explicitly provide an OpenAI API key to use the OpenAI connector. See the [connector reference](/reference/project-files/connectors#openai) for details. For details on pushing and pulling credentials between environments, see [Configure Local Credentials](/developers/build/connectors/credentials#rill-env-push). --- ####### Slack Integration --- title: Slack Integration description: Connect to Slack for data extraction and analytics sidebar_label: Slack sidebar_position: 40 --- ## Overview [Slack](https://slack.com/) is a popular messaging platform that allows teams to communicate and collaborate in real-time. Rill supports sending notifications to Slack channels using the [Slack API](https://api.slack.com/). This can be useful for sending alerts and reports to your team members. ## Setting up the Slack integration Rill Cloud can send alert notifications to channels and/or as direct messages. This will require a Slack application to first be created and configured in your workspace with the necessary [permission scopes](https://api.slack.com/scopes). To set up your Slack application, follow the steps provided in the [Slack documentation](https://api.slack.com/start/quickstart) and configure your app within the appropriate permissions depending on the notification type that you wish to use (see below). ### Slack channels Sending notifications to a specific channel (public or private) requires the [`chat:write`](https://api.slack.com/scopes/chat:write) scope. :::info The application will also need to be added to the channel for the notification to be sent. ::: ### Direct messages Sending notifications via a direct message requires the [`chat:write`](https://api.slack.com/scopes/chat:write), [`users:read`](https://api.slack.com/scopes/users:read), and [`users:read.email`](https://api.slack.com/scopes/users:read.email) scopes. :::tip The last two scopes are required to find the user's ID by email. ::: ## Enabling the Slack integration in your project Once the Slack integration has been set up, the Slack destination will need to be enabled on a per project basis (note - alerts can only be configured on projects deployed to Rill Cloud). This requires the `SLACK_BOT_TOKEN` connector variable to be set, which can be configured in Rill in a manner very similar to [setting credentials](/developers/deploy/deploy-credentials) for other connectors. Please use one of the available options below. ### Creating a Slack.yaml connector Please refer to our [connector YAML reference documentation](/reference/project-files/connectors#slack) for more details. ### Updating the `.env` file directly Within your project's `.env` file (i.e. `/.env`), you can set this connector variable with the Slack Bot User OAuth Token: ```shell SLACK_BOT_TOKEN= ``` Afterwards, if the project has already been deployed to Rill Cloud, you can `rill env push` to update your cloud deployment accordingly. ### Using the `rill env set` command Another option to set this connector variable within your project is to use the `rill env set` command, i.e.: ```shell rill env set SLACK_BOT_TOKEN ``` Afterwards, if the project has already been deployed to Rill Cloud, you can `rill env push` to update your cloud deployment accordingly. ### Enabling the Slack connector through `rill.yaml` You can enable the Slack "connector" within your project by updating your project's `rill.yaml` file with the following configuration: ```yaml # Rest of your rill.yaml contents connectors: - name: slack type: slack ``` ## Deploy to Rill Cloud When deploying a project to Rill Cloud, Rill requires you to explicitly provide Slack credentials used in your project. Please refer to our [connector YAML reference docs](/reference/project-files/connectors#slack) for more information. If you subsequently add sources that require new credentials (or if you simply entered the wrong credentials during the initial deploy), you can update the credentials by pushing the `Deploy` button to update your project or by running the following command in the CLI: ``` rill env push ``` --- ###### "Connect to your Data" --- title: "Connect to your Data" description: Import local files or remote data sources sidebar_label: "Connectors" sidebar_position: 0 toc_max_heading_level: 3 className: connect-connect --- import ConnectorIcon from '@site/src/components/ConnectorIcon'; ## Connection Strategies Rill offers flexible connection strategies to fit different data architectures and requirements. - ### _[Rill Managed OLAP + Data Ingestion (Default)](/developers/build/connectors/data-source)_: Use Rill's embedded **ClickHouse / DuckDB** (depending on size of data) as the OLAP engine and ingest data from external sources. Full Rill functionality is available with [some caveats](/developers/build/connectors/data-source#managed-olap-engine-caveats) depending on which embedded engine you select. :::tip Rill Defaults with DuckDB When starting Rill for the first time, Rill will auto-populate the connector with a `duckdb.yaml`. To use ClickHouse, create a managed ClickHouse connector by selecting "Add Data", then ClickHouse, and finally "Rill-managed ClickHouse" in the UI. For more information, see [Rill Managed ClickHouse](/developers/build/connectors/olap/clickhouse#rill-managed-clickhouse). ::: - ### _[Bring Your Own OLAP (BYO OLAP)](/developers/build/connectors/olap)_: For large-scale datasets (100GB+) or existing [OLAP infrastructure](/developers/build/connectors/olap#what-is-olap), connect to existing **ClickHouse**, **Druid**, **Pinot**, or **MotherDuck** instances. Use Rill's "live connectors" to ingest data directly into your OLAP engines. ## OLAP Engines ### ClickHouse ### Databricks ### Druid ### DuckDB ### DuckLake ### Google BigQuery ### MotherDuck ### Pinot ### Snowflake ### StarRocks
} content="High-performance columnar database for real-time analytics and data warehousing." link="/developers/build/connectors/olap/clickhouse" linkLabel="Learn more" referenceLink="clickhouse" /> } header="Databricks" content="Lakehouse platform with native support for metrics views as a live connector." link="/developers/build/connectors/olap/databricks" linkLabel="Learn more" referenceLink="databricks" /> } content="Real-time analytics database designed for high-performance OLAP queries." link="/developers/build/connectors/olap/druid" linkLabel="Learn more" referenceLink="druid" /> } content="DuckDB is the default engine for Rill Developer." link="/developers/build/connectors/olap/duckdb" linkLabel="Learn more" referenceLink="duckdb" /> } content="Open lakehouse format built on DuckDB — catalog in SQL, data in Parquet." link="/developers/build/connectors/olap/ducklake" linkLabel="Learn more" referenceLink="ducklake" /> } content="Connect to Google BigQuery for analytics and data warehousing with service account authentication." link="/developers/build/connectors/olap/bigquery" linkLabel="Learn more" referenceLink="bigquery" /> } content="Cloud-native DuckDB service for scalable analytics and data processing." link="/developers/build/connectors/olap/motherduck" linkLabel="Learn more" referenceLink="motherduck" /> } content="Distributed OLAP datastore for real-time analytics and business intelligence." link="/developers/build/connectors/olap/pinot" linkLabel="Learn more" referenceLink="pinot" /> } header="Snowflake" content="Cloud data warehouse with native support for metrics views as a live connector." link="/developers/build/connectors/olap/snowflake" linkLabel="Learn more" referenceLink="snowflake" /> } content="Distributed OLAP datastore for real-time analytics and business intelligence." link="/developers/build/connectors/olap/starrocks" linkLabel="Learn more" referenceLink="starrocks" />
:::tip Missing an OLAP Engine? Rill is continually evaluating additional OLAP engines that can be added. For a full list of OLAP engines that we support, refer to our [OLAP Engines](/developers/build/connectors/olap) page. If you don't see an OLAP engine that you'd like to use, please don't hesitate to [reach out](/contact)! ::: ## Data Warehouses ### Athena ### BigQuery ### Databricks ### Redshift ### Snowflake
} header="Athena" content="Connect to Amazon Athena for serverless querying of data stored in S3 using standard SQL." link="/developers/build/connectors/data-source/athena" linkLabel="Learn more" referenceLink="athena" /> } header="BigQuery" content="Connect to Google BigQuery for analytics and data warehousing with service account authentication." link="/developers/build/connectors/data-source/bigquery" linkLabel="Learn more" referenceLink="bigquery" /> } header="Databricks" content="Connect to Databricks SQL warehouses with personal access token authentication." link="/developers/build/connectors/data-source/databricks" linkLabel="Learn more" referenceLink="databricks" /> } header="Redshift" content="Connect to Amazon Redshift data warehouse with AWS credentials and support for both provisioned and serverless clusters." link="/developers/build/connectors/data-source/redshift" linkLabel="Learn more" referenceLink="redshift" /> } header="Snowflake" content="Connect to Snowflake data warehouse with support for individual credentials and JWT authentication." link="/developers/build/connectors/data-source/snowflake" linkLabel="Learn more" referenceLink="snowflake" />
## Databases ### MySQL ### PostgreSQL ### SQLite ### Supabase
} header="MySQL" content="Connect to MySQL databases with support for various authentication methods and SSL connections." link="/developers/build/connectors/data-source/mysql" linkLabel="Learn more" referenceLink="mysql" /> } header="PostgreSQL" content="Connect to PostgreSQL databases with support for SSL connections and various authentication methods." link="/developers/build/connectors/data-source/postgres" linkLabel="Learn more" referenceLink="postgresql" /> } header="SQLite" content="Connect to SQLite databases for lightweight, file-based data storage and querying." link="/developers/build/connectors/data-source/sqlite" linkLabel="Learn more" /> } header="Supabase" content="Connect to Supabase's managed PostgreSQL databases with SSL support and standard connection methods." link="/developers/build/connectors/data-source/supabase" linkLabel="Learn more" referenceLink="supabase" />
## Object Storage ### Amazon S3 ### Google Cloud Storage ### Microsoft Azure Blob Storage
} header="Amazon S3" content="Connect to Amazon S3 buckets to read data files including CSV, JSON, Parquet, and compressed formats." link="/developers/build/connectors/data-source/s3" linkLabel="Learn more" referenceLink="s3" /> } header="Google Cloud Storage" content="Google Cloud Storage provides scalable object storage and data lakes." link="/developers/build/connectors/data-source/gcs" linkLabel="Learn more" referenceLink="gcs" /> } header="Azure" content="Connect to Microsoft Azure Blob Storage to read data files with support for various formats." link="/developers/build/connectors/data-source/azure" linkLabel="Learn more" referenceLink="azure" />
## Table Formats ### Delta Lake ### Apache Iceberg
} header="Delta Lake" content="Read Delta tables directly from object storage through compatible query engines." link="/developers/build/connectors/data-source/delta" linkLabel="Learn more" /> } header="Apache Iceberg" content="Read Iceberg tables directly from object storage through compatible query engines." link="/developers/build/connectors/data-source/iceberg" linkLabel="Learn more" />
## Other Data Connectors ### External DuckDB ### Google Sheets ### HTTPS ### Local File ### Salesforce
} header="DuckDB" content="Attach your local DuckDB database to Rill's embedded database." link="/developers/build/connectors/data-source/duckdb" linkLabel="Learn more" referenceLink="external-duckdb" /> } header="Google Sheets" content="Connect to public Google Sheets to read data from spreadsheets with support for multiple sheets." link="/developers/build/connectors/data-source/googlesheets" linkLabel="Learn more" /> https://

} header="HTTPS" content="Download data from HTTP/HTTPS URLs with support for various authentication methods." link="/developers/build/connectors/data-source/https" linkLabel="Learn more" referenceLink="https" /> } header="Local File" content="Read data from local files including CSV, JSON, Parquet, and compressed formats." link="/developers/build/connectors/data-source/local-file" linkLabel="Learn more" /> } header="Salesforce" content="Connect to Salesforce to extract data from objects and queries using the Salesforce API." link="/developers/build/connectors/data-source/salesforce" linkLabel="Learn more" />
:::tip Missing a connector? We're constantly adding new data connectors. If you don't see what you need, [let us know](/contact) and we'll help you get connected. ::: ## Service Integrations ### Claude ### Gemini ### OpenAI ### Slack
} header="AI" content="Create and define a Claude Connector with your own API key." link="/developers/build/connectors/services/claude" linkLabel="Learn more" referenceLink="claude" /> } header="Gemini" content="Create and define a Gemini Connector with your own API key." link="/developers/build/connectors/services/gemini" linkLabel="Learn more" referenceLink="gemini" /> } header="AI" content="Create and define an OpenAI Connector with your own API key." link="/developers/build/connectors/services/openai" linkLabel="Learn more" referenceLink="openai" /> } header="Slack" content="Connect to Slack to send alerts and messages from Rill." link="/developers/build/connectors/services/slack" linkLabel="Learn more" referenceLink="slack" />
--- ###### Configure Local Credentials --- title: Configure Local Credentials sidebar_label: Configure Local Credentials sidebar_position: 15 --- Rill requires credentials to connect to remote data sources such as private buckets (S3, GCS, Azure), data warehouses (Snowflake, BigQuery), OLAP engines (ClickHouse, Apache Druid), or other DuckDB sources (MotherDuck). Please refer to the appropriate [connector](/developers/build/connectors) and [OLAP engine](/developers/build/connectors/olap) page for instructions to configure credentials accordingly. At a high level, configuring credentials and credential management in Rill can be broken down into three categories: - Setting credentials for Rill Developer - [Setting credentials for a Rill Cloud project](/developers/deploy/deploy-credentials) - [Pushing and pulling credentials to / from Rill Cloud]( /guide/administration/project-settings/variables-and-credentials) ## Setting credentials for Rill Developer When reading from a data source (or using a different OLAP engine), Rill will attempt to use credentials in the following order of priority: :::warning **Highly Recommended: Use .env for credentials** While Rill **can** infer credentials from your local environment (AWS CLI, Azure CLI, Google Cloud CLI), **we HIGHLY recommend explicitly configuring credentials in your `.env` file** for better security, reliability, and portability. Environment-inferred credentials may vary across different setups and may not work consistently across different environments, team members, or when deploying to Rill Cloud. ::: 1. **Credentials referenced in connection strings or DSN within YAML files (RECOMMENDED)** - The UI creates YAML configurations that reference credentials from your `.env` file using templating (see [Connector YAML](/reference/project-files/connectors) for more details) 2. **Credentials passed in as variables** - When starting Rill Developer via `rill start --env key=value` (see [passing environment variables](/developers/build/connectors/templating) for more details) 3. **Credentials configured via CLI** - For [AWS](/developers/build/connectors/data-source/s3#method-4-local-aws-credentials-local-development-only) / [Azure](/developers/build/connectors/data-source/azure#method-5-azure-cli-authentication-local-development-only) / [Google Cloud](/developers/build/connectors/data-source/gcs#method-4-local-google-cloud-cli-credentials) - **NOT RECOMMENDED for production use** For more details, please refer to the corresponding [connector](/developers/build/connectors) or [OLAP engine](/developers/build/connectors/olap) page. :::note Ensuring security of credentials in use If you plan to deploy a project (to Rill Cloud), it is not recommended to pass in credentials directly through the local connection string or DSN as your credentials will then be checked in directly to your Git repository (and thus accessible by others). To ensure better security, credentials should be passed in as a variable, configured locally, or specified in the project's local `.env` file (which is part of `.gitignore` and thus won't be included). ::: ## Variables Project variables work exactly the same way as credentials and can be defined when starting rill via `--env key=value`, set in the `.env` file in the project directory, or defined in the rill.yaml. ### Passing Environment Variables Environment variables in your local shell are not automatically passed to Rill Developer. You can pass these to Rill Developer using the `--env` flag. This is useful for referencing existing environment variables without exposing their values in your command history or configuration files. For example, to use the `AWS_ACCESS_KEY_ID` environment variable: ```bash rill start --env AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID ``` This will pull in the value of AWS_ACCESS_KEY_ID from your environment and make it available to your Rill project without exposing the credential value in your command or .env file. In Rill, you would reference it using: `"{{ .env.AWS_ACCESS_KEY_ID }}"`. :::warning Development Only Passing environment variables via rill start --env is intended for local development purposes only. These credentials are not automatically passed to Rill Cloud when you deploy your project. For production deployments, you must configure credentials separately in Rill Cloud using the project settings or rill env push. ::: ### What is a `.env` file? A `.env` file is a plain text file that stores environment variables and credentials for your Rill project. It's located in your project's root directory and follows a simple `key=value` format. The `.env` file serves several important purposes: - **Security**: Keeps sensitive credentials out of your codebase and Git repository (`.env` files are automatically ignored by `.gitignore`) - **Consistency**: Provides a standardized way to manage credentials across different environments (local development, staging, production) - **Integration**: Works seamlessly with Rill's templating system, allowing YAML files to reference credentials using `"{{ .env.VARIABLE_NAME }}"` syntax Example `.env` file: ```bash # AWS S3 credentials AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY # Google Cloud credentials GOOGLE_APPLICATION_CREDENTIALS={"type":"service_account","project_id":"my-project"} # Database connections POSTGRES_PASSWORD=mypassword SNOWFLAKE_PASSWORD=mysnowflakepassword # Custom variables my_custom_variable=some_value ``` When creating any connector in Rill via the UI, these will be **automatically generated** in the `.env` file. Additional variables can then be usable and referenceable for [templating](/developers/build/connectors/templating) purposes in the local instance of your project. ### Credentials Naming Schema When you create a connector through Rill's UI, credentials are automatically saved to your `.env` file using a standardized naming convention: #### Generic Credentials (Shared Across Connectors) Common cloud provider credentials use standard names without a driver prefix: | Property | Environment Variable | |----------|---------------------| | Google Application Credentials | `GOOGLE_APPLICATION_CREDENTIALS` | | AWS Access Key ID | `AWS_ACCESS_KEY_ID` | | AWS Secret Access Key | `AWS_SECRET_ACCESS_KEY` | | Azure Storage Connection String | `AZURE_STORAGE_CONNECTION_STRING` | | Azure Storage Key | `AZURE_STORAGE_KEY` | | Azure Storage SAS Token | `AZURE_STORAGE_SAS_TOKEN` | | Snowflake Private Key | `PRIVATE_KEY` | #### Driver-Specific Credentials Credentials specific to a database driver use the `DRIVER_PROPERTY` format: | Driver | Property | Environment Variable | |--------|----------|---------------------| | PostgreSQL | password | `POSTGRES_PASSWORD` | | PostgreSQL | dsn | `POSTGRES_DSN` | | MySQL | password | `MYSQL_PASSWORD` | | Snowflake | password | `SNOWFLAKE_PASSWORD` | | ClickHouse | password | `CLICKHOUSE_PASSWORD` | #### Handling Multiple Connectors When you create multiple connectors that use the same credential type, Rill automatically appends a numeric suffix to avoid conflicts: ```bash # First BigQuery connector GOOGLE_APPLICATION_CREDENTIALS={"type":"service_account",...} # Second BigQuery connector GOOGLE_APPLICATION_CREDENTIALS_1={"type":"service_account",...} # Third BigQuery connector GOOGLE_APPLICATION_CREDENTIALS_2={"type":"service_account",...} ``` This ensures each connector can reference its own credentials without overwriting existing ones. #### Referencing Variables in YAML Use the `{{ .env.VARIABLE_NAME }}` syntax to reference environment variables in your connector YAML files: ```yaml google_application_credentials: "{{ .env.GOOGLE_APPLICATION_CREDENTIALS }}" password: "{{ .env.POSTGRES_PASSWORD }}" aws_access_key_id: "{{ .env.AWS_ACCESS_KEY_ID }}" ``` #### Case-Insensitive Variable Lookups The `{{ env "VAR_NAME" }}` function (note: not `{{ .env.VAR_NAME }}`) provides case-insensitive variable lookups, which can be useful when variable names may have inconsistent casing: ```yaml # All of these will match POSTGRES_PASSWORD in your .env file: password: '{{ env "POSTGRES_PASSWORD" }}' password: '{{ env "postgres_password" }}' password: '{{ env "Postgres_Password" }}' ``` Note that `{{ .env.VAR_NAME }}` is **case-sensitive** — the variable name must match exactly. :::note Legacy Naming Convention Older projects may use the `connector..` syntax (e.g., `connector.druid.dsn`, `connector.clickhouse.dsn`). This format is still supported for backwards compatibility. ::: :::tip Avoid committing sensitive information to Git It's never a good idea to commit sensitive information to Git and it goes against security best practices. Similar to credentials, if there are sensitive variables that you don't want to commit publicly to your `rill.yaml` configuration file (and thus potentially accessible by others), it's recommended to set them in your `.env` file directly and/or use `rill env set` via the CLI (and then optionally push / pull them as necessary). ::: ## Cloning an Existing Project from Rill Cloud If you cloned the project using `rill project clone ` and are an admin of that project, the credentials will be pulled automatically. Note that there are some limitations with monorepos where credentials may not be pulled correctly. In those cases, credentials are also pulled when running `rill start`, assuming you have already authenticated via the CLI with `rill login`. For a detailed guide, see our [clone a project guide](/developers/tutorials/clone-a-project). ## Pulling Credentials and Variables from a Deployed Project on Rill Cloud If you are making changes to an already deployed instance from Rill Cloud, it is possible to **pull** the credentials and variables from the Rill Cloud to your local instance of Rill Developer. If you've made any changes to the credentials, don't forget to run `rill env push` to push the variable changes to the project, or manually change these in the project's settings page. ### rill env pull For projects that have been deployed to Rill Cloud, an added benefit of our Rill Developer-Cloud architecture is that credentials that have been configured can be pulled locally for easier reuse (instead of having to manually reconfigure these credentials in Rill Developer). To do this, you can run `rill env pull` from your project's root directory to retrieve the latest credentials (after cloning the project's git repository to your local environment). ![Rill Environment Pull](/img/build/credentials/rill-env-pull.png) :::info Overriding local credentials Please note when you run `rill env pull`, Rill will *automatically override any existing credentials or variables* that have been configured in your project's `.env` file if there is a match in the key name. This may result in unexpected behavior if you are using different credentials locally. ::: ### rill env push As a project admin, you can use `rill env push` to push your credentials to your Rill Cloud project. - Rill Cloud will use the specified credentials and variables in this `.env` file for the deployed project. - Other users will also be able to use `rill env pull` to retrieve these defined credentials for local use (with Rill Developer). :::warning Overriding Cloud credentials If a credential and/or variable has already been configured in Rill Cloud, Rill will warn you about overriding if you attempt to push a new value in your `.env` file. This is because overriding credentials can impact your deployed project and/or other users (if they pull these credentials locally). ![Rill Environment Push](/img/build/credentials/rill-env-push.png) ::: --- ###### Development/Production Connector Environments --- title: Development/Production Connector Environments description: Development and Production Setup sidebar_label: Development/Production Connectors sidebar_position: 19 --- Templating serves several important purposes in your data workflow: - **Cost Management**: Avoid running expensive queries on production data during development - **Environment Separation**: Keep development and production data sources completely separate - **Testing Safety**: Test your models and transformations without affecting production data :::note Rill Developer defaults to 'dev' Unless explicitly defined, Rill Developer will use a `dev` environment. If you want to emulate production locally, you can do so via `rill start --environment prod`. ::: ## Setting Up Environment Variables You can set up environment variables in several locations in Rill. Please review our [configure local credentials documentation](/developers/build/connectors/credentials#setting-credentials-for-rill-developer) for more information. ## Referencing Environment Variables Reference environment variables in your YAML files using the `{{ .env.VARIABLE_NAME }}` syntax: ```yaml password: "{{ .env.POSTGRES_PASSWORD }}" google_application_credentials: "{{ .env.GOOGLE_APPLICATION_CREDENTIALS }}" aws_access_key_id: "{{ .env.AWS_ACCESS_KEY_ID }}" ``` :::tip Case-Insensitive Lookups The `'{{ env "VAR_NAME" }}'` function provides case-insensitive variable lookups, so `'{{ env "my_var" }}'` will match `MY_VAR` in your `.env` file. **Note:** If your `.env` file contains multiple variables that differ only by case (e.g., both `my_var` and `MY_VAR`), the behavior is undefined. Avoid defining variables with the same name in different cases. ::: ## Environment-Specific Connectors The most common use case for connector templating is defining separate databases for your development and production operations. This approach gives you the freedom to experiment, test, and iterate on your models without the risk of accidentally modifying or corrupting your production data. ### Example: ClickHouse Connector with Environment Separation Here's how you can configure a ClickHouse connector to use different environments: ```yaml type: connector driver: clickhouse dev: dsn: "clickhouse://user:password@localhost:9000/dev_database" # ClickHouse connection DSN # Production environment configuration prod: host: "{{ .env.CLICKHOUSE_HOST }}" port: "{{ .env.CLICKHOUSE_PORT }}" database: "{{ .env.CLICKHOUSE_DATABASE }}" username: "{{ .env.CLICKHOUSE_USERNAME }}" password: "{{ .env.CLICKHOUSE_PASSWORD }}" ssl: true cluster: "{{ .env.CLICKHOUSE_CLUSTER }}" ``` In this example: - **Development**: Uses a managed ClickHouse instance that Rill starts locally - **Production**: Connects to your existing ClickHouse cluster using environment variables for secure configuration :::warning Managing development and production credentials Some connectors will reference two unique databases and require two unique credentials for development and production. In this case, you can either define a unique environmental variable for both and reference them separately in the connector. For example: ```yaml "{{ .env.DEV_CLICKHOUSE_USERNAME }}" "{{ .env.PROD_CLICKHOUSE_USERNAME }}" ``` Or, by creating a separate connector altogether. ```yaml # /connectors/dev_clickhouse.yaml # /connectors/prod_clickhouse.yaml ``` ::: ## Environment-Specific Data Source Location Rill allows you to configure different data source locations for different environments (development, staging, production) using templating. This enables you to work with development datasets locally while pointing to production data sources in cloud deployments. ```yaml type: connector driver: bigquery dev: project_id: rilldata_dev project_id: rilldata google_application_credentials: "{{ .env.GOOGLE_APPLICATION_CREDENTIALS }}" ``` ```yaml type: connector driver: postgres dev: dsn: "postgres://user:password@localhost:5432/dev_database" dsn: "postgres://user:password@localhost:5432/database" ``` ## Additional Resources - [Performance Optimization Guide](/developers/tutorials/performance) - [Official docs](https://pkg.go.dev/text/template) (Go) - [Learn Go Template Syntax](https://developer.hashicorp.com/nomad/tutorials/templates/go-template-syntax) (HashiCorp) - [Sprig Function Documentation](http://masterminds.github.io/sprig/) --- #### Custom Apis --- ###### Calling Custom APIs --- title: Calling Custom APIs description: How to call and consume your custom APIs from external applications sidebar_label: Calling APIs sidebar_position: 70 --- Custom APIs are exposed as HTTP endpoints that return JSON. You can call them from any language or tool that supports HTTP requests. ## API endpoints ### Rill Cloud ``` https://api.rilldata.com/v1/organizations//projects//runtime/api/ ``` This targets the project's production deployment. To call the API on the deployment for a specific branch (e.g. a dev/preview deployment), insert `/branch/` before `/runtime`: ``` https://api.rilldata.com/v1/orgs//projects//branch//runtime/api/ ``` ### Local development ``` http://localhost:9009/v1/instances/default/api/ ``` Where `` is the name of your API file without the `.yaml` extension (e.g., `my-api.yaml` → `my-api`). ## Making requests Custom APIs accept both GET and POST requests. ### GET with query parameters ```bash curl "https://api.rilldata.com/v1/organizations//projects//runtime/api/my-api?domain=google.com&limit=10" \ -H "Authorization: Bearer " ``` ### POST with JSON body ```bash curl -X POST "https://api.rilldata.com/v1/organizations//projects//runtime/api/my-api" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"domain": "google.com", "limit": 10}' ``` Both methods produce the same result. If you provide both query parameters and a JSON body, query parameters take precedence. ### Response format APIs return a JSON array of objects: ```json [ {"publisher": "Facebook", "domain": "google.com", "total": 15234}, {"publisher": "Google", "domain": "google.com", "total": 12876} ] ``` ## Testing locally Local development does not require authentication: ```bash # GET request curl "http://localhost:9009/v1/instances/default/api/my-api?domain=google.com" # POST request curl -X POST http://localhost:9009/v1/instances/default/api/my-api \ -H "Content-Type: application/json" \ -d '{"domain": "google.com"}' ``` :::note User attributes (`{{ .user.* }}`) are not available during local testing since no authentication token is provided. To test with user attributes, deploy to Rill Cloud and use a service token with [custom attributes](/developers/build/custom-apis/security#custom-attributes-on-service-tokens). ::: ## Authentication Rill Cloud APIs require a bearer token in the `Authorization` header. ### For development and testing Create a [user token](/guide/administration/access-tokens/user-tokens) (inherits your personal permissions): ```bash rill token issue --display-name "API Testing" # Returns: rill_usr_... curl "https://api.rilldata.com/v1/organizations//projects//runtime/api/my-api" \ -H "Authorization: Bearer rill_usr_..." ``` ### For production systems Create a [service token](/guide/administration/access-tokens/service-tokens) with optional custom attributes: ```bash rill service create my-api-service \ --project my-project \ --project-role viewer \ --attributes '{"customer_id": "acme-corp"}' # Returns: rill_svc_... ``` Custom attributes on the token are available in your API templates as `{{ .user.customer_id }}`. See [Security & Access Control](/developers/build/custom-apis/security) for details on how to use custom attributes to build multi-tenant APIs. :::note If you need short-lived tokens with custom attributes for end users, use the [ephemeral token feature](/developers/build/custom-apis/security#issuing-ephemeral-tokens) instead of creating a service token per user. ::: :::tip Token Documentation For full guidance on token types, roles, and management: - **[User Tokens](/guide/administration/access-tokens/user-tokens)** — Personal access tokens for development - **[Service Tokens](/guide/administration/access-tokens/service-tokens)** — Long-lived tokens for production systems - **[Roles and Permissions](/guide/administration/users-and-access/roles-permissions)** — Understand access levels ::: ## OpenAPI schema Rill automatically generates an OpenAPI spec for your project. Download it to generate typed clients: ```bash # From Rill Cloud curl "https://api.rilldata.com/v1/organizations//projects//runtime/api/openapi" \ -H "Authorization: Bearer " \ -o openapi.json # Locally curl http://localhost:9009/v1/instances/default/api/openapi -o openapi.json ``` See [OpenAPI Documentation](/developers/build/custom-apis/openapi) for how to add request and response schemas to your API definitions. --- ###### Custom APIs --- title: Custom APIs description: Expose your Rill data as HTTP API endpoints sidebar_label: Custom APIs sidebar_position: 10 --- Rill lets you create custom API endpoints that return data from your project as JSON over HTTP. Define a YAML file, write a SQL query, and you have an API — no backend code required. Custom APIs are ideal for: - **Powering internal tools** — feed Rill data into dashboards, Slack bots, or scripts - **Building customer-facing integrations** — expose filtered data to external applications - **Automating workflows** — pull data into CI/CD pipelines, scheduled jobs, or ETL processes - **Multi-tenant data access** — serve different data to different customers using [custom attributes](/developers/build/custom-apis/security) ## API types Rill supports two types of custom APIs: | Type | Best for | Query target | |------|----------|-------------| | [**SQL API**](/developers/build/custom-apis/sql) | Querying models, tables, or external databases directly | Any model, table, or external connector (DuckDB, BigQuery, Snowflake, etc.) | | [**Metrics SQL API**](/developers/build/custom-apis/metrics-sql) | Querying metrics views using dimension and measure names | Metrics views (inherits security policies automatically) | ## Your first custom API ### 1. Create an API file Create a YAML file in your project's `apis/` directory. For example, `apis/top-publishers.yaml`: ```yaml type: api sql: | SELECT publisher, COUNT(*) as total_records FROM ad_bids GROUP BY publisher ORDER BY total_records DESC LIMIT 10 ``` ### 2. Test it locally With Rill Developer running (`rill start`), call your API at: ```bash curl "http://localhost:9009/v1/instances/default/api/top-publishers" ``` You'll get a JSON response: ```json [ {"publisher": "Facebook", "total_records": 15234}, {"publisher": "Google", "total_records": 12876}, {"publisher": "Microsoft", "total_records": 9541} ] ``` :::note Local development does not require authentication. When deployed to Rill Cloud, all API calls require a bearer token. ::: ### 3. Deploy and call from Rill Cloud After deploying your project, call the API with authentication: ```bash curl "https://api.rilldata.com/v1/organizations//projects//runtime/api/top-publishers" \ -H "Authorization: Bearer " ``` To test against the deployment for a specific branch (e.g. a dev/preview deployment), insert `/branch/` before `/runtime`: ```bash curl "https://api.rilldata.com/v1/orgs//projects//branch//runtime/api/top-publishers" \ -H "Authorization: Bearer " ``` See [Calling APIs](/developers/build/custom-apis/calling) for full details on authentication and calling APIs. ## Make it dynamic Add [templating](/developers/build/custom-apis/templating) to make your API accept parameters: ```yaml type: api sql: | SELECT publisher, COUNT(*) as total_records FROM ad_bids WHERE domain = '{{ .args.domain }}' GROUP BY publisher ORDER BY total_records DESC LIMIT {{ default 10 .args.limit }} ``` Call it with query parameters: ```bash curl "http://localhost:9009/v1/instances/default/api/top-publishers?domain=google.com&limit=5" ``` ## Add access control Use [security rules](/developers/build/custom-apis/security) to control who can access your API and what data they see: ```yaml type: api sql: | SELECT publisher, domain, COUNT(*) as total_records FROM ad_bids WHERE customer_id = '{{ .user.customer_id }}' GROUP BY publisher, domain security: access: true ``` Each customer sees only their own data, based on the [custom attributes](/developers/build/custom-apis/security#custom-attributes-on-service-tokens) in their token. ## Next steps - [**Templating**](/developers/build/custom-apis/templating) — Dynamic arguments, user attributes, and conditional logic - [**Security & Access Control**](/developers/build/custom-apis/security) — Custom attributes, multi-tenant patterns, and access rules - [**OpenAPI Documentation**](/developers/build/custom-apis/openapi) — Document your APIs with OpenAPI specs - [**Calling APIs**](/developers/build/custom-apis/calling) — HTTP endpoints, authentication, and client generation --- ###### Metrics SQL APIs --- title: Metrics SQL APIs description: Create custom APIs that query metrics views using dimension and measure names sidebar_label: Metrics SQL APIs sidebar_position: 30 --- Metrics SQL APIs let you query [metrics views](/developers/build/metrics-view) using the dimension and measure names you've already defined. Instead of writing raw SQL against underlying tables, you write queries using your metrics view's semantic layer — and security policies are inherited automatically. ## Basic syntax Create a YAML file in your project's `apis/` directory: ```yaml type: api metrics_sql: SELECT publisher, domain, total_records FROM ad_bids_metrics ``` ## How Metrics SQL works Metrics SQL transforms your query by replacing dimension and measure names with their underlying expressions. Consider this metrics view: ```yaml # metrics/ad_bids_metrics.yaml type: metrics_view title: Ad Bids model: ad_bids timeseries: timestamp dimensions: - name: publisher expression: toUpper(publisher) - name: domain column: domain measures: - name: total_records expression: COUNT(*) - name: avg_bid_price expression: AVG(bid_price) ``` When you write: ```sql SELECT publisher, domain, total_records FROM ad_bids_metrics ``` Rill translates this to: ```sql SELECT toUpper(publisher) AS publisher, domain AS domain, COUNT(*) AS total_records FROM ad_bids GROUP BY publisher, domain ``` This means you write simple queries using business-friendly names, while Rill handles the underlying SQL complexity. ## Why use Metrics SQL over raw SQL? | Benefit | Description | |---------|-------------| | **Reuse definitions** | Query using dimension/measure names defined once in your metrics view | | **Automatic security** | Row-level security policies from the metrics view are applied automatically | | **Simpler queries** | No need to remember complex expressions — just use names like `total_records` | | **Consistency** | All APIs and dashboards use the same metric definitions | ## Examples ### Filtering with WHERE ```yaml type: api metrics_sql: | SELECT publisher, total_records FROM ad_bids_metrics WHERE domain = 'google.com' ORDER BY total_records DESC LIMIT 10 ``` ### Aggregation with HAVING ```yaml type: api metrics_sql: | SELECT publisher, total_records, avg_bid_price FROM ad_bids_metrics HAVING total_records > 1000 ORDER BY avg_bid_price DESC ``` ### Pagination ```yaml type: api metrics_sql: | SELECT publisher, domain, total_records FROM ad_bids_metrics ORDER BY total_records DESC LIMIT 20 OFFSET 40 ``` ## Security inheritance When you use Metrics SQL, any [security policies](/developers/build/metrics-view/security) defined on the metrics view are automatically enforced. For example, if your metrics view has: ```yaml # In your metrics view security: access: true row_filter: "domain = '{{ .user.domain }}'" ``` Then a Metrics SQL API querying this view will automatically filter rows based on the user's domain — no additional configuration needed in the API definition. This is one of the key advantages of Metrics SQL over raw SQL APIs. See [Security & Access Control](/developers/build/custom-apis/security) for more on how security works with custom APIs. ## Supported SQL features For a full reference on supported SQL syntax (SELECT, WHERE, HAVING, ORDER BY, LIMIT, OFFSET) and current limitations, see the [Metrics SQL language reference](/developers/build/metrics-view/metrics-sql). ## Adding dynamic behavior Metrics SQL APIs support the same [templating](/developers/build/custom-apis/templating) as SQL APIs: ```yaml type: api metrics_sql: | SELECT publisher, total_records FROM ad_bids_metrics {{ if hasKey .args "domain" }} WHERE domain = '{{ .args.domain }}' {{ end }} ORDER BY total_records DESC LIMIT {{ default 25 .args.limit }} ``` See [Dynamic Queries with Templating](/developers/build/custom-apis/templating) for the full guide. --- ###### OpenAPI Documentation --- title: OpenAPI Documentation description: Add OpenAPI specs to your custom APIs for documentation and client generation sidebar_label: OpenAPI Documentation sidebar_position: 60 --- Rill automatically generates an OpenAPI specification for your project that combines built-in APIs with your custom API definitions. You can add request and response schemas to your APIs for better documentation and typed client generation. ## Adding an OpenAPI spec to your API Add an `openapi` block to your API definition with `request_schema` and `response_schema`: ```yaml type: api sql: | SELECT publisher, COUNT(*) as total_records FROM ad_bids WHERE domain = '{{ .args.domain }}' {{ if hasKey .args "publisher" }} AND publisher = '{{ .args.publisher }}' {{ end }} ORDER BY total_records DESC LIMIT {{ default 25 .args.limit }} OFFSET {{ default 0 .args.offset }} openapi: summary: Get ad bid statistics by publisher, filtered by domain request_schema: type: object required: - domain properties: domain: type: string description: Domain to filter results by publisher: type: string description: Optional publisher filter limit: type: integer description: Number of results to return (default 25) offset: type: integer description: Offset for pagination (default 0) response_schema: type: object properties: publisher: type: string description: Publisher name total_records: type: integer description: Total number of ad bid records ``` ### OpenAPI fields | Field | Description | |-------|-------------| | `openapi.summary` | Short description of what the API does (appears in OpenAPI docs) | | `openapi.request_schema` | JSON Schema describing the request parameters | | `openapi.response_schema` | JSON Schema describing a single row in the response array | Schemas follow the [JSON Schema](https://json-schema.org/) format. You can use `type`, `required`, `properties`, `description`, and other standard JSON Schema keywords. ## Downloading the OpenAPI spec ### Locally ```bash curl http://localhost:9009/v1/instances/default/api/openapi -o openapi.json ``` ### From Rill Cloud ```bash curl "https://api.rilldata.com/v1/organizations//projects//runtime/api/openapi" \ -H "Authorization: Bearer " \ -o openapi.json ``` To download the spec for the deployment of a specific branch (e.g. a dev/preview deployment), insert `/branch/` before `/runtime`: ```bash curl "https://api.rilldata.com/v1/orgs//projects//branch//runtime/api/openapi" \ -H "Authorization: Bearer " \ -o openapi.json ``` The generated spec includes all your custom APIs with their schemas, plus Rill's built-in API endpoints. ## Generating typed clients Use the downloaded OpenAPI spec with any code generation tool to create typed clients for your language: **JavaScript/TypeScript** (using [openapi-typescript](https://github.com/openapi-ts/openapi-typescript)): ```bash npx openapi-typescript openapi.json -o ./src/api-types.ts ``` **Python** (using [openapi-python-client](https://github.com/openapi-generators/openapi-python-client)): ```bash openapi-python-client generate --path openapi.json ``` **Go** (using [oapi-codegen](https://github.com/oapi-codegen/oapi-codegen)): ```bash oapi-codegen -package api openapi.json > api/client.go ``` ## Full example Here's a complete API with OpenAPI documentation, templating, and security: ```yaml type: api metrics_sql: | SELECT publisher, domain, total_records, avg_bid_price FROM ad_bids_metrics WHERE domain = '{{ .args.domain }}' {{ if hasKey .args "publisher" }} AND publisher = '{{ .args.publisher }}' {{ end }} ORDER BY total_records DESC LIMIT {{ default 25 .args.limit }} OFFSET {{ default 0 .args.offset }} security: access: true openapi: summary: Query ad bid metrics by domain with optional publisher filter request_schema: type: object required: - domain properties: domain: type: string description: Domain to filter metrics by (e.g., "google.com") publisher: type: string description: Optional publisher to filter by (e.g., "Facebook") limit: type: integer description: Max results to return (default 25, max 1000) offset: type: integer description: Pagination offset (default 0) response_schema: type: object properties: publisher: type: string description: Publisher name domain: type: string description: Domain name total_records: type: integer description: Total number of records avg_bid_price: type: number description: Average bid price in USD ``` --- ###### Security & Access Control --- title: Security & Access Control description: Control who can access your APIs and what data they see using security rules and custom attributes sidebar_label: Security & Access Control sidebar_position: 50 --- Rill's custom APIs support fine-grained access control through security rules and custom attributes on tokens. You can restrict who can call an API, and filter the data each caller sees — all without writing backend code. ## API access rules Control who can access an API using the `security` block: ### Allow all authenticated users ```yaml type: api sql: SELECT publisher, COUNT(*) as total FROM ad_bids GROUP BY publisher security: access: true ``` ### Restrict to admins only ```yaml type: api sql: | SELECT publisher, SUM(revenue) as total_revenue FROM ad_bids GROUP BY publisher security: access: "{{ .user.admin }}" ``` Only users with admin permissions on the project can call this API. Non-admins receive a 403 Forbidden response. ### Restrict by custom attribute ```yaml type: api sql: SELECT * FROM internal_reports security: access: "{{ eq .user.tier \"enterprise\" }}" ``` Only users whose token has `tier: "enterprise"` can access this endpoint. ## Custom attributes on service tokens Custom attributes are key-value pairs you attach to [service tokens](/guide/administration/access-tokens/service-tokens). When a service token is used to call an API, its attributes are available in templates as `{{ .user. }}`. ### Creating a service token with attributes ```bash rill service create acme-api \ --project my-project \ --project-role viewer \ --attributes '{"customer_id": "acme-corp", "region": "us-west", "tier": "premium"}' ``` This creates a token with three custom attributes: `customer_id`, `region`, and `tier`. ### Updating attributes on an existing service ```bash rill service edit acme-api \ --attributes '{"customer_id": "acme-corp", "region": "eu-central", "tier": "enterprise"}' ``` ### Common attribute patterns | Attribute | Use case | |-----------|----------| | `customer_id` | Multi-tenant data isolation | | `region` | Geographic data filtering | | `department` | Departmental access control | | `tier` | Feature gating (free, premium, enterprise) | | `environment` | Environment-specific data (production, staging) | ## How attributes flow through the system When an API is called with a service token, here's what happens: ``` 1. Service token created with attributes: {"customer_id": "acme"} ↓ 2. API call with bearer token ↓ 3. Rill extracts attributes from the token into JWT claims ↓ 4. Template engine makes attributes available as {{ .user.customer_id }} ↓ 5. SQL query is rendered with the actual values ↓ 6. Query executes and returns filtered results ``` ## End-to-end example: multi-tenant API This walkthrough shows how to build an API that serves different data to different customers. ### Step 1: Create the API Create `apis/customer-orders.yaml`: ```yaml type: api sql: | SELECT order_id, product_name, quantity, total_price, order_date FROM orders WHERE customer_id = '{{ .user.customer_id }}' ORDER BY order_date DESC LIMIT {{ default 50 .args.limit }} OFFSET {{ default 0 .args.offset }} security: access: true ``` ### Step 2: Create service tokens for each customer ```bash # Token for Acme Corp rill service create acme-api \ --project my-project \ --project-role viewer \ --attributes '{"customer_id": "acme-corp"}' # Returns: rill_svc_abc123... # Token for Globex Inc rill service create globex-api \ --project my-project \ --project-role viewer \ --attributes '{"customer_id": "globex-inc"}' # Returns: rill_svc_def456... ``` ### Step 3: Call the API **Acme sees only their orders:** ```bash curl "https://api.rilldata.com/v1/organizations/my-org/projects/my-project/runtime/api/customer-orders" \ -H "Authorization: Bearer rill_svc_abc123..." ``` ```json [ {"order_id": "A-1001", "product_name": "Widget Pro", "quantity": 50, "total_price": 2500, "order_date": "2025-01-15"}, {"order_id": "A-1002", "product_name": "Gadget Plus", "quantity": 25, "total_price": 1250, "order_date": "2025-01-14"} ] ``` **Globex sees only their orders:** ```bash curl "https://api.rilldata.com/v1/organizations/my-org/projects/my-project/runtime/api/customer-orders" \ -H "Authorization: Bearer rill_svc_def456..." ``` ```json [ {"order_id": "G-2001", "product_name": "Sprocket X", "quantity": 100, "total_price": 5000, "order_date": "2025-01-16"}, {"order_id": "G-2002", "product_name": "Bolt Kit", "quantity": 200, "total_price": 800, "order_date": "2025-01-13"} ] ``` Same API, same endpoint — different data based on the token's `customer_id` attribute. ## Admin vs non-admin patterns Use `{{ .user.admin }}` to expose different data or behavior based on the user's role: ### Show extra columns for admins ```yaml type: api sql: | SELECT publisher, domain, COUNT(*) as impressions {{ if .user.admin }} , SUM(revenue) as total_revenue , AVG(cost_per_click) as avg_cpc {{ end }} FROM ad_bids GROUP BY publisher, domain ORDER BY impressions DESC LIMIT 50 ``` ### Remove filters for admins ```yaml type: api sql: | SELECT publisher, domain, COUNT(*) as total FROM ad_bids WHERE 1=1 {{ if (not .user.admin) }} AND customer_id = '{{ .user.customer_id }}' {{ end }} GROUP BY publisher, domain ``` Admins see all data across all customers; non-admins see only their customer's data. ## Metrics SQL security inheritance When using [Metrics SQL APIs](/developers/build/custom-apis/metrics-sql), security policies defined on the metrics view are automatically enforced. You don't need to add a `security` block to the API — it inherits the metrics view's policies: ```yaml # metrics/ad_bids_metrics.yaml type: metrics_view model: ad_bids security: access: true row_filter: "customer_id = '{{ .user.customer_id }}'" ``` ```yaml # apis/customer-metrics.yaml type: api metrics_sql: | SELECT publisher, total_records FROM ad_bids_metrics ORDER BY total_records DESC ``` The `row_filter` from the metrics view is automatically applied — each customer only sees their own data, even though the API definition doesn't mention security at all. ## Skipping nested security By default, when an API queries a metrics view, Rill enforces the security policies on both the API itself and the underlying metrics view. In some cases, you may want the API to handle all access control itself and skip checks on nested resources: ```yaml type: api sql: | SELECT * FROM sensitive_model WHERE access_level <= {{ .user.access_level }} security: access: true skip_nested_security: true ``` Use `skip_nested_security: true` when your API already handles all necessary access control in its own query logic. ## Issuing ephemeral tokens For applications that need to issue short-lived tokens to end users (e.g., for embedded dashboards or temporary API access), service tokens can issue ephemeral tokens with custom user attributes: ```bash curl -X POST "https://api.rilldata.com/v1/orgs//projects//credentials" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "attributes": { "email": "user@acme.com", "customer_id": "acme-corp", "department": "engineering" }, "ttl_seconds": 3600 }' ``` The response contains a short-lived JWT that can be used to call APIs with those attributes. See [Service Tokens](/guide/administration/access-tokens/service-tokens#issuing-ephemeral-tokens) for details. --- ###### SQL APIs --- title: SQL APIs description: Create custom APIs with SQL queries against models, tables, and external databases sidebar_label: SQL APIs sidebar_position: 20 --- SQL APIs let you write SQL queries and expose them as HTTP endpoints. By default, queries execute against your project's default [OLAP connector](/developers/build/connectors/olap) (e.g., DuckDB, ClickHouse). You can also query external databases like BigQuery, Snowflake, and Postgres by specifying a different `connector`. ## Basic syntax Create a YAML file in your project's `apis/` directory: ```yaml type: api sql: SELECT publisher, domain, bid_price FROM ad_bids LIMIT 100 ``` This queries your default OLAP connector and returns the results as JSON. ### Multi-line queries For longer queries, use YAML multi-line syntax: ```yaml type: api sql: | SELECT publisher, domain, COUNT(*) as impressions, AVG(bid_price) as avg_bid FROM ad_bids WHERE timestamp >= '2024-01-01' GROUP BY publisher, domain ORDER BY impressions DESC LIMIT 50 ``` ## Querying models and tables By default, SQL APIs execute against your project's default [OLAP connector](/developers/build/connectors/olap). You can query any [model](/developers/build/models) or source table: ```yaml # Query a model type: api sql: SELECT * FROM my_model WHERE status = 'active' LIMIT 100 ``` ```yaml # Query with joins across models type: api sql: | SELECT o.order_id, o.total, c.name as customer_name FROM orders o JOIN customers c ON o.customer_id = c.id ORDER BY o.total DESC LIMIT 25 ``` ## Querying external databases You can query external databases directly by specifying a `connector`. This lets you access data in real-time without ingesting it into Rill. ### Data warehouses **Athena:** ```yaml type: api connector: athena sql: SELECT * FROM s3_data_table WHERE event_date >= '2024-01-01' LIMIT 100 ``` **BigQuery:** ```yaml type: api connector: bigquery sql: SELECT * FROM `my-project.my_dataset.my_table` WHERE region = 'us' LIMIT 100 ``` **Redshift:** ```yaml type: api connector: redshift sql: SELECT * FROM transactions WHERE transaction_date >= '2024-01-01' LIMIT 100 ``` **Snowflake:** ```yaml type: api connector: snowflake sql: SELECT * FROM my_database.my_schema.events WHERE created_at >= '2024-01-01' LIMIT 100 ``` ### Databases **MySQL:** ```yaml type: api connector: mysql sql: SELECT * FROM orders WHERE order_date >= '2025-01-01' LIMIT 100 ``` **Postgres:** ```yaml type: api connector: postgres sql: SELECT * FROM events WHERE created_at >= '2025-01-01' LIMIT 100 ``` :::warning External database costs Queries to external connectors execute directly on your data source and incur costs based on your provider's billing model: - **Athena / BigQuery** — charged per TB of data scanned - **Redshift / Snowflake** — charged for compute time - **MySQL / Postgres** — may incur costs based on instance compute and IOPS To minimize costs: use `LIMIT` clauses, apply filters to reduce data scanned, and consider materializing frequently accessed queries as [models](/developers/build/models) in DuckDB. ::: ## When to use external connectors vs your OLAP engine | Factor | OLAP engine (default) | External connector | |--------|-------------------|--------------------| | **Query speed** | Fast — data is local | Depends on source (network + query time) | | **Data freshness** | As of last refresh | Real-time from the source | | **Cost** | No additional cost | Per-query costs from your provider | | **Best for** | Low-latency APIs, pre-modeled data | Real-time access, ad-hoc queries | **Use your OLAP engine** when you need fast, low-cost queries against data already modeled in Rill. Your data refreshes on a schedule and is optimized for analytical queries. **Use external connectors** when you need real-time access to the latest data, your data lives in the source database, or you're building internal tools where query costs are acceptable. ## Adding dynamic behavior SQL APIs support [templating](/developers/build/custom-apis/templating) for dynamic arguments, user attributes, and conditional logic: ```yaml type: api sql: | SELECT publisher, COUNT(*) as total FROM ad_bids WHERE domain = '{{ .args.domain }}' {{ if .user.admin }} AND internal_flag IS NOT NULL {{ end }} GROUP BY publisher LIMIT {{ default 100 .args.limit }} ``` See [Dynamic Queries with Templating](/developers/build/custom-apis/templating) for the full guide. --- ###### Dynamic Queries with Templating --- title: Dynamic Queries with Templating description: Use templating to make your custom APIs dynamic with arguments, user attributes, and conditional logic sidebar_label: Templating sidebar_position: 40 --- Rill's custom APIs support Go-style templating to make your SQL queries dynamic. You can accept parameters from API callers, reference user attributes from authentication tokens, and use conditional logic to build flexible endpoints. Templating works with both [SQL APIs](/developers/build/custom-apis/sql) and [Metrics SQL APIs](/developers/build/custom-apis/metrics-sql). ## Template context reference Every template has access to the following context: | Variable | Type | Description | |----------|------|-------------| | `{{ .args. }}` | `any` | Runtime arguments passed via query parameters or POST body | | `{{ .user.email }}` | `string` | Authenticated user's email address | | `{{ .user.domain }}` | `string` | Email domain of the authenticated user | | `{{ .user.name }}` | `string` | Display name of the authenticated user | | `{{ .user.admin }}` | `bool` | Whether the user has admin permissions on the project | | `{{ .user.groups }}` | `[]string` | User groups the authenticated user belongs to | | `{{ .user. }}` | `any` | Custom attributes from service tokens (e.g., `customer_id`, `region`) | | `{{ .export }}` | `bool` | `true` when the API is being resolved for export (CSV, Excel, Parquet) | :::note When testing locally (`localhost:9009`), `.user` attributes are not available since no authentication is required. To test with user attributes, deploy to Rill Cloud and use a service token with [custom attributes](/developers/build/custom-apis/security#custom-attributes-on-service-tokens). ::: ## Dynamic arguments Pass arguments to your API via query parameters (GET) or JSON body (POST). Reference them with `{{ .args. }}`: ```yaml type: api sql: | SELECT publisher, domain, COUNT(*) as total FROM ad_bids WHERE domain = '{{ .args.domain }}' GROUP BY publisher, domain ``` **Calling the API:** ```bash # Via query parameter curl "http://localhost:9009/v1/instances/default/api/my-api?domain=google.com" # Via POST body curl -X POST http://localhost:9009/v1/instances/default/api/my-api \ -H "Content-Type: application/json" \ -d '{"domain": "google.com"}' ``` ### Multiple arguments ```yaml type: api sql: | SELECT publisher, domain, COUNT(*) as total FROM ad_bids WHERE domain = '{{ .args.domain }}' AND publisher = '{{ .args.publisher }}' GROUP BY publisher, domain ``` ```bash curl "http://localhost:9009/v1/instances/default/api/my-api?domain=google.com&publisher=Facebook" ``` ## User attributes Reference the authenticated user's attributes with `{{ .user. }}`. These come from the user's identity (for user tokens) or from custom attributes (for service tokens). ### Built-in attributes ```yaml type: api sql: | SELECT * FROM reports WHERE owner_email = '{{ .user.email }}' LIMIT 50 ``` ### Custom attributes from service tokens [Service tokens with custom attributes](/developers/build/custom-apis/security#custom-attributes-on-service-tokens) are available in templates as `{{ .user. }}`: ```yaml type: api sql: | SELECT order_id, product, total FROM orders WHERE customer_id = '{{ .user.customer_id }}' AND region = '{{ .user.region }}' ``` See [Security & Access Control](/developers/build/custom-apis/security) for creating tokens and a full multi-tenant walkthrough. ## Conditional logic Use Go template `{{ if }}` / `{{ else }}` / `{{ end }}` blocks to conditionally include SQL: ### Admin-only columns ```yaml type: api sql: | SELECT publisher, COUNT(*) as total_records {{ if .user.admin }} , SUM(revenue) as total_revenue , AVG(bid_price) as avg_bid {{ end }} FROM ad_bids GROUP BY publisher ORDER BY total_records DESC ``` Admins see revenue and bid data; non-admins see only publisher and record count. ### Admin-only filters ```yaml type: api sql: | SELECT publisher, domain, COUNT(*) as total FROM ad_bids WHERE timestamp >= '{{ .args.start_date }}' {{ if (not .user.admin) }} AND domain = '{{ .user.domain }}' {{ end }} GROUP BY publisher, domain ``` Non-admins see only data for their domain; admins see everything. ## Optional parameters Use the `hasKey` function to check whether an argument was provided, making parameters optional: ```yaml type: api sql: | SELECT publisher, COUNT(*) as total_records FROM ad_bids WHERE 1=1 {{ if hasKey .args "publisher" }} AND publisher = '{{ .args.publisher }}' {{ end }} {{ if hasKey .args "domain" }} AND domain = '{{ .args.domain }}' {{ end }} GROUP BY publisher ORDER BY total_records DESC ``` **Without parameters** — returns all publishers: ```bash curl "http://localhost:9009/v1/instances/default/api/my-api" ``` **With one parameter** — filters by publisher: ```bash curl "http://localhost:9009/v1/instances/default/api/my-api?publisher=Google" ``` **With both** — filters by publisher and domain: ```bash curl "http://localhost:9009/v1/instances/default/api/my-api?publisher=Google&domain=news.google.com" ``` ## Pagination pattern Build paginated APIs using `LIMIT` and `OFFSET` with the `default` function for sensible defaults: ```yaml type: api sql: | SELECT publisher, domain, bid_price, timestamp FROM ad_bids ORDER BY timestamp DESC LIMIT {{ default 25 .args.limit }} OFFSET {{ default 0 .args.offset }} ``` **Page 1 (first 25 results):** ```bash curl "http://localhost:9009/v1/instances/default/api/my-api" ``` **Page 2:** ```bash curl "http://localhost:9009/v1/instances/default/api/my-api?offset=25" ``` **Custom page size:** ```bash curl "http://localhost:9009/v1/instances/default/api/my-api?limit=10&offset=20" ``` ## Sprig utility functions Rill uses standard Go templating together with [Sprig](http://masterminds.github.io/sprig/), which provides many utility functions. Commonly used ones: | Function | Example | Description | |----------|---------|-------------| | `default` | `{{ default 100 .args.limit }}` | Use a default value if the argument is empty | | `hasKey` | `{{ if hasKey .args "name" }}` | Check if an argument was provided | | `lower` | `{{ lower .args.status }}` | Convert to lowercase | | `upper` | `{{ upper .args.code }}` | Convert to uppercase | | `trim` | `{{ trim .args.name }}` | Trim whitespace | | `ne` | `{{ if (ne .user.domain "") }}` | Not equal comparison | | `eq` | `{{ if (eq .user.region "us") }}` | Equal comparison | ### Example: combining multiple patterns Here's a real-world example combining optional params, defaults, user attributes, and conditionals: ```yaml type: api sql: | SELECT publisher, domain, COUNT(*) as impressions, AVG(bid_price) as avg_bid {{ if .user.admin }} , SUM(revenue) as total_revenue {{ end }} FROM ad_bids WHERE 1=1 {{ if hasKey .args "publisher" }} AND publisher = '{{ .args.publisher }}' {{ end }} {{ if hasKey .args "start_date" }} AND timestamp >= '{{ .args.start_date }}' {{ end }} {{ if (not .user.admin) }} AND domain = '{{ .user.domain }}' {{ end }} GROUP BY publisher, domain ORDER BY impressions DESC LIMIT {{ default 50 .args.limit }} OFFSET {{ default 0 .args.offset }} ``` This single endpoint handles: - Optional filtering by publisher and date range - Admin users see revenue; non-admins don't - Non-admin users are scoped to their domain - Pagination with sensible defaults --- #### Dashboards --- ##### Canvas Widgets --- ####### All Canvas Widgets --- title: All Canvas Widgets description: Complete guide to all available widgets in Rill Canvas Dashboards toc_max_heading_level: 3 className: connect-connect --- import ComponentTile from '@site/src/components/ComponentTile'; Rill Canvas dashboards are built using a variety of widgets that can display data, create visualizations, and add rich content. Each component can be created dynamically through the visual Canvas dashboard editor or defined in individual YAML files. For more information, refer to our [Components reference doc](/reference/project-files/component). ## Data ### KPI ### Leaderboard ### Pivot / Table
} /> } /> } />
## Chart ### Bar ### Line ### Stacked Area ### Stacked Bar ### Stacked Bar Normalized ### Donut ### Funnel ### Heat Map ### Combo ### Scatter Plot
} /> } /> } /> } /> } /> } /> } /> } /> } /> } />
## Miscellaneous ### Markdown Text ### Image ### Component
} /> } /> } />
--- ####### "Chart Widgets" --- title: "Chart Widgets" sidebar_label: "Chart" sidebar_position: 10 --- import ImageCodeToggle from '@site/src/components/ImageCodeToggle'; Chart widgets in Rill Canvas allow you to visualize your data in various formats. You can create charts dynamically in the Canvas Dashboard or through individual component files. For more information, refer to our [Components reference doc](/reference/project-files/component). ## Bar Chart Bar charts are ideal for comparing values across different categories. ## Line Chart Line charts are perfect for showing trends over time. ## Stacked Area Chart Area charts show the magnitude of change over time with filled areas. ## Stacked Bar Chart Stacked bar charts show multiple data series stacked on top of each other. ## Stacked Bar Normalized Normalized stacked bars show proportions instead of absolute values. ## Donut Chart Donut charts display data as segments of a circle with a hollow center. ## Funnel Chart Funnel charts show the flow through a process with decreasing/increasing values at each stage. ## Heat Map Heat maps visualize data density using color intensity across two dimensions. ## Combo Chart Combo charts combine different chart types (like bars and lines) in a single visualization. ## Scatter Plot Scatter plots visualize the relationship between two measures, with each point representing a data value. Points can be colored by a dimension and optionally sized by a measure. Scatter plots support interactive zooming and panning. **Interactive Controls:** - **Shift + Drag**: Pan the plot - **Scroll Wheel**: Zoom in/out - **Double-click**: Reset zoom and pan to default view --- ####### "Data Widgets" --- title: "Data Widgets" sidebar_label: "Data" sidebar_position: 00 --- import ImageCodeToggle from '@site/src/components/ImageCodeToggle'; Data widgets in Rill Canvas allow you to display raw data in various formats. These widgets are perfect for showing detailed information, metrics, and tabular data. For more information, refer to our [Components reference doc](/reference/project-files/component). ## KPI Grid KPI grids display key performance indicators in a compact grid format with comparison capabilities. You can select up to 10 concurrent measures to display in a single widget ## Leaderboard Leaderboards show ranked data with the top performers highlighted. ## Pivot/Table Tables display detailed data in a structured format with customizable columns. ## Navigation All Data widgets also provide a button to "Go to explore" that can navigate to the Explore dashboard if available. ![Go To Explore](/img/build/dashboard/canvas/go-to-explore.png) --- ####### "Miscellaneous Widgets" --- title: "Miscellaneous Widgets" sidebar_label: "Miscellaneous" sidebar_position: 20 --- import ImageCodeToggle from '@site/src/components/ImageCodeToggle'; Miscellaneous widgets in Rill Canvas provide additional functionality for text, images, and other non-data elements. These widgets help enhance your dashboards with rich content. For more information, refer to our [Components reference doc](/reference/project-files/component). ## Text/Markdown Text widgets allow you to add formatted text, markdown content, and documentation directly to your dashboards. You can also use `metrics_sql` and `metrics_sql_rows` to display data from Metrics SQL queries. ### Dynamic Markdown The `metrics_sql` template function allows you to execute a Metrics SQL query and display the results as formatted text in your dashboard. This is useful for displaying aggregated data or custom metrics. The `metrics_sql_rows` template function allows you to execute a Metrics SQL query and iterate over the results, displaying each row with custom formatting. This is useful for displaying multiple data points in a structured format. Use `metrics_sql_rows` to get query results and then iterate over them using Go template syntax. You can access columns using dot notation (`.column_name`) or by assigning a variable name in the range: {{ $row.publisher }}
Domain: {{ $row.adomain }}
Total: {{ $row.total_bids }}
{{ end }} width: 6`} codeLanguage="yaml" /> Alternatively, you can use dot notation directly without assigning a variable: ```yaml - markdown: content: |- {{ $rows := metrics_sql_rows "SELECT publisher, total_records FROM ad_bids_metrics LIMIT 5" }} {{ range $rows }}
{{ .publisher }}: {{ .total_records }}
{{ end }} ``` The `metrics_sql_rows` function returns an array of rows that you can iterate over with `{{ range }}`. Access columns using either `{{ $row.column_name }}` (when using named variables) or `{{ .column_name }}` (when using dot notation). For more information on Metrics SQL syntax and capabilities, see the [Metrics SQL API documentation](/developers/build/metrics-view/metrics-sql). ## Image Image widgets let you embed images, logos, and visual elements into your dashboards. Put files in your public/ folder and reference them directly in the `url` as `public/image.png`. ## Component Reference a reusable component created outside of the canvas dashboard. --- ###### Canvas Dashboards --- title: Canvas Dashboards description: Create custom dashboards by assembling visualizations and components sidebar_label: Canvas Dashboards sidebar_position: 05 --- While Rill's signature **[Explore dashboards](/developers/build/dashboards/explore)** let you slice, dice, and interact with your data in our suggested layout, **Canvas dashboards** let you define your own visualizations and arrange them into your preferred layout. Canvas dashboards are built using various components that can display data, create visualizations, and add rich content from independent metrics views. You can create components dynamically through the visual Canvas Dashboard interface or define them in individual YAML files. ## Available Components Canvas dashboards support three main types of components: - **[Data components](/developers/build/dashboards/canvas-widgets/data)** - Display structured data and key metrics (KPIs, leaderboards, tables) - **[Chart components](/developers/build/dashboards/canvas-widgets/chart)** - Create rich visualizations (bar charts, line charts, heat maps, etc.) - **[Miscellaneous components](/developers/build/dashboards/canvas-widgets/misc)** - Add text, images, and other content Each component page shows both the visual result and the corresponding YAML code, making it easy to understand how to implement them in your own dashboards. For a complete overview of all available components, see our [**Canvas components**](/developers/build/dashboards/canvas-widgets) reference. ## Creating a Canvas Dashboard ### A Visual Editing Experience To modify any single widget, click to select it and use the right-hand panel to change its associated properties. Click outside the widget to view global properties associated with the overall Canvas. ![Selected Widget](/img/build/dashboard/canvas/selected-widget.png) ### Filters Optionally toggle on the **global filter bar** under Canvas properties to give dashboard viewers access to the same time and dimension filters available on Explore dashboards. ![Global Filter Bar](/img/build/dashboard/canvas/global-filter-bar.png) **Local filters** for a single KPI, Chart, or Table can be separated from the global filters via the "Filters" tab in the properties panel, where you can set filters that are local to just that widget. ![Local Filters](/img/build/dashboard/canvas/local-filters.png) ### Making changes to the YAML While we encourage creating Canvas dashboards via the visual editing experience described above, you can always edit the YAML file directly using the code view by toggling the switch next to the filename at the top of the page. Please see our [customization page](/developers/build/dashboards/customization) and [reference documentation](/reference/project-files/canvas-dashboards) for more information. ![Code Toggle](/img/build/dashboard/canvas/code-toggle.png) :::tip Customize default time ranges Set project-wide default time ranges and available options for all canvas dashboards. [Learn more about canvas defaults →](/developers/build/project-configuration#canvas-defaults) ::: ## Default Filters Dashboard creators can configure default filters to establish a consistent starting point for viewers. Filters are defined as Metrics SQL WHERE expressions, keyed by the metrics view name they apply to. ```yaml defaults: filters: # Key is the metrics view name; value is a Metrics SQL WHERE expression my_metrics_view: "country IN ('US', 'CA') AND revenue > 1000" another_metrics_view: "status = 'active'" ``` This lets you pre-filter data across one or more metrics views used in the canvas, ensuring users begin their analysis with the most relevant context. For detailed YAML configurations, see the [`defaults`](/reference/project-files/canvas-dashboards#defaults) section in our reference documentation. ## Example Canvas Dashboards Here are a few deployed examples of Canvas dashboards that you can check out! - **[E-commerce demo dashboard](https://ui.rilldata.com/demo/ezcommerce-demo/canvas/canvas)** - **[Programmatic advertising demo dashboard](https://ui.rilldata.com/demo/rill-openrtb-prog-ads/canvas/executive_overview)** - **[New York City demo dashboard](https://ui.rilldata.com/demo/nyc-canvas-jam/canvas/scorecard%20canvas)** - **[NYC party demo dashboard 🎉](https://ui.rilldata.com/demo/nyc-canvas-jam/canvas/Leaderboard)** --- ###### "Customization & Themes" --- title: "Customization & Themes" description: Alter dashboard look and feel sidebar_label: "Customization & Themes" sidebar_position: 30 --- Below are some common customizations and dashboard configurations available for end users. :::info Dashboard properties For a full list of available dashboard properties and configurations, please see our [Dashboard YAML](/reference/project-files/explore-dashboards) reference page. ::: ## Define Dashboard Access Along with [metrics views security policies](/developers/build/metrics-view/security), you can set access on the dashboard level. Access policies will be combined with metrics view policies using a logical AND, so if a user doesn’t pass both, they won’t get access to the dashboard. Only the `access` key can be set in the dashboard. ```yaml security: access: "{{ .user.admin }} OR '{{ .user.domain }}' == 'example.com'" ``` ## Setting Default Views for Dashboards ### Default Time Range Default time range controls the data analyzed on initial page load. Setting the default time range improves user experience by setting it to the most frequently used period— in particular, avoiding `all time` if you have a large data source but only analyze more recent data. The value must be either a valid [ISO 8601 duration](https://en.wikipedia.org/wiki/ISO_8601#Durations) (for example, `PT12H` for 12 hours, `P1M` for 1 month, or `P26W` for 26 weeks) or one of the [Rill ISO 8601 extensions](/reference/time-syntax/rill-iso-extensions#extensions). ### Default Dimensions For dashboards with wide tables, setting default dimensions is a good way to make sure that users can focus on the primary analyses and ensure a positive first experience. Each dimension listed under the `dimensions` setting will appear on the screen, while the remainder of the dimensions will be hidden (and still available for selection under filters). Common use cases for setting default fields include simplifying dashboards on initial load, narrowing the dashboard to the most used fields, and avoiding high cardinality fields (that may take longer to load, but are used less often, so this improves performance). An example addition to the `dashboard.yaml` file is below. ```yaml defaults: dimensions: - column1 - column2 ``` :::warning Column vs. Name Usage The `column` property is used by default from the column name in your underlying source. If you decide to use the `name` property, you'd replace the column above with the field name. ::: ### Default Measures A list of measures that should be visible by default. Operates the same as the `default_dimensions` configuration. When selecting measures, by default, consider hiding more computationally intensive measures like `count distinct` or other complicated expressions to improve performance. ```yaml defaults: measures: - measure_1 - measure_1 ``` ### Default Comparison Modes It is also possible to set up a default comparison mode for your dashboard. In Rill, we support both [time comparison](/guide/dashboards/time-series#time-comparisons) and [dimension comparison.](/guide/dashboards/filters#filter-by-dimensions) Note that only one of these comparisons can be set as default. ```yaml defaults: comparison_mode: time # comparison_mode: dimension # comparison_dimension: action ``` ## Time Ranges One of the more important configurations, available time ranges allow you to change the defaults in the time dropdown for periods to select. Updating this list allows users to quickly change between the most common analyses, like day over day, recent weeks, or period to date. The range must be a valid [ISO 8601 duration](https://en.wikipedia.org/wiki/ISO_8601#Durations) or one of the [Rill ISO 8601 extensions](/reference/time-syntax/rill-iso-extensions#extensions). ```yaml time_ranges: - PT15M - PT1H - P7D - P4W - rill-TD ## Today - rill-WTD ## Week-To-date ``` ## Time Zones Rill will automatically select several time zones that should be pinned to the top of the time zone selector. It should be a list of [IANA time zone identifiers](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). You can add or remove relevant time zones for your team from this list. ```yaml time_zones: - America/Los_Angeles - America/Chicago - America/New_York - Europe/London - Europe/Paris - Asia/Jerusalem - Europe/Moscow - Asia/Kolkata - Asia/Shanghai - Asia/Tokyo - Australia/Sydney ``` ## Changing Themes & Colors In your Rill project directory, create a `.yaml` file in any directory. Rill will automatically ingest the theme next time you run `rill start` or deploy to Rill Cloud and change the color scheme of your dashboard. All properties in the file are optional—any values you omit will fall back to Rill's standard theme defaults. First, create the YAML file as below. You can define separate color schemes for light and dark modes: ```yaml type: theme light: primary: "#4F46E5" # Indigo-600 secondary: "#8B5CF6" # Purple-500 variables: # Sequential palette - for ordered data that progresses from low to high color-sequential-1: "hsl(211deg 79% 94%)" color-sequential-2: "hsl(211deg 63% 86%)" color-sequential-3: "hsl(211deg 75% 77%)" color-sequential-4: "hsl(210deg 73% 64%)" color-sequential-5: "hsl(208deg 76% 51%)" color-sequential-6: "hsl(210deg 100% 43%)" color-sequential-7: "hsl(212deg 100% 36%)" color-sequential-8: "hsl(214deg 100% 29%)" color-sequential-9: "hsl(217deg 100% 22%)" # Diverging palette - for data that diverges from a critical midpoint color-diverging-1: "hsl(353deg 87% 48%)" color-diverging-2: "hsl(12deg 100% 62%)" color-diverging-3: "hsl(27deg 100% 70%)" color-diverging-4: "hsl(40deg 96% 82%)" color-diverging-5: "hsl(59deg 48% 94%)" color-diverging-6: "hsl(194deg 100% 86%)" color-diverging-7: "hsl(199deg 91% 73%)" color-diverging-8: "hsl(202deg 83% 57%)" color-diverging-9: "hsl(207deg 100% 44%)" color-diverging-10: "hsl(217deg 100% 39%)" color-diverging-11: "hsl(237deg 69% 34%)" # Qualitative palette - for categorical data (showing first 12 of 24) color-qualitative-1: "hsl(156deg 56% 52%)" color-qualitative-2: "hsl(27deg 100% 65%)" color-qualitative-3: "hsl(195deg 100% 46%)" color-qualitative-4: "hsl(289deg 61% 76%)" color-qualitative-5: "hsl(109deg 56% 64%)" color-qualitative-6: "hsl(41deg 83% 69%)" color-qualitative-7: "hsl(349deg 76% 71%)" color-qualitative-8: "hsl(217deg 49% 61%)" color-qualitative-9: "hsl(165deg 100% 36%)" color-qualitative-10: "hsl(16deg 95% 70%)" color-qualitative-11: "hsl(236deg 65% 74%)" color-qualitative-12: "hsl(75deg 43% 66%)" # ... up to color-qualitative-24 dark: primary: "#818CF8" # Indigo-400 secondary: "#A78BFA" # Purple-400 variables: # Sequential palette - adjusted for dark backgrounds color-sequential-1: "hsl(210deg 20% 25%)" color-sequential-2: "hsl(210deg 25% 30%)" color-sequential-3: "hsl(210deg 30% 35%)" color-sequential-4: "hsl(210deg 35% 40%)" color-sequential-5: "hsl(210deg 40% 45%)" color-sequential-6: "hsl(210deg 45% 50%)" color-sequential-7: "hsl(210deg 50% 55%)" color-sequential-8: "hsl(210deg 55% 60%)" color-sequential-9: "hsl(210deg 60% 65%)" # ... (diverging and qualitative palettes also available) ``` The `light` and `dark` properties allow you to customize: - **primary**: Primary color used for charts, buttons, and interactive elements - **secondary**: Secondary color (used for loading spinners and accents) - **variables**: Custom CSS variables for complete control over color palettes - **Sequential palette** (color-sequential-1 through 9): For ordered data progressions - **Diverging palette** (color-diverging-1 through 11): For data diverging from a midpoint - **Qualitative palette** (color-qualitative-1 through 24): For categorical data Colors can be specified using: - Hex values (with or without the '#' character, e.g., `4F46E5`. `"#4F46E5"`) - Named colors (e.g., `plum`, `violet`) - HSL format (e.g., `hsl(180, 100%, 50%)`) Once you have created that file, update the `dashboard.yaml` with the following configuration (we typically add this at the top along with time zones, time series and other configurations): `theme: ` :::info Theme properties For more details about configuring themes, you can refer to our [Theme YAML](/reference/project-files/themes) reference page. ::: ## Example ```yaml # Explore YAML # Reference documentation: https://docs.rilldata.com/reference/project-files/explores type: explore display_name: "Programmatic Ads Auction" metrics_view: auction_metrics dimensions: expr: "*" measures: - requests - avg_bid_floor - 1d_qps defaults: measures: - avg_bid_floor - requests dimensions: - app_site_cat - app_site_domain - app_site_name - pub_name comparison_mode: time time_range: P7D time_ranges: - rill-TD - rill-WTD - rill-MTD - rill-QTD - rill-YTD - rill-PDC - rill-PWC - rill-PMC - rill-PQC - rill-PYC theme: light: primary: "#14B8A6" # Teal-500 secondary: "#10B981" # Emerald-500 variables: color-sequential-1: "hsl(180deg 80% 95%)" color-sequential-5: "hsl(180deg 80% 50%)" color-sequential-9: "hsl(180deg 80% 25%)" dark: primary: "#2DD4BF" # Teal-400 secondary: "#34D399" # Emerald-400 security: access: "{{ .user.admin }} OR '{{ .user.domain }}' == 'example.com'" # only access can be set on dashboard level, see metric view for detailed access policies '{{ .user.domain }}' == 'example.com'" # only access can be set on dashboard level, see metric view for detailed access policies ``` --- ###### Dashboards in Rill --- title: Dashboards in Rill description: Understanding Explore vs Canvas dashboards and when to use each sidebar_label: Overview sidebar_position: 00 --- Rill offers two distinct dashboard experiences, each optimized for different use cases and user needs. While both are used to visualize your data, the key difference lies in how they handle [**metrics views**](/developers/build/metrics-view) - the data sources that power your dashboards. - **Explore Dashboards** - Standardized _opinionated_ dashboards with consistent data components, visualizations, and layout structure that promote slice-and-dice discovery and interactive data exploration. These are built over a **single metrics view** using the [One Big Table approach](/developers/build/models/models-101#one-big-table-and-dashboarding). - **Canvas Dashboards** - Traditional dashboards that allow users to customize visualizations and layout according to their specific requirements utilizing **multiple metrics views**. ## Explore Dashboards **[Explore dashboards](/developers/build/dashboards/explore)** are designed for interactive data exploration and analysis. They provide a powerful "slice and dice" experience focused on a single metrics view. Some key features of our Explore dashboards include: - [Pivot Tables](/guide/dashboards/explore/pivot) - [Time Dimension Detail](/guide/dashboards/explore/tdd) - [Leaderboards](/guide/dashboards/explore/multi-metrics) ![Explore](/img/build/dashboard/explore.png) ### Live Demo See both dashboard types in action with our interactive demos: - **[Programmatic Advertising Explore Dashboard](https://ui.rilldata.com/demo/rill-openrtb-prog-ads/explore/auction_explore?tr=-24h+to+ref+as+of+latest%2FD&tz=UTC&grain=day&measures=1d_qps%2Cavg_bid_floor%2Crequests&dims=pub_name%2Capp_site_name%2Cad_size%2Capp_site_domain%2Cdevice_osv&leaderboard_measures=1d_qps%2Cavg_bid_floor%2Crequests)** - Real-time bidding and auction data exploration - **[Mobile App Engagement Explore Dashboard](https://ui.rilldata.com/demo/rill-app-engagement/explore/mobile_events_explore)** - Mobile app user behavior and engagement metrics - **[GitHub Analytics Explore Dashboard](https://ui.rilldata.com/demo/rill-github-analytics/explore/rill_commits_explore?tr=P12M&grain=week&dims=username%2Cfile_path%2Cfilename%2Cfile_extension%2Cfirst_directory%2Csecond_directory%2Cprevious_file_path%2Cis_merge_commit)** - Code repository and commit analysis - **[E-commerce Explore Dashboard](https://ui.rilldata.com/demo/ezcommerce-demo/explore/data_metrics_explore)** - Interactive exploration of sales data ## Canvas Dashboards **[Canvas dashboards](/developers/build/dashboards/canvas)** provide a traditional dashboard experience where you can combine multiple metrics views and create custom layouts. See our [list of components](/developers/build/dashboards/canvas-widgets) to start creating your custom dashboard. ![RTBads Canvas 2](/img/build/dashboard/canvas/RTBAds-Canvas-2.png) ### Live Demo See both dashboard types in action with our interactive demos: - **[E-commerce demo dashboard](https://ui.rilldata.com/demo/ezcommerce-demo/canvas/canvas)** - Sales analytics and revenue tracking with multiple visualizations - **[Programmatic advertising demo dashboard](https://ui.rilldata.com/demo/rill-openrtb-prog-ads/canvas/executive_overview)** - Real-time bidding metrics and campaign performance analysis - **[New York City demo dashboard](https://ui.rilldata.com/demo/nyc-canvas-jam/canvas/scorecard%20canvas)** - Urban analytics and city performance metrics - **[NYC party demo dashboard 🎉](https://ui.rilldata.com/demo/nyc-canvas-jam/canvas/Leaderboard)** - Fun leaderboard showcasing various data insights ## Key Features Both dashboard types share common capabilities that make them powerful for data analysis: - **Time filtering and comparisons** - Navigate through time periods and compare different time ranges - **Dimension filtering** - Filter data by specific dimensions and values ## Learn More About Using Dashboards For comprehensive guidance on using dashboards as an analyst, see our [Analyst's Guide to Dashboards](/guide/dashboards/explore), which covers: - [Dashboard navigation and interface](/guide/dashboards/explore) - [Filtering and comparisons](/guide/dashboards/filters) - [Pivot tables](/guide/dashboards/explore/pivot) - [Time dimension details](/guide/dashboards/explore/tdd) - [Bookmarks and sharing](/guide/dashboards/bookmarks) - [Alerts and notifications](/guide/alerts) --- ###### Create Dashboards --- title: Create Dashboards description: Create and customize interactive dashboards for data exploration and analysis sidebar_position: 00 --- Rill provides powerful dashboard capabilities to visualize and explore your data. - [**Dashboard Overview**](/developers/build/dashboards/dashboards-101) - Compare Explore vs Canvas dashboards and choose the right one for your needs - [**Create an Explore Dashboard**](/developers/build/dashboards/explore) - Interactive, slice-and-dice interfaces for dynamic data exploration and analysis. Built on top of a single metrics view, they provide real-time filtering, drilling, and comparison capabilities. - [**Create a Canvas Dashboard**](/developers/build/dashboards/canvas) - Traditional dashboards that can be built on multiple metrics views, allowing you to consolidate key measures in a single view. - [**Canvas Widgets**](/developers/build/dashboards/canvas-widgets) - Browse all available widgets - [**Customization & Themes**](/developers/build/dashboards/customization) - Customize your dashboard's available dimensions and measures, and create default views for your viewers. --- ###### Explore Dashboards --- title: Explore Dashboards description: Explore dashboard overview sidebar_label: Explore Dashboards sidebar_position: 00 --- Explore dashboards are interactive, slice-and-dice interfaces that transform your metrics view data into powerful visualizations for data exploration and analysis. Built on top of a single metrics view, they provide an intuitive way for users to interact with your data through real-time filtering, drilling, and comparison capabilities. ## Creating an Explore Dashboard in Rill Developer ### Using the Code Editor In the Explore dashboard YAML, you can define dashboard level parameters to customize the capabilities. For a full list, see our [explore dashboard reference](/reference/project-files/explore-dashboards) doc. * _**metrics_view**_ - A single metrics view that powers the dashboard * _**measures**_ - `*` Which measures to include or exclude from the metrics view; using a wildcard will include all. * _**dimensions**_ - `*` Which dimensions to include or exclude from the metrics view; using a wildcard will include all. In some cases, a specific dashboard will not need to include all of the underlying metrics view's measures and/or dimensions. In this case, you can use the `measures` and `dimensions` parameters to filter these out. Rill supports providing a single value, list, or regex to filter out unnecessary measures and dimensions. ```yaml type: explore title: Title of your Explore Dashboard description: a description for your explore dashboard metrics_view: my_metricsview dimensions: expr: "^public_.*$" measures: - total_downloads - total_impressions defaults: comparison_mode: time time_range: P3M measures: - total_downloads - total_impressions dimensions: - show_name - season - program_name ``` :::tip Customize default time ranges Set project-wide default time ranges and available options for all explore dashboards. [Learn more about dashboard defaults →](/developers/build/project-configuration#explore-defaults) ::: ### Using AI In various locations throughout the platform, you have the opportunity to fast-track your dashboard creation via AI. This feature [creates the underlying metrics view with AI](/developers/build/metrics-view/what-are-metrics-views#creating-a-metrics-view-with-ai) and generates your explore.yaml similar to the example above with the required components. **Directly from your connector's tables:** ![Explorable Metrics](/img/build/dashboard/explorable-metrics.png) **From an ingested model:** ![Create With AI](/img/build/metrics-view/create-with-ai.png) ## Preview a Dashboard in Rill Developer Once a dashboard is ready to preview, before [deploying to Rill Cloud](/developers/deploy/deploy-dashboard), you can preview the dashboard in Rill Developer. ![Preview](/img/build/dashboard/preview.png) ### Setting Up Dashboard Access If you are setting up [dashboard policies](/developers/build/dashboards/customization#define-dashboard-access), it is recommended to preview and test the dashboard before deploying. This option will be available for testing if you have set up access policies at the [project level](/developers/build/project-configuration#testing-security), [metrics view level](/developers/build/metrics-view/security), or [dashboard level](/developers/build/dashboards/customization#define-dashboard-access). ![Preview Dashboard](/img/build/dashboard/preview-dashboard.png) --- #### Debugging --- ###### "Debugging Rill Projects" --- title: "Debugging Rill Projects" description: "Tools and techniques for debugging Rill projects" sidebar_label: "Debugging" sidebar_position: 30 --- When building Rill projects, you'll encounter various debugging scenarios—from understanding project logs to tracing resource reconciliation. This section covers the tools and techniques available for troubleshooting your Rill projects. - **[Understanding Project Logs](#understanding-project-logs)** - Learn the basics of reading and interpreting logs - **[Troubleshooting Common Errors](#troubleshooting-common-errors)** - Resolve common error patterns - **[Advanced Debugging Techniques](#advanced-debugging-techniques)** - Use debug flags, trace viewer, and cloud logs ## Understanding Project Logs Whether you start Rill from the terminal or your favorite IDE, the terminal window will output the project logs. From reconciling items to partition ingestion and beyond, browsing the project logs is a great place to start when troubleshooting errors or slow-loading models. ### Log Format Rill logs follow a structured JSON format. Here are some common log entries: ```bash Reconciled resource {"name": "commits__ (copy)_metrics_explore", "type": "Explore", "elapsed": "1ms"} Executed model partition {"model": "CH_incremental_commits_directory", "key": "55454ed4ad31cd3266988fe523103637", "data": {"path":"github-analytics/Clickhouse/2025/08","uri":"gs://rilldata-public/github-analytics/Clickhouse/2025/08"}, "elapsed": "283.188333ms"} Executed model partition {"model": "staging_to_CH", "key": "0030406e528b3799c8cbad6bfe609e83", "trace_id": "3073a89ac5cee9e7e3433ce0a34d291a", "span_id": "c3cb402d7b4af9b6", "data": {"day":"2022-12-20T00:00:00Z"}} ``` ### Generic Logging - **`name`** – Filename or YAML-defined name of the Rill object. - **`type`** – Resource type (e.g., `Connector`, `Model`, `MetricsView`, `Explore`, `API`, `Alert`, `Theme`, `Component`, `Canvas`). - **`elapsed`** – Time taken to reconcile, execute, or otherwise process the resource. - **`error`** – Error message generated during reconciliation or execution. - **`dependency_error`** – Boolean flag indicating that the resource failed due to another resource's error. - **`deleted`** – Boolean flag indicating that the resource was deleted. - **`path`** – Filesystem path to the resource YAML file. - **`logger_name`** – Name of the logger emitting the message. - **`message`** – Log message content. ### Partitioning - **`model`** – Name of the model associated with a partition operation. - **`partitions`** – Number of partitions resolved for a model. - **`key`** – Partition key (usually an MD5-like hash). - **`data`** – Partition-specific metadata or parameters. ### Embedded ClickHouse - **`addr`** – Host and port address for the embedded ClickHouse server. - **`sql`** – SQL statement executed by ClickHouse during query execution. ### Debug - **`sql`** – SQL statement executed by DuckDB during model evaluation or metrics computation. - **`args`** – SQL query parameters (if any). - **`trace_id`** – Unique trace identifier for the operation (used for distributed tracing). - **`span_id`** – Unique span identifier within the trace. ### Logging Examples #### Simple: Project Creation When you first initialize a Rill project, you'll see Rill reconcile a resource "duckdb" of type "Connector". This is expected as we explicitly create this file to initialize a connection to our embedded DuckDB. ```bash Rill will create project files in "~/Desktop/GitHub/testing-folder/dsn". Do you want to continue? Yes INFO Serving Rill on: http://localhost:9009 INFO Reconciling resource {"name": "duckdb", "type": "Connector"} INFO Reconciled resource {"name": "duckdb", "type": "Connector", "elapsed": "90ms"} ``` #### Simple: Connecting to a Data Source When connecting to a data source via a connector, you'll see a "Connector" being reconciled. In the case of any errors, you'll see this in both the UI and the logs. ```bash INFO Reconciling resource {"name": "gcs", "type": "Connector"} INFO Reconciled resource {"name": "gcs", "type": "Connector", "elapsed": "39ms"} WARN Reconcile failed {"name": "gcs", "type": "Connector", "elapsed": "1ms", "error": "failed to resolve templated property \"google_application_credentials\": template: :1:6: executing \"\" at <.env.GOOGLE_APPLICATION_CREDENTIALS>: map has no entry for key \"GOOGLE_APPLICATION_CREDENTIALS\""} ``` Once connected, you'll likely create a model and see this also reconciling in the logs. Similar to the above, if there are any issues, you'll see it both in the logs and UI. ```bash INFO Reconciling resource {"name": "commits__", "type": "Model"} INFO Reconciled resource {"name": "commits__", "type": "Model", "elapsed": "944ms"} # or WARN Reconcile failed {"name": "commits__", "type": "Model", "elapsed": "682ms", "error": "blob (key \"github-analytics/Clickhouse/2025/06/commits_2025_0.parquet\") (code=Unknown): storage: object doesn't exist: googleapi: Error 404: No such object: rilldata-public/github-analytics/Clickhouse/2025/06/commits_2025_0.parquet, notFound", "errorVerbose": "blob (key \"github-analytics/Clickhouse/2025/06/commits_2025_0.parquet\") (code=Unknown):\n gocloud.dev/blob.(*Bucket).Attributes\n /Users/runner/go/pkg/mod/gocloud.dev@v0.36.0/blob/blob.go:913\n - storage: object doesn't exist: googleapi: Error 404: No such object: rilldata-public/github-analytics/Clickhouse/2025/06/commits_2025_0.parquet, notFound"} ``` #### Intermediate: Creating Rill Objects The next section of logs shows the creation of a metrics view and explore dashboard. You'll see some errors thrown in the metrics view that get resolved in Rill Developer. ```bash INFO Reconciling resource {"name": "commits___metrics", "type": "MetricsView"} WARN Reconcile failed {"name": "commits___metrics", "type": "MetricsView", "elapsed": "15ms", "error": "measure \"earliest_commit_date_measure\" is of type CODE_TIMESTAMP, but must be a numeric type\nmeasure \"latest_commit_date_measure\" is of type CODE_TIMESTAMP, but must be a numeric type"} INFO Reconciling resource {"name": "commits___metrics", "type": "MetricsView"} WARN Reconcile failed {"name": "commits___metrics", "type": "MetricsView", "elapsed": "21ms", "error": "measure \"earliest_commit_date_measure\" is of type CODE_TIMESTAMP, but must be a numeric type"} INFO Reconciling resource {"name": "commits___metrics", "type": "MetricsView"} INFO Reconciled resource {"name": "commits___metrics", "type": "MetricsView", "elapsed": "15ms"} INFO Reconciling resource {"name": "commits___metrics_explore", "type": "Explore"} INFO Reconciled resource {"name": "commits___metrics_explore", "type": "Explore", "elapsed": "1ms"} ``` #### Advanced: Dependency Errors When a resource fails, dependent resources will also fail with a `dependency_error` flag. This helps you trace the root cause of cascading failures. In the example below, the `orders` model fails, which causes `orders_customers_model` to fail with a dependency error, which in turn causes `orders_customers_metrics` and `orders_customers_explore` to fail. ```bash INFO Reconciling resource {"name": "duckdb", "type": "Connector"} INFO Reconciling resource {"name": "gcs", "type": "Connector"} INFO Reconciling resource {"name": "orders", "type": "Model"} INFO Reconciling resource {"name": "customers", "type": "Model"} INFO Reconciled resource {"name": "gcs", "type": "Connector", "elapsed": "1ms"} INFO Reconciled resource {"name": "duckdb", "type": "Connector", "elapsed": "96ms"} INFO Reconciled resource {"name": "customers", "type": "Model", "elapsed": "10.8s"} WARN Reconcile failed {"name": "orders", "type": "Model", "elapsed": "17.5s", "error": "failed to create model: Cannot open file \"/path/to/project/tmp/default/duckdb/orders/data.db.wal\": No such file or directory"} INFO Reconciling resource {"name": "orders_customers_model", "type": "Model"} INFO Reconciled resource {"name": "orders_customers_model", "type": "Model", "error": "dependency error: resource \"orders\" (rill.runtime.v1.Model) has an error", "dependency_error": true} INFO Reconciling resource {"name": "orders_customers_metrics", "type": "MetricsView"} WARN Reconcile failed {"name": "orders_customers_metrics", "type": "MetricsView", "elapsed": "11ms", "error": "table \"orders_customers_model\" does not exist"} INFO Reconciling resource {"name": "orders_customers_explore", "type": "Explore"} INFO Reconciled resource {"name": "orders_customers_explore", "type": "Explore", "error": "dependency error: resource \"orders_customers_metrics\" (rill.runtime.v1.MetricsView) has an error", "dependency_error": true} ``` #### Advanced: Partitioned Models The main takeaway for partitioned models is that you'll be able to see the number of partitions that Rill will start ingesting. This is especially important when creating [dev/prod](/developers/build/connectors/templating) environments and you're trying to avoid ingesting large amounts of data locally. ```bash Resolved model partitions {"model": "staging_to_CH", "partitions": 16} INFO Executed model partition {"model": "staging_to_CH", "key": "0030406e528b3799c8cbad6bfe609e83", "data": {"day":"2022-12-20T00:00:00Z"}} ``` ## Troubleshooting Common Errors When debugging errors, start by checking the project logs and understanding the error messages. Here are common error patterns and how to resolve them: ### Model Errors Model errors typically occur when there are issues with credentials, data processing, SQL syntax, or data type mismatches. Common error messages and their solutions: - **`Failed to connect to ...`**: Issue with your connector. Check your credentials and [firewall settings](/developers/build/connectors/data-source#externally-hosted-services) if using externally hosted services - **`Table with name ... does not exist!`**: Verify the table exists by running `rill query --sql "select * from {table_name} limit 1"` or checking your data source - **`IO Error: No files found that match the pattern...`**: Check that your cloud storage folder path is correct and files exist - **`some partitions have errors`**: Run `rill project refresh --model {model_name} --errored-partitions` to refresh errored partitions - **`Out of Memory Error: ...`**: Contact [support](/contact) for assistance with memory issues ### Metrics View and Dashboard Errors Metrics view and dashboard errors often stem from issues with the underlying models or configuration problems: - **Model Dependencies:** Dashboards failing because their underlying models have errors. Check the [dependency errors](/developers/build/debugging#model-errors) section above - **Missing Dimensions/Measures:** References to fields that don't exist in the underlying model. Verify that measures and dimensions in your metrics YAML match existing columns in your data - **Type Mismatches:** Measures must be numeric types. Check that timestamp fields aren't being used as measures ### Checking Resource Status To understand what's failing in your project: 1. **Check project logs** - Review the terminal output or use `rill project logs` for Rill Cloud projects 2. **Use the Trace Viewer** - Visualize resource reconciliation and trace execution paths 3. **Check resource status** - Use the `Status` tab in Rill Developer or [`rill project status`](/reference/cli/project/status) CLI command :::tip Check upstream dependencies The surfaced error might not be the root cause. A dashboard error could stem from an underlying model timeout. Always trace errors to their source by checking dependent resources. ::: ## Advanced Debugging Techniques When standard logs aren't providing enough detail, Rill offers several advanced debugging options to help you diagnose issues more effectively. ### Using Debug and Verbose Flags Rill Developer provides two flags for increasing log verbosity: **`--verbose`**: Sets the log level to debug, showing more detailed information about what Rill is doing internally. This includes: - More granular resource reconciliation steps - Additional context about operations - Extended error details **`--debug`**: Collects additional debug information beyond just log verbosity. This flag enables: - Enhanced debugging metadata - More detailed trace information - Additional diagnostic data useful for troubleshooting complex issues ```bash # Increase log verbosity rill start --verbose # Collect additional debug info rill start --debug # Combine both for maximum detail rill start --debug --verbose ``` :::tip When to use each flag - Use `--verbose` when you need more detail about what Rill is doing but don't need deep debugging info - Use `--debug` when troubleshooting complex issues that require additional diagnostic data - Use both together when you need the most comprehensive debugging information ::: ### Log Format Options By default, Rill outputs logs in a human-readable console format. For programmatic processing or filtering, you can output logs in JSON format: ```bash rill start --log-format json ``` JSON format is useful when: - Parsing logs with scripts or tools - Filtering logs programmatically - Integrating with log aggregation systems ### Viewing Rill Cloud Logs For projects deployed to Rill Cloud, you can view logs directly from the CLI: ```bash # View recent logs rill project logs # Follow logs in real-time (like tail -f) rill project logs --follow # Show only the last N lines rill project logs --tail 100 # Filter by log level rill project logs --level DEBUG ``` The `rill project logs` command provides the same structured log output you see in Rill Developer, making it easy to debug issues in production deployments. ### Checking Project Status Use the `rill project status` command to get a quick overview of your project's health: ```bash # Check status of a deployed project rill project status # Check status of locally running project rill project status --local ``` This command shows: - Resource reconciliation status - Error states for individual resources - Dependency relationships - Overall project health ### Trace Viewer For complex debugging scenarios involving multiple resources and dependencies, use the [Trace Viewer](/developers/build/debugging/trace-viewer) to visualize resource reconciliation and trace execution paths across your project. The Trace Viewer helps you: - Understand resource dependency chains - Identify bottlenecks in reconciliation - Visualize execution flows - Debug cascading failures To use the Trace Viewer, start Rill with the `--debug` flag: ```bash rill start --debug ``` Then access the Trace Viewer through the Rill Developer UI to see a visual representation of your project's resource reconciliation. --- ###### "Trace Viewer in Rill Developer" --- title: "Trace Viewer in Rill Developer" description: Visualize resource reconciliation and trace execution sidebar_label: "Trace Viewer in Rill Developer" sidebar_position: 00 --- Rill Developer provides a built-in trace viewer, enabling users to visually inspect operations performed when reconciling resources or fetching data for dashboards. This helps in diagnosing performance and operational issues. ![Trace Viewer Overview](/img/build/debugging/trace-viewer-overview.png) ## How to Use the Trace Viewer ### Step 1: Start Rill Developer with Debug Mode Launch Rill Developer with the debug flag enabled: ```bash rill start --debug ``` ### Step 2: Access the Trace Viewer Open your web browser and navigate to: ``` http://localhost:9009/traces ``` ### Step 3: Visualize Traces There are two ways to visualize traces: - **By Trace ID:** Enter a specific Trace ID to inspect a particular operation. - **By Resource Reconciliation:** Browse operations associated with reconciling specific resources. For example, to view the operations performed during the reconciliation of the `bids` model, enter `bids` in the resource name field. ## Understanding the Trace Graph - **Horizontal Bars:** Each horizontal bar represents a distinct operation performed by Rill. - **Bar Length:** The length of each bar indicates the duration of the operation. - **Operation Details:** Click on any bar to view detailed tags and metadata in the pane on the right. - **Nested Operations:** Operations may trigger other sub-operations, displayed as nested bars beneath the parent operation. - **Parallel Operations:** Bars displayed side by side indicate operations executed concurrently. ## Retrieving the Trace ID for Dashboard Data Fetch To find the Trace ID associated with fetching data for a dashboard: 1. **Open Browser Developer Tools:** Use your browser's developer tools (usually `F12` or `Cmd+Option+I`). 2. **Inspect API Requests:** Select the API call you're interested in from the network tab. 3. **Find `X-trace-id`:** Check the response headers to locate the `X-trace-id`. 4. **Use Trace ID:** Enter this Trace ID into the trace viewer to inspect the operation details. ![Capture Trace Id](/img/build/debugging/capture-trace-id.png) ## Inner Workings - The Trace Viewer UI uses OpenTelemetry (OTEL) traces. - Traces are captured in JSON format and stored locally. - The UI retrieves and filters traces based on query parameters and trace type. --- #### Ide --- ###### Build Rill Projects with Your Favorite IDE --- title: Build Rill Projects with Your Favorite IDE description: Use VS Code, IntelliJ, or any IDE to create and edit Rill projects with real-time feedback sidebar_label: External IDE Integration sidebar_position: 00 --- ## Use Any IDE for Rill Development Rill projects are just files and folders that you can edit with any code editor or IDE. Whether you prefer VS Code, IntelliJ, Vim, or any other editor, you can create and modify Rill projects directly from your favorite development environment. ### How It Works Rill projects consist of: - **SQL files** (`.sql`) for models and metrics views - **YAML files** (`.yml`) for project configuration - **Data files** in various formats You can edit these files in any IDE, and Rill will automatically detect changes and provide real-time feedback. ![](https://cdn.rilldata.com/docs/release-notes/36_hot_reload.gif) ## Using AI Agents to Build Rill Projects Rill ships built-in instructions that teach AI coding agents like **Claude Code** and **Cursor** how to build Rill projects. A single `rill init` command scaffolds everything your agent needs — resource schemas, best practices, and development conventions. ```bash # Add Claude Code instructions to your project rill init --template claude # Or add Cursor rules rill init --template cursor ``` --- #### Metrics View --- ##### Dimensions --- ####### "Clickable Dimension Links" --- title: "Clickable Dimension Links" description: "Make dimension values clickable by adding URI parameters to create interactive links in your dashboards" sidebar_label: "Clickable Dimension Links" sidebar_position: 50 --- You can make dimension values clickable by adding a `uri` parameter to your dimension configuration. This enables users to click directly on dimension values in the dashboard to navigate to external URLs, making your dashboards more interactive and useful for data exploration. ![Clickable Dimension](/img/build/dashboard/clickable-dimension.png) ## Simple Setup: Column Already Contains URLs If your column values are already URLs, simply add `uri: true` to the dimension: ```yaml dimensions: - display_name: Company URL column: company_url uri: true ``` ## Advanced Setup: Dynamic URL Generation For more advanced use cases, you can dynamically create URLs using expressions. The dimension displays the generated URL and uses it as the clickable link: ```yaml dimensions: - name: profile_url display_name: Bluesky Profile Link expression: CONCAT('https://bsky.app/profile/', profile_id) uri: true ``` --- ####### "Define Your Dimensions" --- title: "Define Your Dimensions" description: "Learn how to create and configure dimensions for data segmentation, filtering, and analysis in your metrics views" --- Dimensions are fundamental components of metrics views that enable data segmentation, filtering, and analysis. They represent categorical or descriptive attributes of your data that allow you to break down measures into meaningful segments. Dimensions are the "who," "what,", and "where" of your data. They provide context and categorization for your measures, enabling you to: - **Segment data**: Break down metrics by categories like region, product type, or customer segment - **Filter analysis**: Focus on specific subsets of your data - **Enable drill-downs**: Navigate from high-level summaries to detailed breakdowns - **Create comparisons**: Analyze performance across different groups Explore these advanced capabilities to enhance your dimensions: - [**Time Dimensions**](/developers/build/metrics-view/dimensions/time-dimensions) - Add multiple time-based dimensions for filtering across different date columns - [**Clickable Dimension URI**](/developers/build/metrics-view/dimensions/dimension-uri) - Generate navigable URIs for your dimension values - [**Unnesting Arrays**](/developers/build/metrics-view/dimensions/unnesting) - Work with array data by expanding nested structures - [**Lookup Dimensions**](/developers/build/metrics-view/dimensions/lookup) - Create dimensions that reference data from other sources --- ####### "Query-Time Dimension Joins" --- title: "Query-Time Dimension Joins" description: "Use lookup dimensions to enrich your metrics view data with reference information at query time" sidebar_label: "Query-Time Joins" sidebar_position: 55 --- Query-time joins or more simply, lookup dimensions, provide a powerful way to incorporate reference tables into your dashboard without requiring upstream SQL transformations. This feature enables data enrichment with reference information at query time, eliminating the need for complex ETL processes. :::note The lookup function is dependent on the type of [OLAP engine](/developers/build/connectors/olap) that you are using for your project. ::: ## ClickHouse Lookups ClickHouse provides powerful dictionary functions for lookup operations through `dictGet` functions. The table needs to be defined as a dictionary. ### Using `expression` You can use `dictGet` directly in a dimension's `expression`: ```yaml dimensions: - name: user_email display_name: "User Email" # Query-time lookup in the 'users' dictionary expression: dictGet('users', 'email', user_id) description: "User email from users dictionary" - name: product_category display_name: "Product Category" expression: dictGet('products', 'category', product_id) description: "Product category information" # With default fallback - name: user_email_safe display_name: "User Email (Safe)" expression: dictGetOrDefault('users', 'email', user_id, 'unknown@example.com') description: "User email with fallback for missing users" # Multiple attributes from same dictionary - name: user_full_name display_name: "User Full Name" expression: concat(dictGet('users', 'first_name', user_id), ' ', dictGet('users', 'last_name', user_id)) description: "Concatenated full name from user dictionary" ``` This approach is straightforward but has a performance drawback: when filtering by a lookup dimension, ClickHouse must evaluate `dictGet` for every row before applying the filter, which prevents it from using indexes on the key column. ### Using `lookup_*` properties The `lookup_*` properties provide a declarative way to define lookup dimensions. Rill automatically rewrites filter queries to use an optimized subquery pattern, enabling ClickHouse to leverage indexes on key columns. ```yaml dimensions: - name: country_name display_name: "Country" # column in the fact table to match with the dictionary key column: country_code lookup_table: country_dict lookup_key_column: code lookup_value_column: name # With a default value for missing keys - name: region display_name: "Region" # column in the fact table to match with the dictionary key column: country_code lookup_table: country_dict lookup_key_column: code lookup_value_column: region lookup_default_expression: "'Unknown'" ``` The properties are: - **`lookup_table`** — the dictionary name. Use `database.dictionary_name` if the dictionary is in a non-default database (e.g. `mydb.my_dict`). - **`lookup_key_column`** — the primary key column in the dictionary that matches the fact table's `column`. - **`lookup_value_column`** — the attribute column in the dictionary to display. - **`lookup_default_expression`** _(optional)_ — a SQL expression used as a fallback when no match is found (maps to `dictGetOrDefault`). #### How the query optimization works When you filter on a lookup dimension defined with `expression: dictGet(...)`, the generated query looks like: ```sql SELECT ... FROM fact_table WHERE dictGet('dict', 'name', publisher_id) IN ('Alice', 'Bob') ``` ClickHouse must call `dictGet` on every row to evaluate the filter, making it unable to use the index on `publisher_id`. With `lookup_*` properties, Rill rewrites the filter to a subquery pattern: ```sql SELECT ... FROM fact_table WHERE publisher_id IN ( SELECT code FROM dict WHERE name IN ('Alice', 'Bob') ) ``` This allows ClickHouse to first resolve the small set of matching keys from the dictionary, then use the index on `publisher_id` to efficiently scan only the relevant rows in the fact table. :::tip Optimizing GROUP BY with INJECTIVE If the key-to-value mapping in your dictionary is 1:1 (e.g. every `code` maps to exactly one `name`), you can additionally mark the attribute columns as `INJECTIVE` in the ClickHouse dictionary definition. This tells ClickHouse that the mapping preserves uniqueness, allowing it to optimize `GROUP BY` queries by grouping on the key column instead of evaluating `dictGet` for every group. Combined with `lookup_*` properties, this gives you optimized performance for both filtering (via Rill's subquery rewrite) and grouping (via ClickHouse's INJECTIVE optimization). ::: ## Druid Lookups For those looking to add ID to name mappings with Druid (as an OLAP engine), you can utilize expressions in your **Dimension** settings. Simply use the `lookup` function and provide the name of the lookup and ID, i.e. `lookup(city_id, 'cities')`. Be sure to include the lookup table name in single quotes. ```yaml dimensions: - name: city_name display_name: "Cities" expression: lookup(city_id, 'cities') description: "City names from lookup table" ``` ## DuckDB/MotherDuck Lookups Unfortunately, DuckDB does not have a dimension lookup **function** and instead relies on upstream modeling to join the lookup table and create a [one big table](/developers/build/models/models-101#one-big-table-and-dashboarding) for dashboarding. Take a look through [DuckDB docs](https://duckdb.org/docs/stable/sql/introduction) for further information! ```sql SELECT o.*, u.user_name FROM orders o LEFT JOIN users u ON u.email = o.email; ``` Alternatively, you can use DuckDB's `map` function to create a lookup table by mapping values from one column to another. This approach creates an in-memory mapping that can be referenced without using a joining SQL: ```yaml - expression: (SELECT map(list(email), list(user_name)) FROM users_dataset)[email] name: user_name ``` ## Performance Implications While joins can kill the performance of [OLAP engines](/developers/build/connectors/olap), lookups (key-value pairs) are common to reduce data size and improve query speeds. Lookups can be done during ingestion time (a static lookup to enrich the source data) or at query time (dynamic lookups). **Static Lookups** Static lookups are lookups that are ingested at processing time. When a record is being processed, if a match is found between the record and lookup's key, the lookup's corresponding value at that moment in time is extracted and carbon-copied into your OLAP engine for the records it processed. Static lookups are best suited for: - Dimensions with values that require a historical record for how they have changed over time - Values that are never expected to change (leverage Dynamic Lookups if the values are expected to change) - Extremely large lookups (hundreds of thousands of records or >50MB lookup file) to improve query performance Customers typically store lookup values in S3 or GCS, and the lookup file is then updated by customers as needed and consumed by ETL logic. **Dynamic Lookups** Since static lookups transform and store the data permanently, any changes to the mapping would require reprocessing the entire dataset to ensure consistency. To address the case when values in a lookup are expected to change with time, we developed dynamic lookups. Dynamic lookups, also known as Query Time Lookups, are lookups that are retrieved at query time, as opposed to being used at ingestion time. Benefits of dynamic lookups include: - Historical continuity for dimensions that change frequently without reprocessing the entire dataset - Time savings, because there is no dataset reprocessing required to complete the update - Dynamic lookups are kept separate from the dataset. Thus, any human errors introduced in the lookup do not impact the underlying dataset - Ability for users to create new dimension tables from metadata associated with a dimension table. For example, account ownership can change during the course of a quarter. In such cases, a dynamic lookup ensures that the most current ownership information is always available without requiring data reprocessing. --- ####### "Time Dimensions" --- title: "Time Dimensions" description: "Add multiple time dimensions to enable time-based filtering and analysis across different temporal columns" sidebar_label: "Time Dimensions" sidebar_position: 52 --- While every metrics view has a primary `timeseries` column that powers the main time series chart, you can define additional time dimensions using `type: time`. This enables users to filter and analyze data across multiple temporal columns. ## When to Use Time Dimensions Time dimensions are useful when your data contains multiple date or timestamp columns that users may want to filter by: - **Order date vs. Ship date**: Filter orders by when they were placed or when they were shipped - **Created vs. Updated timestamps**: Analyze records by creation date or last modification - **Event time vs. Processing time**: Distinguish between when events occurred and when they were recorded - **Multiple business dates**: Handle scenarios like invoice date, due date, and payment date ## Adding a Time Dimension To create a time dimension, add `type: time` to your dimension definition: ```yaml version: 1 type: metrics_view model: orders_model timeseries: order_date # Primary time series for charts dimensions: - name: order_date display_name: Order Date column: order_date type: time - name: ship_date display_name: Ship Date column: ship_date type: time - name: customer_region column: region ``` ## Time Dimension vs. Timeseries Understanding the difference between these two concepts is important: | Feature | `timeseries` (top-level) | `type: time` (dimension) | |---------|--------------------------|--------------------------| | Purpose | Powers main time series chart | Enables time-based filtering | | Chart display | Shows trends over time | Used in filter panel | | Required | No (but recommended) | No | | Multiple allowed | No (one per metrics view) | Yes | The primary `timeseries` column determines which dates appear on the x-axis of your time series visualizations. Time dimensions provide additional temporal filtering options in the dashboard filter panel. ## Full Example Here's a complete example with multiple time dimensions: ```yaml version: 1 type: metrics_view model: sales_model timeseries: transaction_date dimensions: # Time dimensions - name: transaction_date display_name: Transaction Date column: transaction_date type: time - name: fulfillment_date display_name: Fulfillment Date column: fulfillment_date type: time - name: return_date display_name: Return Date column: return_date type: time # Categorical dimensions - name: product_category display_name: Product Category column: category - name: store_location display_name: Store Location column: store_id measures: - name: total_sales display_name: Total Sales expression: SUM(amount) - name: order_count display_name: Order Count expression: COUNT(*) ``` ## Using Expressions You can also create time dimensions using expressions to transform or derive time values: ```yaml dimensions: - name: order_month display_name: Order Month expression: DATE_TRUNC('month', order_date) type: time - name: fiscal_quarter_start display_name: Fiscal Quarter Start expression: DATE_TRUNC('quarter', order_date + INTERVAL '3 months') - INTERVAL '3 months' type: time ``` :::tip Time dimensions work with columns of type `TIMESTAMP`, `TIME`, or `DATE`. If your source data stores dates in a different format (like strings), use an expression to convert them to a proper date type. ::: --- ####### "Unnest Dimensions" --- title: "Unnest Dimensions" description: Tips & Tricks for Measure Formatting sidebar_label: "Unnest Dimensions" sidebar_position: 50 --- For multi-value fields, you can set the `unnest` property within the dimension. If `true`, this property allows a multi-valued dimension to be unnested (such as lists), and filters will automatically switch to "contains" instead of exact match. ## Example In this example, the data contains an array column that has the value `['deal_one', 'deal_two', 'deal_three']`. Setting the unnest property enables the user to filter on each value in the array. Measures split by unnested values are non-additive, so that in this example the “Total Impressions” measure is applied equally across each value. Totals in Pivot Tables and the Time Dimension Detail view are calculated correctly, avoiding issues with double counted values when splitting multi-value dimensions. ![Unnested Dimension](/img/build/metrics-view/examples/unnested-dimension.png) ```yaml - display_name: "Deal Name" column: deal_name description: "Unnested Column" unnest: true ``` In another example, we are provided with a directory_path column that gives us information on which path was edited. Using DuckDB's `regexp_split_to_array`, we have converted the string into an array. Using `unnest`, we can see which top directories are being updated more than others. ![Tutorial Unnest](/img/build/metrics-view/examples/tutorial-unnest.png) ```yaml - expression: regexp_split_to_array(directory_path, '/') display_name: "The directory unnested" description: "The directory path" name: directory_path_unnested unnest: true ``` ## Demo [See this project live in our demo!](https://ui.rilldata.com/demo/my-rill-tutorial/explore/advanced_explore?f=directory_path_unnested+IN+%28%27docs%27%29) --- ##### Measures --- ####### "Case Statements and Filters" --- title: "Case Statements and Filters" description: Tips & Tricks for Case Statements sidebar_label: "Case Statements and Filters" sidebar_position: 20 --- One of the most common advanced measure expressions is the use of [`case`](https://duckdb.org/docs/stable/sql/expressions/case.html) statements and [`filters`](https://duckdb.org/docs/stable/sql/query_syntax/filter.html) to filter or apply logic to part of the result set. Use cases for case statements include filtered sums (e.g., only sum if a flag is true) and bucketing data (e.g., if between threshold x and y, then apply an aggregate). While similar, case statements give you a bit more flexibility as they allow you to set a custom value depending on the case. See below for some examples! ![Case Example](/img/build/metrics-view/examples/case-example.png) Please review the reference documentation, [here.](/reference/project-files/metrics-views) ## Examples ### Case Statements The following expression sums of the values of Global_active_power only when considered to be a lower value. ```yaml - name: total_low_active_power_measure display_name: Total Low Global Active Power description: Total sum of Global Active Power where considered Low expression: SUM(CASE WHEN GAP_category = 'Low' THEN Global_active_power ELSE 0 END) format_preset: humanize valid_percent_of_total: true ``` The following expression only considers the total value of users who are identified. ```yaml - name: total_value_for_identified_users display_name: Total Value for Identified Users description: Total Sum of Value for Identified Users expression: SUM(CASE WHEN user_id != '' OR user_id IS NOT NULL THEN value ELSE 0 END) format_preset: humanize valid_percent_of_total: true ``` The following expression modifies the value of the column based on the value of column XX ```yaml - name: modify_value display_name: Arithmetic on Value description: Arithmetic on Value based on XX expression: | SUM( CASE WHEN XX = 'multiply_10' THEN Value * 10 WHEN XX = 'multiply_2' THEN Value * 2 WHEN XX = 'divide_5' THEN Value / 5 END ) format_preset: humanize valid_percent_of_total: true ``` ### Filters Similar to the above case statements, you can use the filter expression to filter the data on a specific column's value. However, in the example where we are explicitly changing the value in the CASE statement, this is not possible using only a filter. ```yaml - name: total_low_active_power_measure display_name: Total Low Global Active Power description: Total sum of Global Active Power where considered Low expression: sum(Global_active_power) FILTER (WHERE GAP_category = 'Low') format_preset: humanize valid_percent_of_total: true ``` ```yaml - name: total_value_for_identified_users display_name: Total Value for Identified Users description: Total Sum of Value for Identified Users expression: SUM(value) FILTER (WHERE user_id != '' OR user_id IS NOT NULL) format_preset: humanize valid_percent_of_total: true ``` ## Demo [See this project live in our demo!](https://ui.rilldata.com/demo/rill-kaggle-elec-consumption/explore/household_power_consumption_metrics_explore) --- ####### "Fixed Measures" --- title: "Fixed Measures" description: Tips & Tricks for Fixed Measures sidebar_label: "Fixed Measures" sidebar_position: 35 --- Some measures may be at a different level of granularity where a sum across the measure is no longer accurate. As an example, perhaps you have a campaign with a daily budget of $5000 across five line items. Summing the `daily_budget` column would give an inaccurate total of $25,000 budget per day. For those familiar with Tableau, this is referred to as a `FIXED metric`. ![Incorrect Sum](/img/build/metrics-view/examples/incorrect-sum.png) To create the correct value, you can utilize DuckDB's unnest functionality. In the example below, you would be pulling a single value of `daily_budget` based on `campaign_id` to get the sum of budget for the day by campaign ids. Note that you can use multiple keys if your granularity is defined by multiple dimensions. ```yaml expression: | select sum(a.val) as value from ( select unnest( list(distinct { key: campaign_id, val: daily_budget }) ) a ) ``` :::note The syntax for fixed metrics is specific to DuckDB as an OLAP engine as it requires DuckDB-specific commands. However, you can create a similar SQL expression using a different OLAP engine, too! ::: Please review the reference documentation, [here.](/reference/project-files/metrics-views) ## Example In the following example, each publishing company has a monthly minimum guarantee. As you'll see in the measure, `incorrect_sum_of_guarantee`, you'll get an incorrect value as this will sum multiple values as there are multiple shows and days per publisher. Another workaround would be to use MIN, MAX, or AVG, but when selecting multiple publishers, the values will not be accurate. ![Selecting Publishers](/img/build/metrics-view/examples/selecting-publishers.png) ```yaml - name: incorrect_sum_of_guarantee expression: sum(min_guarantee_usd) format_preset: currency_usd valid_percent_of_total: false - name: guarantee_usd_measure_monthly display_name: Monthly Minimum Guarantee USD description: Total minimum guarantee in USD recorded in the dataset. expression: > SELECT SUM(a.val) AS value FROM ( SELECT unnest( list(distinct { key: publisher_id, month: date_trunc('month', date), val: min_guarantee_usd }) ) a ) format_preset: currency_usd valid_percent_of_total: false ``` ## Demo [See this project live in our demo!](https://ui.rilldata.com/demo/sample-podcast-project/explore/podcast_explore) --- ####### "Formatting your Measures" --- title: "Formatting your Measures" description: Tips & Tricks for Measure Formatting sidebar_label: "Measure Formatting" sidebar_position: 15 --- When creating your measures in Rill, you have the option to pick from a preset of formats that we provide to you or use the [d3-format](https://d3js.org/d3-format) parameter to format your data in any way you like. While the big number in the explore dashboard won't apply all the decimals changes (it will add currency or percentage if that is the type), you will be able to see the changes in the dimension leaderboard and pivot tables. ![Metrics Editor](/img/build/metrics-view/metrics-editor.png) Using `format_d3` to control the format of a measure in the metrics view allows for further customization. :::tip Invalid format Strings If an invalid format string is supplied, measures will be formatted with `format_preset: humanize`. Measures cannot have both `format_preset` and `format_d3` entries. If neither `format_preset` nor `format_d3` is supplied, measures will be formatted with the `humanize` preset. ::: Please review the reference documentation, [here.](/reference/project-files/metrics-views) ## Customization For further customization of your measures, you can switch to the YAML view and, with our [metrics view reference documentation](/reference/project-files/metrics-views), use the [format_d3_locale](https://d3js.org/d3-format#formatLocale) parameter to create specific formatting. ```yaml format_d3: format_d3_locale: grouping: currency: ``` ## Examples As explained in the introduction, you'll notice that in each of the screenshots the Big Number doesn't always follow the exact formatting, but will change based on percentage/currency formatting. This is as designed, as there is a fixed width that the number has to be displayed in. Instead, you'll see these values in the dimension leaderboard, TDD, and pivot tables. If you have any questions, please review our [reference documentation.](/reference/project-files/metrics-views) ### Format a measure to include specific amount of decimals ![Decimal Example](/img/build/metrics-view/examples/decimal-example.png) In the case that you need to view more granular values of your data, you can set the decimal places to whatever value you need. In the above example, we are setting the average voltage measure to 4 decimal places to get a more accurate representation for each dimension. ```yaml format_d3: ".4f" ``` ### Format currency with different ',' locations. IE: Indian Rupee ![Currency Example](/img/build/metrics-view/examples/currency-example.png) ```yaml format_d3: "$," format_d3_locale: grouping: [3, 2, 2] currency: ["₹", ""] ``` As Indian Rupees are formatted in a different way than USD and EUR, you'll need to use the `format_d3_locale` parameter to set the exact grouping and currency. Likewise, if the currency symbol is written after the numeric value, you can set the currency to `["", "$"]`. ### Percentages ![Percent Example](/img/build/metrics-view/examples/percent-example.png) ```yaml format_d3: '.4%' ``` While our `format_preset: percentage` will automatically apply `.2%`, you can manually set the value in format_d3 if you are looking for a more specific measure format. ## Demo [See this project live in our demo!](https://ui.rilldata.com/demo/rill-kaggle-elec-consumption/explore/household_power_consumption_metrics_explore) --- ####### "Define Your Measures" --- title: "Define Your Measures" description: "Learn how to create and configure measures for quantitative analysis and business insights in your metrics views" --- Measures are the quantitative metrics that power your dashboards and reports. They represent numeric calculations derived from your underlying data through SQL aggregation functions and expressions. These measures transform raw data into meaningful business insights, such as total revenue, average order value, or customer count. Measures are the "how much" and "how many" of your data. They provide the numerical foundation for your analysis, enabling you to: - **Quantify performance**: Track key business metrics like revenue, growth, and efficiency - **Support decision-making**: Provide concrete numbers for business decisions ## Supported SQL Functions * Standard SQL [numeric](https://www.w3schools.com/sql/sql_operators.asp) operators and functions * Common SQL [aggregates](https://www.w3schools.com/sql/sql_aggregate_functions.asp): `AVG`, `COUNT`, `MAX`, `MIN`, `SUM`, `STDDEV`, `VARIANCE` * Advanced aggregates (engine-dependent): `APPROX_COUNT_DISTINCT`, `APPROX_QUANTILE`, `STDDEV_POP`, `STDDEV_SAMP`, `VAR_POP`, `VAR_SAMP` * [Filtered aggregates](https://www.postgresql.org/docs/current/sql-expressions.html#SYNTAX-AGGREGATES) can be used to filter the set of rows fed to the aggregate functions (syntax may vary by engine) :::info Engine-specific SQL dialects Different OLAP engines support varying SQL dialects and functions. While standard SQL functions work across engines, some advanced features may be engine-specific. For engine-specific documentation, see: - **DuckDB**: [DuckDB SQL documentation](https://duckdb.org/docs/sql/introduction.html) - **ClickHouse**: [ClickHouse SQL documentation](https://clickhouse.com/docs/en/sql-reference) - **Druid**: [Druid SQL documentation](https://druid.apache.org/docs/latest/querying/sql.html) - **Pinot**: [Pinot SQL documentation](https://docs.pinot.apache.org/users/user-guide-query/pinot-query-language) ::: As an example, if you have a table of sales events with the sales price and customer ID, you could calculate the following measures with these aggregates and expressions: * Number of sales: `COUNT(*)` * Total revenue: `SUM(sales_price)` * Revenue per customer: `CAST(SUM(sales_price) AS FLOAT)/CAST(COUNT(DISTINCT customer_id) AS FLOAT)` * Number of orders with order value more than $100: `COUNT(*) FILTER (WHERE order_val > 100)` (syntax may vary by engine) Explore these advanced capabilities to enhance your measures: - **[Measure Formatting](/developers/build/metrics-view/measures/measures-formatting)** - Learn how to format and display your measures effectively - **[Case Statements and Filters](/developers/build/metrics-view/measures/case-statements)** - Use conditional logic and filtering in your measures - **[Referencing Measures](/developers/build/metrics-view/measures/referencing)** - Reference and combine existing measures in your calculations - **[Quantiles](/developers/build/metrics-view/measures/quantiles)** - Calculate percentiles and quantiles for statistical analysis - **[Fixed Measures](/developers/build/metrics-view/measures/fixed-measures)** - Create measures with fixed values and constants - **[Window Functions](/developers/build/metrics-view/measures/windows)** - Apply window functions for advanced analytical operations --- ####### "Quantiles" --- title: "Quantiles" description: Tips & Tricks for Measure Quantiles sidebar_label: "Quantiles" sidebar_position: 30 --- In addition to common aggregates, you may wish to look at the value of a measure within a certain band or quantile. In the example below, we can measure the P95 of a given measure using `QUANTILE_CONT`. ![Percentile Visual](/img/build/metrics-view/examples/percentile-visual.png) Using [DuckDB aggregate function](https://duckdb.org/docs/stable/sql/functions/aggregates.html#quantile_contx-pos), you can easily calculate various quantiles. :::tip Not on DuckDB? If you are using a different OLAP engine to power your dashboard, simply use the correct function for quantile calculation. E.g.: [ClickHouse quantile](https://clickhouse.com/docs/sql-reference/aggregate-functions/reference/quantile), [Pinot percentile](https://docs.pinot.apache.org/configuration-reference/functions/percentile) ::: Please review the reference documentation, [here.](/reference/project-files/metrics-views) ## Examples ![Percentile Example](/img/build/metrics-view/examples/percentile-example.png) In this example we see the values of P95 and P99 are calculated using the following expressions: ```yaml - name: p95_quantile_global_intensity expression: QUANTILE_CONT(Global_intensity, 0.95) format_d3: ".3f" description: P95 of Global Intensity - name: p99_quantile_global_intensity expression: QUANTILE_CONT(Global_intensity, 0.99) format_d3: ".4f" description: P99 of Global Intensity ``` ## Demo [See this project live in our demo!](https://ui.rilldata.com/demo/rill-kaggle-elec-consumption/explore/household_power_consumption_metrics_explore) --- ####### "Referencing Measures" --- title: "Referencing Measures" description: Tips & Tricks for Referencing Measure sidebar_label: "Referencing Measures" sidebar_position: 25 --- Within a metrics view, it is possible for a measure to reference another by using the `requires` array parameter. By doing this, you can easily aggregate already existing measures to simplify the expressions. For example, get a percentage of two already summed values. ![Requires Example](/img/build/metrics-view/examples/requires-example.png) Please review the reference documentation, [here.](/reference/project-files/metrics-views) ## Examples ### Simple Aggregation In the following example, `percentage_reactive_to_active_measure` uses the already defined measures `total_global_active_power_measure` and `total_global_reactive_power_measure` to calculate the percentage without having to recalculate the sum of the respective columns. ![Explore Percent](/img/build/metrics-view/examples/explore-percent.png) ```yaml - name: percentage_reactive_to_active_measure display_name: Percent Reactive to Active Power requires: [total_global_active_power_measure, total_global_reactive_power_measure] expression: total_global_reactive_power_measure / total_global_active_power_measure format_preset: percentage ``` ### Window Function If you are using a [window function](./windows), you'll need to define the measure that you are building the window for. In this example, we are getting the rolling sum of average voltage measurements for all time— that's a lot of volts! You can modify the frame to include fewer rows based on the order column. ![Window Example](/img/build/metrics-view/examples/window-example.png) ```yaml - name: rolling_sum_avg_voltage_all_time display_name: Rolling Sum Windowed Voltage Average expression: SUM(average_voltage_measure) requires: [average_voltage_measure] window: order: Date frame: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW treat_nulls_as: -1 ``` ## Demo [See this project live in our demo!](https://ui.rilldata.com/demo/rill-kaggle-elec-consumption/explore/household_power_consumption_metrics_explore) --- ####### "Window Functions" --- title: "Window Functions" description: Tips & Tricks for Window Measures sidebar_label: "Window Functions" sidebar_position: 40 --- In addition to standard metrics, it is possible to define running window calculations on your data, whether you are looking to monitor a cumulative trend, smooth out fluctuations, etc. You'll need to navigate to the Code view in order to create a windowed measure. Please review the reference documentation, [here.](/reference/project-files/metrics-views) ## Example In the example below, bids is another measure defined in the metrics view, and we are getting the previous week and current date's values and averaging them. This allows us to remove any short-term trends to detect real patterns. You'll need to add the [`requires`](./referencing) array parameter to reference another measure. ![Explore Percent](/img/build/metrics-view/examples/explore-percent.png) ```yaml - name: bids_7day_rolling_avg display_name: 7 Day Bid rolling avg expression: AVG(total_bids) requires: [total_bids] window: order: "__time" frame: RANGE BETWEEN INTERVAL 6 DAY PRECEDING AND CURRENT ROW ``` Another example is using a rolling sum with no bounding preceding rows, also known as your whole data. This will be a cumulative sum of all of your measure's data; in this case, it is the average voltage measure. ![Window Example](/img/build/metrics-view/examples/window-example.png) ```yaml - name: rolling_sum_avg_voltage_all_time display_name: Rolling Sum Windowed Voltage Average expression: SUM(average_voltage_measure) requires: [average_voltage_measure] window: order: Date frame: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ``` ## Demo [See this project live in our demo!](https://ui.rilldata.com/demo/rill-kaggle-elec-consumption/explore/household_power_consumption_metrics_explore) --- ##### Time Series --- ####### Time Series in Metrics Views --- title: Time Series in Metrics Views description: "Configure time-based dimensions and aggregations for comprehensive temporal analysis" sidebar_label: Time Series sidebar_position: 07 --- Time is the most critical dimension in analytics and powers our dashboards. Understanding not just the "what," but how metrics evolve over hours, days, and months provides the narrative arc for decision-making. ## Defining Your Time Series Column Your time series must be a column from your data model of type `TIMESTAMP`, `TIME`, or `DATE`. If your source has a date in a different format, you can apply time functions to transform these fields into valid time series types. The specific functions available depend on your OLAP engine. ```yaml # Metrics View YAML # Reference documentation: https://docs.rilldata.com/reference/project-files/metrics_views version: 1 type: metrics_view model: example_model # Choose a table to underpin your metrics timeseries: timestamp_column # Choose a timestamp column from your table ``` ### Time Configuration Parameters You can customize time behavior using the following parameters: **`first_day_of_week`** Specifies which day should be considered the start of the week. Valid values are 1 through 7, where Monday=1 and Sunday=7. **`first_month_of_year`** Determines which month should be treated as the beginning of the year. Valid values are 1 through 12, where January=1 and December=12. ```yaml timeseries: transaction_date first_month_of_year: 7 # July start first_day_of_week: 2 # Monday start ``` These parameters enable you to define non-standard reporting periods. For example, if you set June as the starting month, "past year" calculations will span from June to May instead of January to December. :::tip Full YAML Configurations Please refer to our [reference page](/reference/project-files/metrics-views) for all the available parameters to define in a metrics view. ::: ### Time Grain Configuration The `smallest_time_grain` parameter controls the minimum temporal resolution available in your dashboards. Limiting granularity provides several benefits: - **Performance**: Reduces query complexity and improves dashboard responsiveness - **Consistency**: Ensures all users see data at the same level of detail - **Focus**: Prevents analysis paralysis from overly granular data ```yaml timeseries: order_timestamp smallest_time_grain: day ``` :::note Time Grain Availability The time picker automatically adjusts available time grains based on your selected time range. For example, when viewing "Last 24 hours," only day, and hour grains are available, while "Last 30 days" offers day, week, and month grains. This ensures meaningful time-based analysis appropriate to your data range. ::: ### `watermark` The `watermark` parameter defines the data freshness threshold for your metrics view. It determines the latest point in time where data is considered "complete" and reliable for analysis. **Purpose:** - Prevents analysis of incomplete or partial data - Ensures consistent reporting across different time zones - Provides a clear boundary between complete and incomplete data **Configuration:** ```yaml timeseries: event_time watermark: "MAX(__TIME) - INTERVAL 3 DAYS" ``` **How it works:** - `MAX(__TIME)` gets the latest timestamp in your data - `INTERVAL 3 DAYS` subtracts 3 days from that timestamp - If your latest data is September 5th, complete data extends only to September 2nd - Queries for September 3rd-5th data will return empty or incomplete results - This prevents misleading metrics from partial data **Common watermark expressions:** - `"MAX(__TIME) - INTERVAL 1 DAY"` - For daily batch processing (most common) - `"MAX(__TIME) - INTERVAL 1 HOUR"` - For real-time data with hourly completeness - `"MAX(__TIME) - INTERVAL 1 WEEK"` - For weekly aggregated data - `"MAX(__TIME)"` - No watermark (use with caution) :::tip Best practices Set your watermark based on your data pipeline's processing time. If your ETL takes 2 hours to complete daily data, set watermark to "2 hours" or "1 day" to ensure you only analyze complete datasets. ::: --- ###### Annotations --- title: Annotations description: Enrich your metrics view with external context using annotations sidebar_label: Annotations sidebar_position: 20 --- Annotations allow you to enrich your metrics view with external data sources or context that isn't directly part of the primary model. This is often used to overlay events, holidays, or deployment markers onto your time series charts. ![Annotations](/img/build/metrics-view/annotations.png) ## Requirements The underlying table or model used for annotations must strictly follow this schema: 1. **`time` column (Required)**: Used to position the annotation on the time series chart. 2. **`description` column (Required)**: The text displayed when hovering over the annotation. 3. **`time_end` column (Optional)**: Timestamp - If present, defines the end time for a range-based annotation. 4. **`duration` column (Optional)**: String/Enum - If present, defines the granularity (e.g., 'day', 'hour', 'minute').selected dashboard time grain. It also forces `time` and `time_end` in the UI to be truncated to the selected grain. ## Configuring Annotations To add an annotation, you need to define a reference to a `table` or `model` in your metrics view YAML. ```yaml annotations: - model: annotations_auction name: auction_annotations measures: ['requests'] ``` ### Configuration Properties - **`model`** or **`table`**: Reference to the data source containing the annotation data. - **`name`**: A unique identifier for the annotation set. - **`measures`** (optional): A list of measures to display these annotations alongside. If not specified, the annotation will appear for all measures. ### Visual Appearance Annotations appear as markers or ranges on the time series charts in your dashboard. - **Point Annotations**: Events with a single `time` timestamp appear as point markers. - **Range Annotations**: Events with both `time` and `time_end` timestamps appear as shaded regions spanning the duration. - **Hover Details**: Hovering over an annotation marker reveals the text from the `description` column. :::info Refer to the [`annotations` section](/reference/project-files/metrics-views#annotations) in Metrics View YAML reference for more details on how to implement annotations. ::: --- ###### Derived Metrics Views --- title: Derived Metrics Views description: Create derived metrics views that inherit dimensions, measures, and properties from a parent metrics view sidebar_label: Derived Metrics Views sidebar_position: 06 --- A derived metrics view inherits its dimensions, measures, and data source configuration from an existing parent metrics view. It can only expose a subset of the parent's dimensions and measures — it cannot define new ones. This lets you create focused views for specific teams or use cases without duplicating definitions. The parent metrics view should contain the superset of all dimensions and measures that any derived view might need. ## Basic Example Given a parent metrics view `ad_bids.yaml`: ```yaml version: 1 type: metrics_view model: ad_bids_model timeseries: timestamp smallest_time_grain: hour dimensions: - column: domain - column: city - column: country - column: device measures: - name: total_bids expression: count(*) - name: total_revenue expression: sum(bid_price) - name: avg_bid expression: avg(bid_price) ``` A derived metrics view `ad_bids_summary.yaml` that inherits everything: ```yaml type: metrics_view parent: ad_bids ``` This creates a metrics view with the same dimensions, measures, timeseries, and data source as `ad_bids`. An explore dashboard is automatically emitted for it. ## Selecting Dimensions and Measures Use `parent_dimensions` and `parent_measures` to control which fields are inherited. When omitted, all fields are inherited (equivalent to `'*'`). ### Select All (Wildcard) ```yaml type: metrics_view parent: ad_bids parent_dimensions: '*' parent_measures: '*' ``` ### Select by Name ```yaml type: metrics_view parent: ad_bids parent_dimensions: - domain - country parent_measures: - total_bids - total_revenue ``` ### Exclude Specific Fields Use `exclude` to inherit everything except specific fields: ```yaml type: metrics_view parent: ad_bids parent_dimensions: exclude: - city parent_measures: exclude: - avg_bid ``` ### Select by Regex ```yaml type: metrics_view parent: ad_bids parent_dimensions: regex: "^(domain|country)$" parent_measures: regex: "^total_.*" ``` ### Select by DuckDB Expression ```yaml type: metrics_view parent: ad_bids parent_dimensions: expr: "* EXCLUDE (city)" ``` ## Overriding Inherited Properties A derived metrics view can override certain properties from the parent. Properties you set on the child take precedence; properties you omit are inherited. ### Overridable Properties | Property | Behavior | |---|---| | `display_name` | Child's value if set, otherwise defaults to the resource name (not inherited from parent) | | `description` | Child's value if set, otherwise empty (not inherited from parent) | | `timeseries` | Child's value if set, otherwise parent's | | `smallest_time_grain` | Child's value if set, otherwise parent's. Must be >= parent's grain | | `first_day_of_week` | Child's value if set, otherwise parent's | | `first_month_of_year` | Child's value if set, otherwise parent's | | `watermark` | Child's value if set, otherwise parent's | | `ai_instructions` | Child's value if set, otherwise parent's | ### Always Inherited (Cannot Override) These properties are always taken from the parent: - `model` / `table` (data source) - `connector`, `database`, `database_schema` - Cache settings (`cache.enabled`, `cache.key_sql`, `cache.key_ttl`) ### Example: Override Time Grain The parent uses hourly data, but this derived view restricts to daily granularity: ```yaml type: metrics_view parent: ad_bids display_name: Ad Bids Daily Summary smallest_time_grain: day parent_dimensions: - domain - country parent_measures: '*' ``` :::info When overriding `smallest_time_grain`, the value must be equal to or coarser than the parent's grain. For example, if the parent uses `hour`, the child can use `day` but not `minute`. ::: ## Dimensions and Measures A derived metrics view can only inherit dimensions and measures from its parent — it cannot define its own `dimensions` or `measures` directly. The parent metrics view must contain the superset of all dimensions and measures that any derived view might need. Use `parent_dimensions` and `parent_measures` to select which subset to expose. :::caution Setting `dimensions` or `measures` on a derived metrics view will produce a validation error. All dimensions and measures must be defined on the parent. ::: ## Security Rules All security rules from the parent are inherited by the derived metrics view. The only exception is the `access` rule: if the derived view defines its own `access` rule, the parent's `access` rule is skipped. All other parent rules (`row_filter`, `field_access`, `include`, `exclude`) are always appended to the child's rules regardless. This means a derived view can narrow who has access (by overriding `access`), and can add additional row filters or field restrictions on top of whatever the parent already enforces. ```yaml type: metrics_view parent: ad_bids parent_dimensions: '*' parent_measures: '*' security: access: "'{{ .user.domain }}' = 'partner.com'" row_filter: "country = 'US'" ``` In this example, the child overrides the parent's `access` rule with its own, but the parent's `row_filter`, `include`, `exclude`, and `field_access` rules are still inherited and appended to the child's rules. The child also adds its own `row_filter` on top of the parent's. ## Inline Explore Configuration By default, a derived metrics view automatically emits an explore dashboard. You can customize it using the `explore` key, or disable it entirely. :::note Auto-emission of the explore dashboard only happens when the YAML file does not set `version` (or sets `version: 0`). If `version: 1` is set, no explore is emitted automatically — you must either define one inline with the `explore` key or create a separate explore file. ::: ### Default Behavior When no `explore` key is specified and `version` is not set, an explore dashboard is created with all inherited dimensions and measures. ### Custom Explore ```yaml type: metrics_view parent: ad_bids parent_dimensions: '*' parent_measures: '*' explore: display_name: Ad Bids Partner View time_ranges: - P7D - P30D - range: P90D comparison_offsets: - P90D defaults: dimensions: - domain measures: - total_bids - total_revenue time_range: P7D comparison_mode: time ``` ### Disable Explore ```yaml type: metrics_view parent: ad_bids parent_dimensions: '*' parent_measures: '*' explore: skip: true ``` ### Explore Options The inline `explore` key supports the same configuration as a standalone [explore dashboard](/reference/project-files/explore-dashboards): | Property | Description | |---|---| | `skip` | Set to `true` to disable the explore dashboard | | `name` | Custom name for the explore resource | | `display_name` | Display name shown in the UI | | `description` | Description for the explore | | `banner` | Custom banner at the header of the explore | | `theme` | Theme name or inline theme object | | `time_ranges` | Available time range selections | | `time_zones` | Pinned time zones (IANA identifiers) | | `lock_time_zone` | Lock to the first time zone in `time_zones` | | `allow_custom_time_range` | Allow custom time range selection (default: `true`) | | `defaults` | Default UI state: `dimensions`, `measures`, `time_range`, `comparison_mode`, `comparison_dimension` | | `embeds` | Embed configuration, e.g. `hide_pivot: true` | ## Complete Example A parent metrics view and two derived views for different teams: **`metrics/ad_bids.yaml`** — the shared parent: ```yaml version: 1 type: metrics_view model: ad_bids_model timeseries: timestamp smallest_time_grain: hour dimensions: - column: domain - column: city - column: country - column: device - column: publisher measures: - name: total_bids expression: count(*) - name: total_revenue expression: sum(bid_price) - name: avg_bid expression: avg(bid_price) - name: unique_domains expression: count(distinct domain) security: access: true row_filter: "bid_price > 0" ``` **`metrics/ad_bids_sales.yaml`** — for the sales team: ```yaml type: metrics_view parent: ad_bids display_name: Ad Bids - Sales parent_dimensions: - domain - country parent_measures: - total_revenue - avg_bid smallest_time_grain: day explore: display_name: Sales Dashboard defaults: time_range: P30D measures: - total_revenue ``` **`metrics/ad_bids_engineering.yaml`** — for the engineering team: ```yaml type: metrics_view parent: ad_bids display_name: Ad Bids - Engineering parent_dimensions: '*' parent_measures: exclude: - avg_bid security: access: "'{{ .user.department }}' = 'engineering'" explore: defaults: time_range: P7D comparison_mode: time ``` ## Validation Rules The parser enforces these constraints on derived metrics views: - `parent` must reference an existing metrics view with a valid state. - `dimensions` and `measures` cannot be defined on a derived view — use `parent_dimensions` and `parent_measures` to select from the parent instead. - `model`, `table`, `database`, `database_schema`, and cache settings cannot be set on a derived view (they come from the parent). - `parent_dimensions` and `parent_measures` can only be used when `parent` is set. - `smallest_time_grain`, if specified, must be coarser than or equal to the parent's grain. - Deprecated top-level explore fields (`default_time_range`, `available_time_zones`, etc.) cannot be used; use the `explore` key instead. --- ###### Metrics SQL --- title: Metrics SQL description: Query metrics views using SQL syntax sidebar_label: Metrics SQL --- You can write a SQL query referring to metrics definitions and dimensions defined in a metrics view. It should have the following structure: ```yaml type: api metrics_sql: SELECT publisher, domain, total_records FROM ad_bids_metrics ``` ## Querying Fundamentals Metrics SQL transforms queries that reference `dimensions` and `measures` within a `metrics view` into their corresponding database columns or expressions. This transformation is based on the mappings defined in a metrics view YAML configuration, enabling reuse of dimension or measure definitions. Additionally, any security policies defined in the metrics view are also inherited. ## Example: Crafting a Metrics SQL Query Consider a metrics view configured as follows: ```yaml #metrics/ad_bids_metrics.yaml type: metrics_view title: Ad Bids model: ad_bids timeseries: timestamp dimensions: - name: publisher expression: toUpper(publisher) - name: domain column: domain measures: - name: total_records display_name: Total records expression: COUNT(*) ``` To query this view, a user might write a Metrics SQL query like: ```sql SELECT publisher, domain, total_records FROM ad_bids_metrics ``` This Metrics SQL is internally translated to a standard SQL query as follows: ```sql SELECT toUpper(publisher) AS publisher, domain AS domain, COUNT(*) AS total_records FROM ad_bids_metrics GROUP BY publisher, domain ``` ## Security and Compliance Queries executed via Metrics SQL are subject to the security policies and access controls defined in the metrics view YAML configuration, ensuring data security and compliance. ## Supported SQL Features ### SELECT Reference dimensions and measures by name. The `date_trunc` function can be used to group a time dimension by a specific grain (and optionally aliased with `AS`): ```sql SELECT date_trunc('MONTH', timestamp) AS month, publisher, total_records FROM ad_bids_metrics ``` Supported grains: `SECOND`, `MINUTE`, `HOUR`, `DAY`, `WEEK`, `MONTH`, `QUARTER`, `YEAR`. ### FROM A single metrics view name. Joins and subqueries in the FROM clause are not supported. ### WHERE and HAVING `WHERE` filters on dimensions; `HAVING` filters on aggregated measures. Both support the same operators and functions. ### ORDER BY, LIMIT, OFFSET Standard SQL sorting and pagination clauses are supported: ```sql SELECT publisher, total_records FROM ad_bids_metrics ORDER BY total_records DESC LIMIT 20 OFFSET 40 ``` ## Operators The following operators are supported in `WHERE` and `HAVING` clauses: | Operator | |----------| | `=`, `!=`, `<`, `<=`, `>`, `>=` | | `AND`, `OR`, `(` `)` | | `IN (...)`, `NOT IN (...)` | | `LIKE`, `NOT LIKE` (case-insensitive, `%` wildcard) | | `BETWEEN ... AND ...` | | `IS NULL`, `IS NOT NULL` | | `IS TRUE`, `IS FALSE`, `IS NOT TRUE`, `IS NOT FALSE` | ```sql SELECT publisher, total_records FROM ad_bids_metrics WHERE (publisher IS NOT NULL AND domain LIKE '%google%') OR publisher IN ('Yahoo', 'Microsoft') ``` ## Functions ### Time range functions `time_range_start` and `time_range_end` resolve a Rill time expression against the metrics view's watermark and time range. They must be compared against the time dimension: ```sql SELECT publisher, total_records FROM ad_bids_metrics WHERE timestamp > time_range_start('7D as of watermark/D+1D') AND timestamp <= time_range_end('7D as of watermark/D+1D') ``` ### Interval arithmetic Add or subtract an interval from a timestamp literal using `INTERVAL amount UNIT` syntax. Supported units: `SECOND`, `MINUTE`, `HOUR`, `DAY`, `WEEK`, `MONTH`, `YEAR`. ```sql SELECT publisher, total_records FROM ad_bids_metrics WHERE timestamp > '2024-07-30' - INTERVAL 90 DAY ``` ### now() Returns the current timestamp: ```sql SELECT publisher, total_records FROM ad_bids_metrics WHERE timestamp > now() - INTERVAL 7 DAY ``` ### CAST Casting to `DATETIME` or `TIMESTAMP` is supported (other target types are not): ```sql SELECT publisher, total_records FROM ad_bids_metrics WHERE timestamp > CAST('2024-01-01' AS TIMESTAMP) ``` ## Subqueries Subqueries are supported inside `IN` expressions. The subquery must select exactly one dimension from the same metrics view and can include its own `WHERE` and `HAVING` clauses: ```sql SELECT publisher, total_records FROM ad_bids_metrics WHERE publisher IN ( SELECT publisher FROM ad_bids_metrics HAVING total_records > 100 ) ``` Subqueries do not support `ORDER BY`, `LIMIT`, `DISTINCT`, window functions, joins, or CTEs. ## Limitations - Only one metrics view can be queried per statement (no joins). - `SELECT *` is not supported; list dimensions and measures explicitly. - `GROUP BY` is implicit based on selected dimensions and cannot be specified manually. - Aggregate functions like `COUNT()` or `SUM()` cannot be used directly; reference predefined measures instead. - Set operations (`UNION`, `INTERSECT`, `EXCEPT`) and CTEs (`WITH`) are not supported. :::warning The Metrics SQL feature is currently evolving. We are dedicated to enhancing the syntax by introducing additional SQL features, while striving to maintain support for existing syntax. However, please be advised that backward compatibility cannot be guaranteed at all times. Additionally, users should be aware that there may be untested edge cases in the current implementation. We appreciate your understanding as we work to refine and improve this feature. ::: ## Using Metrics SQL in custom APIs To expose Metrics SQL queries as HTTP API endpoints, see the [Metrics SQL APIs](/developers/build/custom-apis/metrics-sql) guide. You can also add [dynamic templating](/developers/build/custom-apis/templating), [security rules](/developers/build/custom-apis/security), and [OpenAPI documentation](/developers/build/custom-apis/openapi) to your Metrics SQL APIs. --- ###### Create Metrics Views --- title: Create Metrics Views description: Create metrics views using source data and models with time, dimensions, and measures sidebar_label: Create Metrics Views sidebar_position: 00 --- Metrics Views are a key part of defining your measures and dimensions. ### Metrics 101 - [**What are Metrics Views?**](/developers/build/metrics-view/what-are-metrics-views) - Define your time series, measures, and dimensions using your OLAP engine aggregate functions - [**Underlying Model/Table**](/developers/build/metrics-view/underlying-model) - When to use `model` or `table` in a metrics view - [**Time Series**](/developers/build/metrics-view/time-series) - What is a time series column and how to use it in your metrics view - [**Derived Metrics Views**](/developers/build/metrics-view/derived-metrics-views) - Create focused subsets by inheriting from a parent metrics view ### Dimensions - [**Dimensions**](/developers/build/metrics-view/dimensions) - Categorize and segment your data for analysis and filtering - [**Navigatable Dimensions**](/developers/build/metrics-view/dimensions/dimension-uri) - Generate navigable URIs for your dimension values - [**Unnesting Arrays**](/developers/build/metrics-view/dimensions/unnesting) - Work with array data by expanding nested structures - [**Time Dimensions**](/developers/build/metrics-view/dimensions/time-dimensions) - Add multiple time-based dimensions for filtering across different date columns - [**Lookup Dimensions**](/developers/build/metrics-view/dimensions/lookup) - Create dimensions that reference data from other sources ### Measures - [**Measures**](/developers/build/metrics-view/measures) - Define calculated metrics and aggregations for your data analysis - [**Format Measures**](/developers/build/metrics-view/measures/measures-formatting) - Learn how to format and display your metrics effectively - [**Case Statements and Filters**](/developers/build/metrics-view/measures/case-statements) - Use conditional logic and filtering in your measures - [**Referencing Measures**](/developers/build/metrics-view/measures/referencing) - Reference and combine existing measures in your calculations - [**Quantiles**](/developers/build/metrics-view/measures/quantiles) - Calculate percentiles and quantiles for statistical analysis - [**Fixed Measures**](/developers/build/metrics-view/measures/fixed-measures) - Create measures with fixed values and constants - [**Window Functions**](/developers/build/metrics-view/measures/windows) - Apply window functions for advanced analytical operations ### Performance - [**Rollups**](/developers/build/metrics-view/rollups) - Accelerate queries by routing to pre-aggregated rollup tables ### Security - [**Data Access Policies**](/developers/build/metrics-view/security) - Control access to your metrics data with role-based permissions --- ###### Rollups --- title: Rollups description: Accelerate metrics view queries by routing to pre-aggregated rollup tables sidebar_label: Rollups sidebar_position: 25 --- Rollups let a metrics view be backed by one or more pre-aggregated tables in addition to the base table. When a query's time grain, dimensions, measures, time range, and filters all match a rollup, Rill transparently rewrites the query to read from the rollup table instead of the base. ## Defining a Rollup A rollup is defined as a separate table in the external olap engine or as a model that pre-aggregates the base model, plus a `rollups` entry in the metrics view YAML that points to it. The rollup model must produce columns with the same names as the base model's time dimension, the dimensions you list, and the measure expression inputs. The simplest case — a daily rollup of an hourly fact table: ```yaml # metrics_views/events.yaml type: metrics_view version: 1 model: events timeseries: timestamp dimensions: - name: publisher column: publisher - name: domain column: domain - name: country column: country measures: - name: total_impressions expression: SUM("impressions") - name: total_clicks expression: SUM("clicks") rollups: - model: events_daily time_grain: day dimensions: [publisher, domain] measures: [total_impressions, total_clicks] ``` Note that the rollup omits `country`. Queries that group by or filter on `country` will fall back to the base table; queries on `publisher` and/or `domain` at day grain or coarser will use the rollup. ### Multiple Rollups You can define several rollups at different grains. Rill will pick the most efficient one that can answer a given query: ```yaml rollups: - model: events_daily time_grain: day dimensions: [publisher, domain] measures: [total_impressions, total_clicks] - model: events_monthly time_grain: month dimensions: [publisher, domain] measures: [total_impressions, total_clicks] ``` A query at month grain over the full year will be served from `events_monthly`; a query at day grain over a single month will be served from `events_daily`. ### Field Selectors `dimensions` and `measures` accept the standard field-selector forms — an explicit list, a wildcard, a regex, or an exclusion, if not defined then all dimensions and measures are included. For example, this rollup includes all dimensions and all measures except `total_clicks` measure: ```yaml rollups: - model: events_daily time_grain: day measures: exclude: [total_clicks] # all measures except total_clicks ``` ### Declaring Coverage with `data_time_range` By default Rill discovers a rollup's time coverage at query time by running `SELECT min(time), max(time)` against the rollup table. If you'd rather declare coverage statically — to skip the probe, or to scope a rollup to a specific window — set `data_time_range` on the rollup. The value is a [rilltime expression](/reference/time-syntax): ```yaml rollups: - model: events_hourly time_grain: hour data_time_range: -1Y to now # rolling last 12 months dimensions: [publisher, domain] measures: [total_impressions] - model: events_daily_archive time_grain: day data_time_range: -5Y to -1Y # 1 to 5 years ago dimensions: [publisher, domain] measures: [total_impressions] ``` A query for last week routes to `events_hourly`; a query 18 months back routes to `events_daily_archive`; a query reaching past 5 years falls back to the base. You can declare coverage on the metrics view itself the same way — this skips the probe on the base table: ```yaml data_time_range: -5Y to now ``` When `data_time_range` is set, the rilltime expression is resolved against fixed anchors: `now`/`latest`/`watermark` all resolve to the current wallclock. The start must be bounded — `inf` and `earliest` are rejected, because they resolve start time to the zero time value, which the system treats as "no data". To declare full historical coverage, either omit `data_time_range` so the bounds are probed from the table, or use a concrete early bound such as `-100Y to now`. Mixing declared and undeclared rollups in the same metrics view is fine — each table independently decides whether to probe or to use its declaration. ### Selection Priority and Definition Order When two rollups have the same time grain and both could answer a query, the **first one declared in the YAML wins**. This is the lever for priority-style routing: list narrower / fewer-row rollups first. ```yaml rollups: # Same grain, different dimension sets. Priority is declared by order. - model: events_daily_narrow # selected for queries that only need publisher time_grain: day dimensions: [publisher] measures: [total_impressions] - model: events_daily_wide # selected when publisher + domain are queried time_grain: day dimensions: [publisher, domain] measures: [total_impressions] - model: events_daily_wider # selected when publisher + domain + country are queried time_grain: day dimensions: [publisher, domain, country] measures: [total_impressions] ``` A query on `publisher` alone is eligible against all three — `events_daily_narrow` wins because it's listed first. A query on `publisher` + `domain` knocks `events_daily_narrow` out of eligibility, so `events_daily_wide` wins. ### Configuration Reference - **`model`** (required) — The pre-aggregated table or model. - **`time_grain`** (required) — Grain of the rollup. One of `millisecond`, `second`, `minute`, `hour`, `day`, `week`, `month`, `quarter`, `year`. - **`time_zone`** (optional) — IANA timezone the rollup was bucketed in (e.g. `America/New_York`). For day and coarser grains, queries are routed to the rollup only if their timezone matches. - **`database`**, **`database_schema`** (optional) — Override the OLAP database and schema for the rollup table. - **`dimensions`** (optional) — Field selector for which base-view dimensions are present in the rollup. Defaults to all. - **`measures`** (optional) — Field selector for which base-view measures are present in the rollup. Defaults to all. - **`data_time_range`** (optional) — Rilltime expression describing the rollup's time coverage. When set, Rill skips the OLAP `min/max` probe for this rollup and uses the declared bounds for coverage checks. A metrics view must define a `timeseries` to use rollups. The metrics view itself also accepts a top-level `data_time_range` to declare the base table's coverage. The full schema is documented in the [metrics view reference](/reference/project-files/metrics-views#rollups). ## How Rollup Selection Works For each query, Rill walks through three phases: a quick disqualification, a per-rollup eligibility check, and a selection step among the eligible rollups. ### 1. Quick Disqualification The whole rollup system is skipped — and the base table is used — when: - The query asks for raw rows rather than aggregates. - The query has a comparison time range. (Time comparison queries always read the base table.) - The query's time range is on a time dimension that isn't the metrics view's primary `timeseries`. ### 2. Eligibility A rollup is eligible for a given query only if **all** of the following hold: 1. **Grain derivable.** The query's time grain can be aggregated up from the rollup's grain. For example, a `month` query can be derived from a `day` rollup, but a `week` query cannot be derived from a `month` rollup, and a `month` query cannot be derived from a `week` rollup. Sub-day grains form one chain (`ms → s → min → hour → day`); calendar grains form another (`day → month → quarter → year`); `week` sits on its own branch and can only be derived from `day` or finer. 2. **Timezone matches** (day grain and coarser). The query's timezone must equal the rollup's `time_zone`. UTC variants (`""`, `"UTC"`, `"Etc/UTC"`) are treated as equivalent. Sub-day grains are timezone-agnostic, so this check is skipped for them. 3. **Start aligned.** The query's time-range start must fall exactly on a rollup-grain boundary in the rollup's timezone. A query starting at `2024-01-01 12:00` cannot use a daily rollup, because the noon boundary doesn't line up with a day bucket. 4. **All queried dimensions present.** Every dimension in the query (group-by, time floor, or WHERE filter) must be in the rollup's `dimensions` list. The primary time dimension is always considered present. 5. **All queried measures present.** Every measure named in the query must be in the rollup's `measures` list. Computed measures like `COUNT(*)` or `COUNT(DISTINCT …)` are rejected outright — they would produce wrong results when applied on top of pre-aggregated rows. ### 3. Time Coverage For each eligible rollup, Rill checks that the rollup actually contains data for the requested range: - **With a time range.** The query range is first clamped to the base table's `[min, max]` (so a query that extends past the base data isn't penalized for the rollup also stopping there). The rollup must then cover the clamped start and end. - **Without a time range** ("all data"). The rollup must cover the base table's full `[min, max]`. - **End alignment.** If the base table has data beyond the query's end time, the end must also be aligned to the rollup grain. Otherwise the last rollup bucket would pull in data from outside the requested range. Queries whose end falls past the latest base data don't need to be end-aligned, because there is no extra data to pull in. ### 4. Selection Among rollups that pass eligibility and coverage: 1. Prefer the **coarsest grain** — fewer rows to scan. 2. On a tie, prefer the rollup **declared earlier** in the `rollups` list — this is the lever for explicit priority among same-grain rollups. The base table is used if no rollup is eligible. ## Limitations and Edge Cases - **Time grain must be derivable.** `week` and the calendar grains (`month`, `quarter`, `year`) live on separate branches. A weekly rollup cannot answer monthly queries, and a monthly rollup cannot answer weekly queries. Define rollups at the grain you actually query. - **Day-and-above rollups are timezone-specific.** A rollup bucketed in UTC cannot serve a dashboard query in `America/New_York`, because the day boundaries are different. If your users query in multiple timezones, either materialize a rollup per timezone or keep the rollup at hour grain (which is timezone-agnostic). - **Misaligned starts disqualify the rollup.** A query starting mid-bucket (e.g. `2024-01-01 12:00` against a daily rollup) silently falls back to the base table. Dashboard time-range pickers typically snap to grain boundaries; ad-hoc API queries may not. - **Computed measures fall back.** `count` and `count_distinct` measures bypass rollups even if the rollup looks otherwise suitable, because counting pre-aggregated rows is not the same as counting raw rows. Define an explicit `SUM(...)` measure on a pre-aggregated counter column in the rollup if you want this case to route. - **Derived measures fall back.** A measure of type `derived` (one with `requires` or `per`) cannot match a rollup's measure list — only `simple` measures can. The base table is used. - **Rollups require a `timeseries`.** Metrics views without a primary time dimension cannot define rollups. - **Filters on missing dimensions disqualify the rollup.** A WHERE clause on `country` will skip a rollup that doesn't include `country`, even if the query's group-by columns are all in the rollup. - **The rollup is responsible for being correct.** Rill does not validate that the rollup's measure values are consistent with the base — it trusts the model. If the rollup model uses the wrong aggregation (e.g. `AVG` where the base measure is `SUM`), queries routed to it will return wrong numbers. - **Rollups are assumed to be roughly caught up with the base table.** Coverage is measured against the base table's latest timestamp. A rollup that lags behind the base will be silently skipped for any query that reaches the tail of the data — including common "last 24 hours" queries and queries without a time range — even if it has the right grain, dimensions, and measures. Refresh rollups in step with the base model so selection actually happens. - **Rollups must not extend beyond the base table.** Routing assumes the rollup's max timestamp is no later than the base's. A rollup that gets ahead of the base (e.g. ingested through a separate path) may return incorrect results at the tail of the data. :::info The full configuration schema is in the [metrics view reference](/reference/project-files/metrics-views#rollups). ::: --- ###### Who Can Access Your Data --- title: Who Can Access Your Data description: Control who can view your metrics and data sidebar_label: Data Access Control sidebar_position: 50 --- Rill supports **granular access policies** that let you control: - **Who can access your data** - **What rows they can see** - **Which dimensions and measures are visible** Policies are based on user attributes such as **email address**, **domain**, or **custom attributes**. This avoids dashboard sprawl — instead of creating multiple dashboards for each audience, you can build _**one dashboard**_ and tailor it for many teams and use cases. ## How Does It Work? Access policies are defined in the **metrics view** and/or **[dashboard YAML](/developers/build/dashboards/customization#define-dashboard-access)**. There are three types of rules: **General Access:** (`access`) A boolean expression deciding if a user can access the metrics view ```yaml security: access: "{{ .user.admin }} OR '{{ .user.domain }}' == 'example.com'" ``` :::info Dashboard access `access` can be set on both the dashboard YAML and metrics view YAML and policies are combined using logical AND operations. If no policies are defined on the dashboard, they are derived from the metrics view. For most set-ups, setting the access on the metrics view is sufficient. ::: **Row-level access** (`row_filter`) – a SQL expression that will be injected into the WHERE clause of all dashboard queries to restrict access to a subset of rows ```yaml security: row_filter: region = '{{ .user.region }}' ``` **Column-level access** (`include` or `exclude`) – lists of boolean expressions that determine which dimension and measure names will be available to the user ```yaml security: exclude: - if: "'{{ .user.domain }}' != 'example.com'" names: - ssn - id ``` When a user loads a dashboard, the policies are resolved in two phases: 1. The templating engine first replaces expressions like `{{ .user.domain }}` with actual values ([Templating reference](/developers/build/connectors/templating)) 2. The resulting expression is then evaluated contextually: - The `access` and `if` values are evaluated as SQL expressions and resolved to a `true` or `false` value - The `row_filter` value is injected into the `WHERE` clause of the SQL queries used to render the dashboard :::info What about MCP, and APIs? Metrics views limit data access for all requests, including MCP integrations and custom APIs. When creating a token or copying from the AI tab, the user's attributes (such as email, domain, groups, and custom attributes) are automatically included in the request context. This ensures that the same security policies that apply to dashboard users also apply to programmatic access, maintaining consistent data governance across all access methods. For more details, see [Service Tokens](/guide/administration/access-tokens/service-tokens). ::: Typical use cases include: - [**Granting or Restricting Access**](#restrict-data-access-to-users-matching-specific-criteria) to data and, as a result, dashboards - [**Hiding specific dimensions and measures**](#conditionally-hide-a-dashboard-dimension-or-measure) from specific groups of users, creating a tailored dashboard experience - [**Restricting Access to Internal users**](#hide-dimensions-or-measures-for-members-of-a-certain-group) of your organization, allowing specific dashboards to be viewed by internal users only - [**Partner-filtered Dashboards**](#show-only-data-from-the-users-own-domain) where external users can only access the subset of their data - [**Embedded**](#advanced-example-custom-attributes-embed-dashboards) use cases, passing custom attributes to Rill :::tip Project Access Required Access Policies assume that the user already has access to the project in Rill Cloud. For more information on user management, see our [User Management](/guide/administration/users-and-access/user-management) and [Project Management](/guide/administration/project-settings) documentation. ::: ## Creating Access Policies There are two locations that control data access in Rill. ### Project Level Defaults By default, when a user is granted access to your project, they have access to all metrics views and, if there is [no dashboard policy](/developers/build/dashboards/customization#define-dashboard-access), all dashboards. While this is the default behavior, it can be easily changed in the project's `rill.yaml`. This will lock down all metrics views and block all users who are not Rill Administrators or do not have 'example.com' as their domain. :::tip Set project-wide security defaults Configure default security policies for all metrics views and dashboards in your project. [Learn more about security defaults →](/developers/build/project-configuration#metrics-views-security-policy) ::: ```yaml metrics_views: security: access: "{{ .user.admin }} OR '{{ .user.domain }}' == 'example.com'" row_filter: "partner_id IN (SELECT id FROM mapping WHERE partner_domain = '{{ .user.domain }}') OR '{{ .user.domain }}' = 'example.com'" exclude: - if: "'{{ .user.domain }}' != 'example.com'" names: - ssn - id ``` ### Object Specific Policies You can define policies directly in a specific metrics view or dashboard YAML to override the project-level defaults. ```yaml security: access: '{{ has "partners" .user.groups }}' row_filter: "domain = '{{ .user.domain }}'" exclude: - if: "'{{ .user.domain }}' != 'example.com'" names: - ssn - id ``` :::tip Access Policy Behavior When combining access policies from project defaults and object-specific policies, remember that the object level policies will overwrite the project level ones. Dashboard and metrics view policies are combined using logical AND operations. ::: ## Dashboard Access Dashboards also have an `access` key that can add additional security to the metrics view. Both [explore](/developers/build/dashboards/customization#define-dashboard-access) and [canvas](/developers/build/dashboards/customization#define-dashboard-access) dashboards can set the following: ```yaml security: access: "'{{ .user.domain }}' == 'example.com'" ``` This will logically AND with your metrics view's access so ensure that a user who needs access to the dashboard passes **both** conditions. :::tip complicated set-ups Access Policies can get quite complicated as your use case grows and having to navigate multiple files to figure out why a user is able to or unable to access certain dashboards. A few recommendations: 1. Only change project level access if absolutely necessary. (They get overwritten by object level security) 2. Dashboard access can be derived from the metrics view, only add extra policies on the dashboard if absolutely necessary as this gets combined with the metrics view using logical AND operations anyway. 3. Solve project access issues higher up in the [user](/guide/administration/users-and-access/user-management) / [usergroup](/guide/administration/users-and-access/usergroup-management) settings, and keep default project security rules. ::: ## User Attributes - `.user.email` – the current user's email address, for example john.doe@example.com (string) - `.user.domain` – the domain of the current user's email address, for example example.com (string) - `.user.name` - the current user's name, for example John Doe (string) - `.user.admin`– a boolean value indicating whether the current user is an org or project admin, for example true (bool) - `.user.groups` - a list of user groups the user belongs to in the project's org (list of strings), e.g. ["marketing","sales","finance"] - `.user.attribute` - where `attribute` is a custom variable that you can pass via an embedded dashboard from your application Note: Rill requires users to confirm their email address before letting them interact with the platform, so a user cannot fake an email address or email domain. ## Testing Policies in Rill Developer In development (on `localhost`), you can test your policies by adding "mock users" to your project and viewing the dashboard as one of them. :::tip Test policies in Rill Developer Use `mock_users` in rill.yaml to test your security policies before deploying. [Learn more about testing security →](/developers/build/project-configuration#testing-security) ::: In your project's `rill.yaml` file, add a `mock_users` section. Each mock user must have an `email` attribute and can optionally have `name` and `admin` attributes. For example: ```yaml # rill.yaml mock_users: - email: john@yourcompany.com name: John Doe admin: true - email: jane@partnercompany.com groups: - partners - email: anon@unknown.com ``` On the dashboard page (provided you've added a policy), you'll see a "View as" button in the top right corner. Click this button and select one of your mock users. You'll see the dashboard as that user would see it. ### Rill Cloud If you want to test what your users are seeing in Rill Cloud after deploying, you can find this in the dropdown of your account. You will see the actual users in the dropdown of this list, not the mock users defined in the rill.yaml file. ![Rill Cloud View As](/img/manage/access-policies/rill-cloud-view-as.png) ### Embedded Dashboards When [requesting an embedded dashboard from Rill](/developers/embed/iframe) from your frontend, you can pass the `attributes` parameter with custom names to ensure that the resulting dashboard displays the correct information. For more information, see [our embedding docs](/developers/embed/iframe#2-build-the-iframe-url-backend). ## Examples ### Restrict data access to users matching specific criteria Let's say you want to restrict dashboard access to admin users or users whose email domain is `example.com`. Add the following clause to your metrics view's YAML: ```yaml security: access: "{{ .user.admin }} OR '{{ .user.domain }}' == 'example.com'" ``` :::note DEFAULT SECURITY IS FALSE If the `security` section is defined and `access` is not, then `access` will default to `false`, meaning that it won't be accessible to anyone and users will need to be invited individually. ::: ### Restrict dashboard access to specific user groups Group membership can be utilized to specify which users have access to a specific dashboard (using the templating function `has`). For example: ```yaml security: access: '{{ has "partners" .user.groups }}' ``` ### Show only data from the user's own domain You can limit the data available to the dashboard by applying a filter on the underlying data. Assuming the dashboard's underlying model has a `domain` column, adding the following clause to the dashboard's YAML will only show dimension and measure values for the current user's email domain: ```yaml security: access: true row_filter: "domain = '{{ .user.domain }}'" ``` :::note FILTERS SHOULD BE VALID SQL The `row_filter` value needs to be valid SQL syntax for a `WHERE` clause. It will be injected into every SQL query used to render the dashboard. ::: ### Conditionally hide a dashboard dimension or measure You can include or exclude dimensions and measures based on a boolean expression. For example, to exclude dimensions named `ssn` and `id` for users whose email domain is not `example.com`: ```yaml security: access: true exclude: - if: "'{{ .user.domain }}' != 'example.com'" names: - ssn - id ``` Alternatively, you can explicitly define the dimensions and measures to include using the `include` key. It uses the same syntax as `exclude` and automatically excludes all names not explicitly defined in the list. See the [Dashboard YAML](/reference/project-files/explore-dashboards) reference for details. ### Use wildcards to select all dimensions and measures When defining inclusion policies, you can easily and automatically select all columns by using `names: '*'` as a wildcard. For example: ```yaml security: access: true include: - if: true names: - ssn - id - if: "{{ .user.admin }}" names: '*' ``` Note that the `'*'` must be quoted (using single or double quotes) and **must** be provided as a scalar value, not as an entry in a list. ### Filter queries based on the user's groups You can directly inject the groups that a user belongs to into the row filter itself, such as: ```yaml security: access: true row_filter: "groups IN ('{{ .user.groups | join \"', '\" }}')" ``` ### Hide dimensions or measures for members of a certain group You can check group membership using the templating function `has`. For example: ```yaml security: access: true exclude: - if: '{{ has "partners" .user.groups }}' names: - cost - profit ``` ### Advanced Example: Mapping Dimensions and Attributes For some use cases, the built-in user attributes do not provide sufficient context to correctly restrict access. For example, a dashboard for a multi-tenant SaaS application might have a `tenant_id` column, and external users should only be able to see data for the tenant they belong to. To support this, ingest a separate data [source](/developers/build/connectors) containing mappings of user email addresses to tenant IDs and reference it in the row-level filter. This can be a locally created CSV file or any hosted data source. For example, a locally created `mappings.csv` file in the `data` directory of your Rill project with the following contents: ```csv email,tenant_id john.doe@example.com,1 jane.doe@example.com,2 ``` This needs to be ingested as a source in Rill like any other data source: ```yaml # sources/mappings.yaml type: local_file path: data/mappings.csv ``` (In practice, you would probably ingest the data from a regularly updated export in S3 with a source refresh.) We can now refer to the mappings data using a SQL sub-query as follows: ```yaml security: access: true row_filter: "tenant_id IN (SELECT tenant_id FROM mappings WHERE email = '{{ .user.email }}')" ``` ### Advanced Example: Custom attributes (Embed Dashboards) Another use case for row access policies is to ensure that your embedded dashboard provides a specific view for your end users. During the [embed dashboard request](/developers/embed/iframe), you can pass custom attributes (other than the ones provided out-of-the-box) that map directly to a value within your Rill explore dashboard. ```yaml security: access: true row_filter: > dimension_1 = '{{ .user.custom_variable_1 }}' AND dimension_2 = '{{ .user.custom_variable_2 }}' ``` In order to test the view of your embedded dashboard, you can add the same custom variables to [your mock users](#testing-policies-in-rill-developer) as seen below: ```yaml - email: embed@rilldata.com name: embed custom_variable_1: Value_1 custom_variable_2: Value_2 ``` ### Advanced Example: Access to Dashboard in Rill and Embedded While not common, there are use cases where a dashboard is used both in the Rill Cloud UI and as an embedded dashboard. In this case, passing a similar user attribute could suffice, but if you need to pass a custom attribute, you'll need to add an extra layer of logic to your dashboard. ```yaml security: access: "{{ .user.admin }} OR '{{ .user.domain }}' == 'example.com' {{- if .user.custom_variable_1 }} OR true {{- end }}" ``` --- ###### Powering your Metrics View --- title: Powering your Metrics View sidebar_label: Underlying Model/Table sidebar_position: 05 --- Once you have finished [building your model](/developers/build/models), you can create a metrics view to define measures and dimensions for your dashboard. The way you specify the underlying data source depends on your OLAP engine. ## Choosing Your Data Source Rill supports [multiple OLAP engines](/developers/build/connectors/olap), and the engine you're using determines which YAML property you'll use in your metrics view: - **Use `model`** for DuckDB and Rill-managed ClickHouse - **Use `table`** for self-managed live connectors ## DuckDB and Rill-Managed ClickHouse For DuckDB (the default engine) and Rill-managed ClickHouse, use the `model` property to reference your data model: ```yaml # Metrics View YAML # Reference documentation: https://docs.rilldata.com/reference/project-files/metrics-views version: 1 type: metrics_view model: example_model # Choose a model to underpin your metrics view ``` ## Self-Managed Live Connectors For self-managed live connectors (like your own ClickHouse, MotherDuck, or Druid instance), use the `table` property and specify connection details: ```yaml # Metrics View YAML # Reference documentation: https://docs.rilldata.com/reference/project-files/metrics-views version: 1 type: metrics_view database: default connector: clickhouse database_schema: billing table: events # Choose a table to underpin your metrics view ``` For more information, refer to our [metrics view YAML configuration](/reference/project-files/metrics-views). ### Caching Query Results By default, dashboard queries against metrics views backed by an external, self-managed connector (Snowflake, BigQuery, Databricks, your own ClickHouse, Druid, MotherDuck, Pinot, etc.) run live against the source on every interaction. For dashboards with many concurrent users or repeated drill-downs, this can drive up compute costs and add latency. Enable caching to reuse query results between users until the underlying data changes. Caching is configured under the `cache` block on the metrics view. Because tables in live connectors are externally managed, caching is **off by default** — opt in by setting `cache.enabled: true`. By default, Rill invalidates the cache when the max timestamp of the metrics view's `timeseries` column changes. If you have a better indicator for when to invalidate the cache (for example, an ingest version column), use `cache.key_sql` to set a custom expression for the cache key. ```yaml type: metrics_view connector: snowflake database: MY_DATABASE database_schema: MY_SCHEMA table: MY_TABLE timeseries: created_at dimensions: - column: region measures: - name: total_revenue expression: SUM(revenue) cache: enabled: true key_sql: SELECT MAX(created_at) FROM MY_DATABASE.MY_SCHEMA.MY_TABLE key_ttl: 5m ``` Rill runs `key_sql` against the metrics view's connector (re-evaluated at most once per `key_ttl`) and uses the returned scalar value as the cache key (together with a hash of the incoming metrics query). When the value changes — for example because a new row landed — the cache is invalidated and the next query repopulates it. **Pros** - **Lower source spend.** Repeat queries (multiple users on the same dashboard, back-and-forth filtering) are served from Rill's in-memory cache instead of hitting the source. On BigQuery this means fewer bytes scanned; on Snowflake and Databricks it means less warehouse compute; on ClickHouse, Druid, and Pinot it means fewer broker queries. - **Faster cache hits.** Cache lookups skip warehouse warm-up, query queue time, and per-query slot allocation. End-to-end latency depends on your Rill deployment topology. - **Tunable freshness.** `key_sql` ties cache invalidation to your data — typically a max event timestamp, an ingest version column, or a partition metadata lookup. `key_ttl` controls how often that signal is checked. **Cons** - **Up to `key_ttl` of staleness.** Between `key_sql` evaluations, new data in the source will not appear on the dashboard. Match `key_ttl` to your tolerable staleness; setting it lower means more frequent `key_sql` queries against the source. - **`key_sql` itself runs on the source.** Make sure it's a cheap query: a `MAX()` on a clustered/partitioned column on Snowflake or Databricks, or a metadata lookup against `INFORMATION_SCHEMA.PARTITIONS` on BigQuery. A `MAX()` without a partition filter on BigQuery scans the whole table on every refresh. - **Cache is split per security context.** Row-level security policies and user attributes are part of the cache key, so a dashboard that filters per user does not share cached results across users. - **Memory usage.** Cached results live in Rill's in-memory cache; very high-cardinality dashboards with many distinct queries will evict older entries. ## Visual Metrics Editor If you're using the UI to select your table, choosing a live connector will automatically configure the YAML with the correct `table`, `connector`, and `database_schema` fields. ![ClickHouse Metrics View](/img/build/metrics-view/clickhouse-metrics-view.png) --- ###### Get Started with Metrics Views --- title: Get Started with Metrics Views description: Create metrics view using source data and models with time, dimensions, and measures sidebar_label: What are Metrics Views? sidebar_position: 00 --- A metrics view is a 'centralized framework' used to define and organize **key measures and dimensions** for your organization. Having a centralized layer allows an organization to easily manage and reuse calculations across various reports, dashboards, and data tools. Each metrics view is powered by a single [model or table](/developers/build/metrics-view/underlying-model). Rill metrics views are analogous to the **semantic layer** and **metrics layer** concepts found in other business intelligence and analytics tools. It provides a consistent, reusable abstraction over your underlying data, enabling standardized definitions of business metrics that can be shared across different dashboards and reports. ![Metrics View Components](/img/concepts/metrics-view/metrics-view-components.png) In Rill, your metrics view is defined by _metric definitions_. Metric definitions are composed of: * [_**model/table**_](/developers/build/metrics-view/underlying-model) - A data model or underlying table created with the concept of [One Big Table](/developers/build/models/models-101#one-big-table-and-dashboarding) that will power the metrics view. * [_**timeseries**_](/developers/build/metrics-view/time-series) - A column from your model that will underlie x-axis data in Rill's Explore dashboards and Canvas dashboards. Time can be truncated into different time periods. * [_**dimensions**_](/developers/build/metrics-view/dimensions) - Categorical columns from your data model whose values are shown in _leaderboards_ in explore dashboard and allow you to look at segments or attributes of your data (and filter/slice accordingly) as well as selectable axis in Canvas dashboard components. * _[**measures**](/developers/build/metrics-view/measures)_ - Numerical aggregates of columns from your data model shown on the y-axis of the explore charts and canvas components and the "big number" summaries. ## Creating a Metrics view Once your [model or underlying table](/developers/build/metrics-view/underlying-model) is ready to visualize, you'll need to create a metrics view to define your measures and dimensions. This can be done in a few ways. Either create a blank YAML file, use the Add metrics view button, or "Generate Metrics with AI" from the model. ### Create a Metrics view with Code Copy the below into a blank YAML or use the Add -> metrics view to create a blank metrics view. Here you can start to define dimensions and measures as seen below. ```yaml # Metrics View YAML # Reference documentation: https://docs.rilldata.com/reference/project-files/metrics-views version: 1 type: metrics_view model: example_model # Choose a model to underpin your metrics view timeseries: timestamp_column # Choose a timestamp column (if any) from your table dimensions: - column: category display_name: "Category" description: "Description of the dimension" measures: - expression: "SUM(revenue)" display_name: "Total Revenue" description: "Total revenue generated" ``` :::tip Set project-wide time defaults Configure default time modifiers like first day of week and smallest time grain for all metrics views. [Learn more about metrics view defaults →](/developers/build/project-configuration#metrics-views-time-modifiers) ::: ### Using the Visual Metrics Editor When you add a metrics definition using the UI, a code definition will automatically be created as a YAML file in your Rill project within the metrics directory by default. ![Visual Metrics Editor](/img/build/metrics-view/visual-metrics-editor.png) ### Creating a Metrics View with AI In order to streamline the process and get to a dashboard as quickly as possible, we've added the "Create Metrics with AI" and "Create Dashboard with AI" options! This will pass your schema to OpenAI to suggest measures and dimensions to get started with Rill. ![Create With AI](/img/build/metrics-view/create-with-ai.png) :::tip Skipped creating a model? You can now create dashboards directly for the connector panel. This will create a model, metrics view and dashboard all in one step. ![Explorable Metrics](/img/build/dashboard/explorable-metrics.png) ::: You can define your own OpenAI key by creating a [connector file](/reference/project-files/connectors#openai). If you want to disable metrics view generation with AI from your environment, please set the following in the `rill.yaml`: ```yaml features: ai: false ``` --- #### Models --- ###### Data Quality Tests --- title: Data Quality Tests sidebar_label: Data Quality Tests sidebar_position: 14 --- ## Overview Data quality tests allow you to define automated checks that run whenever your models refresh. These tests ensure your data meets expectations and catch issues early in your pipeline. Tests are defined in your model's YAML file using the `tests:` property. Each test runs a SQL query against your model's output. If the query returns any rows, the test fails and the error is recorded in the model's state. ## When to Use Data Quality Tests :::tip Using live connectors? Use alerts instead Data quality tests run when models refresh, which means they only work with models that Rill manages. If you're using [live connectors](/developers/build/connectors/olap) (ClickHouse, Druid, Pinot, StarRocks, etc.) where data lives in external systems, use [alerts](/developers/build/alerts) to monitor data quality on a schedule instead. ::: Data quality tests are useful for: - **Data Quality Checks** - Verify that your data meets business rules and constraints - **Schema Validation** - Ensure expected columns exist and have correct data types - **Referential Integrity** - Check relationships between tables - **Range Validation** - Verify numeric values fall within expected ranges - **Null Checks** - Ensure required fields don't contain null values - **Uniqueness Constraints** - Verify primary keys and unique columns - **Business Logic** - Validate complex business rules and calculations ## Defining Tests Tests are defined in your model's YAML file under the `tests:` property. Each test requires: - `name` - A descriptive name for the test - Either `assert` or `sql` - The validation logic ### Basic Syntax ```yaml type: model sql: SELECT * FROM my_source tests: - name: Test Name assert: column > 0 # OR sql: SELECT * FROM model WHERE condition_is_bad ``` ## Test Types ### Assert Tests Assert tests use a WHERE clause condition that should be true for all rows. If any row fails the assertion (the condition is false), the test fails. **Syntax:** ```yaml - name: Test Name assert: ``` The assertion is internally converted to: ```sql SELECT * FROM model WHERE NOT () ``` **Examples:** ```yaml tests: # Check for null values - name: No Null Campaign ID assert: campaign_id IS NOT NULL # Range validation - name: Valid Bid Price assert: bid_price >= 0 AND bid_price <= 100 # Value constraints - name: Valid Ad Status assert: status IN ('active', 'paused', 'completed', 'draft') # Multiple conditions - name: Valid Impression Count assert: impressions >= 0 AND impressions <= 1000000000 ``` ### SQL Tests SQL tests use a complete SQL query that should return zero rows. If any rows are returned, the test fails. **Syntax:** ```yaml - name: Test Name sql: SELECT * FROM model WHERE ``` **Examples:** ```yaml tests: # Row count validation - name: Minimum Impression Count sql: SELECT 'Too few impressions' WHERE (SELECT COUNT(*) FROM model) < 1000 # Aggregate validation - name: Positive Total Spend sql: SELECT 'Negative spend detected' WHERE (SELECT SUM(spend) FROM model) < 0 # Duplicate detection - name: No Duplicate Impression IDs sql: | SELECT impression_id, COUNT(*) as count FROM model GROUP BY impression_id HAVING COUNT(*) > 1 # Referential integrity - name: Valid Campaign References sql: | SELECT i.campaign_id FROM model i LEFT JOIN campaigns c ON i.campaign_id = c.id WHERE c.id IS NULL # Data completeness - name: All Expected Ad Formats Present sql: | SELECT missing_format FROM ( VALUES ('banner'), ('video'), ('native') ) AS expected(missing_format) WHERE missing_format NOT IN (SELECT DISTINCT ad_format FROM model) ``` ## Complete Example Here's a comprehensive example showing various validation patterns: ```yaml # models/ad_impressions.yaml type: model sql: SELECT * FROM raw_impressions tests: # Basic null checks - name: Impression ID Not Null assert: impression_id IS NOT NULL - name: Campaign ID Not Null assert: campaign_id IS NOT NULL # Range validations - name: Valid Bid Price assert: bid_price > 0 AND bid_price < 100 - name: Valid Impression Date assert: impression_date >= '2020-01-01' AND impression_date <= CURRENT_DATE # Status validation - name: Valid Campaign Status assert: status IN ('active', 'paused', 'completed', 'draft') # Uniqueness check - name: No Duplicate Impression IDs sql: | SELECT impression_id, COUNT(*) as count FROM ad_impressions GROUP BY impression_id HAVING COUNT(*) > 1 # Row count validation - name: Minimum Impressions Present sql: SELECT 'Too few impressions' WHERE (SELECT COUNT(*) FROM ad_impressions) < 1000 # Aggregate validation - name: Positive Total Spend sql: SELECT 'Negative spend' WHERE (SELECT SUM(spend) FROM ad_impressions) < 0 # Complex business logic - name: Clicks Must Have Impressions sql: SELECT * FROM ad_impressions WHERE clicks > 0 AND impressions = 0 - name: Click Timestamp After Impression sql: SELECT * FROM ad_impressions WHERE click_timestamp < impression_timestamp # Referential integrity - name: Valid Campaign References sql: | SELECT i.campaign_id FROM ad_impressions i LEFT JOIN campaigns c ON i.campaign_id = c.campaign_id WHERE c.campaign_id IS NULL ``` ## Test Execution Tests are executed automatically when your model is refreshed: 1. **Model Refresh** - The model's SQL query runs and produces output 2. **Test Execution** - Each test query runs against the model's output table 3. **Result Recording** - Test failures are recorded in the model state 4. **Model Status** - The model remains available even if tests fail ### Test Behavior - Tests run **after** successful model refresh - A failing test does **not** prevent the model from being available for queries - Test results are stored in the model's state and visible in the [Rill logs](/reference/cli/project/logs) - All tests run independently - one failure doesn't stop other tests - Tests can reference the model's output using the model name ## Viewing Test Results Test results are stored in the model state and visible in: - **Model State** - Access via the runtime API in the `test_errors` field - **Logs** - Test failures are logged during model reconciliation When a test fails, the error message includes: - Test name - Number of rows that failed the test - Sample of the failing rows (if applicable) ## Best Practices ### Test Naming Use descriptive names that clearly indicate what's being validated: ```yaml # Good names - name: No Null Campaign IDs - name: Spend Within Valid Range - name: All Impressions Have Valid Status # Less clear names - name: Test 1 - name: Check Data - name: Validation ``` ### Test Organization Group related tests together and add comments: ```yaml tests: # Null checks - name: Impression ID Not Null assert: impression_id IS NOT NULL - name: Campaign ID Not Null assert: campaign_id IS NOT NULL # Range validations - name: Valid Bid Price assert: bid_price > 0 - name: Valid Date Range assert: impression_date >= '2020-01-01' ``` ### Understanding Assert vs SQL Syntax **Assert Syntax** - Define conditions that should be true for all rows: - You write: `assert: value > 0` - Rill converts this to: `SELECT * FROM model WHERE NOT (value > 0)` - Tests **pass** if no rows are returned (all rows satisfy the condition) - Tests **fail** if any rows are returned (violations found) **SQL Syntax** - Write custom queries that return failing rows: - You write: `sql: SELECT * FROM model WHERE value <= 0` - Your query should explicitly return rows that represent failures - Tests **pass** if the query returns an empty result set - Tests **fail** if the query returns any rows :::tip Key Difference With `assert`, you define what should be **true**. With `sql`, you query for what is **wrong**. ::: ### Choosing Between Assert and SQL **Use Assert when:** - Testing row-level conditions - Checking simple constraints - The logic is straightforward - You want Rill to handle the "NOT" logic for you **Use SQL when:** - Testing aggregate values (COUNT, SUM, AVG) - Checking relationships between tables - Implementing complex validation logic - You need more control over the error message - You prefer to explicitly write the failure query ### Performance Considerations - Tests add time to your model refresh cycle - Complex tests with joins or aggregations can be expensive - Consider the trade-off between test coverage and refresh time - Use indexes on columns referenced in test queries when possible ### Testing Strategy **Start with critical validations:** 1. Null checks on required fields 2. Uniqueness constraints 3. Referential integrity 4. Range validations **Add domain-specific tests:** 1. Business rules 2. Data quality checks 3. Completeness validations **Monitor and iterate:** 1. Review test failures regularly 2. Add tests when data issues are discovered 3. Remove or update tests that are no longer relevant ## Working with Incremental Models Tests work with incremental models and run after each incremental refresh: ```yaml type: model incremental: true sql: SELECT * FROM raw_impressions WHERE impression_timestamp > (SELECT MAX(impression_timestamp) FROM {{ .self }}) tests: - name: No Null Impression Timestamps assert: impression_timestamp IS NOT NULL - name: No Future Impression Timestamps assert: impression_timestamp <= CURRENT_TIMESTAMP ``` The tests validate the **entire model output**, not just the newly added rows. ## Working with Partitioned Models Tests also work with partitioned models: ```yaml type: model incremental: true partitions: sql: SELECT DISTINCT impression_date FROM raw_impressions sql: SELECT * FROM raw_impressions WHERE impression_date = '{{ .partition.impression_date }}' tests: - name: Valid Impression Dates assert: impression_date IS NOT NULL - name: Partition Date Matches assert: impression_date = '{{ .partition.impression_date }}' ``` Tests run against the complete model output after all partitions are processed. ## Examples ### Checking for Duplicates ```yaml - name: No Duplicate Impression IDs sql: | SELECT impression_id, COUNT(*) as count FROM model GROUP BY impression_id HAVING COUNT(*) > 1 ``` ### Validating Relationships ```yaml - name: Valid Campaign References sql: | SELECT i.campaign_id FROM model i LEFT JOIN campaigns c ON i.campaign_id = c.id WHERE c.id IS NULL ``` ### Checking Completeness ```yaml - name: All Expected Ad Formats Present sql: | SELECT expected_format FROM (VALUES ('banner'), ('video'), ('native')) AS expected(expected_format) WHERE expected_format NOT IN (SELECT DISTINCT ad_format FROM model) ``` ### Aggregate Validations ```yaml - name: CTR Within Expected Range sql: | SELECT 'CTR out of range' as error WHERE (SELECT SUM(clicks) * 1.0 / NULLIF(SUM(impressions), 0) FROM model) > 0.5 ``` ### Date Range Checks ```yaml - name: Valid Impression Date Range assert: impression_date >= '2020-01-01' AND impression_date <= CURRENT_DATE - name: Impressions Within Last Year assert: impression_date >= CURRENT_DATE - INTERVAL '1 year' ``` :::warning Limitations - Tests only run when the model is refreshed - Failing tests do not prevent the model from being queryable - Tests cannot modify data - they are read-only validations - Test queries should complete reasonably quickly to avoid long refresh times ::: --- ###### Schedule Your Data Refresh --- title: Schedule Your Data Refresh description: Manage refresh schedules for models deployed to Rill Cloud sidebar_label: Scheduled Refreshes sidebar_position: 15 --- When creating or updating a model in Rill Cloud, you also have the option to configure how often the underlying model is refreshed (and thus ingested into the underlying OLAP layer powering Rill models and dashboards). By default, models are refreshed manually, but this can also be automated to a predefined schedule. This is handled through the underlying [model settings](/reference/project-files/models) and/or [project YAML](/reference/project-files/rill-yaml#project-wide-defaults) using standard cron or Go duration syntax. :::tip Configuring model refreshes for Cloud deployments It is generally strongly recommended to configure model refreshes when [deploying a project](/developers/deploy/deploy-dashboard) to Rill Cloud to ensure that your production data (and dashboards) _remains up-to-date_. The interval that you should set really depends on how often your own data is being refreshed. Furthermore, while it is technically possible to configure model refreshes for Rill Developer as well, Rill Developer is primarily used for local development and thus typically does not require working with the most up-to-date data (local model refreshes that occur too often could also lead to resource constraints on your local machine). For more details, please see our pages on [environments](/developers/build/connectors/credentials#variables), [templating](/developers/build/connectors/templating), and [performance optimization](/developers/build/models/performance). ::: ## Configuring Model Refresh Individually To specify a model refresh schedule for a particular model, this can be handled using the `refresh` property in the underlying YAML file. For example, to set a daily refresh for a model, you can do the following: ```yaml refresh: every: 24h ``` Similarly, if you would like to utilize cron syntax, the following example would update a model every 15 minutes: ```yaml refresh: cron: '*/15 * * * *' ``` :::note Model settings For more details about available model configurations and properties, check our [model YAML](/reference/project-files/models) reference page. ::: ## Configuring a Project-Wide Default You can also specify a project-wide refresh schedule that will apply to all models by default. This can be done through the `rill.yaml` file. More details can be found [here](/reference/project-files/rill-yaml#project-wide-defaults). :::tip Set project-wide defaults You can set a default refresh schedule for all models in your project. [Learn more about project defaults →](/developers/build/project-configuration#model-refresh-schedule) ::: Using the same example as above, the following sets a project-wide default of refreshing models every 24 hours: ```yaml models: refresh: every: 24h ``` Similarly, the following would use cron syntax to set a project-wide configuration of refreshing models by default every 15 minutes (unless overridden at the individual model level): ```yaml models: refresh: cron: '*/15 * * * *' ``` :::info Did you know? If you have both a project-wide default and model-specific refresh schedule _configured in the same project_, the model-specific refresh will **override** the project default based on how inheritance works in Rill. Otherwise, if not specified, the project-wide default will be used instead! ::: ## Running Scheduled Source Refreshes in Development As an exception, scheduled source refreshes specified using `refresh:` are not applied in the `dev` environment by default. If you want to run or test scheduled refreshes in local development, you can override this behavior using the `run_in_dev` property: ```yaml refresh: cron: 0 * * * * run_in_dev: true ``` --- ###### Incremental Models --- title: Incremental Models description: Create Incremental Models sidebar_label: Incremental Models sidebar_position: 20 --- Incremental models help with the ingestion of large datasets by allowing a dataset to be broken down into smaller sections for ingestion, rather than reading the entire dataset at once. Unlike [standard SQL models](/developers/build/models/sql-models) that are created via a .sql file, incremental models are defined in a YAML file and are used when a large dataset needs to be incrementally ingested to improve ingestion costs and time. :::note Take a look at the Reference! If you are unsure about the required parameters, please review the [reference page for Advanced Models](/reference/project-files/models). ::: Rill supports incremental models on either cloud storage or data warehouses, but the parameters to set these up will be different. Cloud storage requires the `glob` parameter while data warehouses will need to use `sql`. See [our reference documentation](/reference/project-files/models) for more information. :::tip Need help setting up Incremental Models? Please [reach out to us](/contact) if you have any questions regarding incremental modeling! ::: ## Creating an Incremental Model In order to enable an incremental model, you will need to set the following: `incremental: true`. ```yaml type: model incremental: true sql: # some SQL query from source_table ``` :::warning Duplicate Data Incremental models default to an append strategy, and with neither `state` nor `partition` defined, your data will append data per incremental refresh from the source table. This will result in duplicate data and is not recommended. Instead, use the `merge_strategy` with a `unique_key` to ensure duplicate data is not ingested. ::: :::warning Late Arriving Data If you have late arriving data, you will need to keep this in mind when designing your incremental model. If you simply use max(date) from the source, you may risk leaving out late arriving data. Depending on your specific use case, you might consider a larger time difference and use a `merge` as your `incremental_strategy`. ::: ### Incremental Models with State Defined (Optional) If your data is not [partitioned](/developers/build/models/partitioned-models), you can define the incremental model with a predefined `state` parameter. This is only useful for multi-connector incremental ingestion such as BigQuery to DuckDB. ```yaml type: model incremental: true connector: bigquery state: sql: SELECT MAX(date) as max_date sql: | SELECT ... FROM events {{ if incremental }} WHERE event_time > '{{.state.max_date}}' {{end}} output: connector: duckdb ``` Once state is defined in an incremental model, its value can be used as a variable in your SQL statement. In the above example, the state gets the most recent date from the model and when incrementally refreshing, ingests data for events that are more recent than the state's max_date. :::tip You can verify the current value of your state in the left-hand panel under Incremental Processing. ::: ### Refreshing an Incremental Model When you are testing with incremental models in Rill Developer, you will notice a change in the refresh functionality. Instead of a full refresh, you are given the option for `incremental refresh`. ![Now Incremental](/img/tutorials/advanced-models/now-incremental.png) :::tip What's the difference? Once increments are enabled on a model, this grants you the ability to refresh the model in increments, instead of loading the full data each time. This is handy when your data is massive and re-ingesting the data may take time. For a project in production, this allows for less downtime when needing to update your dashboards when the source data is updated. There are times when a full refresh may be required. In these cases, running the full refresh is equivalent to running a normal refresh with incremental disabled. ::: When selecting to refresh incrementally, what is being run in the CLI is: ```bash rill project refresh --local --model ``` Keep in mind that if you select `Full refresh`, this will start the ingestion of **all of your data** from scratch. Only use this when absolutely required. When running a full refresh, the CLI command is: ```bash rill project refresh --local --model --full ``` ## Model Change Modes Configure how changes to your model specifications are applied: ```yaml # model.yaml change_mode: reset # Options: reset (default), manual, patch ``` - `reset`: changing the model automatically leads to a full refresh (default) - `manual`: changing the model stops refreshes until a manual incremental or full refresh is run - `patch`: changing the model automatically changes to the new logic without a reset (only works for models with `incremental: true`) --- ###### Incremental Partitioned Models --- title: Incremental Partitioned Models description: Create incremental partitioned models sidebar_label: Incremental + Partitioned Models sidebar_position: 30 --- Putting the two concepts together, it is possible to create an incremental partitioned model. Doing so will allow you to not only partition the model but also refresh only the partition that you need and incrementally ingest partitions. :::note Need help? If you need any assistance with setting up an incremental partitioned model, [reach out](/contact) to us for assistance! ::: :::tip Looking for an example? If you're looking for a working example, take a look at [my-rill-tutorial in our examples' repository](https://github.com/rilldata/rill-examples). ::: As we already know how to set these up separately, let's see what changes in the UI when we enable both on a single model. In the following example, note that incremental modeling is enabled and partitions are defined by the Google Cloud Storage directory. ```yaml type: model incremental: true refresh: cron: "0 8 * * *" partitions: glob: path: gs://rilldata-public/github-analytics/Clickhouse/2024/*/* partition: directory sql: | SELECT * FROM read_parquet('{{ .partition.uri }}/commits_*.parquet') WHERE '{{ .partition.uri }}' IS NOT NULL ``` ### Refreshing Partitions in Incremental Models When this model loads, you will be able to both view the partitions and select a specific partition to refresh via the UI in Rill Developer. Unlike **partitioned-only** models, a new button is added in each of the partitions. ![Incremental Partitions Developer](/img/build/advanced-models/incremental-partitions-developer.png) Likewise, if you refresh using the **CLI**: ```bash rill project refresh --model CH_incremental_commits_directory --local --partition ba9f71625de8e042cabf3333576d502c Refresh initiated. Check the project logs for status updates. ``` ## How Incremental Partitioned Models Work ### Initial Ingestion: When a model is first created, an initial ingestion will occur to bring in all the data, also known as a `Full Refresh`. All refreshes after this will be considered an `incremental refresh`. Note in the below image, the source table writes each section of data to a specific partition as mapped in the YAML file. ![Initial Ingestion](/img/build/advanced-models/initial-ingestion.png) ### Additional Partition: If you add an additional partition to the source table, on the next manual or automatic refresh, Rill will detect the new partition and **only** add the additional partition to the model, as you can see in the diagram, the **blue** additional partition is added in its own partition in the partitioned model. If the other partitions have not been modified, these will not be touched. ![Additional Partition](/img/build/advanced-models/additional-partition.png) ### Modify Existing Partition: If you modify any of the already existing partitions, **yellow**, Rill will re-ingest just the modified file during the scheduled refresh by checking the `last_modified_date` parameter. ![Modified Partition](/img/build/advanced-models/modified-partition.png) --- ###### Models in Rill --- title: Models in Rill sidebar_label: Build sidebar_position: 0 --- Rill provides users the option to build various types of models based on different use cases. Take a look through the list of available types of models! ### Fundamentals - [**Getting Started with Models**](/developers/build/models/models-101) - What are models and important topics to remember as you create your own. - [**Differences between SQL and YAML Models**](/developers/build/models/model-differences) - Compare SQL and YAML models and learn when to use each. - [**SQL Models**](/developers/build/models/sql-models) - Transform and prepare your data with a basic SQL file ### Data Ingestion - [**Source Models**](/developers/build/models/source-models) - Ingest data from various sources - [**Data Quality Tests**](/developers/build/models/data-quality-tests) - Add data quality tests to validate your models - [**Scheduled Refreshes**](/developers/build/models/data-refresh) - Schedule your data refresh ### Advanced YAML Features - [**Incremental Models**](/developers/build/models/incremental-models) - Increment your model to avoid large ingestion times - [**Partitioned Models**](/developers/build/models/partitioned-models) - Split your large data into smaller sizes - [**Incremental + Partitioned Models**](/developers/build/models/incremental-partitioned-models) - Combine the concepts of Incremental Models and Partitioned Models into a single file - [**Staging Models**](/developers/build/models/staging-models) - When data is not directly available to ingest ### Environment Management - [**Environment Templating**](/developers/build/models/templating) - Separate your production and development settings ### Performance Optimization - [**Optimize your Models**](/developers/build/models/performance) - Optimize model performance and data refresh strategies --- ###### When to use SQL vs YAML --- title: When to use SQL vs YAML description: Create models from source data and apply SQL transformations sidebar_label: When to use SQL vs YAML sidebar_position: 03 --- In Rill, there are two types of data models: - [SQL models](/developers/build/models/model-differences#sql-models) - [YAML models](/developers/build/models/model-differences#yaml-models) For most use cases, SQL models, _the default_, are sufficient to transform your data to prepare for visualization. SQL models are built using SQL `SELECT` statements applied to your source data. Under the hood, SQL models are created as views in DuckDB and can be [materialized](/developers/build/models/performance#materialization) as tables when needed. For more complex modeling and [data ingestion](/developers/build/models/source-models), YAML models are used. By using a YAML approach, we are able to fine-tune the model's settings to enable partitions, incremental modeling, refreshes, and more. :::tip Avoid Pre-aggregated Metrics Rill works best for slicing and dicing data, meaning keeping data closer to raw to retain that granularity for flexible analysis. When loading data, be careful with adding pre-aggregated metrics like averages, as that could lead to unintended results like a sum of an average. Instead, load the two raw metrics and calculate the derived metric in your model or dashboard. ::: ## SQL Models ### When to use SQL Models? For most users working with DuckDB-backed Rill projects, SQL models provide everything needed to transform and prepare data for visualizations. These models are the default option when using the UI and offer full functionality for data transformation. ### Creating a SQL Model When using the UI to create a new model, you'll see something similar to the screenshot below. You can also create a model directly from the connector UI in the bottom left by selecting the "...". This will create a `select * from underlying_table` as a SQL model file. ![Model](/img/build/model/model.png) ## YAML Models Unlike SQL models, YAML file models allow you to fine-tune a model to perform additional capabilities such as pre-exec, post-exec SQL, partitioning, and incremental modeling. This is an important addition to modeling, as it allows users to customize the model's build process. In the case of partitions and incremental modeling, this will reduce the amount of data ingested into Rill at each interval and provide insight into specific issues per partition. Another use case is when using [multiple OLAP engines](/developers/build/connectors/olap/multiple-olap), which allows you to define where a SQL query is run. ### When to use YAML Models For the majority of users on a DuckDB-backed Rill project, YAML models are not required. When a project grows larger and refreshing entire datasets becomes a time-consuming and costly task, we introduce incremental ingestion to help alleviate the problem. Along with incremental modeling, we use partitions to divide a dataset into smaller, more manageable partitions. After enabling partitions, you will be able to refresh individual partitions of data when required. Another use case is when using multiple OLAP engines. This allows you to specify where your SQL query is running. When both DuckDB and ClickHouse are enabled in a single environment, you will need to define `connector: duckdb/clickhouse` in the YAML to tell Rill where to run the SQL query, as well as define the `output` location. For more information, refer to the [YAML reference](/reference/project-files/models). ### Types of YAML Models 1. [Source Models](/developers/build/models/source-models) 2. [Incremental Models](/developers/build/models/incremental-models) 3. [Partitioned Models](/developers/build/models/partitioned-models) 4. [Incremental + Partitioned Models](/developers/build/models/incremental-partitioned-models) 5. [Staging Models](/developers/build/models/staging-models) ### Creating a YAML Model You can get started with an advanced model using the following code block: ```yaml # Model YAML # Reference documentation: https://docs.rilldata.com/reference/project-files/models type: model connector: duckdb sql: select * from output: connector: duckdb table: output_name ``` Please refer to [our reference documentation](/reference/project-files/models) linked above for the available parameters to set in your model. --- ###### Introduction to Models --- title: Introduction to Models description: Create models from source data and apply SQL transformations sidebar_label: Models 101 sidebar_position: 00 --- Models in Rill enable data transformation, preparation, and enrichment through SQL queries and YAML configurations. They serve as the foundation for creating clean, structured datasets that power your metrics views and dashboards. Data models are built using SQL SELECT statements applied to your source data. They allow you to join, transform, and clean data. ### SQL Transformations By default, data transformations in Rill Developer are powered by DuckDB and its dialect of SQL (DuckDB SQL). Please visit the [DuckDB SQL documentation](https://duckdb.org/docs/sql/introduction) to learn how to write your queries. You can change the default [OLAP engine](https://docs.rilldata.com/developers/build/connectors/olap) for [the entire project](https://docs.rilldata.com/reference/project-files/rill-yaml#configuring-the-default-olap-engine) or [a specific metrics view](https://docs.rilldata.com/reference/project-files/metrics-views). You will need to define the connector credentials within your Rill project or via environment variables. :::tip Supported OLAP engines for modeling We support modeling on [ClickHouse\*](/developers/build/connectors/olap/clickhouse), [DuckDB](/developers/build/connectors/olap/duckdb), and [MotherDuck\*](/developers/build/connectors/olap/motherduck). For more information, see each OLAP engine page for further details. \* indicates some caveats with modeling, and we encourage you to read the documentation before getting started. ::: For additional tips on advanced expressions (either in models or measure definitions), visit our [advanced expressions page](/developers/build/metrics-view). ### Intermediate Processing Models can also be cross-referenced with each other to produce the final output for your dashboard. This approach enables more complex, intermediate data transformations to achieve your final data source. Common modeling patterns include: - Lookups for id/name joins - Unnesting and merging complex data types - Combining multiple sources with data cleansing or transformation requirements ## Data Preview and Validation ### Table Preview Rill automatically generates a preview of your data (first 150 rows) to help verify that the output table structure is correct and identify any potential issues that need to be addressed in the SQL configuration, such as data type detection problems. ### Schema Details The right panel displays comprehensive information about your dataset and column contents: - **Dataset Overview**: Total row and column counts - **Data Quality Metrics**: Number of dropped rows and columns - **Column Analysis**: - Column names and data types - Distinct value counts for string columns - Basic numeric statistics (minimum, maximum, median, etc.) This information helps you validate your model configuration and ensure data quality before proceeding with the full data ingestion. ![Preview](/img/build/model/preview.png) ## One Big Table and Dashboarding The power of this approach lies in translating many ad hoc questions into a data framework that can answer a class of questions at scale. For example, high-level company insights (how much revenue did we make last week?) become more actionable for employees when contextualized to their role (how did my campaign increase revenue last week?). To experience the full potential of Rill, model your data sources into "One Big Table" – a granular resource that contains as much information as possible and can be rolled up in a meaningful way. This flexible OBT can be combined with a generalizable [metrics definition](/developers/build/metrics-view) to enable ad hoc slice-and-dice discovery through Rill's interactive dashboard. :::tip Materializing metrics-powered models We recommend materializing the model that powers your [metrics view](/developers/build/metrics-view). You can materialize a SQL model by adding this to the top of the file. This will greatly improve the performance of your dashboards. ```sql -- @materialize: true ``` ::: --- ###### Partitioned Models --- title: Partitioned Models description: Create Partitioned Models sidebar_label: Partitioned Models sidebar_position: 25 --- ## What are Partitions? In Rill, partitions are a special type of state that allows you to explicitly partition the model into parts. Depending on whether your data is in cloud storage or a data warehouse, you can use the `glob` or `sql` parameters. This is useful when a specific partition is failing to ingest; you can specify to reload only that specific partition. ### Defining a Partition in a Model Under the `partitions:` parameter, you will define the pattern in which your data is stored. Both SQL and glob patterns support [templating](/developers/build/connectors/templating) and can be used to separate `dev` and `prod` instances. ### SQL When defining your SQL partitions, it is important to understand the data that you are querying and creating a partition that makes sense. For example, you might select a distinct customer_name per partition, or partition the SQL by a chronological partition, such as month. #### Using DuckDB for Partition Queries By default, partition queries use DuckDB (Rill's embedded OLAP engine): ```yaml partitions: sql: SELECT range AS num FROM range(0,100) #num is the partition variable and can be referenced as {{partition.num}} #sql: SELECT DISTINCT customer_name as cust_name from table #results in {{partition.cust_name}} dev: partitions: sql: SELECT range AS num FROM range(0,10) sql: SELECT * from table where column = {{partition.num}} ``` #### Using Other Connectors for Partition Queries You can query partitions directly from data sources like **Athena**, **BigQuery**, **MySQL**, **Postgres**, **Redshift**, or **Snowflake** by specifying a `connector` in the `partitions` section. This is particularly useful when: - You want to leverage native partitioning features (like BigQuery's `_PARTITIONTIME`) - You need to query large tables that benefit from the warehouse's optimization **Data Warehouses** Athena: ```yaml type: model partitions: connector: athena sql: | SELECT DISTINCT year, month FROM s3_data_partitioned WHERE year >= 2024 connector: athena sql: | SELECT * FROM s3_data_partitioned WHERE year = {{ .partition.year }} AND month = {{ .partition.month }} output: connector: duckdb ``` BigQuery: ```yaml type: model partitions: connector: bigquery sql: | SELECT DISTINCT _PARTITIONTIME AS partition_time FROM `project.dataset.table` WHERE TIMESTAMP_TRUNC(_PARTITIONTIME, MONTH) = TIMESTAMP("2025-08-01") connector: bigquery sql: | SELECT * FROM `project.dataset.table` WHERE _PARTITIONTIME = '{{ .partition.partition_time }}' output: connector: duckdb ``` Redshift: ```yaml type: model partitions: connector: redshift sql: | SELECT DISTINCT DATE_TRUNC('month', transaction_date) AS month FROM transactions WHERE transaction_date >= '2024-01-01' connector: redshift sql: | SELECT * FROM transactions WHERE DATE_TRUNC('month', transaction_date) = '{{ .partition.month }}' output: connector: duckdb ``` Snowflake: ```yaml type: model connector: snowflake partitions: connector: snowflake sql: | select DISTINCT date_trunc('YEAR', release_date) as "year" from rillqa.public.horror_movies where "year" > '1999-01-01' limit 3 sql: select * from rillqa.public.horror_movies where date_trunc('YEAR', release_date) = '{{ .partition.year }}' output: connector: duckdb ``` **OLTP Databases** MySQL: ```yaml type: model partitions: connector: mysql sql: | SELECT DISTINCT DATE(order_date) AS order_day FROM orders WHERE order_date >= '2025-01-01' connector: mysql sql: | SELECT * FROM orders WHERE DATE(order_date) = '{{ .partition.order_day }}' output: connector: duckdb ``` Postgres: ```yaml type: model partitions: connector: postgres sql: | SELECT DISTINCT DATE_TRUNC('day', created_at) AS partition_day FROM events WHERE created_at >= '2025-01-01' connector: postgres sql: | SELECT * FROM events WHERE DATE_TRUNC('day', created_at) = '{{ .partition.partition_day }}' output: connector: duckdb ``` :::tip Why use multiple connectors? Using Athena, BigQuery, MySQL, Postgres, Redshift, or Snowflake for partition discovery and data extraction, then outputting to DuckDB, gives you: - **Best of both worlds**: Leverage your warehouse's partitioning and scale for extraction - **Fast dashboards**: DuckDB provides extremely fast query performance for end-user dashboards - **Cost optimization**: Only query what you need from your warehouse, reducing scan costs ::: :::tip Using the SQL partition in the YAML Depending on the column name of the partition, you can reference the partition using `{{ .partition. }}` in the model's SQL query. ```yaml partitions: sql: SELECT range AS num FROM range(0,10) sql: SELECT {{ .partition.num }} AS num, now() AS inserted_on {{if dev}} limit 1000 {{end}} ``` ::: ### glob When defining the glob pattern, you will need to consider whether you'd partition the data by folder or file. In the first example, we are partitioning by each file with the suffix data.csv. ```yaml partitions: glob: gs://my-bucket/y=2025/m=03/d=15/*data.csv #glob: gs://my-bucket/{{if dev}}y=2025/m=03/d=15{{else}}**{{end}}/*data.csv ``` Or, you can define each glob separately. ```yaml partitions: glob: path: 'gs://my-bucket/**/*.parquet' dev: partitions: glob: path: 'gs://my-bucket/y=2025/m=03/d=15/*.parquet' ``` If you'd prefer to partition it by folder, you can add the partition parameter and define it as `directory`. ```yaml glob: path: gs://rendo-test/**/*data.csv partition: directory #hive ``` :::tip Using the glob partition in the YAML The glob partition has a predefined `{{ .partition.uri }}` reference to use in the model's SQL query. ```yaml partitions: glob: connector: gcs path: gs://path/to/file/**/*.parquet sql: SELECT * FROM read_parquet('{{ .partition.uri }}') ``` ::: ### Windowed Glob Partitions Use `start` and `end` to filter which partitions are processed based on their path — useful for backfills, time-bounded ingestion, or limiting how much data is reprocessed. - **`start`** — lower bound (inclusive): only partitions with paths ≥ this value are included. - **`end`** — upper bound (exclusive): only partitions with paths < this value are included. - **`last`** — limit to the last N partitions (by lexicographic path order). This hard limit always applies, including on the first run when no partitions have been ingested yet. Once partitions have been ingested, it also raises the lower bound to the Nth-from-last successfully ingested partition, forming a rolling window that re-lists recent partitions. ```yaml # Fixed window: only process a specific month partitions: glob: path: s3://bucket/year=*/month=*/day=* partition: directory start: s3://bucket/year=2026/month=03 end: s3://bucket/year=2026/month=04 sql: SELECT * FROM read_parquet('{{ .partition.uri }}/*.parquet') ``` ```yaml # Rolling window: reprocess last 30 days partitions: glob: path: s3://bucket/year=*/month=*/day=* partition: directory last: 30 sql: SELECT * FROM read_parquet('{{ .partition.uri }}/*.parquet') ``` :::note `last` is not compatible with `transform_sql`. ::: ### Viewing Partitions in Rill Developer Once `partitions:` is defined in your model, a new button will appear in the right-hand panel: `View Partitions`. When selecting this, a new UI will appear with all of your partitions and more information on each. Note that these can be sorted by all, pending, and errors. ![Partitions Developer](/img/build/advanced-models/partitions-developer.png) You can sort the view by `all partitions`, `pending partitions`, and `error partitions`. - **all partitions**: shows all the available partitions in the model. - **pending partitions**: shows the partitions that are waiting to be processed. - **error partitions**: displays any partitions that errored during ingestion. ### Viewing Partitions in the CLI Likewise to the UI, you can view the partitions of a model within the CLI. ``` rill project partitions List partitions for a model Usage: rill project partitions [] [flags] Flags: --project string Project Name --path string Project directory (default ".") --model string Model Name --pending Only fetch pending partitions --errored Only fetch errored partitions --local Target locally running Rill --page-size uint32 Number of partitions to return per page (default 50) --page-token string Pagination token ``` If running locally, you will need to add the `--local` flag to the command. ```bash rill project partitions model_name [--local] KEY (10) DATA EXECUTED ON ELAPSED ERROR ---------------------------------- ----------- ---------------------- --------- ------- ff7416f774dfb086006d0b4696c214e1 {"num":0} 2024-11-12T22:48:49Z 95ms ... ``` ### Refreshing Partitions via the CLI :::note Incremental not enabled If you try to refresh a partition using the following command on a partitioned but not incremental model, you will experience the following error: ``` rill project refresh --model [--local] --partition ff7416f774dfb086006d0b4696c214e1 Error: can't refresh partitions on model "model_name" because it is not incremental ``` ::: You will need to enable [incremental modeling](/developers/build/models/incremental-partitioned-models) in order to individually refresh a partition. --- ###### Optimize Your Models --- title: Optimize Your Models sidebar_label: Performance Optimization sidebar_position: 45 --- Model performance is critical for maintaining responsive dashboards and ensuring users have access to the most current data. This guide covers strategies for optimizing your Rill models to deliver fast query results while keeping data fresh and up-to-date. By following these best practices, you can create models that provide both speed and accuracy, delivering insights when your users need them most. ## Local Development / Rill Developer As discussed in the [templating section](/developers/build/models/templating), there are a few key recommendations to increase model performance in Rill Developer: 1. [Limiting the source data](/developers/build/models/templating#applying-a-one-week-sample-to-the-source-bucket-for-local-development) to a smaller time range (e.g., one week's worth of data instead of the full year) 2. Creating smaller models for development by [applying a raw limit](/developers/build/models/templating#example-conditional-sql-limiting-dev-rows), which will then serve as the starting point for your actual downstream models/modeling 3. [Applying a filter](/developers/build/models/templating#example-leveraging-variables-to-apply-a-filter-and-row-limit-dynamically-to-a-model) to the data to work with only a subset of data ## Model Performance ### Consider which models to materialize By default, models will be materialized as views (in DuckDB). This allows for a dynamic and highly interactive experience when modeling, such as keystroke-by-keystroke profiling. However, since views are logical in nature, as the complexity and size of your data models continue to grow (especially if the underlying data is very large), this can start to significantly impact performance as these complex queries will need to be continuously re-executed along with a number of profiling queries that the Rill runtime will send in the backend. In such scenarios, we recommend materializing these models as tables. However, there are some tradeoffs to consider: - **Pros:** Materializing a model will generally ensure significantly improved performance for downstream dependent models and dashboards. - **Cons:** Enabling materialization for a model can severely impact or break the "keystroke-by-keystroke" experience and these models may also take longer to update (because the results are being written to a table vs. remaining a view). It can also lead to _degraded_ performance for very specific operations, such as when you need to perform cross joins. :::tip Materialize models powering dashboards We strongly recommend materializing final models that are being used directly in dashboards to ensure this data is served more quickly. ::: ### Materialization Model materialization is something to consider when creating intermediate models. Other than [source models](/developers/build/models/source-models), intermediate models are not, by default, materialized and are views in your underlying database engine. There are some pros and cons to enabling it during the development process. ```sql -- model.sql -- @materialize: true ``` ```yaml # model.yaml materialize: true ``` The pros include improved performance for downstream models and dashboards, especially with complex logic and/or large data sizes. Some cons are certain edge cases like cross joins might have a degraded keystroke-by-keystroke experience, and materialized models are billable. If you are seeing degraded performance, the first recommendation you'll hear from us is to materialize the metrics-powered model. ### Default Model Materialization If you want, you can change the default behavior of all models in Rill by setting the default model behavior in the rill.yaml file. ```yaml models: materialize: true ``` To override this on a per-model basis, simply set the specific model.sql to false. ```sql -- @materialize: false ``` :::info To materialize or not to materialize? There are both pros and cons to materializing your models. - Pros can include improved performance for downstream models and dashboards, especially when the SQL is complex and/or the data size is large. We generally recommend _materializing_ final models that power dashboards. - Cons can include a degraded keystroke-by-keystroke modeling experience or for specific edge cases, such as when using cross joins. If unsure, we would generally recommend leaving the defaults and/or [reaching out](/contact) for further guidance! ::: ## Query Optimization Query optimization is crucial for maintaining high performance and efficiency, especially when working with data-intensive applications. As Rill dashboards are powered by [OLAP engines](/developers/build/connectors/olap), designed for analytical queries, ensuring that our queries are well-optimized can help maximize the responsiveness and speed of our dashboards. There are also additional potential second-order benefits to optimizing queries in Rill, such as improving ingestion times, how long it takes to build models, how resource-intensive it is to build models, how fast profiling queries run, and more. ### Use appropriate data types and avoid casting when possible Casting can be expensive, especially when the underlying models are views and not [materialized](#consider-which-models-to-materialize) as a table. For example, if a timestamp column is actually incorrectly typed as a string, then for timeseries charts, Rill ends up having to iterate across each row to try to infer the timestamp and a lot of time parsing has to occur. Similarly, for incorrectly typed or cast columns that are used in calculations, the calculations will have to be constantly looped through, which can be both inefficient and expensive over time (and simply make everything slower). Similarly, choosing the right data type for each column is also important. Smaller data types, when applicable, consume less memory and can improve query performance. For example, use `INT` instead of `BIGINT` if your data range permits. ### Select the columns you need and avoid `SELECT *` when possible Because most [OLAP databases](/developers/build/connectors/olap) store data in a columnar format, including [DuckDB](/developers/build/connectors/olap/duckdb), selecting only the columns that you need during the modeling phase ensures that DuckDB will only ingest and store the data _it actually needs_ (speeding up model build times and reducing footprint). Furthermore, columnar formats are optimized for analytical queries, so by selecting only the columns that you need (instead of a blanket `SELECT *`), this will help to minimize data processing times and improve the query execution speed. ### Consider sorting your data by an appropriate timestamp column Generally speaking, if possible, it is recommended to make sure that your upstream data is relatively well organized and/or sorted by timestamp before being ingested into Rill. This helps to ensure that timeseries queries are efficient when they execute against resulting models and can result in an order of magnitude difference in query performance. This can also help improve the effectiveness of filters by reducing I/O. :::info When to sort vs. not to sort? Sorting, especially in DuckDB, _can also be computationally intensive_ and most input data is generally sorted enough (by time). If the data ingested is completely unsorted or sorted by a different, non-timestamp column, it could be worth the computational overhead to sort by a timestamp column (especially if used in a dashboard). If you're unsure, please feel free to [reach out](/contact) and we'd be happy to help you assess the best path forward! ::: ### Use joins efficiently Plan your joins carefully, especially when working with large datasets. Most [OLAP engines](/developers/build/connectors/olap), DuckDB included, will optimize join operations, but ensuring the join keys are well chosen and considering the size of the datasets being joined can reduce processing time. For example, if you're looking to perform a cross or cartesian join across a very wide table, be sure it's necessary as it can otherwise explode the size of your result set. ### Apply filters early and use WHERE clauses wisely When possible, it can be good practice to apply filtering early in your queries with `WHERE` clauses to reduce the amount of data being processed in subsequent steps (or downstream models/queries). This can both help to reduce the amount of data being scanned and, given the columnar nature of most [OLAP engines](/developers/build/connectors/olap), significantly speed up queries. ### Optimize your subqueries to leverage joins or CTEs when possible Subqueries can very often prove to be inefficient and result in suboptimal query execution plans. When possible, it is generally better practice to rewrite subqueries as joins or use Common Table Expressions (CTEs) to make them more readable and potentially more efficient. ### Rather than UNION, consider using UNION ALL when possible Depending on the [OLAP engine](/developers/build/connectors/olap), `UNION` can be a very expensive operation and much more computationally intensive than `UNION ALL`. For example, when using [DuckDB](/developers/build/connectors/olap/duckdb), a `UNION` will require performing full duplicate eliminations _across all columns_ while a `UNION ALL` will simply concatenate the tables together. If a concatenation is sufficient (for the query), this will be both much quicker and significantly less resource intensive for the query to complete. ## Time Series Transformation If your time series column is quite granular, this may affect your dashboards as the grain will define how granular the dashboards can be viewed. Instead of handling this in the metrics view by adding a `smallest_time_grain` key, you can use the modeling layer to roll up your data. :::note Query-time vs Model processing There are benefits to pre-processing the data in the model layer, but for some quick processing, this can be done in the metrics view. **Query-time processing** (in metrics views): - Flexible and dynamic - No storage overhead - Slightly slower for complex calculations **Model-level processing** (in SQL models): - Pre-computed and optimized - Faster query performance - Requires model refresh for updates ::: ### DuckDB Time Functions DuckDB provides a comprehensive toolkit for temporal data manipulation: - **`DATE_TRUNC`**: Normalize timestamps to consistent intervals (day, week, month, quarter, year) - **`EXTRACT`**: Extract specific time components (year, quarter, month, day of week, hour) - **`LAG/LEAD`**: Reference prior or future rows for period-over-period comparisons - **`DATE_ADD/DATE_SUB`**: Perform date arithmetic for dynamic time ranges - **`STRFTME`**: Extract strings from a time column. For comprehensive documentation on all available time functions, see the [DuckDB time functions documentation](https://duckdb.org/docs/stable/sql/functions/timestamp.html). :::tip not using DuckDB? Each engine has slightly different functions and syntax for rolling up your data, see your OLAP engine's documentation for more examples. ::: ### Time Aggregation (Roll-ups) Roll-ups aggregate granular events into coarser intervals. For example, if your data arrives hourly but daily analysis suffices: ```sql SELECT DATE_TRUNC('day', timestamp_column) AS time_series_column, ... FROM your_model ``` --- ###### Source Models --- title: Source Models sidebar_label: Source Models sidebar_position: 10 --- After [creating a connector to your data source](/developers/build/connectors/data-source), you'll need to create a model to bring that data into Rill. This can be implemented as either a SQL model with [defined connector parameters](/developers/build/models/sql-models#specifying-the-data-source-connector) or as a YAML configuration file. This guide focuses on YAML-based source models, which are auto-generated when using the UI. ```yaml # Model YAML # Reference documentation: https://docs.rilldata.com/reference/project-files/models type: model materialize: true connector: snowflake sql: | select * from database_name.schema_name.table_name ``` ### YAML Structure The YAML configuration file contains several key parameters: - **`type: model`**: Explicitly defines the file type. While Rill automatically detects the file type based on the parent folder, this parameter provides explicit definition. - **`connector`**: Defines the connector type used to create the model (e.g., `bigquery`, `athena`, `snowflake`, etc.). - **`sql`**: The actual SQL query to be executed. When nested under `dev:`, the query runs in the Rill Developer environment. - **`dev`**: Configuration for development mode. Rill Developer runs in dev mode by default, but when deployed to Rill Cloud, the root-level SQL configuration executes. See [Environment Templating](/developers/build/models/templating) for more information. ## Retry Configuration By default, a model will retry if the initial load fails. This helps ensure reliable data processing by automatically retrying failed operations. The default retry settings are: The default `if_error_matches` values are exactly: `".*OvercommitTracker.*"`, `".*Bad Gateway.*"`, `".*Timeout.*"`, and `".*Connection refused.*"`. If you set `retry.if_error_matches`, it overrides these defaults rather than appending to them. ```yaml retry: attempts: 3 delay: 5s exponential_backoff: true if_error_matches: - ".*NetException.*" # overrides defaults ``` You can customize the retry behavior to better suit your specific needs. For example, you might want to increase the number of attempts for critical models, adjust the delay between retries, or only retry on specific error types. Use the following configuration in your source YAML: ```yaml retry: attempts: 5 delay: 10s exponential_backoff: true if_error_matches: - ".*OvercommitTracker.*" - ".*Bad Gateway.*" - ".*Timeout.*" - ".*Connection refused.*" - ".*NetException.*" ``` ## Examples ### BigQuery Model ```yaml # Model YAML # Reference documentation: https://docs.rilldata.com/reference/project-files/models type: model materialize: true connector: bigquery dev: sql: select * from project_id.dataset_name.table_name limit 10000 sql: select * from project_id.dataset_name.table_name ``` ### Snowflake Model ```yaml # Model YAML # Reference documentation: https://docs.rilldata.com/reference/project-files/models type: model materialize: true connector: "snowflake" dev: sql: select * from database_name.schema_name.table_name limit 10000 sql: select * from database_name.schema_name.table_name ``` ### S3 Model ```yaml # Model YAML # Reference documentation: https://docs.rilldata.com/reference/project-files/models type: model materialize: true connector: "duckdb" dev: sql: | select * from read_csv('s3://my-bucket/path/to/file.csv', auto_detect=true, ignore_errors=1, header=true) limit 10000 sql: | select * from read_csv('s3://my-bucket/path/to/file.csv', auto_detect=true, ignore_errors=1, header=true) ``` For more information, see our [model reference documentation](/reference/project-files/models)! ## Next Steps Rill provides automatic refresh capabilities for your source models at specified intervals to maintain data currency. This functionality enables you to establish scheduled data ingestion without manual intervention, ensuring your analytics dashboards remain current with the latest information from your data sources. For additional details, see [Scheduled Refreshes](/developers/build/models/data-refresh). After validating your source model configuration and confirming the data preview appears correct, you can move forward to create your first metrics view. If no additional data transformations are needed, you can choose [Generate Metrics View with AI](/developers/build/metrics-view) from the top-right corner of the interface to initiate Rill's AI-powered dashboard generation. --- ###### SQL Models --- title: SQL Models description: Create models from source data and apply SQL transformations sidebar_label: SQL Models sidebar_position: 05 --- ## Adding a data model Add a new data model by either clicking 'model' in the 'Add' menu or selecting the '...' in any connector view or existing model. When you add a model, a code definition will automatically be created as a `.sql` file in the `models` folder of your Rill project. You can also create a model outside of the application and add it to Rill by placing a `.sql` file in the `models` directory containing a DuckDB SQL `SELECT` statement. Rill will automatically detect and parse the model the next time you run `rill start`. ## Annotating your models with properties In most cases, objects are represented in Rill as YAML files. Models are unique in that any `.sql` file can be considered a model resource in Rill, representing a SQL transformation that you would like to perform using a set of inputs and outputting a view or table (depending on the materialization type). For most other resources, available properties can be set directly via the corresponding YAML file. In the case of a model SQL file though, configurable properties should be set by annotating the top of the file using the following syntax: ```sql -- @property: value select * from users ``` We will cover different available configurable properties in the sections below. ### Marking your model SQL file as a model resource type By default, any new model that is created in a Rill project will populate a corresponding `.sql` file representing the model. Similarly, a `.sql` file that is directly created in the project directory will also be _automatically assumed_ by Rill to be a model by default. Therefore, it is not necessary to annotate the model resource with the `type` property. For consistency or documentation purposes, if you'd like to annotate your model resource as well with the `type` property, you can do so by adding the following to the top of your `.sql`: ```sql -- @type: model select * from orders ``` ### Specifying the Data Source Connector The `@connector` annotation explicitly defines which data source connector to use when executing the SQL query. This parameter is essential when working with multiple connectors of the same type, as it determines which connection credentials and source path to utilize. By default, query results are written to the project's default [OLAP engine](/developers/build/connectors/olap#olap-engines). ```sql -- @connector: bigquery SELECT * FROM project_id.dataset_name.table_name ``` ```sql -- @connector: postgres SELECT * FROM mydb.public.users ``` For projects configured with [multiple OLAP engines](/developers/build/connectors/olap/multiple-olap), you can specify both input and output connectors for a SQL model. Nested parameter syntax uses dot notation (e.g., `output.connector`). ```sql -- @connector: clickhouse -- @output.connector: clickhouse select from clickhouse_table ``` ## Working with Pivots Pivots deserve their own section, as using the [Pivot](https://duckdb.org/docs/sql/statements/pivot) statement while modeling requires special consideration. Notably, there are a few existing DuckDB limitations to consider: - DuckDB's [SQL to JSON serializer](https://duckdb.org/docs/extensions/json.html#serializing-and-deserializing-sql-to-json-and-vice-versa) doesn't support `PIVOT` without the `IN` [filter](https://duckdb.org/docs/sql/statements/pivot#in-filter-for-on-clause) - DuckDB doesn't support creating views based on `PIVOT` without an `IN` filter (and all models are materialized as views by default in Rill) Fortunately, there are workarounds available to address these limitations. #### Passing the `IN` Filter with Your `PIVOT` Statement If you know the exact values you want to pivot on, you can specify them using an `IN` filter with your `ON` clause. For example, instead of: ```sql PIVOT table_name ON column_name USING SUM(measure) ``` You can use the following `PIVOT` statement: ```sql PIVOT table_name ON column_name IN (value_a, value_b, value_c) USING SUM(measure) ``` --- ###### Staging Models --- title: Staging Models description: When the source ingestion is not directly supported, use staging models sidebar_label: Staging Models sidebar_position: 35 --- As the name suggests, a staging model is one that uses a staging platform to ingest data from one source to another output that is not directly supported. An example of this is ingesting from Snowflake to ClickHouse. In order to support our users' varying setups, we've devised staging models. ## When to use a Staging Model? Staging models are required for situations where the input and output types are not directly supported. However, by using a staging platform that is supported by both platforms, it allows users to temporarily write data to the staging platform and ingest directly to the output platform. :::note Supported Paths Staging Models are in ongoing development. While we do have support for the following, please [reach out to us](/contact) if you have any specific requirements. - Snowflake → S3 → ClickHouse - BigQuery → GCS → ClickHouse ::: ### Example Let's expand further on the Snowflake to ClickHouse example. ```mermaid sequenceDiagram Snowflake->>Staging (S3): write data to temporary Staging Staging (S3)->>ClickHouse: write temporary data to ClickHouse Staging (S3) ->> Staging (S3): remove temporary data from Staging ``` In the above example, during the ingestion from Snowflake to ClickHouse, we use the temporary staging table in S3 to write from Snowflake to S3, then from S3 to ClickHouse. Once this procedure is complete, we clear the temporary data from S3. ### Sample YAML: ```yaml # Use DuckDB to generate a range of days from 1st Jan to today partitions: connector: duckdb sql: SELECT range as day FROM range(TIMESTAMPTZ '2024-01-01', now(), INTERVAL 1 DAY) # Don't reload previously ingested partitions on every refresh incremental: true # Query Snowflake for all events belonging to the current partition connector: snowflake sql: SELECT * FROM events WHERE date_trunc('day', event_time) = '{{ .partition.day }}' # Since ClickHouse can't ingest from Snowflake or vice versa, we use S3 as a temporary staging connector stage: connector: s3 path: s3://bucket/temp-data # Produce the final output into ClickHouse, requires a clickhouse.yaml connector defined. output: connector: clickhouse ``` --- ###### Model Environment Templating --- title: Model Environment Templating description: Dev/Prod Setup sidebar_label: Environment Templating sidebar_position: 40 --- Along with [connector templating](/developers/build/connectors/templating), you can also optimize your development environment for performance and cost-effectiveness at the model layer. This approach is particularly useful when you don't have a separate development database or file system, as it allows you to limit the amount of data processed in source models. ## Why Use Templating? Templating serves several important purposes in your data workflow: - **Cost Management**: Avoid running expensive queries on production data during development - **Performance in dev**: Optimize development environment performance by limiting data volume - **Testing Safety**: Test your models and transformations without affecting production data :::note Rill Developer is a dev environment Unless explicitly defined, Rill Developer will use a `dev` environment. If you want to emulate production locally, you can do so via `rill start --environment prod`. ::: ## Separate Dev and Prod SQL Queries Another common templating pattern is using the `dev:` partition in your model definitions. This tells Rill to use a different data source or query when running in development mode, typically with a smaller dataset or different data location. ### Example: Ingest Data from Specific Folder Path in Dev ```yaml # Model YAML # Reference documentation: https://docs.rilldata.com/reference/project-files/models type: model connector: "duckdb" dev: sql: "select * from read_parquet('gs://rilldata-public/github-analytics/Clickhouse/2025/06/commits_2025_06.parquet')" sql: "select * from read_parquet('gs://rilldata-public/github-analytics/Clickhouse/*/*/commits_*.parquet')" ``` :::info Why is the connector type duckdb and not s3 or gcs? In this case, we are using the [embedded DuckDB engine](/developers/build/connectors/olap/duckdb) to execute a [SELECT](https://duckdb.org/docs/sql/statements/select.html) statement while leveraging DuckDB's native [read_parquet](https://duckdb.org/docs/data/parquet/overview.html) function. Therefore, the `connector` type ends up being `duckdb` instead of `s3`. For more details, see our [Source YAML](/reference/project-files/sources) reference documentation. ::: ![GCS Environment Example](/img/deploy/templating/gcs-env-example.png) In this example: - **Development**: Reads from a single file (`2025/06/commits_2025_06.parquet`) for faster processing - **Production**: Reads from all files in the folder using the `/*/*/` wildcard pattern; you can also use `/**/` if you're unsure of the number of directories ## Inline SQL Templating You can also use Go template syntax directly within your SQL queries to conditionally include or exclude parts of your query based on the environment. ### Example: Conditional SQL Limiting Dev Rows ```sql select * from read_csv('gs://your-bucket/folder/**/file.csv', auto_detect=true, ignore_errors=1, header=true) {{ if dev }} limit 10000 {{ end }} ``` ```sql select * from read_csv('gs://your-bucket/folder/**/file.csv', auto_detect=true, ignore_errors=1, header=true) {{ if dev }} where column = 'value' {{ end }} ``` In this example: - **Development**: Adds a where/limit clause, filtering the data by a specific value to restrict the dataset size - **Production**: No limits ### Example: If/Else Conditions ```sql select * from read_csv('gs://your-bucket/folder/**/file.csv', auto_detect=true, ignore_errors=1, header=true) {{ if dev }} limit 10000 {{ else }} where column = 'value' {{ end }} ``` In this example: - **Development**: Adds a `LIMIT 10000` clause to restrict the dataset size - **Production**: Applies a where clause to filter the data in production ### Applying a One-Week Sample to the Source Bucket for Local Development In another example, let's say we had an [S3](/developers/build/connectors/data-source/s3) source defined that happens to be reading a very large amount of parquet data. Following best practices, we'll want to read in a subset of this source data for local modeling in Rill Developer rather than using the full dataset for development purposes. Furthermore, we'll make the assumption that the upstream data is not partitioned and thus the S3 bucket is not partitioned (where we could then simply filter the `path` by using a glob pattern potentially in conjunction with [environment-specific logic](/developers/build/models/templating)). So what can we do? Fortunately, we can leverage DuckDB's ability to read from S3 files directly and _apply a filter post-download_ using templating logic in the SQL. In this case, because there is an existing `updated_at` timestamp column, we can use it to filter and retrieve only one week's worth of data. For example, our `source.yaml` file may end up looking something like this: ```yaml type: model connector: "duckdb" sql: > SELECT * FROM read_parquet('s3://bucket/path/*.parquet') {{ if dev }} where updated_at > CURRENT_DATE - INTERVAL 7 DAY {{ end }} ``` :::tip Dynamic dates Depending on the OLAP engine of your project, you can set the dates dynamically using `CURRENT_DATE` (DuckDB) or `now()` (ClickHouse) and subtract days. ::: ### Example: Leveraging Variables to Apply a Filter and Row Limit Dynamically to a Model Our last example will highlight how the same templating concepts can be applied with [variables](/developers/build/connectors/credentials#setting-credentials-for-rill-developer). In this case, we have a source dataset about horror movies that came out in the past 50 years, which includes various characteristics, attributes, and metrics about each horror movie as separate columns. For example, we know the release date, how many people saw a movie, what the budget was, its popularity, the original language of the movie, the genres, and much more. Let's say that we wanted to apply a filter on the resulting model based on the `original_language` of the movie and also limit the number of records that we retrieve, which will be based on the `language` and `local_limit` variables we have defined. Taking a quick look at our project's `rill.yaml` file, we can see the following configuration (to return only English movies and apply a limit of 5): ```yaml env: local_limit: 5 language: "en" ``` Furthermore, our `model.sql` file contains the following SQL: ```sql SELECT * FROM {{ ref "snowflake" }} WHERE original_language = '{{ .env.language }}' {{if dev}} LIMIT {{ .env.local_limit }} {{end}} ``` :::warning When applying templated logic to model SQL, make sure to leverage the `ref` function If you use templating in SQL models, you must replace references to tables/models created by other sources or models with `ref` tags. This ensures that the native Go templating engine used by Rill is able to resolve and correctly compile the SQL syntax during runtime (to avoid any potential downstream errors). ::: If we simply run Rill Developer using `rill start`, our model will look like the following (this will also reflect our data model in production, i.e., Rill Cloud, after we've [pushed the changes for the project to GitHub](/developers/deploy/deploy-dashboard)): ![Variables Example](/img/deploy/templating/vars-example.png) **Now**, just to illustrate what a local override might look like, let's say we stop Rill Developer and then restart Rill via the CLI with the following command: ```bash rill start --env language="ja" --env local_limit=100 ``` Even though we have defaults set in `rill.yaml` (and this will be used by any downstream models and dashboards on Rill Cloud), we will instead see these local overrides come into effect with our templated logic to return Japanese movies and the model limit is now 100 rows. ![Variables Override Example](/img/deploy/templating/vars-override-example.png) --- #### Structure --- ###### Structure your project --- title: Structure your project sidebar_label: Organize your Code Files sidebar_position: 00 --- After creating your initial set of sources, models, and dashboards, you may have noticed the following _native_ folders that exist in your Rill project directory: - [Models](/reference/project-files/models) - [Metrics Views](/reference/project-files/metrics-views) - [Dashboards](/reference/project-files/explore-dashboards) By default, any new sources, models, metrics views, and dashboards will be created in their respective native folders. However, this does not necessarily have to be the case, and Rill Developer allows for a flexible project directory structure, including nested folders or even storing objects in non-native folders. This is a powerful feature that allows you, as a developer, to organize your project to meet your team's specific needs. ## Adding new resources or parent folders Within Rill Developer, from the left-hand side (file explorer), you should be able to click on the `Add` button to add a new resource, such as a new source, model, or dashboard. Furthermore, you will also have the ability to add a new parent folder to store groups of resources (which can be mixed). If you choose to add a new folder, you should see the folder structure reflected when you check the project directory via the CLI. ![Adding Objects](/img/build/structure/adding-objects.png) :::warning Make sure to include the `type` property For backward compatibility purposes, any resource that belongs in the `sources`, `models`, and `dashboards` native folders is assumed to be a source, model, or dashboard respectively (including nested folders that belong within a native folder). However, if you'd like to create a resource outside one of these native folders, make sure to include the `type` property in the resource definition, or Rill will not be able to properly resolve the resource type! For more details, see our [reference documentation](/reference/project-files/rill-yaml). ::: ## Navigating Upstream / Downstream Objects ![Breadcrumb](/img/build/structure/breadcrumb.png) When selecting between a source, model, metrics view, and dashboard, you can view the upstream/downstream objects to the current view. For example, if you are selecting a metrics view, you can see all of the dashboards (in a dropdown) that are built on the metrics view. Likewise, if your model references several sources, these will be available to select. ## Moving resources within your project From the UI, within the file explorer, you should be able to drag resources/objects around and move them to or from folders as necessary. :::info Using the CLI For developers who prefer to use the CLI, the project structure can still be controlled or adjusted directly via the CLI or your preferred IDE (e.g., VS Code). ::: ## Adding a nested folder Rather than creating a parent folder and moving it into another folder manually, there is a shortcut to create a nested folder by directly hovering over an existing folder in the file explorer, clicking on the "triple dots," and selecting `New folder`. ![Adding Nested Folder](/img/build/structure/adding-nested-folder.png) --- ##### "AI Configuration" --- title: "AI Configuration" description: "Configure AI instructions for your Rill project" sidebar_label: "AI Configuration" sidebar_position: 55 --- # AI Configuration Rill's AI capabilities, including [AI Chat](/guide/ai/ai-chat) and the [MCP Server](/guide/ai/mcp), rely on context to provide accurate and relevant answers. You can provide additional context using the `ai_instructions` field in your project configuration files. LLMs give their best results when they have good context. For a conversation with Rill Data, this means things like clarifying project-specific terms, routing questions to the correct metrics view, or defining business rules. Rather than expecting the user to provide this context every time, you can add `ai_instructions` to your project. This adds the context automatically for every conversation. There are two places to add `ai_instructions`: 1. **`rill.yaml`**: Project-wide instructions that apply to all queries across your entire project. 2. **`.yaml`**: Metrics view-specific instructions for individual dashboards. ## Automatic Context Inclusion In addition to `ai_instructions`, Rill automatically includes the following in the AI context: - **Measure and dimension descriptions**: Any `description` fields you add to measures and dimensions in your metrics view YAML files are automatically included in the AI context. This helps the AI understand what each metric or dimension represents without requiring you to duplicate that information in `ai_instructions`. - **Metrics view metadata**: The metrics view name, display name, and description are included to help route questions to the correct dashboard. This means you can document your measures and dimensions directly in your metrics view YAML, and that documentation will be available to the AI automatically. ## Project-Level Instructions ([`rill.yaml`](/developers/build/project-configuration)) Use the `ai_instructions` field in `rill.yaml` to provide information that is **unique to your project**. This helps the AI agent deliver more relevant and actionable insights tailored to your specific needs. **What to include:** - Guidance on which metrics views are most important or should be prioritized for your project - Any custom business logic, definitions, or terminology unique to your data or organization - Preferences for aggregations, filters, or dimensions that are especially relevant to your use case - Specific business context that helps the AI understand your domain **Example:** Here's an example of how you might configure `ai_instructions` in your `rill.yaml` to provide project context, metrics routing, and business definitions: ```yaml ai_instructions: | # Project Context This project tracks e-commerce metrics for our multi-brand retail business. # Metrics View Routing - For questions about overall sales, revenue, or order volume → use `company_sales_metrics` - For questions about customer behavior, retention, or cohorts → use `customer_analytics` - For questions about product performance or inventory → use `product_metrics` - For questions about marketing campaigns or attribution → use `marketing_performance` - For questions about fulfillment, shipping, or logistics → use `operations_metrics` # Business Rules & Definitions - "Revenue" always refers to net revenue (after returns and discounts) - "Conversion rate" is calculated as orders/sessions, not users - Our fiscal year starts in February, not January - "Active customer" means a purchase within the last 90 days - Weekend traffic patterns are anomalous due to our B2B focus # Company Acronyms - GMV = Gross Merchandise Value - AOV = Average Order Value - ROAS = Return on Ad Spend - SKU = Stock Keeping Unit - NDR = Net Dollar Retention - CLTV = Customer Lifetime Value # Known Data Quirks - Mobile web data before March 2024 is incomplete due to tracking migration - European region data excludes VAT (use `revenue_with_vat` dimension if needed) - Refunds are processed with a 2-3 day delay, so recent data may shift ``` ## Metrics View-Level Instructions ([`.yaml`](/developers/build/metrics-view/what-are-metrics-views)) You can provide context and instructions for AI tools interacting with a specific metrics view using the `ai_instructions` field in the metrics view's YAML file. This is useful for clarifying specific metrics, dimensions, or data quirks that apply only to that specific view. :::tip Use descriptions for measure and dimension documentation Instead of (or in addition to) adding definitions in `ai_instructions`, you can use the `description` key in the measure and dimension definitions in your metrics view YAML to document what each metric or dimension represents. These descriptions are automatically included in the AI context, making your metrics view self-documenting. ::: **Example:** ```yaml ai_instructions: | # Analysis Guidance - When analyzing "Revenue", always breakdown by "Region" to see currency impacts. - For questions about user growth, prioritize the "monthly_active_users" measure over "daily_active_users". - When comparing time periods, account for the fact that data for the "Legacy Plan" is static and will not update after Dec 2023. # Data Context - Mobile web data before March 2024 is incomplete due to tracking migration. - Refunds are processed with a 2-3 day delay, so recent data may shift. - Weekend traffic patterns are anomalous due to our B2B focus. ``` ## Visualization Tips When using the [Rill MCP Server](/guide/ai/mcp) with external AI clients like Claude, you can provide specific instructions on how to visualize data. Since the MCP server returns structured data, the AI client is responsible for rendering it. :::note Visualization tips affect all AI interactions Visualization instructions added to `rill.yaml` will affect both [Rill Chat](/guide/ai/ai-chat) responses and external AI clients via the MCP Server. If you only want visualization tips to apply to external AI clients (like Claude Desktop), consider adding them to your client-specific configuration files instead: - **Claude Desktop**: Add to `claude_desktop_config.json` or `Claude.md` in your project - **Cursor**: Add to `.cursorrules` or `AGENT.md` in your project - **Other AI clients**: Check your client's documentation for where to add custom instructions This way, visualization formatting will only apply when using external clients, while Rill Chat maintains its default formatting. ::: You can add instructions to your `rill.yaml` to guide the AI in presenting data more effectively (note that this will affect both Rill Chat and MCP clients): ```yaml ai_instructions: | # Visualization Guidelines - When presenting time series data, use sparklines or trend indicators (e.g. 📈/📉) to show direction. - For comparisons, clearly state the percentage change and absolute difference. - Use bar charts for categorical comparisons when there are fewer than 10 categories. - When showing tables, always include a header row and align numeric columns to the right. # Example Formatting - Bar Charts using block characters: Q1 ████████░░ 411 Q2 ██████████ 514 Q3 ██████░░░░ 300 Q4 ████████░░ 400 - Horizontal progress bars: Project Progress: Frontend ▓▓▓▓▓▓▓▓░░ 80% Backend ▓▓▓▓▓▓░░░░ 60% Testing ▓▓░░░░░░░░ 20% - Using different block densities: Trends: Jan ▁▂▃▄▅▆▇█ High Feb ▁▂▃▄▅░░░ Medium Mar ▁▂░░░░░░ Low - Sparklines with Unicode Basic sparklines: Stock prices: ▁▂▃▅▂▇▆▃▅▇ Website traffic: ▁▁▂▃▅▄▆▇▆▅▄▂▁ CPU usage: ▂▄▆█▇▅▃▂▄▆█▇▄▂ - Trend indicators: AAPL ▲ +2.3% GOOG ▼ -1.2% MSFT ► +0.5% TSLA ▼ -3.1% - Simple trend arrows: Sales ↗️ (+15%) Costs ↘️ (-8%) Profit ⤴️ (+28%) ``` --- ##### Alerts --- title: Alerts description: Define alerts as code for automated monitoring and notifications sidebar_label: Alerts sidebar_position: 45 --- ## Overview Alerts in Rill allow you to monitor your data and receive notifications when specific conditions are met. While alerts can be created through the UI, defining them as code in YAML files provides version control, reproducibility, and the ability to manage complex alerting logic programmatically. When you create an alert via a YAML file, it appears in the UI marked as `Created through code`. :::tip Using live connectors? If you're using [live connectors](/developers/build/connectors/olap) (ClickHouse, Druid, Pinot, StarRocks, etc.), **alerts are your primary tool for data quality monitoring**. Since live connectors don't create local models, [data quality tests](/developers/build/models/data-quality-tests) won't run. Use alerts instead to validate your data on a schedule. ::: ## Alert Structure An alert YAML file has the following core components: ```yaml type: alert display_name: My Alert Name description: A brief description of what this alert monitors # When to check the alert refresh: cron: "0 * * * *" # Every hour # What data to check data: sql: SELECT * FROM my_model WHERE condition_is_bad # Where to send notifications notify: email: recipients: - team@example.com ``` ## Scheduling Alerts The [`refresh`](/reference/project-files/alerts#refresh) property defines when and how often the alert runs. ### Cron Schedule Use standard `cron` expressions to define the schedule: ```yaml refresh: cron: "0 * * * *" # Every hour time_zone: "America/New_York" # Optional timezone ``` ### Interval-Based Monitoring Use `intervals` when you need to check data across multiple time windows, such as validating metrics for each hour or day. This is useful for time-series monitoring where you want to ensure data quality across a rolling window of time periods. Interval-based monitoring is more flexible than simple cron schedules when you need to check multiple historical periods on each evaluation. ```yaml refresh: cron: "5 * * * *" # 5 minutes past each hour intervals: duration: PT1H # 1 hour intervals limit: 24 # Check last 24 intervals check_unclosed: false ``` ## Data Sources Alerts support multiple data source types to query your data. ### SQL Query Execute raw SQL against your models: ```yaml data: sql: | SELECT * FROM orders WHERE created_at < NOW() - INTERVAL '24 hours' AND status = 'pending' ``` The alert triggers when the query returns **any rows**. ### Metrics SQL Use `metrics_sql` when you want to query a [metrics view](/developers/build/metrics-view) using its defined dimensions and measures, rather than writing raw SQL against the underlying model. This approach leverages the metrics view's security policies and allows you to reference measures and dimensions by name. For details on the `metrics_sql` syntax, see [Metrics SQL APIs](/developers/build/custom-apis/metrics-sql). ```yaml data: metrics_sql: | SELECT * FROM sales_metrics WHERE total_revenue < 1000 ``` ### Custom API Use a custom API when you want to reuse complex query logic that's already defined as a [Custom API](/developers/build/custom-apis) in your project. This approach is useful for sharing validation logic between alerts and other integrations, or when you need to pass dynamic arguments to your alert queries. ```yaml data: api: my_custom_validation_api args: threshold: 100 date_range: "7d" ``` ### Resource Status Monitor the health of your Rill resources to catch pipeline failures and reconciliation errors. This is useful for monitoring pipeline health and catching reconciliation failures before they impact downstream processes. ```yaml data: resource_status: where_error: true ``` This triggers when any resource in your project has a reconciliation error. ## Notification Configuration Configure where and how you receive notifications when alerts trigger. You can send notifications via email, Slack, or both. Notifications are sent when the alert condition is met (when the data query returns rows), and optionally when the alert recovers or encounters evaluation errors. ### Email Notifications ```yaml notify: email: recipients: - alice@example.com - bob@example.com - data-team@example.com ``` ### Slack Notifications Before using Slack notifications, you must [configure the Slack integration](/developers/build/connectors/services/slack) for your project. ```yaml notify: slack: channels: - "#data-alerts" - "#engineering" users: - "U1234567890" # Slack user IDs webhooks: - "https://hooks.slack.com/services/..." ``` ### Combined Notifications Send to multiple destinations: ```yaml notify: email: recipients: - team@example.com slack: channels: - "#alerts" ``` ## Alert Behavior ### Recovery Notifications Control when you receive notifications about alert state changes. Use `on_recover` to confirm issues are resolved and get peace of mind that problems have been fixed. Use `on_error` to catch alert evaluation failures (e.g., query syntax errors) that prevent the alert from running properly. ```yaml on_recover: true # Notify when alert recovers on_fail: true # Notify when alert triggers (default) on_error: false # Notify on evaluation errors ``` ### Re-notification (Snooze) Control how often you're notified for ongoing issues. This prevents alert fatigue while ensuring ongoing issues aren't forgotten. Instead of receiving notifications on every evaluation cycle, you'll only be re-notified after the specified duration if the alert is still failing. ```yaml renotify: true renotify_after: "24h" # Re-notify every 24 hours if still failing ``` ## Working Examples ### Data Freshness Alert This example demonstrates a data freshness check that queries the maximum timestamp from an events model and triggers when data is older than 24 hours. It uses both email and Slack notifications, includes recovery notifications to confirm when data freshness is restored, and implements re-notification every 6 hours to prevent alert fatigue while ensuring ongoing issues are tracked. ```yaml # alerts/data_freshness.yaml type: alert display_name: Data Freshness Check description: Alert when event data is stale refresh: cron: "0 * * * *" # Check every hour data: sql: | SELECT 'Data is stale' AS error_message FROM ( SELECT MAX(event_timestamp) AS latest_event FROM events_model ) WHERE latest_event < NOW() - INTERVAL '24 hours' notify: email: recipients: - data-ops@example.com slack: channels: - "#data-alerts" on_recover: true renotify: true renotify_after: "6h" ``` ### Project Health Monitor This example monitors the overall health of your Rill project by checking for any resource reconciliation errors. It runs every 10 minutes for rapid detection of pipeline failures, uses the `resource_status` data source to automatically detect errors across all resources, and sends notifications to both Slack and email channels. Recovery notifications ensure you're alerted when issues are resolved. ```yaml # alerts/project_health.yaml type: alert display_name: Project Health Monitor description: Alert when any resource has a reconciliation error refresh: cron: "*/10 * * * *" # Every 10 minutes data: resource_status: where_error: true notify: slack: channels: - "#rill-alerts" email: recipients: - platform-team@example.com on_recover: true ``` ### Interval-Based Monitoring Example This example shows how to use interval-based monitoring to validate metrics across multiple time periods. It checks hourly aggregates for the last 24 hours, looking for any hours with zero event counts. The alert runs 5 minutes past each hour to ensure the previous hour's data is complete, and uses the `intervals` configuration to systematically check each hour in the rolling window. This pattern is ideal for time-series data quality monitoring where you need to validate multiple periods on each evaluation. ```yaml # alerts/hourly_metrics.yaml type: alert display_name: Hourly Metrics Check description: Validate metrics for each hour refresh: cron: "5 * * * *" # 5 minutes past each hour intervals: duration: PT1H # 1 hour intervals limit: 24 # Check last 24 intervals check_unclosed: false data: sql: | SELECT * FROM hourly_aggregates WHERE hour_start = DATE_TRUNC('hour', NOW() - INTERVAL '1 hour') AND event_count = 0 notify: slack: channels: - "#monitoring" ``` ## Reference For the complete specification of all available properties, see the [Alert YAML Reference](/reference/project-files/alerts). :::note Advanced Properties For advanced properties like `glob`, `for`, `watermark`, and `timeout`, see the [Alert YAML Reference](/reference/project-files/alerts). ::: --- ##### "Your First Rill Project" --- title: "Your First Rill Project" description: "Get started with Rill by understanding what happens when you create a project" sidebar_label: "Your First Project" sidebar_position: 1 --- # Your First Rill Project When you create a new Rill project, you're setting up a complete data pipeline that transforms raw data into interactive dashboards. This guide explains what happens during project initialization and introduces you to the key files you'll work with. ## What Happens When You Create a Rill Project? A Rill project consists of several data assets that work together to create a dashboard. The data pipeline begins with [connecting](/developers/build/connectors) to your data sources, transforms raw data through [models](/developers/build/models), defines metrics and dimensions in [metrics views](/developers/build/metrics-view), and results in interactive [dashboards](/developers/build/dashboards) for data analysis. When you create a new Rill project, the following files are automatically generated: - **`rill.yaml`** - Central configuration hub for your entire project - **`connectors/.yaml`** - Connector configuration files for the project's default OLAP engine (e.g., `duckdb.yaml`, `clickhouse.yaml`) - **`.gitignore`** - Git ignore rules for the project ## Initial Project Files These are the foundational files that Rill creates for you. As you build your project, you'll add many more files including sources, models, metrics views, and dashboards. ### `rill.yaml` The central configuration file that controls project-wide settings. You rarely need to modify `rill.yaml` when starting out - the defaults work great! This file enables you to set [project-wide defaults](/developers/build/project-configuration#model-defaults), [configure variables](/developers/build/project-configuration#variable-management), define [connector settings](/developers/build/project-configuration#olap-connector), create [test users](/developers/build/project-configuration#testing-security), and establish [security policies](/developers/build/project-configuration#metrics-views-security-policy). ### `connectors/.yaml` Configuration for your default OLAP engine (DuckDB, ClickHouse, Druid, or Pinot). When starting a blank project, this always defaults to `duckdb.yaml`. This file defines how Rill connects to your analytical database. Connectors enable Rill to connect to various data sources and OLAP engines. You can configure [data source connectors](/developers/build/connectors/data-source) (like S3, GCS, BigQuery, Snowflake) to ingest data, and [OLAP connectors](/developers/build/connectors/olap) to power your analytics. See our [connectors documentation](/developers/build/connectors) for the full list of supported connections. ### `.gitignore` Specifies which files and directories should be ignored by Git version control. Rill automatically generates this file with appropriate rules to exclude sensitive files like `.env`, temporary files, and build artifacts from being committed to your repository. ## Preview Mode Preview mode starts Rill Developer with a dashboard-only interface — no file editor, no code. It's designed for sharing a local Rill instance with stakeholders who don't need the development environment. ```bash rill start my-project --preview ``` ![Preview Mode](/img/get-started/preview-mode.png) Preview mode sets the application to **read-only**, so dashboards cannot be modified through the UI. :::tip For a full team deployment with authentication and access controls, [deploy to Rill Cloud](/developers/deploy/deploy-dashboard). ::: ## Next Steps Now that you understand the basics, you can: 1. **[Connect to your data](/developers/build/connectors)** - Set up connections to your data sources 1. **[Create your first model](/developers/build/models)** - Transform your raw data 2. **[Build a metrics view](/developers/build/metrics-view)** - Define your metrics and dimensions 3. **[Create a dashboard](/developers/build/dashboards)** - Visualize your data 4. **[Deploy to Rill Cloud](/developers/deploy)** - Deploy your project to Rill Cloud When you need more control over your project configuration, see our [project configuration guide](/developers/build/project-configuration) for advanced settings and customization options. --- ##### Building Your Data Analytics Pipeline in Rill --- title: Building Your Data Analytics Pipeline in Rill sidebar_label: Build sidebar_position: 0 --- Rill provides a comprehensive platform for building end-to-end data analytics solutions. Once you've connected to your source data or connected to your own OLAP database, you can start building with Rill. Don't forget to check out the [reference documentation!](/reference/project-files) ### What is a Rill project? - [**Getting Started with your Rill Project**](/developers/build/getting-started) - Understand Rill project structure and configuration ### Connect to your Data - [**Connectors Overview**](/developers/build/connectors) - Learn about connecting Rill to data sources and OLAP engines - [**OLAP Engines**](/developers/build/connectors/olap) - Configure DuckDB, ClickHouse, Druid, or Pinot for analytics - [**Data Sources**](/developers/build/connectors/data-source) - Connect to S3, GCS, BigQuery, Snowflake, and more - [**Configure Local Credentials**](/developers/build/connectors/credentials) - Set up credentials for local development - [**Dev/Prod Connectors**](/developers/build/connectors/templating) - Separate your production and development data sources ### Modeling and Last Mile ETL - [**Models Overview**](/developers/build/models) - Overview of models available in Rill - [**Getting Started with Models**](/developers/build/models/models-101) - What are models and important topics to remember as you create your own. - [**Differences between SQL and YAML Models**](/developers/build/models/model-differences) - Compare SQL and YAML models and learn when to use - [**SQL Models**](/developers/build/models/sql-models) - Transform and prepare your data - [**Data Refresh**](/developers/build/models/data-refresh) - Schedule your data refresh - [**YAML Models**](/developers/build/models/model-differences#yaml-models) - Incremental ingestion, data partitions, and more - [**Additional Model Templating**](/developers/build/models/templating) - Separate your production and development settings - [**Performance Optimization**](/developers/build/models/performance) - Optimize model performance and data refresh strategies ### Define Measures and Dimensions - [**What are Metrics Views?**](/developers/build/metrics-view/what-are-metrics-views) - Learn about Metrics Views! - [**Underlying Model/Table**](/developers/build/metrics-view/underlying-model) - Powering your metrics view with a `model` or `table` - [**Time Series**](/developers/build/metrics-view/time-series) - The backbone of Rill Explore Dashboards, time - [**Dimensions**](/developers/build/metrics-view/dimensions) - Expand your dimensions capabilities further in your metrics view - [**Measures**](/developers/build/metrics-view/measures) - Aggregate your measures further in your metrics view - [**Define Data Access**](/developers/build/metrics-view/security) - Define access and row access policies ### Build Visualizations - [**Dashboards Types**](/developers/build/dashboards/dashboards-101) - What are the visualization options in Rill, and how are they different? - [**Slice-and-Dice Dashboards**](/developers/build/dashboards/explore) - Explore and find insights in your data - [**Traditional Visualizations**](/developers/build/dashboards/canvas) - Visualize your data with various chart types - [**Canvas Components**](/developers/build/dashboards/canvas-widgets) - See all of our supported components! - [**Define Dashboard Access**](/developers/build/dashboards/customization#define-dashboard-access) - Set a SQL boolean query that defines access to the dashboard ### Build Integrations with Custom APIs - [**Custom APIs Overview**](/developers/build/custom-apis) - Create HTTP API endpoints to expose your Rill data - [**SQL APIs**](/developers/build/custom-apis/sql) - Query models, tables, and external databases - [**Metrics SQL APIs**](/developers/build/custom-apis/metrics-sql) - Query metrics views with automatic security inheritance - [**Dynamic Queries & Templating**](/developers/build/custom-apis/templating) - Add parameters, user attributes, and conditional logic - [**Security & Access Control**](/developers/build/custom-apis/security) - Multi-tenant APIs with custom attributes on tokens ### Project Settings - [**Project Configuration**](/developers/build/project-configuration) - Configure your Rill Project and set default behavior - [**Structure your Project**](/developers/build/structure) - Structure folder architecture in Rill - [**Use your favorite IDE**](/developers/build/ide) - Utilize your favorite IDE to build Rill projects - [**Debugging Rill Developer**](/developers/build/debugging/trace-viewer) - Troubleshoot dashboard access, trace your reconciled resources, and understand project logs --- ##### "Project Configuration" --- title: "Project Configuration" description: "Complete guide to configuring your Rill project with rill.yaml and other project files" sidebar_label: "Project Configuration" sidebar_position: 50 --- # Project Configuration This guide covers all aspects of configuring your Rill project, from basic settings to advanced security and testing configurations. ## OLAP Connector When you add an OLAP connector to your project, Rill automatically updates the `olap_connector` field in `rill.yaml` with the new connector name (e.g., `clickhouse`, `druid`). If you create multiple connectors of the same type, Rill will number them sequentially. For example, if you create three ClickHouse connectors, they will be named: - `clickhouse` (first connector) - `clickhouse_2` (second connector) - `clickhouse_3` (third connector) The most recently created connector becomes the default and is referenced in `rill.yaml`. ```yaml olap_connector: clickhouse ``` The default OLAP connector is used as the default `output` for all of your models unless otherwise specified. ## Model Defaults Configure default behaviors for all models in your project. These settings control refresh schedules and materialization behavior that apply across all models unless overridden in individual model files. ### Model Refresh Schedule Set up your project's model refresh schedule. You can override this in the model's YAML file if needed. ```yaml models: refresh: cron: '0 * * * *' ``` For more information on configuring model refreshes, see the [models documentation](/developers/build/models) and [model refresh reference](/reference/project-files/models#refresh). ### Model Materialization By default, models in Rill are created as views rather than materialized tables. While views enable a dynamic keystroke-by-keystroke modeling experience, materializing models as tables can significantly improve performance for complex queries and large datasets. You can set a default materialization behavior for all models in your project: ```yaml models: materialize: true ``` This is particularly useful in production environments where performance is prioritized over the interactive modeling experience. Individual models can still override this setting using `-- @materialize: false` in their SQL file, or `materialize: false` in the YAML file. :::tip When to materialize - **Recommended:** Final models that power dashboards should be materialized for better performance - **Consider:** Complex intermediate models with large datasets - **Avoid:** Models with cross joins or when you need real-time keystroke-by-keystroke updates For more details, see our [model performance guide](/developers/build/models/performance#materialization). ::: ## Metrics Views Defaults Configure default settings for all metrics views in your project. These settings control time-related behavior and security policies that apply across all metrics views unless overridden in individual metrics view files. For comprehensive information on metrics views, see the [metrics view documentation](/developers/build/metrics-view). ### Metrics Views Time Modifiers Set default time modifiers for all metrics views, such as `first_day_of_week` or `smallest_time_grain` as shown below. For more parameters, see the [metrics view reference page](/reference/project-files/metrics-views). ```yaml metrics_views: first_day_of_week: 1 smallest_time_grain: month ``` ### Metrics Views Security Policy By default, Rill is open to access (to your organization users), unless otherwise defined. To add project-level access to the Rill project, you can add a default metrics view security policy in the `rill.yaml` file. Like in a metrics view file, you can define the security as shown below. For more information, read our [data access documentation](/developers/build/metrics-view/security#examples). ```yaml metrics_views: security: access: '{{ has "partners" .user.groups }}' row_filter: "domain = '{{ .user.domain }}'" ``` :::tip Security Policy Rules Rill YAML settings < (Metrics View YAML AND Dashboard YAML) For detailed guide on security policies, review our [data access policies](/developers/build/metrics-view/security) doc. ::: ## Dashboard Defaults Rill supports two types of dashboards: **Explores** (metrics-focused dashboards) and **Canvases** (custom visualization dashboards). You can set default configurations for each type. Learn more about creating and customizing dashboards in our [dashboard documentation](/developers/build/dashboards). ### Dashboard Security Policy Similar to metrics views, you can set [security for a dashboard](/developers/build/dashboards/customization#define-dashboard-access). (Note that only `access` can be set at the dashboard level.) ```yaml explores: security: access: "'{{ .user.domain }}' == 'example.com'" canvases: security: access: '{{ has "dev" .user.groups }}' ``` ### Explore Defaults You are also able to set the `defaults` parameter in the explore dashboard to define your default time range, as well as the available `time_zones` and `time_ranges` in an Explore dashboard. :::note Time Range Format Time ranges use [ISO 8601 duration format](https://en.wikipedia.org/wiki/ISO_8601#Durations). Common examples: - `PT24H` = 24 hours - `P7D` = 7 days - `P3M` = 3 months - `P24M` = 24 months ::: ```yaml explores: defaults: time_range: P3M time_zones: - UTC time_ranges: - PT24H - P7D - P14D - P3M ``` ### Canvas Defaults Similarly, you can configure defaults for canvas dashboards: ```yaml canvases: defaults: time_range: P7D time_zones: - UTC time_ranges: - PT24H - P7D - P14D - P3M ``` :::tip Understanding Dashboard Default Behavior When setting dashboard defaults, keep in mind that Rill Cloud saves each user's last state on dashboards. This means users will see the view they last used rather than the defaults configured above. **Settings precedence (lowest to highest):** Rill YAML settings < Dashboard YAML < Bookmarks in Rill Cloud < User Last State Defaults are most useful for new users accessing a dashboard for the first time or when viewing in Rill Developer. ::: ## Environment Configuration ### Differentiating Dev and Prod Environments Rill comes with default `dev` and `prod` properties defined, corresponding to Rill Developer and Rill Cloud, unless otherwise specified in the `rill start --environment (dev/prod)` command for Rill Developer. You can use these keys to set environment-specific YAML overrides or SQL logic. For example, the following `rill.yaml` file explicitly sets the default materialization setting for models to `false` in development and `true` in production: ```yaml dev: models: materialize: false prod: models: materialize: true ``` :::note Specifying a custom environment When using Rill Developer, instead of defaulting to `dev`, you can run your project in production mode using the following command: ```bash rill start --environment prod ``` ::: ## Variable Management Variables in Rill enable dynamic templating throughout your project files. They can be used in SQL queries, YAML configurations, and security policies to make your project more flexible and maintainable. ### Setting Variables Variables are defined in your `rill.yaml` file using the `env` key. This allows you to set variables that will be available in your Rill Cloud deployments while maintaining the ability to use different values locally during development. ```yaml env: numeric_var: 10 string_var: "string_value" api_endpoint: "https://api.example.com" ``` Once defined, these variables can be referenced in your project files using template syntax: `{{ .env.variable_name }}` **Example usage in a model:** ```sql SELECT * FROM users WHERE status = '{{ .env.string_var }}' AND score > {{ .env.numeric_var }} ``` ## AI Configuration You can provide context and instructions for AI tools (like [AI Chat](/guide/ai/ai-chat) and [MCP](/guide/ai/mcp)) using the `ai_instructions` field. This helps the AI agent deliver more relevant and actionable insights tailored to your specific needs. For detailed instructions and examples on how to configure AI instructions at both the project and metrics view levels, please refer to our [AI Configuration guide](/developers/build/ai-configuration). ## Testing Security ### Test Access Policies in Rill Developer Testing access policies in your local environment is a crucial step before deploying to Rill Cloud. This is done via the `mock_users` in the project file. You can create pseudo-users with specific domains, admin/non-admin roles, or user groups to ensure that access policies work as intended. For comprehensive information on security policies, see our [data access policies documentation](/developers/build/metrics-view/security). Let's assume that the following security policy is applied to the metrics view. ```yaml security: access: "{{ .user.admin }} OR '{{ .user.domain }}' == 'rilldata.com'" row_filter: "region = '{{ .user.region }}'" ``` In order to test both access to the dashboard, as well as the row filter, you can create the following in the project YAML. ```yaml mock_users: - email: royendo@rilldata.com admin: true region: us-west - email: your_email@domain.com groups: - tutorial-admin region: us-east - email: your_email2@another_domain.com region: europe ``` See our embedded example, [here](https://rill-embedding-example.netlify.app/security/filter-by-user). ### Custom Attributes Embedded dashboards allow passing custom attributes (variables) from your application to control access and filtering. These attributes are set when generating the embed JWT token in your application code. For more information on embedding dashboards, see our [embedding documentation](/developers/embed/iframe). To test embedded dashboards locally with custom attributes, add them to `mock_users`: ```yaml - email: embed@rilldata.com name: embed custom_variable_1: Value_1 #this is passed at embed creation custom_variable_2: Value_2 #this is passed at embed creation ``` See our [Custom Attributes Embedded Dashboard](https://rill-embedding-example.netlify.app/security/filter-by-custom-attributes) live! Let's assume a similar setup to the above example. Within the metrics view, we define: ```yaml security: access: true row_filter: > app_site_name = '{{ .user.app_site_name }}' AND pub_name = '{{ .user.pub_name }}' ``` Then within the application we are passing ```yaml app_site_name='Sling' pub_name='MobilityWare' ``` You can create a test mock user to ensure that this dashboard is working as designed with the following: ```yaml - email: embed@rilldata.com name: embed app_site_name: 'Sling' pub_name: 'MobilityWare' ``` ![Custom Attribute Mock User](/img/tutorials/admin/custom-attribute-mock-user.png) ## Feature Flags :::warning Experimental Features Feature flags enable experimental functionality that may be unstable or change before general availability. Use with caution in production environments. ::: If you are interested in testing our upcoming features and experimental functionality, you can enable feature flags in your `rill.yaml` file. These flags allow you to access beta features and provide early feedback on new capabilities before they become generally available. To enable feature flags, add the `features` section to your `rill.yaml`: ```yaml features: - cloudDataViewer ``` **Available feature flags:** - `cloudDataViewer`: Enables the cloud data viewer interface for exploring data directly in the browser (default: `false`) - `dimensionSearch`: Enables advanced dimension search functionality (default: `false`) - `twoTieredNavigation`: Enables two-tiered navigation interface (default: `false`) - `rillTime`: Enables Rill-specific time functionality (default: `false`) - `hidePublicUrl`: Hides public URL sharing options (default: `false`) - `exportHeader`: Enables export header functionality (default: `false`) - `alerts`: Enables alerting features (default: `true`) - `reports`: Enables reporting functionality (default: `true`) - `chat`: Enables chat functionality (default: `true`) - `dashboardChat`: Enables chat features within dashboards (default: `false`) **How to enable**: Add the `features` section to your `rill.yaml` **Reporting issues**: If you encounter issues with feature flags, please [contact support](/contact) with details about the flag and behavior. For a complete list of available feature flags and their current status, see our [feature flags reference](https://github.com/rilldata/rill/blob/main/web-common/src/features/feature-flags.ts#L36) in the codebase. ## Complete Example Here is an example YAML that uses many of our features. ```yaml compiler: rillv1 display_name: Rill Project Dev # The project's default OLAP connector. # Learn more: https://docs.rilldata.com/reference/olap-engines olap_connector: duckdb # Project variables for templating env: numeric_var: 10 string_var: "string_value" # Project Defaults models: refresh: cron: '0 * * * *' run_in_dev: true metrics_views: first_day_of_week: 1 smallest_time_grain: month explores: defaults: time_range: P24M time_zones: - UTC - America/Los_Angeles - America/New_York - Europe/London - Europe/Paris - Asia/Tokyo - Australia/Sydney time_ranges: - PT24H - P6M - P12M canvases: defaults: time_range: P24M time_zones: - UTC - America/Los_Angeles - America/New_York - Europe/London - Europe/Paris - Asia/Tokyo time_ranges: - PT24H - P7D - P14D - P30D - P3M - P6M - P12M # These are example mock users to test your security policies. # Learn more: https://docs.rilldata.com/developers/build/metrics-view/security mock_users: - email: john@yourcompany.com - email: jane@partnercompany.com - email: your_email@domain.com groups: - tutorial-admin - email: embed@rilldata.com name: embed custom_variable_1: Value_1 #this is passed at embed creation custom_variable_2: Value_2 #this is passed at embed creation features: - cloudDataViewer ``` --- ### Deploy --- #### Deploy Dashboard --- ###### Deploy Dashboards --- title: Deploy Dashboards sidebar_label: Deploy Dashboards sidebar_position: 00 --- ## Overview Deploying dashboards from Rill Developer allows you to share dashboards with other users, leverage [Rill Cloud capabilities](/guide/dashboards/explore), [embed Rill](/developers/embed/iframe) into other applications, and more! The flow diagram below shows two options for deploying an existing project. **Deploy via the UI or CLI using `rill project deploy`**: ```mermaid graph LR; A(Local code files); B(Rill Cloud); A--deploy (once)/update via UI -->B; ``` --- **Deploy via the CLI via `rill project connect-github`**: ```mermaid graph LR; A(Local code files); B(Rill Cloud); C(GitHub); A--1. deploy via CLI (once) -->B; C-- Continuous Deployment-->B; A--2. Push changes to GitHub-->C; ``` ## Deploying a project from Rill Developer Starting from **v0.48**, we have introduced the possibility to push dashboards _directly from Rill Developer to Rill Cloud_. On the dashboard page, you can select the `Deploy` button and follow the steps to deploy to Rill Cloud. ![Deploy UI](/img/deploy/existing-project/deploy-ui.gif) Now that your project has been deployed to Rill Cloud, you will need to ensure that your users have access! Please refer to the [user management](/guide/administration/users-and-access/user-management) section. If you make changes locally on Rill Developer, you will need to push the contents to Rill Cloud by selecting the `Update` button. ![Redeploy](/img/deploy/existing-project/redeploy.gif) :::tip On an older version of Rill? You can easily check the version of Rill that you are using in Rill Developer by running the following command: ```bash rill --version ``` If you are on an older version of Rill, it is **strongly recommended** to [upgrade](/developers/get-started/install#upgrade-to-the-newest-version-of-rill-developer) to the latest version. ::: ### Syncing your GitHub Repository :::note GitHub app permissions This assumes that the installed GitHub app in your organization has write access. If unsure, please check with your GitHub admin. The required permissions are: - Read access to metadata and pull requests - Read and write access to administration and code ::: At this point, you have the option to connect your Rill project to a GitHub Repository. Navigating to the Settings page and selecting `Connect to GitHub` will prompt you to login and create a repository for your project. If you've already created a repository, check the box 'I've created a GitHub Repo' and add the permissions for Rill to access the repository. :::info Check with your GitHub organization admin If you're not the admin of your GitHub organization, they will likely need to first install the Rill Cloud app in your organization before you can proceed with deploying a project. After the Rill Cloud app is installed, it should have the following privileges: ::: ![Install Rill Cloud](/img/deploy/existing-project/install-rill-cloud.png) Once the permissions to the repository have been confirmed and set, you can continue to select the repository in the dropdown. ![Select Repo](/img/deploy/existing-project/select-repo.png) Once completed, you'll see the newly updated repository on the UI of your settings page! ![Finished](/img/deploy/existing-project/finished.png) :::warning Still unable to connect? If you encounter issues, check that the app installation is not pending. Go to your organization's settings and click on Installed GitHub Apps. You will see a section of Pending GitHub Apps installation requests. If you're an Owner or App Manager, grant access to the Rill app if it is pending." ::: ## Deploying a project via the CLI :::note Starting from v0.49, we have deprecated `rill deploy` in favor of `rill project deploy` and `rill project connect-github`. For more information on the `rill deploy` command click [here](#deprecated-rill-deploy). ::: ### Deploy project without GitHub Repository You can add a GitHub Repository later. ``` rill project deploy Using org "Rill_Learn". Starting upload. All files uploaded successfully. Created project "Rill_Learn/my-rill-tutorial". Use `rill project rename` to change name if required. ... Your project can be accessed at: https://ui.rilldata.com/Rill_Learn/my-rill-tutorial Opening project in browser... ``` If you have not already [configured your connections' credentials](https://docs.rilldata.com/developers/build/connectors/credentials), you will be reminded here which connections are required. **First deployment** If this is your first deployment to Rill Cloud, you will get prompted to either sign up or log in (if you have an existing account on [Rill Cloud](https://ui.rilldata.com/)). Proceed with the sign up and email verification process for new users or authorization process for existing users. As a new user, you can expect to see the following page: ![Rill Cloud Sign In](/img/deploy/existing-project/rill-cloud-sign-in.png) **Project Uploaded Successfully** Once the project has been uploaded to Rill Cloud, you should be able to see the following page: ![Status](/img/deploy/existing-project/status.png) ### Deploy Project with Repository Follow the instructions in the Terminal to login to GitHub (if not already done so), and select your repository. If you do not set any parameters, Rill will infer the project name based on the folder path and use this as both the repository and project name. If there are any overlaps, we will request for a new name. ```bash rill project connect-github No git remote was found. ? Do you want to create a repo? Yes ? Select a GitHub account for the new repository royendo Repository name "my-rill-tutorial" is already taken ? Please provide alternate name my-rill-tutorial-cli Request submitted for creating repository. Checking completion status Successfully created repository on "https://github.com/royendo/my-rill-tutorial-cli" Pushing local project to GitHub Successfully pushed your local project to GitHub Using org "Rill_Learn". Created project "Rill_Learn/my-rill-tutorial-cli". Use `rill project rename` to change name if required. Rill projects deploy continuously when you push changes to GitHub. ... Your project can be accessed at: https://ui.rilldata.com/Rill_Learn/my-rill-tutorial-cli Opening project in browser... ``` Once completed, you will see the following in the settings page. Note that the GitHub repository is already set up! ![Cli Upload](/img/deploy/existing-project/cli-upload.png) ## Continuous Deployment Whether you decide to manage your Rill projects using GitHub or by re-running `rill project deploy`, Rill should automatically detect changes that you have pushed locally and update your deployed project accordingly. Depending on the changes, this may result in a project reconciliation. If you are experiencing issues with the project after pushing changes with the CLI, please refer to the project's status page for more information, or run the following command: ``` rill project status ``` Likewise, if using the UI by selecting the `Update` button, Rill will detect the changes in files and update your deployed project accordingly. Along with the above CLI command, you can view the status of the objects in the Status page. :::tip Interested in using GitLab? Check out our documentation on deploying a [Rill project using GitLab](/developers/deploy/deploy-dashboard/deploy-from-cli)! ::: ## Change your production branch By default, Rill deploys from the [default branch](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-branches#about-the-default-branch) of your Git repository. You can change this to any branch you want. To deploy your project from a different branch, run the following command: ```bash rill connect-github --prod-branch [PROD-BRANCH] ``` ## Deploy from a monorepo If your Rill project is in a sub-directory of a Git repository, use the `--subpath` option when creating your project: ``` rill connect-github --subpath path/to/rill/project ``` :::warning Note that you must run `rill connect-github` from the root of your Git repository, **not** the root of your Rill project. ::: ## Deprecated Rill Deploy When running `rill deploy` you have two options: 1. Enable automatic deploys to Rill Cloud via GitHub 2. Disable automatic deploys to Rill Cloud via GitHub ``` rill deploy ? Enable automatic deploys to Rill Cloud from GitHub? ``` ### Enable Automatic deploys Like running `rill project connect-github`, you will be [prompted to create a github repository](#deploy-project-with-repository). Once created, Rill will deploy the project. You can confirm that the project has the correct repository linked from the UI on the settings page. ### Disable Automatic deploys In this case, the project will be deployed to Rill Cloud without a GitHub repository connected. You can always [add a repository via the UI](#syncing-your-github-repository) at a later time. --- ###### Deploy to Rill Cloud from GitLab --- title: Deploy to Rill Cloud from GitLab description: How to set up continuous deploys to Rill Cloud from GitLab sidebar_label: Deploy from GitLab sidebar_position: 10 --- While Rill Cloud natively integrates with [GitHub](https://github.com), you can also deploy your Rill project from [GitLab](https://about.gitlab.com/) using direct uploads from a [GitLab CI/CD pipeline](https://docs.gitlab.com/ee/ci/quick_start/). Follow these steps to set up continuous deployment from GitLab to Rill Cloud: 1. Create a new GitLab repository and push your Rill project to it. 2. On your local, [authenticate with Rill Cloud](/guide/administration/users-and-access/user-management#logging-into-rill-cloud) and create an organization (replace `my-org-name` with your desired name): ```bash rill login rill org create my-org-name ``` 3. Create the project in Rill Cloud ```bash rill project deploy ``` :::note Multiple branches If your repo contains multiple branches ensure the branch you want to deploy from via ```bash rill project edit --project my-project-name --prod-branch my-branch-name ``` ::: 4. Provision a Rill Cloud [service account](/reference/cli/service/create) called `gitlab-ci` and copy its access token: ``` rill service create gitlab-ci ``` 5. Set the service token as a CI/CD variable called `RILL_SERVICE_TOKEN` in GitLab (from the repository page, it's under _Settings > CI/CD > Variables_). 6. Create a file named `.gitlab-ci.yml` at the root of the repository containing your Rill project. Paste the following contents into it (replace `my-org-name` and `my-project-name` with your desired names): ```yaml deploy-rill-cloud: stage: deploy script: - curl -L -o $HOME/rill.zip https://cdn.rilldata.com/rill/latest/rill_linux_amd64.zip - unzip -d $HOME $HOME/rill.zip - git checkout -B "$CI_COMMIT_REF_NAME" "$CI_COMMIT_SHA" - $HOME/rill project deploy --org my-org-name --project my-project-name --interactive=false --api-token $RILL_SERVICE_TOKEN ``` Your Rill project should now automatically deploy to `ui.rilldata.com/my-org-name/my-project-name` each time changes are pushed to GitLab! :::note File size limits We enforce a file size limit of 100mb so ensure you do not unpack the rill binary in the repo root or add it to your .gitignore ::: --- ###### "GitHub Basics" --- title: "GitHub Basics" description: GitHub Basics for Deployment sidebar_label: "GitHub Basics" sidebar_position: 20 --- ## Overview :::info New to Git? This page makes the deploy process easier for those who are less familiar with GitHub by walking you through the steps to use the UI rather than the GitHub command line interface. Much simpler for those who prefer a graphic interface! ::: To share dashboards with other users, Rill utilizes GitHub as a means of version control - effectively creating BI-as-code. There are several advantages to Git as the backend for dashboards: develop locally, integrate into existing development workflows, and manage versions + change control. While many users are familiar with Git and basic commands, there are other Rill users who are relatively new to Git. In this section, we will outline: - Installing the GitHub Desktop app (recommended for those less familiar with Git versus using the command line) - Setting up your repository (repo) and cloning to your local - Pushing changes to Git - Deploying Rill to your new org to share dashboards, set up alerts and receive scheduled reports ![Process](/img/deploy/github-101/process.png) ## Installing Git -To download the GitHub app for a UI driven workflow, visit the [install site here](https://docs.github.com/en/desktop/installing-and-authenticating-to-github-desktop/installing-github-desktop). - In your computer's Downloads folder, double-click the GitHub Desktop zip file. - After the file has been unzipped, double-click the GitHub Desktop application file. - GitHub Desktop will launch after installation is complete. :::note Creating a GitHub account If you do not already have a GitHub account, you'll need to create one as part of this process. For those looking to deploy to a company repo, we would suggest requesting access to a specific repo where you can get started. ::: ## Cloning Locally Once you have downloaded GitHub and connected your account, you'll be able to create a new repository. Think of the repository as a shared cloud drive for the configuration files (sources, models, dashboards) that power Rill. - First, go to GitHub.com and create a new repository - clicking the ```New``` button on the homepage\ ![New](/img/deploy/github-101/new.png) - After you name your repository and click save, you'll see the option to ```Set up in Desktop```. This will open the GitHub desktop app directly to "clone" the repo to your local computer. - ![Repo](/img/deploy/github-101/repo.png) - Select the folder on your computer where you'd like to build with Rill. Confirm the location you cloned for the next step. ![Clone](/img/deploy/github-101/clone.png) - Open the Terminal and go to your local copy of the repo (e.g. the above would be ```cd Documents/GitHub/rill-poc/testrepo```) - Run the Rill install command ```curl https://rill.sh | sh``` (or run ```rill start``` if you've already installed) Now - any changes you make locally can be synced directly with your GitHub repo in one click! You can revisit our [build section](/developers/build/connectors) for refreshers on connecting to new sources, modeling data, or building dashboards. ## Pushing Changes After installing Rill and adding sources, your local project will now contain a variety of new files - source definitions, SQL models for transformation, and dashboards for analysis. GitHub automatically recognizes those files as new and you can now push those changes to your cloud repository. First, open the GitHub app and select your repo. You should now see a variety of new changes (that can then be committed): ![Commit](/img/deploy/github-101/commit.png) Now, in order to actually commit these changes to your repo, you can take the following steps: - On the bottom left, you can describe the changes (or add something like "initial commit") - Select the now blue _Commit to main_ button on the bottom left - Click _Publish Branch_ on the top right - and that's it. Any future changes would follow the same two steps - add a comment, commit and publish! :::warning Avoid Syncing Data Files During the commit process, make sure to **uncheck** files you do not wish to sync to your cloud repo (or add the appropriate exclusions in your `.gitignore` file). In particular, if you were working with local data files, those will now be in the repo including ```tmp``` files and/or sources (e.g. parquet or CSV files). GitHub has a limit on cloud storage so instead consider adding that data to a [cloud object storage location](/developers/build/connectors) (S3, GCS, Azure). ::: :::tip Make small changes directly in github For any larger changes, we would strongly suggest developing locally to see the effects of that change. However, for smaller changes (add a simple metric, changing the name of a dimension, etc.), some users _may_ prefer to edit directly inline on [GitHub](https://github.com) directly. ::: ## Deploying Rill Now that your dashboards should be fully synced to GitHub, version controlled, and available to be edited by others. To make the dashboard fully cloud-enabled, you can return to your terminal and run ```rill project connect-github```. With everything synced, you should now be able to create a [new organization](/guide/administration/organization-settings#organization) within Rill Cloud and push your dashboard to the cloud for shared collaboration. Any future changes should automatically be present in your deployed dashboards on [Rill Cloud](https://ui.rilldata.com) once committed and new dashboards will appear automatically as well (no deploy command needed). More details on [deployment here](/developers/deploy/deploy-dashboard). :::info Have further questions? We'd love to hear from you! We know that utilizing Git may be new to some - please don't hesitate to [reach out](/contact) to us with any questions and we'd be happy to assist. ::: --- ##### Rill Cloud vs Rill Developer --- title: Rill Cloud vs Rill Developer sidebar_label: Rill Cloud vs Rill Developer sidebar_position: 10 hide_table_of_contents: false --- ## What is Rill Cloud and Rill Developer? Rill offers two unique but complementary experiences within our broader product suite, **Rill Cloud** and **Rill Developer**. As the name suggests, Rill _Developer_ is designed with the developer in mind, where project development actually occurs. Rill Developer is meant for the primary developers of project assets and dashboards, allowing them to import, wrangle, iterate on, and explore the data before presenting it for broader consumption by the team. Rill Developer is meant to run on your local machine - see here for some [recommendations and best practices](/developers/tutorials/performance#local-development--rill-developer) - but it is a simple process to [deploy a project](/developers/deploy/deploy-dashboard) once ready to Rill Cloud. Rill Cloud, on the other hand, is designed for our dashboard consumers and allows broader team members to easily collaborate. Once the developer has deployed the dashboard onto Rill Cloud, these users will be able to utilize the dashboards to interact with their data, set alerts / bookmarks, investigate nuances / anomalies, or otherwise perform everyday tasks for their business needs at Rill speed. ## Is Rill Cloud a higher offering than Rill Developer? Based on the naming, it might be confusing and easy to assume that Rill Cloud is our "higher" offering but **that is not the case!** Similarly, Rill Developer is _not meant to be used as a standalone tool either_. Rill Developer and Rill Cloud are to be used together. Rill Developer provides a space for our developers to define and test any new or needed changes to the data and/or dashboards before pushing to our Rill Cloud users, who need stable access to working dashboards. Then, once finalized, these dashboards are deployed to the Rill Cloud project for broader consumption and to power business use cases. :::info Isn't Rill Developer enough? Please note that a common **misnomer** is that Rill Developer can be a sufficient replacement for Rill Cloud. They both serve different purposes but are meant to be used _in conjunction_. Rill enables speed of exploration and is easy to use for developers, allowing the project to be iterated on quickly. Rill Cloud then allows for shared collaboration at scale, especially for production deployments. ::: ### Why deploy to Rill Cloud? Rill Developer is an extremely strong tool for deep-diving into your data, as it allows users to import sources from many destinations and join these tables together to create something useful in a slice-and-dice visualization. Many times the feedback we receive is, "`I didn't even know my data had an issue in it,`" or, "`In a few minutes, I was able to make new insights into my data that would've taken me hours.`" This is great, but if that's possible in Rill Developer, why publish the dashboard to Rill Cloud?

## Rill Developer Rill Developer is designed around developers. Using a familiar IDE-like interface, developers are able to import data, create SQL models, and create metrics views. Many of the underlying files in Rill Developer are either SQL or YAML files. Once data is imported into Rill (and the underlying OLAP engine), developers are able to perform last-mile ETL changes using one or a series of SQL models (as their own [DAG](https://en.wikipedia.org/wiki/Directed_acyclic_graph#:~:text=A%20directed%20acyclic%20graph%20is,a%20path%20with%20zero%20edges)). You can then create and materialize your ["One Big Table"](/developers/build/models/models-101#one-big-table-and-dashboarding) for your dashboard needs. Finally, any specifications for your dimensions and measures can be defined and tested in Developer's dashboard preview. ![Empty Project](/img/concepts/rcvsrd/empty-project.png) ## Rill Cloud Once the dashboard has been [deployed to Rill Cloud](/developers/deploy/deploy-dashboard), the dashboard can be shared with others and viewed by other members of your Rill Cloud organization. As you can see below, the UI is different from Developer. Upon accessing Rill Cloud, a user will be able to view all the projects they have been granted access to by project admins. ![Rill Cloud Projects](/img/concepts/rcvsrd/rill-cloud-projects.png) After selecting a specific project, they will be directed to a list of dashboards. From Rill Cloud, the dashboard consumer does not have the ability to make any modifications to sources or models. However, they are given some additional capabilities that are not accessible in Rill Developer, such as alerting, creating bookmarks or shareable public URLs, checking the project status, and more. :::info Dashboard 101 For more details about using a Rill Cloud dashboard, please refer to our [Explore section](/guide/dashboards/explore)! ::: --- ##### Deployment Credential Considerations --- title: Deployment Credential Considerations description: Configuring credentials for your deployed project on Rill Cloud sidebar_label: Configure Deployment Credentials sidebar_position: 10 --- :::tip Development and Production Setup We recommend reviewing the [Development/Production Setup](/developers/build/connectors/templating) documentation before deploying your project to Rill Cloud to ensure that your local development and production environments have been separated. This will ensure that your shared dashboard will be decoupled from your local Rill Developer environment, and you can further develop your dashboards locally without worrying about data availability. ::: When deploying a project, credentials that have been defined in your `.env` file will be automatically passed into your Rill Cloud project. However, for [remote sources](/developers/build/connectors) that are dynamically retrieving your credentials via the CLI, such as S3 and GCS, you will need to ensure that these are [defined in the .env file]( /guide/administration/project-settings/variables-and-credentials#credentials-naming-schema). [Local credentials](/developers/build/connectors/credentials#setting-credentials-for-rill-developer) are used by Rill Developer to connect to sources from your local machine, while deployment credentials are used by Rill Cloud for production workloads. There are a [few ways to set up credentials in Rill Developer](/developers/build/connectors/credentials/#setting-credentials-for-rill-developer); however, you will need to ensure that they are set up in your `.env` file for a seamless experience. If you have defined your connector's credentials in your `.env` file, these will be deployed along with your project. You should see the credentials in [your project's settings page]( /guide/administration/project-settings/variables-and-credentials#modifying-variables-and-credentials-via-the-settings-page). ![Environment Variable UI](/img/tutorials/admin/env-var-ui.png) ## Service Accounts :::info Separating development and production credentials As a general best practice, it is strongly recommended to use service accounts and dedicated service credentials for projects deployed to Rill Cloud, especially when used in a production capacity. This is covered in more detail in our [Development/Production Setup documentation](/developers/build/connectors/templating). ::: Service accounts are non-human user accounts that provide an identity for processes or services running on a server to interact with external resources, such as databases, APIs, and cloud services. Unlike personal user accounts, service accounts are intended for use by software applications or automated tools and do not require interactive login. In the context of Rill, service accounts are credentials that should be used for projects deployed to Rill Cloud. ### Why are service accounts important? Using service accounts for production workflows and pipelines is a general best practice for several reasons: 1. **Improved Security Posture**: Service accounts are specialized accounts used specifically by applications, as opposed to human users, to interact with data sources and other services. They help in implementing the principle of _least privilege_ by restricting permissions to only what is necessary for the application to function. This minimizes the potential damage when an account or set of credentials are compromised. 2. **Auditing and Monitoring**: Using service accounts makes it easier to audit and monitor access and activities. Since these accounts are used exclusively by applications (in this case Rill), any data access or actions performed can be traced back to the specific application, simplifying the process of identifying unusual or unauthorized activities. 3. **Credential Management**: Service accounts facilitate better management of credentials. For instance, when humans manage and share credentials, there's a higher risk of credentials being leaked or mishandled. Service accounts can be managed programmatically, reducing human error and improving security through automated rotations and stricter access controls. 4. **Scalability and Automation**: As organizations grow and deploy more applications, the use of service accounts allows for scalable and automated access management. It's easier to programmatically control access for multiple service accounts across different environments and services, fitting well into infrastructure as code (IaC) practices. Similarly, this ensures deployed projects (on Rill Cloud) don't share credentials with other applications. 5. **Compliance and Governance**: For regulatory compliance, using service accounts helps enforce data governance policies by ensuring that access is granted _according to the defined roles and responsibilities of the application_. It also aids in ensuring that data handling complies with policies and regulations since access patterns and permissions are clearly defined and can be audited. 6. **Reliability**: Applications using service accounts are less likely to experience downtime due to credential changes. Unlike user accounts, which may have passwords that expire or change frequently, service accounts can be configured with long-lived credentials or managed identity solutions that automatically handle authentication, reducing the risk of disruptions. Similarly, service accounts are not tied to the lifecycle of employee accounts, which help ensure that automated processes aren't potentially disrupted by personnel changes (such as offboarding an employee). ### Service Account Best Practices - _Secure Storage of Credentials_: Store service account credentials securely, using encrypted storage solutions and access controls to prevent unauthorized access. - _Regular Rotation of Credentials_: Regularly update service account passwords and keys to reduce the risk of compromise. - _Minimum Necessary Permissions_: Grant **only** the permissions necessary for the specific tasks the service account needs to perform, and review permissions regularly to adapt to changes in application functionality. - _Monitoring and Logging_: Depending on your organization's security needs, you can also consider implementing monitoring and logging of all access and actions taken by service accounts to detect and respond to anomalous activities promptly. :::warning Be careful of overriding local credentials and/or pushing the wrong credentials to Rill Cloud When using service accounts, it is very likely that different or even personal credentials are being used in local development (i.e., Rill Developer). Therefore, it is worth double-checking that the correct credentials are being used or set before [syncing credentials](/developers/build/connectors/credentials#pulling-credentials-and-variables-from-a-deployed-project-on-rill-cloud) between your local instance of [Rill Developer and Rill Cloud](/developers/deploy/cloud-vs-developer) using the `rill env push` and `rill env pull` commands respectively. ::: --- ##### Publish your Dashboards to Rill Cloud --- title: Publish your Dashboards to Rill Cloud sidebar_label: Publish sidebar_position: 0 --- Rill Developer is a great tool for building, testing, and viewing your data locally, but once you're ready to share your findings, you'll need to publish the dashboard to Rill Cloud! ### Concepts - [**Rill Cloud vs Rill Developer**](/developers/deploy/cloud-vs-developer) - Before you deploy, understand the difference between Rill Cloud and Rill Developer ### Credentials - [**Deployment Credentials**](/developers/deploy/deploy-credentials) - Confirm deployment credentials based on your deployment environment ### Deployment - [**Deploy your Dashboard**](/developers/deploy/deploy-dashboard) - Publish your dashboard to Rill Cloud ### Post-Deployment Steps - [**Managing Project Errors**](/developers/deploy/project-errors) - Troubleshooting errors in Rill Cloud --- ##### Managing Project Errors --- title: Managing Project Errors description: Configure alerts and manage errors for deployed projects in Rill Cloud sidebar_label: Managing Project Errors sidebar_position: 25 --- When you deploy to Rill Cloud, projects can encounter errors—from missing credentials to data type mismatches. This guide focuses on managing errors in deployed projects and setting up automated alerts. :::info General Troubleshooting For general troubleshooting guidance, error message explanations, and debugging techniques, see the [Debugging Rill Projects](/developers/build/debugging) documentation. ::: ## How Rill Handles Errors Rill's error management approach ensures visibility and isolation: - **Visibility:** View project status at the resource level via the `Status` tab or [`rill project status`](/reference/cli/project/status) CLI command - **Isolation:** Errors are contained to individual resource trees—if one dashboard fails, others remain available - **Fallback:** Rill attempts to serve from the most recent valid state when possible :::tip Check upstream dependencies The surfaced error might not be the root cause. A dashboard error could stem from an underlying model timeout. Always check the [project status page](/guide/administration/project-settings#checking-deployment-status) to trace errors to their source. ::: ## Deployment-Specific Error Scenarios Most errors will surface during local development in Rill Developer. However, after deploying to Rill Cloud, you may encounter additional issues: 1. **Production configuration missing** - Your YAML files reference `prod:` parameters that have been defined incorrectly. Verify your [dev/prod setup](/developers/build/connectors/templating). 2. **Timeouts, OOM** - Production data volumes may be larger than local development data, leading to timeouts and out-of-memory issues. [Contact us](/contact) if you see any related error messages. To troubleshoot deployment errors: 1. **Check the resource status** in the [project status page](/guide/administration/project-settings#checking-deployment-status) 2. **Review project logs** using `rill project logs` or the Rill Cloud UI 3. **Compare with local behavior** - If it worked locally, check production-specific configuration differences ## Setting Up Error Alerts You can configure alerts to automatically notify you when project errors occur. Once set up, you'll receive notifications (via email or Slack) whenever any resource in your project enters an error state. Besides alerting on project errors, it is possible to configure generic alerts in your dashboards based on specific thresholds or conditions being met. For more details, check out our [alerts documentation](/guide/alerts)! ### Configure an email alert To configure an email alert for project errors, add a file named `project_errors.yaml` to your Rill project with the contents below. Remember to update the `recipients` field to your desired alert recipients. ```yaml type: alert # Check the alert every 10 minutes. refresh: cron: "*/10 * * * *" # Query for all resources with a reconcile error. # The alert will trigger when the query result is not empty. data: resource_status: where_error: true # Send notifications by email notify: email: recipients: [john@example.com] ``` This will give you a good idea of what object has an issue, and you can browse the [status page](/guide/administration/project-settings#checking-deployment-status) for more information. After making these changes, you should commit and [push these changes](/developers/deploy/deploy-dashboard/github-101#pushing-changes) to your git repository. ### Configure a Slack alert To configure a Slack alert for project errors, first follow the Slack configuration steps described on [Configuring Slack integration](/developers/build/connectors/services/slack#setting-up-the-slack-integration). Next, add a file named `project_errors.yaml` to your Rill project with the contents below. Remember to update the `channels` field to your desired destination channel. ```yaml type: alert # Check the alert every 10 minutes. refresh: cron: "*/10 * * * *" # Query for all resources with a reconcile error. # The alert will trigger when the query result is not empty. data: resource_status: where_error: true # Send notifications in Slack. # Follow these steps to configure a Slack token: https://docs.rilldata.com/guide/alerts/alerts/slack. notify: slack: channels: [rill-alerts] ``` After making these changes, you should commit and [push these changes](/developers/deploy/deploy-dashboard/github-101#pushing-changes) to your git repository or update your Rill project via the Deploy button. --- ### Embed --- ##### Embed Dashboards in an Iframe --- id: iframe title: Embed Dashboards in an Iframe description: Embed Rill dashboards in your own applications using iframes sidebar_label: Iframe sidebar_position: 10 --- import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; ## Introduction Rill Cloud provides the ability to embed dashboards as components in your own application using iframes, with a few different options: - Embedding individual dashboards as standalone iframes - Embedding individual dashboards with the ability to navigate to other dashboards (that exist in the _same_ project) - Embedding the dashboard list page present in a Rill project (with the ability to select and navigate between dashboards) When embedding Rill, you need to generate a service token for your backend to request an authenticated iframe URL via the Rill API. Afterwards, the iframe URL can be passed to your frontend application for rendering. Here's a high-level diagram of what this flow looks like: ```mermaid sequenceDiagram participant A as ui.ezcommerce.com participant B as api.ezcommerce.com participant C as api.rilldata.com participant D as ui.rilldata.com/-/embed participant E as node.region.runtime.rilldata.com A ->> B: Get iframe URL B ->> B: Resolve the user's email
using ezcommerce's own auth B ->>+ C: Get iframe URL for:
project="ezcommerce"
user="john@example.com"
(uses Rill service token) Note right of C: 1. Lookup deployment
2. Generate JWT
3. Build iframe URL C ->>- B: iframe URL B ->> A: iframe URL A ->> D: Render iframe URL loop D ->> E: Request data
(gets host and JWT from
iframe URL query) E ->> D: Data with policies applied for
user="john@example.com" end ``` ## Create a service token Use the Rill CLI to create a service token for your current organization using the following command: ```bash # Create with organization role rill service create --org-role admin # Or create with project-specific role rill service create --project --project-role admin ``` :::info For comprehensive documentation on service tokens, including roles, custom attributes, and management, see [Service Tokens](/guide/administration/access-tokens/service-tokens). Also see the [CLI reference](/reference/cli/service) for command details. ::: :::note Why is the admin role required? Service tokens used for iframe embedding require the `admin` role to access the deployment management APIs that generate iframe URLs. Using the `viewer` role will result in a 403 permission error. ::: :::caution Service tokens can have broad permissions and should be handled confidentially. Therefore, the service token itself should **not** be integrated directly in frontend or other user-facing code that can be exposed publicly. ::: ## Backend: Build an iframe URL You should implement an API on your backend that uses the service token to retrieve and return an iframe URL from Rill's API (which is hosted on `api.rilldata.com`). There are multiple reasons why the iframe URL must be constructed on your backend: - To avoid leaking your master Rill service token in the browser - To allow you to use your own authentication and authorization logic to restrict access to the dashboard - To optionally use your backend's context about the authenticated user to include user attributes in the iframe URL for enforcement of row-level security policies Here are examples of how to get an iframe URL using different languages: ```bash curl -X POST --location 'https://api.rilldata.com/v1/orgs//projects//iframe' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ --data-raw '{ "type": "explore", "resource": "", "user_email": "" }' ``` JavaScript (Node.js) with Express.js ```js const express = require('express'); const fetch = require('node-fetch'); const rillServiceToken = ""; const rillOrg = ""; const rillProject = ""; const app = express(); app.use(express.json()); app.post('/api/rill/iframe', async (req, res) => { const dashboardName = req.body.resource; try { const response = await fetch(`https://api.rilldata.com/v1/orgs/${rillOrg}/projects/${rillProject}/iframe`, { method: 'POST', headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${rillServiceToken}`, }, body: JSON.stringify({ type: 'explore', resource: dashboardName, // Optionally include the end user's email address for security policies: // user_email: '', // Optionally set theme and theme mode: // theme: 'my-custom-theme', // theme_mode: 'dark', // Options: 'light', 'dark', 'system' }), }); const data = await response.json(); if (response.ok) { res.json(data); } else { throw new Error(data.message); } } catch (error) { res.status(500).json({ error: error.message }); } }); const PORT = 3000; app.listen(PORT, () => { console.log(`Server running on port ${PORT}`); }); ``` Python with Flask ```python from flask import Flask, request, jsonify import requests app = Flask(__name__) @app.route('/api/rill/iframe', methods=['POST']) def get_rill_iframe(): dashboard_name = request.json.get('resource') try: response = requests.post( 'https://api.rilldata.com/v1/orgs//projects//iframe', headers={ 'Content-Type': 'application/json', 'Authorization': 'Bearer ', }, json={ 'type': 'explore', 'resource': dashboard_name, # Optionally include the end user's email address for security policies: # 'user_email': '', # Optionally set theme and theme mode: # 'theme': 'my-custom-theme', # 'theme_mode': 'dark', # Options: 'light', 'dark', 'system' } ) response.raise_for_status() data = response.json() return jsonify(data) except requests.RequestException as e: return jsonify(error=str(e)), 500 if __name__ == '__main__': app.run(port=3000) ``` Go with net/http ```go package main import ( "bytes" "encoding/json" "fmt" "io/ioutil" "net/http" ) func getRillIframe(w http.ResponseWriter, r *http.Request) { var reqBody map[string]string json.NewDecoder(r.Body).Decode(&reqBody) dashboardName := reqBody["resource"] requestBody, err := json.Marshal(map[string]string{ "type": "explore", "resource": dashboardName, // Optionally include the end user's email address for security policies: // "user_email": "", }) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } resp, err := http.Post("https://api.rilldata.com/v1/orgs//projects//iframe", "application/json", bytes.NewBuffer(requestBody)) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } defer resp.Body.Close() body, err := ioutil.ReadAll(resp.Body) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } var respBody map[string]any err = json.Unmarshal(body, &respBody) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } w.Header().Set("Content-Type", "application/json") err = json.NewEncoder(w).Encode(respBody) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } } func main() { http.HandleFunc("api/rill/iframe", getRillIframe) fmt.Println("Server started at port 3000") http.ListenAndServe(":3000", nil) } ``` Java With Spring Boot ```java import org.springframework.web.bind.annotation.PostMapping; import org.springframework.web.bind.annotation.RequestBody; import org.springframework.web.bind.annotation.RestController; import org.springframework.web.client.RestTemplate; import org.springframework.http.ResponseEntity; import org.springframework.http.HttpEntity; import org.springframework.http.HttpHeaders; import java.util.HashMap; import java.util.Map; @RestController public class DashboardController { @PostMapping("/api/rill/iframe") public ResponseEntity getRillIframe(@RequestBody Map payload) { String dashboardName = (String) payload.get("resource"); HttpHeaders headers = new HttpHeaders(); headers.set("Content-Type", "application/json"); headers.set("Authorization", "Bearer "); Map request = new HashMap<>(); request.put("type", "explore"); request.put("resource", dashboardName); // Optionally include the end user's email address for security policies: // request.put("user_email", ""); HttpEntity> entity = new HttpEntity<>(request, headers); RestTemplate restTemplate = new RestTemplate(); ResponseEntity response = restTemplate.postForEntity( "https://api.rilldata.com/v1/orgs//projects//iframe", entity, Map.class ); return ResponseEntity.ok(response.getBody()); } } ``` The API accepts the following parameters: | Parameter | Description | Required | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | | resource | The name of the dashboard to embed | No (if not specified, `navigation` should be set to `true`) | | type | The type of the dashboard identified by `resource` (options: `explore`, `canvas`) | No (defaults to `explore`) | | navigation | Boolean whether to enable navigation and allow users to navigate to other dashboards (`false` will hard embed and allow access to a single dashboard; `true` allows navigation) | No (defaults to `false`) | | theme | If [themes](/developers/build/dashboards/customization#changing-themes--colors) are being used, the specific theme to pass to the embedded dashboard | No (set to the name of the theme) | | theme_mode | The theme mode to use for the embedded dashboard. Valid values: `light`, `dark`, `system`. When set to `system`, the dashboard will follow the user's system preference. | No (defaults to `light`) | | user_email | Email of the end user accessing the embed. Used to populate attributes for use in security policies. | No (only one of `user_email` or `attributes` should be passed in) | | attributes | Raw attributes about the end user. Used to populate attributes for use in security policies. | No (only one of `user_email` or `attributes` should be passed in) | | external_user_id | A stable identifier for an external end user. Enables per-user state. | No | | ttl_seconds | The time to live for the iframe URL | No (Default: 86400) | The response of the POST request will then contain an `iframeSrc` value that can be used to embed the dashboard or set of dashboards in your application. It will also contain a `ttlSeconds` value, which indicates how long the iframe URL will be valid for. _After the TTL has elapsed_, the iframe URL needs to be refreshed as the underlying access token being used will no longer be valid (for security purposes). Here's an example response: ```json { "iframeSrc": "https://ui.rilldata.com/-/embed?access_token=&instance_id=&type=&resource=&runtime_host=", "runtimeHost": "", "instanceId": "", "accessToken": "", "ttlSeconds": 86400 } ``` ## User identity in embeds The `user_email`, `attributes` and `external_user_id` parameters serve two purposes in the embeds: - **User attributes for security policies:** `user_email` and `attributes` determine which attributes (such as `email`, `domain`, and `admin`) are available for use in [security policies](/developers/build/metrics-view/security). - **Per-user state:** `external_user_id` establishes a stable user identity that isolates per-user features such as AI chat history. Without a user identity, these features are not available. Only one of `user_email` or `attributes` can be provided for a given iframe. The `external_user_id` parameter can optionally be combined with either of them. Here is how each parameter works: - `user_email`: Looks up the user in Rill Cloud by email and populates their standard attributes. If no matching user is found, it generates limited attributes with only the fields `email`, `domain` (the part of the email after `@`) and `admin` (set to `false`). The derived `domain` is commonly used in security policies (e.g. `app_site_domain = '{{ .user.domain }}'`). Does not enable per-user state on its own; combine with `external_user_id` to enable per-user state. - `attributes`: Passes the provided attributes through directly. Make sure to include all attributes referenced in your security policies (e.g. `email`, `domain`, `admin`, or custom attributes like `tenant_id`). Does not enable per-user state on its own; combine with `external_user_id` to enable per-user state. - `external_user_id`: Any stable identifier for the end user. This is usually the user's ID in your own database. Setting it enables per-user state such as AI chat history. ## Embedding the project vs embedding an individual dashboard One of the most common differences between how developers may wish to iframe Rill is whether they wish to embed at the project level or individual dashboard level. This behavior can be controlled through the combination of the `resource` and `navigation` properties! If you wish to embed a single dashboard **only**, your payload might look like: ```json { "type": "explore", "resource": "dashboardName" } ``` If you wish to still embed a dashboard _but allow navigation between dashboards_, then your payload should include both parameters: ```json { "type": "explore", "resource": "dashboardName", "navigation": true } ``` Finally, _if you wish to embed the project list view of dashboards instead (what you see when you first open a project in Rill Cloud)_, then you can simply omit the `resource` and appropriately set `navigation` in your payload: ```json { "navigation": true } ``` ## Testing the dashboard While it is possible to create the iframeSrc URL via the CLI or code to _test_ your embedded dashboard, it might be easier to start off using [Rill Developer's mock users](/developers/build/metrics-view/security#advanced-example-custom-attributes-embed-dashboards), especially if you have multiple attribute views that you want to test before deploying to Rill Cloud. You can pass specific custom_attributes as you would during iframe URL generation to view the pre-filtered explore dashboard. ```yaml - email: embed@rilldata.com name: embed custom_variable_1: Value_1 custom_variable_2: Value_2 ``` ## Frontend: Embed the dashboard Your frontend should request an iframe URL from your backend API (which you set up in the previous step) and use the `iframeSrc` value of the response to render an HTML ` ``` Set up message handling and send requests from the parent window: ```js const iframe = document.getElementById("my-iframe"); window.addEventListener("message", (event) => { const { id, result, error, method, params } = event.data; // notifications if (method === "ready") { console.log("Iframe is ready"); } if (method === "stateChanged") { console.log("State changed to:", params.state); } if (method === "navigation") { console.log("Navigated from:", params.from, "to:", params.to); // params.from and params.to will be dashboard names or "dashboardListing" } if (method === "resized") { console.log("Iframe resized to:", params.width, "x", params.height); } // responses if (id && result) { console.log("Response to request:", result); } if (id && error) { console.error("RPC error:", error); } }); ``` ## Supported Methods These methods are called **from the parent** and handled **by the iframe**. Note: if including an `id`, the iframe will respond with a matching `id`. If you do not need a response, you can omit the `id` property. ### `setState(state)` Updates the iframe's dashboard's UI state. ```js iframe.contentWindow.postMessage({ id: 1, method: "setState", params: "view=pivot&tr=PT24H&grain=hour", }, "*"); ``` **Parameters:** - `state` (string): A URL query string describing the dashboard view, filters, time range, etc. Uses the same format as the query strings in URLs on Rill Cloud. **Response:** ```json { "id": 1, "result": true } ``` ### `setValidState({ state, failOnError })` Like `setState`, but validates the state against the dashboard's metrics view and explore specs before applying it. Invalid parameters (for example, an unknown dimension or measure) are reported back, and the applied URL is canonicalized (defaults removed, parameters ordered) the same way it would be if the user navigated there directly. The applied state is returned in `appliedState`. Validation is currently supported for explore dashboards; for other dashboard types the state is applied as-is. ```js iframe.contentWindow.postMessage({ id: 1, method: "setValidState", params: { state: "view=pivot&tr=PT24H&grain=hour", failOnError: true }, }, "*"); ``` **Parameters:** - `state` (string): A URL query string describing the dashboard view, filters, time range, etc. Uses the same format as the query strings in URLs on Rill Cloud. - `failOnError` (boolean, optional): When `false` (the default), the cleaned state is applied even when some parameters were invalid; the dropped parameters are still reported in `errors`. When `true`, the state is applied only if there are no validation errors. **Response:** ```json { "id": 1, "result": { "success": true, "appliedState": "view=pivot&tr=PT24H&grain=hour", "errors": [] } } ``` `success` is `false` when `failOnError` is `true` and validation produced errors, in which case the state is not applied and `appliedState` is omitted. `appliedState` is the canonicalized query string that was actually applied to the dashboard. `errors` contains a message for each invalid parameter. ### `getState()` Returns the iframe's dashboard's current UI state. ```js iframe.contentWindow.postMessage({ id: 2, method: "getState" }, "*"); ``` **Response:** ```json { "id": 2, "result": { "state": "view=pivot&tr=PT24H&grain=hour" } } ``` ### `getThemeMode()` Fetches the current theme mode of the iframe. ```js iframe.contentWindow.postMessage({ id: 3, method: "getThemeMode" }, "*"); ``` **Response:** ```json { "id": 3, "result": {"themeMode": "light"} } ``` The `themeMode` value will be one of: `"light"`, `"dark"`, or `"system"`. ### `setThemeMode(themeMode)` Sets the theme mode inside the iframe. ```js iframe.contentWindow.postMessage({ id: 4, method: "setThemeMode", params: "dark" }, "*"); ``` **Parameters:** - `themeMode` (string): The theme mode to set. Must be one of: `"light"`, `"dark"`, or `"system"`. **Response:** ```json { "id": 4, "result": true } ``` **Error Response (if invalid themeMode):** ```json { "id": 4, "error": { "code": -32603, "message": "Expected themeMode to be one of \"dark\", \"light\", or \"system\"" } } ``` ### `getTheme()` Fetches the current theme name applied to the dashboard. ```js iframe.contentWindow.postMessage({ id: 5, method: "getTheme" }, "*"); ``` **Response:** ```json { "id": 5, "result": {"theme": "my-custom-theme"} } ``` If no theme is set, the response will be: ```json { "id": 5, "result": {"theme": "default"} } ``` The `theme` value will be the name of the theme resource, or `"default"` if no theme is set. ### `setTheme(theme)` Sets the theme name to apply to the dashboard. ```js iframe.contentWindow.postMessage({ id: 6, method: "setTheme", params: "my-custom-theme" }, "*"); ``` **Parameters:** - `theme` (string | null): The theme name to set. Must be the name of an existing theme resource. To clear the theme and use the default, pass `null` or `"default"`. **Response:** ```json { "id": 6, "result": true } ``` **Error Response (if invalid theme):** ```json { "id": 6, "error": { "code": -32603, "message": "Expected theme to be a string or null" } } ``` :::caution The theme name must correspond to an existing theme resource in your Rill project. Setting an invalid theme name will **not** return an error, but the theme will silently not be applied. ::: ### `getAiPane()` Fetches the current visibility state of the AI chat pane. ```js iframe.contentWindow.postMessage({ id: 7, method: "getAiPane" }, "*"); ``` **Response:** ```json { "id": 7, "result": {"open": false} } ``` The `open` value will be `true` if the AI pane is currently visible, or `false` if it is hidden. **Note:** The AI pane is only available for Explore dashboards when the chat feature is enabled. If the AI pane is not available, the method will still return the current state (which will typically be `false`). ### `setAiPane(open)` Sets the visibility state of the AI chat pane. ```js iframe.contentWindow.postMessage({ id: 8, method: "setAiPane", params: true }, "*"); ``` **Parameters:** - `open` (boolean): Whether to show (`true`) or hide (`false`) the AI chat pane. **Response:** ```json { "id": 8, "result": true } ``` **Error Response (if invalid parameter):** ```json { "id": 8, "error": { "code": -32603, "message": "Expected open to be a boolean" } } ``` **Note:** The AI pane is only available for dashboards when the chat feature is enabled. If the AI pane is not available, calling this method will not cause an error, but the pane will not be shown. ## Notifications Notifications are sent **from the iframe** to the parent window. These do not include an `id`. ### `ready()` Fired once when the iframe is initialized and ready to receive messages. ```json { "method": "ready" } ``` ### `stateChanged({ state: string })` Fired whenever the internal state of the iframe changes. ```json { "method": "stateChanged", "params": { "state": "" } } ``` ### `navigation({ from: string, to: string })` Fired whenever a user navigates between dashboards. This event is only emitted when navigation is enabled in the embed configuration. - `from`: The name of the dashboard the user navigated from, or `"dashboardListing"` if navigating from the dashboard listing page - `to`: The name of the dashboard the user navigated to, or `"dashboardListing"` if navigating to the dashboard listing page This event fires for all dashboard navigation scenarios: - Navigating from one explore dashboard to another - Navigating from one canvas dashboard to another - Navigating from an explore dashboard to a canvas dashboard (or vice versa) - Navigating from the dashboard listing page to any dashboard - Navigating from any dashboard to the dashboard listing page ```json { "method": "navigation", "params": { "from": "dashboard-name", "to": "another-dashboard-name" } } ``` Example when navigating from the listing page: ```json { "method": "navigation", "params": { "from": "dashboardListing", "to": "dashboard-name" } } ``` Example when navigating to the listing page: ```json { "method": "navigation", "params": { "from": "dashboard-name", "to": "dashboardListing" } } ``` ### `resized({ width: number, height: number })` Fired whenever the iframe content is resized. ```json { "method": "resized", "params": { "width": 1200, "height": 800 } } ``` ### `aiPaneChanged({ open: boolean })` Fired whenever the AI pane visibility state changes (opened or closed). - `open`: `true` if the AI pane is now open, `false` if it is now closed This event fires when: - The AI pane is opened or closed by the user clicking the AI button - The AI pane state is changed programmatically via the `setAiPane()` method - The AI pane state changes for any other reason ```json { "method": "aiPaneChanged", "params": { "open": true } } ``` Example when the AI pane is closed: ```json { "method": "aiPaneChanged", "params": { "open": false } } ``` **Note:** This notification is only emitted for Explore dashboards when the chat feature is enabled. ## Error Handling All errors follow the JSON-RPC 2.0 structure: ```json { "id": 3, "error": { "code": -32601, "message": "Method not found" } } ``` **Common Error Codes:** | Code | Message | Description | | ------ | ---------------- | -------------------- | | -32600 | Invalid Request | Malformed request | | -32601 | Method Not Found | Unknown method | | -32602 | Invalid Params | Parameters incorrect | | -32603 | Internal Error | Unexpected failure | | -32700 | Parse Error | Malformed JSON | --- ## Full Example ```js const iframe = document.getElementById("my-iframe"); function sendRequest(method, params) { const id = Math.random().toString(36).slice(2, 11); return new Promise((resolve, reject) => { function handler(event) { if (event.data?.id === id) { window.removeEventListener("message", handler); if (event.data.result !== undefined) resolve(event.data.result); else reject(event.data.error); } } window.addEventListener("message", handler); iframe.contentWindow.postMessage({ id, method, params }, "*"); }); } window.addEventListener("message", async (event) => { if (event.data?.method === "ready") { console.log("Iframe ready"); await sendRequest("setState", "view=pivot&tr=PT24H&grain=hour"); const currentState = await sendRequest("getState"); console.log("Current state:", currentState.state); const currentThemeMode = await sendRequest("getThemeMode"); console.log("Current theme mode:", currentThemeMode.themeMode); await sendRequest("setThemeMode", "dark"); const currentTheme = await sendRequest("getTheme"); console.log("Current theme:", currentTheme.theme); await sendRequest("setTheme", "my-custom-theme"); const aiPaneState = await sendRequest("getAiPane"); console.log("AI pane open:", aiPaneState.open); await sendRequest("setAiPane", true); } if (event.data?.method === "stateChanged") { console.log("State changed:", event.data.params.state); } if (event.data?.method === "navigation") { console.log("Navigated from:", event.data.params.from, "to:", event.data.params.to); } if (event.data?.method === "resized") { console.log("Iframe resized:", event.data.params.width, "x", event.data.params.height); } if (event.data?.method === "aiPaneChanged") { console.log("AI pane changed:", event.data.params.open ? "opened" : "closed"); } }); ``` --- ### Get Started --- ##### "Building Rill Projects with AI" --- title: "Building Rill Projects with AI" sidebar_label: "Agentic Quickstart" sidebar_position: 5 --- Rill projects are defined as YAML and SQL files, which makes them a natural fit for AI coding agents. This guide walks through using an AI agent like Claude Code or Cursor to build a Rill project from scratch. ## Prerequisites - [Rill CLI installed](/developers/get-started/install) - An AI coding agent: [Claude Code](https://docs.anthropic.com/en/docs/claude-code/overview), [Cursor](https://www.cursor.com/), or another MCP-compatible tool ## Step 1: Initialize a project with agent instructions Run `rill init` to create a new project. The interactive setup will prompt you for a project name, OLAP engine, and agent instructions: ```bash rill init ? Project name my-rill-project ? OLAP engine clickhouse ? Agent instructions claude ``` This creates a project directory with: - `rill.yaml` — project configuration - `.claude/CLAUDE.md` — entry point that tells Claude Code to load Rill development skills - `.claude/skills/` — detailed instructions for each resource type (models, metrics views, dashboards, etc.) - `.mcp.json` — connects Claude Code to Rill's local [MCP server](/guide/ai/mcp) :::tip Using a different AI agent? Select a different option in the "Agent instructions" prompt, or use the `--agent` flag: ```bash # Cursor rules rill init my-project --agent cursor # Tool-agnostic AGENTS.md format rill init my-project --agent agentsmd # All formats at once rill init my-project --agent all ``` ::: ### Adding agent instructions to an existing project If you already have a Rill project, you can add agent instructions without affecting your existing files: ```bash rill init ./my-existing-project --agent claude ``` ## Step 2: Start Rill in preview mode Launch Rill Developer in [preview mode](/developers/build/getting-started#preview-mode) to get a clean, dashboard-only view while your AI agent handles the code: ```bash rill start my-project --preview ``` This also starts a local MCP server at `http://localhost:9009/mcp`. If you generated agent instructions in Step 1, your AI agent will connect to this server automatically via the `.mcp.json` config — no additional setup required. The MCP server gives your AI agent access to: - **Project status** — see which resources are healthy, errored, or pending - **Table schemas** — inspect columns, types, and sample data - **SQL queries** — run analytical queries against your OLAP engine - **File operations** — read and write project files ## Step 3: Build with your AI agent With Rill running, open your AI agent in the project directory and start building. Here are some examples of what you can ask: ### Connect a data source > "Connect to the parquet file at `gs://rilldata-public/auction_data.parquet`" The agent will create a source YAML file and Rill will automatically ingest the data. ### Create models > "Create a model that cleans the auction data — filter out null bids and add a `bid_bucket` column that groups bids into $0-1, $1-5, $5-10, and $10+ ranges" ### Define metrics > "Create a metrics view on the auction model with measures for total bids, average bid price, and win rate, broken down by dimensions like domain, device type, and bid bucket" ### Build dashboards > "Create an explore dashboard for the auction metrics view" > "Create a canvas dashboard with KPI cards for total bids and win rate, a time series chart, and a breakdown table by domain" ### Iterate The agent has full context on Rill's resource types and YAML schemas. It can fix errors, refactor models, add new measures, and restructure your project — just describe what you want. :::tip Check project status If something isn't working, ask your agent to check the project status. The MCP connection lets it see parse errors, reconciliation failures, and resource health directly. ::: ## Next steps - **[Deploy to Rill Cloud](/developers/deploy/deploy-dashboard)** — share your dashboards with your team - **[AI Chat](/guide/ai/ai-chat)** — ask questions about your data in natural language from Rill Cloud - **[AI Configuration](/developers/build/ai-configuration)** — add `ai_instructions` to improve AI responses for your project - **[Rill MCP Server](/guide/ai/mcp)** — connect Claude Desktop, ChatGPT, or other AI clients to Rill Cloud projects --- ##### Get Started with Rill --- title: Get Started with Rill slug: / sidebar_label: Home sidebar_position: 00 --- import FeatureList from '@site/src/components/FeatureList'; Rill Developer users encompass the data engineers, analytics engineers, BI developers, and platform teams — all building data pipelines, defining metrics, and delivering interactive dashboards with Rill. In addition to Rill Developer for local project development, these docs cover deploying to Rill Cloud, embedding dashboards into your own applications, and integrating with external tools and APIs. Looking to **explore dashboards and data** that your team has already set up? Head over to the [User Guide](/guide). ## Install Install Rill using the command below, then launch your first project to explore core features hands-on. For more installation methods, see our [complete installation instructions](/developers/get-started/install). ```bash curl https://rill.sh | sh rill start my-rill-project ``` New to Rill? Follow the [Quickstart](/developers/get-started/quickstart) to build your first dashboard with a public dataset, the [Agentic Quickstart](/developers/get-started/ai-quickstart) to build a project with an AI coding agent, or browse our [tutorials](/developers/tutorials/). Use [Preview Mode](/developers/build/getting-started#preview-mode) (`rill start --preview`) for a dashboard-only view with no code editor — ideal for demos, stakeholder reviews, or pairing with an AI agent that handles the code for you. ## Start Developing Rill Developer is a local application that makes it easy to build end-to-end analytics pipelines. Connect to data sources, write SQL or YAML models for last-mile transformations, define a metrics layer with measures and dimensions, and preview interactive dashboards — all from your local machine. See the full [Build](/developers/build) docs for details. Looking for hands-on examples? Browse our [tutorials and example projects](/developers/tutorials/) for guided walkthroughs you can clone and explore. ## Deploy to Rill Cloud Rill Developer is great for building and testing locally, but once you're ready to share your work, deploy your project to Rill Cloud. Rill Cloud is a fully managed service where your team can explore dashboards, ask questions with AI Chat, set up alerts, and schedule reports — no local setup required for consumers. ## Integrate into your Application Rill is designed to fit into your existing stack. Embed interactive dashboards directly into your web applications using iframes, programmatically control them with the Embed API, or build custom API endpoints to pull Rill data into external tools and workflows. ## Join the Community Rill is open source — browse the code on [GitHub](https://github.com/rilldata/rill) or join the community on [Discord](https://discord.gg/2ubRfjC7Rh). If you're a customer, reach out directly via your dedicated Slack channel or contact support. --- ##### How to Install Rill Developer --- title: How to Install Rill Developer sidebar_label: Install Rill Developer sidebar_position: 02 --- ## Quick Install You can install `rill` using our installation script on both macOS and Linux: ```bash curl https://rill.sh | sh ``` :::tip Why am I installing locally? Installing Rill Developer locally is required to create a new project, or [clone a project](/developers/tutorials/clone-a-project) from Rill Cloud. For key differences between Rill Cloud and Rill Developer, see [Rill Cloud vs. Rill Developer](/developers/deploy/cloud-vs-developer). ::: To verify that the installation succeeded, run the following command. You will see a list of various commands that you can run via the CLI. ```bash rill --help ``` :::warning macOS users If you previously installed Rill using `brew`, *the brew-managed binary will take precedence*. You can remove it by running `brew uninstall rill`. ::: ### Checking Rill Version You can check the current version of rill from the CLI by running the following command: ```bash rill version ``` ### Upgrade to the newest version of Rill Developer To ensure you're on the latest version of Rill and enjoying all of our newest features, you can upgrade Rill Developer easily via the command line: ```bash rill upgrade ``` :::info What about Rill Cloud? Rill Cloud is always on the latest stable version of Rill. To check the latest version available, please see our [Releases](https://github.com/rilldata/rill/releases) page. ::: ## Nightly Releases On both macOS and Linux, you can install the latest nightly build using the installation script: ```bash curl https://rill.sh | sh -s -- --nightly ``` ### What is a nightly release? The nightly release will give you the most up-to-date version of Rill without having to wait for the official release. As these releases are not fully ready for production, you may encounter some issues. Also, version mismatches between Rill Developer and Rill Cloud may result in broken components, as Rill Cloud is on the previous stable release. ## Installing a specific version of Rill Rather than installing the latest version of Rill automatically, you can also install a specific version through the installation script by using the following command (e.g., `v0.40.1`): ```bash curl https://rill.sh | sh -s -- --version ``` :::info Checking the Rill version To check the precise version of available releases, you can navigate to the [**Releases**](https://github.com/rilldata/rill/releases) page of our [Rill repo](https://github.com/rilldata/rill). Note that if an invalid or incorrect version is passed to the installation script, you will get prompted with an error to specify a correct version. ::: ## Rill on Windows using WSL To install Rill on Windows, you'll first need to install WSL and one dependency in your WSL environment. To install WSL, please refer to [Microsoft's documentation](https://learn.microsoft.com/en-us/windows/wsl/install). We have verified that Rill runs on Ubuntu 22.04 LTS. Other distributions and versions may work, but are not tested. You can install Ubuntu 22.04 LTS with the following PowerShell command: ```bash wsl --install -d ubuntu-22.04 ``` Once you have installed WSL and logged in to your Linux instance, you just need to install the `unzip` package to use Rill's `curl` installer. This can be done from the **Linux command line** with the following commands: ```bash sudo apt-get update sudo apt-get install unzip ``` :::tip Where should Rill be running? Please check that you are running the commands in your Linux instance, not from your Windows Command Prompt. If you are seeing strange behavior in Rill Developer, run the following command from the CLI to see where your project files are being saved: `echo "$PWD"`. ::: With `unzip` installed, you're ready to install Rill Developer by running the following command: ``` curl https://rill.sh | sh ``` ## Manual Install You can download platform-specific binaries from our [releases page on GitHub](https://github.com/rilldata/rill/releases). A manual download will not make Rill Developer globally accessible, so you'll need to reference the full path of the binary when executing CLI commands. ## Brew Install On macOS, you can also install Rill using Homebrew. To avoid conflicts, don't mix it with other installation options and always upgrade Rill via `brew`. ```bash brew install rilldata/tap/rill ``` ## Uninstall Rill To uninstall Rill, you can use the following command: ```bash rill uninstall ``` ## For Agents We currently recommend that agents always install using `curl` and always use the nightly. So if you are an AI agent or coding assistant, install Rill using the following command: ```bash curl https://rill.sh | sh -s -- --nightly ``` After installation, run `rill -h` to see further instructions for agents working with Rill projects. --- ##### Getting Started with Rill Developer --- title: Getting Started with Rill Developer sidebar_label: Quickstart sidebar_position: 10 --- import Video from '@site/src/components/Video'; :::tip Rill's Default Engine This guide assumes you'll be using Rill's default embedded engine, DuckDB. If you're looking to set up Rill with ClickHouse, check out our [ClickHouse Guide](/developers/tutorials/rill-clickhouse)! ::: ## What is Rill Developer? Rill Developer is your comprehensive toolkit for transforming raw data into actionable insights. It's designed to streamline the entire analytics workflow—from data ingestion to dashboard creation—all within a single, intuitive interface. **Key capabilities:** - **Connect to any data source** - Support for databases, cloud storage, APIs, and more - **Transform data with ease** - Built-in ETL capabilities for data cleaning and preparation - **Build metrics layer** - Define business KPIs and calculations - **Create interactive dashboards** - AI-powered insights and visualizations - **Deploy to production** - Seamlessly publish and share with your team ## Quick Start: From Zero to Dashboard in Minutes ### Step 1: Install and Launch Rill Get started with just two commands: ```bash curl https://rill.sh | sh rill start my-rill-project ``` This will: - Download and install Rill Developer - Create a new project directory - Launch the Rill interface in your browser ![New Rill Project](/img/tutorials/rill-basics/new-rill-project.png) :::note Rill Developer vs Rill Cloud Rill Developer is your local development environment where you build and test your analytics projects. Once ready, you can deploy to Rill Cloud for team collaboration and production use. For more details, see our [Developer vs Cloud comparison](/developers/deploy/cloud-vs-developer). ::: ### Step 2: Connect Your Data Rill supports a wide range of data sources through our [connector library](/developers/build/connectors). For this tutorial, we'll use a sample dataset hosted on Google Cloud Storage. Select Add Data in the dropdown, GCS, and use the following dataset: ``` gs://rilldata-public/auction_data.parquet ``` **What happens when you connect data:** - Rill automatically detects your data schema - Provides a preview of the first 150 rows - Analyzes data types, value ranges, and patterns - Creates a foundation for your analytics
### Step 3: Create Your First Dashboard Rill's AI-powered dashboard creation makes it easy to go from data to insights in seconds. The system automatically generates relevant visualizations and suggests key metrics based on your data. Select the model's side menu and select "Generate dashboard with AI". Watch as Rill configures your metrics view and creates an Explore dashboard. Once finished, you can navigate the different features of our [Explore dashboard](/guide/dashboards).
### Step 4: Explore and Analyze Once your dashboard is created, you can: - **Navigate different views** - Pivot tables, time-series analysis, and multi-measure charts - **Drill down into data** - Click on any element to explore deeper insights - **Customize visualizations** - Adjust charts, filters, and layouts - **Export insights** - Share findings with your team ## Preview Mode [Preview Mode](/developers/build/getting-started#preview-mode) gives you a dashboard-only view of your project — no file editor, no code. Start it with `rill start my-rill-project --preview`. ## Build with AI Rill projects are YAML and SQL files, which makes them a natural fit for AI coding agents like Claude Code and Cursor. See the [Agentic Quickstart](/developers/get-started/ai-quickstart) to get started. ## What else can Rill do? This quick start covered the basics, but Rill offers much more: - **[Data Modeling](/developers/build/models)** - Transform and prepare your data with SQL - **[Metrics Layer](/developers/build/metrics-view)** - Define business KPIs and calculations - **[Deployment](/developers/deploy/deploy-dashboard)** - Share your dashboards with your team Ready to build something more complex? Check out our [comprehensive tutorial](/developers/tutorials/rill-basics/launch) for a complete walkthrough of Rill's advanced features. --- ##### Why Rill? --- title: Why Rill? sidebar_label: Why Rill? sidebar_position: 15 --- Data teams often struggle with a fragmented stack: while data lakes and warehouses are cost-effective for large volumes, they often lack the sub-second performance required for interactive, operational analytics. Engineering teams spend too much time managing ETL pipelines and answering ad-hoc requests, while business users are stuck with slow, static dashboards that don't allow for true exploration. Rill bridges this gap by bringing fast, interactive analytics directly to your data lake or warehouse, with a developer-friendly workflow. ## What Makes Rill Different? - **Fast, interactive dashboards on large datasets**: Rill powers sub-second queries and instant interactions, even on hundreds of millions of rows, by optimizing aggregation and pruning. - **BI-as-Code**: Manage your entire analytics stack with code (SQL and YAML), bringing version control, CI/CD, and collaboration to your dashboards. - **Works directly with your data lake**: [Connects directly](/developers/build/connectors) to your cloud data warehouse (BigQuery, Snowflake), lakehouse, or object storage (S3, GCS, Azure) without complex ETL. - **Built-in OLAP**: An [integrated, in-memory OLAP engine](/developers/build/connectors/olap) handles data orchestration and query execution automatically for blazing speed. ## How It Works ![architecture](/img/concepts/architecture/architecture.svg) Rill automatically turns your SQL data models into interactive, exploratory dashboards. It ingests data from your external sources into an embedded, high-performance OLAP engine. By defining metrics and dimensions upfront, Rill aggregates and optimizes the data, creating a responsive interface that allows users to slice and dice billions of records instantly without waiting for query processing. This approach means engineering teams can: 1. **Orchestrate data** out of cloud [data warehouses](/developers/build/connectors/data-source#data-warehouses) or [object stores](/developers/build/connectors#object-storage) into the fast serving layer. 2. **Define metrics & dimensions** in [metrics view](/developers/build/metrics-view) to automatically aggregate and prune raw tables. 3. **Deploy** your [project to Rill Cloud](/developers/deploy/deploy-dashboard) without managing separate BI servers or database infrastructure. ## Built for Operational Analytics Operational intelligence provides decision-making capabilities that are complementary to traditional Business Intelligence (BI). * **Business Intelligence** drives complex decisions that occur daily or weekly, often on historical data (e.g., "How did we perform last quarter?"). * **Operational Intelligence** fuels fast, frequent decisions on real-time and near-time data by hands-on operators (e.g., "Why is site latency spiking right now?" or "Which campaign is driving traffic this hour?"). Rill is built for this "fast" mode. It empowers product managers, operations teams, and engineers to explore data at the speed of thought, identifying trends and anomalies as they happen, without waiting for pre-computed reports. ## AI-Native Analytics Rill's code-first architecture — YAML and SQL files that define your entire analytics stack — makes it uniquely suited for AI-powered workflows. ### Build with AI agents Because Rill projects are just files, AI coding agents like Claude Code and Cursor can build and iterate on your entire project: connect data sources, write transformation models, define metrics views, and create dashboards. Run `rill init` to generate agent instructions and an MCP connection out of the box. See the [Agentic Quickstart](/developers/get-started/ai-quickstart) to get started. ### Chat with your data [AI Chat](/guide/ai/ai-chat) is built into Rill Cloud, letting you ask questions about your data in natural language. Responses are grounded in your predefined measures and dimensions, so answers are accurate and governed — not hallucinated from raw tables. Every response includes links back to your Explore dashboards so you can verify the numbers. ### Connect external AI tools The [Rill MCP Server](/guide/ai/mcp) connects your projects to Claude Desktop, ChatGPT, and other MCP-compatible AI assistants. Your team gets the same governed analytics in whatever AI tool they prefer, with security and access controls enforced by Rill. ### Improve AI with instructions Add [`ai_instructions`](/developers/build/ai-configuration) to your project and metrics view YAML files to give AI agents context about your business logic, terminology, and data quirks — improving the quality of responses across all AI surfaces. ## Key Benefits * **Performance**: Rill achieves performance that end users love by serving dashboards from in-memory data models, often reducing the data footprint by 10–100x compared to raw events. * **Developer Experience**: Define your entire analytics stack—from data models to dashboards—using code. This brings the benefits of version control, collaboration, and automation that software development teams rely on. * **Cost Effective**: Build analytics directly on your storage. Rill lowers costs because data can be transformed in place without duplication in a heavy enterprise warehouse. * **AI-Native**: Build projects with AI coding agents, chat with your data in natural language, and connect external AI tools — all governed by your predefined metrics. --- ### Other --- #### Granting --- ###### "Managed AWS S3 Bucket" --- title: "Managed AWS S3 Bucket" description: Setting up S3 access sidebar_label: "AWS S3 Bucket" sidebar_position: 10 --- :::warning Please note that these instructions were made specifically for Rill Managed Pipelines and used by our data engineering team to set up orchestration from object stores/data warehouses into Rill Managed Database Services. While some of the concepts may apply, please refer to your provider's documentation on the correct permissions required to connect to your service. ::: ## Setup instructions ### Set up S3 bucket * Create a new bucket, in the US Standard S3 region, named `rill-ORG_NAME-share`, where **"ORG_NAME"** is your organizational name. Make sure the bucket name is a DNS-compliant address. For example, the name should not include any underscores; use hyphens instead, as shown in the example above. * Be sure to disable the "Requester Pays" feature. For more information on S3 bucket name guidelines, see [the AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/RequesterPaysBuckets.html). ### Method 1: Bucket ACL Policies We can provide access to the desired buckets using the bucket policies. You can provide access to the root user for Rill Data AWS Account. :::info Rill Data AWS Account arn:aws:iam::248432388601:root ::: #### Using AWS Console 1. Open S3 bucket console: https://s3.console.aws.amazon.com/ 1. Choose the S3 bucket to be shared. 1. Select Permissions -> Bucket Policy 1. Add the following bucket policy for the access: (Replace rill-ORG_NAME-share or bucket-to-be-shared with the bucket name) ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::248432388601:root" }, "Action": [ "s3:GetObject", "s3:PutObject", "s3:PutObjectAcl" ], "Resource": [ "arn:aws:s3:::rill-ORG_NAME-share/*" ] }, { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::248432388601:root" }, "Action": [ "s3:ListBucket", "s3:GetBucketLocation" ], "Resource": [ "arn:aws:s3:::rill-ORG_NAME-share" ] } ] } ``` ![](https://images.contentful.com/ve6smfzbifwz/5pQbl3HSlLZrHEkeikrCk0/e32fbac29841f7f7ef94a9c942a6111d/a2fff1a-Screen_Shot_2020-09-11_at_5.22.23_PM.png) #### Using CLI We can use CLI to put the bucket policy too. Make sure you have access to put bucket policy. ```shell export BUCKET="rill-ORG_NAME-share" #Create a bucket with Name of Bucket to be shared curl -s http://pkg.rilldata.com/aws/cross-account-s3-bucket-policy.json | sed "s/bucket-to-be-shared/${BUCKET}/" | tee policy.json aws s3api put-bucket-policy --bucket ${BUCKET} --policy file://policy.json ``` ### Method 2: AWS IAM Roles We can provide access to the S3 Bucket through an IAM Role which will be assumed by the Rill Data AWS Account to gain the access to the buckets. #### Using Cloudformation Console 1. Open AWS Cloudformation to create a new Stack. https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/template 2. Use Amazon S3 URL: `https://s3.amazonaws.com/cf-templates.rilldata.com/rilldata-s3-bucket-access.yaml` ![](https://images.contentful.com/ve6smfzbifwz/2PLlx4LviVGr8lDwSTMIn7/6b652cf3c578b34ea1f6b60e39b56ec4/6266ce4-Screen_Shot_2020-09-14_at_3.44.05_PM.png) 3. Specify Stack Details ** Stack Name: `rilldata-s3-access` ** Bucket Name: Name of the bucket we want to provide access to. ![](https://images.contentful.com/ve6smfzbifwz/6oyNvy8RYkxRNGEj04g33Z/0112a51eadd29ae3f165d70f4c6d325a/01cef67-c2681ce-Screen_Shot_2020-06-03_at_12.46.58_AM.png) 4. Click Next, Again Next, Acknowledge the Capabilities and Create the Stack. 5. You can check the events and it should create the resources for you. ![](https://images.contentful.com/ve6smfzbifwz/2Klnft4rComRhVBQZOHKah/c2797be0657d535fd614a2e63f71b540/209865b-f84093e-Screen_Shot_2020-06-03_at_1.21.47_AM.png) 6. Share the Outputs with Rill Data ![](https://images.contentful.com/ve6smfzbifwz/6ZwtzmCgUR5qduKnZjxL4r/bb09c4adea36e4107b26774957af96c8/3448175-00dea4a-Screen_Shot_2020-06-03_at_1.31.40_AM.png) ##### CloudFormation Template Reference We would be using the following Cloudformation Template. ```yaml AWSTemplateFormatVersion: '2010-09-09' Metadata: License: Apache-2.0 Description: 'AWS CloudFormation Template for providing Rill Data Access to S3 Bucket. It creates a Role that can be assumed by the RillData AWS Account. The Role has a IAM policy associated with them.' Parameters: BucketName: Type: String Description: S3 Bucket Name. Don't append s3://. NamePrefix: Type: String Description: Name prefix for the IAM Policy and IAM Role. Default: rilldata ExternalID: Type: String Description: External ID for Secured Cross Account Access. Default: r!lld@ta Resources: S3Role: Type: AWS::IAM::Role Properties: Description: 'RillData Access to the S3 Bucket. Managed by: Cloudformation' AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: AWS: - 'arn:aws:iam::248432388601:root' Action: - 'sts:AssumeRole' Condition: StringEquals: 'sts:ExternalId': !Ref ExternalID Policies: - PolicyName: !Join - '' - - !Ref NamePrefix - 'S3AccessPolicy' PolicyDocument: Statement: - Effect: Allow Action: ['s3:*'] Resource: - !Join - '' - - 'arn:aws:s3:::' - !Ref BucketName - !Join - '' - - 'arn:aws:s3:::' - !Ref BucketName - '/*' RoleName: !Join - '-' - - !Ref NamePrefix - s3-access Tags: - Key: Accessor Value: RillData - Key: ManagedBy Value: Cloudformation Outputs: RoleName: Value: !GetAtt [S3Role, Arn] Description: S3 Access Role Arn, to be shared with RillData ExternalID: Value: !Ref ExternalID Description: ExternalID for Secured Access, to be shared with RillData ``` ## S3 Logging As part of your data uploads, Rill Data requires logging to be enabled for the data you intend on sending to us for processing. Rill Data may use these logs for troubleshooting, pipeline optimization, and/or billing. In order to allow Rill Data to access these logs, please follow the instructions below to set up the proper configuration. If you have not enabled logging for the bucket Rill Data is ingesting data from, please follow these steps: 1. Go to http://aws.amazon.com * Login to your AWS account on the AWS Management Console and navigate to the S3 tab * Click the “Properties” icon on the upper-right menu to bring up the “Properties” options on the bucket Rill Data is ingesting data from * Under “Logging”, click “Enabled” * Under “Target Bucket”, select the same name of the bucket Rill Data is processing data from * Under “Target Prefix”, make sure “s3logs/” is entered * Click Save 2. Log into your Amazon S3 console and open AWS Cloudformation to create a new Stack. https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/template *Note: the person implementing this configuration must have permission to manage CloudFormation and IAM resources.* 3. Click on "Create Stack" and under "Choose a template", either upload the template provided by Rill Data, or use the following s3URL: https://s3.amazonaws.com//cf-templates.rilldata.com/RillS3LogRole.yaml 4. The template allows you to set the following parameters: * **"Stack name"** * Name the stack `rill-ORG_NAME-s3Logs` * **"S3LogBucketName"** * Description: "New or existing bucket for S3 logs" 5. You will then click that you understand that the template will create IAM resources, and then click "Create". 6. Once the stack has finished being created, send your Rill Data representative the results on the "Outputs" tab. * **RoleARN**: This IAM role identifier allows Rill Data to ingest the S3 Logs * **S3BucketName**: This is the name of S3 Bucket that the Rill Data role can have access to, in this case, it should be the bucket name you inputted into Cloudformation *Note: If interested, you can set lifecycle policies on the log bucket of interest. You can either leave this variable blank (no lifecycle policy) or 35 days at minimum. This can be applied through the S3 console on the respective bucket. Reach out to your Technical Account Manager or Sales Director if you have any questions.* --- ###### "Managed Azure Storage Container" --- title: "Managed Azure Storage Container" description: Azure Storage Container sidebar_label: "Azure Storage Container" sidebar_position: 40 --- :::warning Please note that these instructions were made specifically for Rill Managed Pipelines and used by our data engineering team to set up orchestration from object stores/data warehouses into Rill Managed Database Services. While some of the concepts may apply, please refer to your provider's documentation on the correct permissions required to connect to your service. ::: ## Overview Rill would need access to the Azure Storage Container to fetch the data. For this, Rill needs: * Storage Account * Container Name * Access Key ## Fetch Access key for a Storage Account 1. Go to Storage Accounts on your Azure Console. ![](https://images.contentful.com/ve6smfzbifwz/01q8c4i8bmP92JSQO7fXBK/81187ee95eacae80f597e888b0874d4e/0aa8f8b-Screen_Shot_2020-10-02_at_1.21.43_AM.png) 2. Choose the storage account you want to share ![](https://images.contentful.com/ve6smfzbifwz/5FV4ysIlKlxFhypnFcU6LD/7c6ceb18dd8db93177505c2d494448ae/43d7355-Screen_Shot_2020-10-02_at_1.22.44_AM.png) 3. Share the access key and the Connection String from the settings. It's recommended to share Key 1 and keep the Key 2 for the rotation. ![](https://images.contentful.com/ve6smfzbifwz/5FGt6CwBzccU5m9k5jtNMb/78e9f3b030281a25779771a4365b36eb/3f0b401-Screen_Shot_2020-10-02_at_1.24.18_AM.png) --- ###### "Managed GCS Bucket" --- title: "Managed GCS Bucket" description: Setting up GCS access sidebar_label: "GCS Bucket" sidebar_position: 20 --- :::warning Please note that these instructions were made specifically for Rill Managed Pipelines and used by our data engineering team to set up orchestration from object stores/data warehouses into Rill Managed Database Services. While some of the concepts may apply, please refer to your provider's documentation on the correct permissions required to connect to your service. ::: ## Setup instructions Follow the instructions below to grant Rill access to your Google Cloud Storage Bucket. 1. Find your Google Cloud Service Account by logging into Rill and clicking on Integrations. Your Google Cloud Service Account will be displayed. It will be of the form `organization`-`workspace`@rilldata.iam.gserviceaccount.com. 2. Go to Storage Console: https://console.cloud.google.com/storage/browser. 3. Click on the bucket to which you want to grant access (click on the bucket name itself, not the checkbox). 4. Select Permissions tab. ![](https://images.contentful.com/ve6smfzbifwz/4YwoXZUqT2BuTwEvBsG6OA/6b70d11103a3921e64d54a05d99746f2/3df9887-bucket_select_permissions.png) 5. Click Add to open the modal to add members to your bucket. ![](https://images.contentful.com/ve6smfzbifwz/2Ki9BiKaHYMivZ5DPTiwbd/762b2a071d3d6fb58a1b08fd13973dc2/8fa34b8-permissions_add.png) 6. In the New members field, enter your google service account. You can find your google service account in the Settings page for your workspace in RCC. This will typically have the form `{workspace}`-`{organization}`@rilldata.iam.gserviceaccount.com. ![](https://images.contentful.com/ve6smfzbifwz/50nIholwjMFJkaMTw8bMjy/c3334709d2eb6c8516e056f72f424957/42d2803-new_members_modal.png) 7. Select the role Cloud Storage -> Storage Object Viewer. ![](https://images.contentful.com/ve6smfzbifwz/7HHypfag0BAVHmLegVeKuJ/2bb32c99aa57abcb9ad10a4de1053b46/46c12ce-select_role_storage_viewer.png) --- ###### "Managed Google BigQuery" --- title: "Managed Google BigQuery" description: Setting up BigQuery access sidebar_label: "Google BigQuery" sidebar_position: 30 --- :::warning Please note that these instructions were made specifically for Rill Managed Pipelines and used by our data engineering team to set up orchestration from object stores/data warehouses into Rill Managed Database Services. While some of the concepts may apply, please refer to your provider's documentation on the correct permissions required to connect to your service. ::: ## Setup instructions Follow the instructions below to grant Rill access to your Google BigQuery datasets. 1. Find your Google Cloud Service Account by logging into Rill and clicking on Integrations. Your Google Cloud Service Account will be displayed. It will be of the form `organization-workspace-name@rilldata.iam.gserviceaccount.com`. 2. Go to your Google Cloud Console and select the project to which you want to grant access. ![](https://images.contentful.com/ve6smfzbifwz/4KskMcw6t4az7qdW5i9YDa/7c8fe66bdd9b02864ffd878a29031ac8/2c3627e-Project_selector.png) 3. Open the sidebar menu by clicking the 3 lines button in the top left, then choose IAM & Admin then click on IAM https://console.cloud.google.com/iam-admin/iam ![](https://images.contentful.com/ve6smfzbifwz/5lkiJLFKP9i0mNJGVcEJpQ/f78c764249c43db1da358df842f3ef0e/8efbbf9-IAM.png) 4. In the IAM menu click the ADD button. This will display a form where you can input the service accounts that can access your project and the permissions with which they can access it. 5. In the New members field, enter your google service account, found in step 1. 6. Select the role `BigQuery Data Viewer`, `BigQuery Read Session User`, and `BigQuery Job User`. This will permit Rill to fetch your projects tables into BigQuery. ![](https://images.contentful.com/ve6smfzbifwz/41T3D34qZmZEzFf91mhKo1/013b627be97a308698e04f50a9dccfef/be5a511-Screen_Shot_2020-10-20_at_7.10.46_PM.png) 7. Click on save. --- ###### "Rill Managed Pipelines" --- title: "Rill Managed Pipelines" description: Rill Managed Pipelines sidebar_label: "Rill Managed Pipelines" sidebar_position: 00 --- ## Rill access for managed pipelines In addition to standard connectors, Rill also manages data pipelines for customers or sets up orchestration for moving data from object stores/data warehouses into Rill's managed database service. Rill's dedicated data engineering services help build out everything from basic orchestration to complete data enrichment and custom data pipelines. To provide access for the Rill team, your team can grant read access: - [Amazon s3](/developers/other/granting/aws-s3-bucket) - [GCS](/developers/other/granting/gcs-bucket) - [Google BigQuery](/developers/other/granting/google-bigquery) - [Microsoft Azure Storage](/developers/other/granting/azure-storage-container) For more information on how Rill can help, please contact our team at [contact@rilldata.com](mailto:contact@rilldata.com). --- ##### FAQ --- title: FAQ sidebar_label: FAQ sidebar_position: 40 --- ## Technical requirements ### Why does macOS say "Rill cannot be opened because it is from an unidentified developer"? This occurs when the Rill binary is downloaded via the browser. You need to change the permissions to make it executable and remove it from Apple's developer identification quarantine. The CLI commands below will help you do that: ```bash chmod a+x rill xattr -d com.apple.quarantine ./rill ``` ### Why am I seeing "This macOS version is not supported. Please upgrade"? Rill uses DuckDB internally, which requires a newer [macOS version](https://github.com/duckdb/duckdb/issues/3824). Please upgrade your macOS version to 10.14 or higher. ### Which browsers work best with Rill? Rill is optimized for Google Chrome. While other browsers may work, we recommend using the latest version of Chrome for the most reliable experience when accessing Rill Developer or Rill Cloud dashboards. ## Rill Developer ![Empty Project](/img/concepts/rcvsrd/empty-project.png) ### What is Rill Developer? Rill Developer is a local application used to preview your project and make any necessary changes before deploying to Rill Cloud. For more information, please review [our documentation](https://docs.rilldata.com/concepts/cloud-vs-developer#rill-developer). Within Rill Developer, you can ingest new datasets, transform the sources into models, build a metrics layer, and finally visualize your data in an explore dashboard. This preview allows you to develop your project before deploying or updating an existing deployment in Rill Cloud. ### How do I do XXX in Rill Developer? Please refer to [our guided tutorial](/developers/tutorials/rill-basics/launch) to get started using Rill. In the tutorials, we walk you through first project creation, modeling, creating a metrics view and explore dashboard, and finally deploying to Rill Cloud. From there, we go through making local changes in Rill Developer and pushing your changes. In more advanced topics, we discuss custom APIs, Embed Dashboards, and more! If you still have any questions, please [contact us!](/contact) ### How do I start more than one instance of Rill Developer? If you try to start two instances of Rill Developer, you will encounter the following error: ```bash Error: serve: server crashed: grpc port 49009 is in use by another process. Either kill that process or pass `--port-grpc PORT` to run Rill on another port ``` In order to run two instances, please use the following flags with a unique port number. ```bash rill start --port 10010 --port-grpc 10011 ``` ### How do I share my dashboard with my colleagues? To share your dashboards with your colleagues, you need to [deploy your dashboard to Rill Cloud](https://docs.rilldata.com/developers/deploy/existing-project). Once deployed, you have various ways to share this dashboard with your team. Since Rill does not charge by number of users, you can simply [add them to your organization](/guide/administration/users-and-access/user-management#how-to-add-an-organization-user) and have them sign up to view the dashboard! Other ways to share the dashboard include [public URLs](/guide/dashboards/public-urls) for a limited view and [project invites](/guide/administration/users-and-access/user-management#how-to-add-a-project-user). ## Rill Cloud ![Rill Cloud](/img/concepts/rcvsrd/Rill-Cloud.png) ### What is Rill Cloud? Rill Cloud is where your deployed Rill project exists and can be shared with your colleagues or end-users. For more information, please review [our documentation](https://docs.rilldata.com/concepts/cloud-vs-developer#rill-cloud). Unlike Rill Developer, which is developer-based, Rill Cloud is where your dashboards are consumed by your end users. Additional features include bookmarks, public URLs, reporting, alerts, and more! ### How do I deploy to Rill Cloud? You can deploy your project directly from the UI by selecting [the Deploy button](/developers/deploy/deploy-dashboard/#deploying-a-project-from-rill-developer). Upon deployment, an organization will be automatically created with your Rill project inside. Each organization can have multiple projects that house multiple sources, models, metrics views, and dashboards. Note that sources are not cross-project compatible. When creating your first project, we will automatically start your 30-day free trial. In order to extend usage, you will need to sign up for a [Team Plan](#rill-team-plan). ![Deploy UI](/img/deploy/existing-project/deploy-ui.gif) ### How do I make changes to my dashboard in Rill Cloud? You can follow the same steps as above. After deploying to Rill Cloud, if you return to Rill Developer, the button will have changed from `deploy` to `update`. When selecting `update`, the objects in your Rill project will be automatically updated. Or, after syncing your Rill project to GitHub, simply push changes directly to the repository and this will automatically update your project on Rill Cloud. ### How do I share my dashboard to other users? You will need to [invite users to your organization](/guide/administration/users-and-access/user-management#how-to-add-an-organization-user) or [project](/guide/administration/users-and-access/user-management#how-to-add-a-project-user), send them a URL for them to [request access to your dashboard](/guide/administration/users-and-access/user-management#user-requests-access-via-url), or if you just want them to see the contents of your dashboard, you can look into using [public URLs](/guide/dashboards/public-urls). ## Rill Cloud Trial ### What is Rill Cloud Trial? We offer a free 30-day trial to anyone interested in testing out Rill Cloud. Simply create an account and deploy your project from Rill Developer. If you haven't already created an account and logged in, you will be prompted during the deployment process. There are no feature limitations in a free trial, but we have set the limit for imported data to 10 GB per project with two projects per deployment. You can check the data usage in the settings page. :::note The banner will show you the remaining days for your trial and will update as the expiration gets closer! Upgrade to a Team plan to continue using Rill! ::: ![Rill Trial Banner](/img/FAQ/rill-trial-banner.png) ### When does my trial start? Your trial will start when you deploy a project to Rill Cloud from Rill Developer. An Organization will be automatically created during this process using your email, and the project will be the folder that your Rill project exists in. You can change the name using [CLI commands](https://docs.rilldata.com/reference/cli/project/rename). ### How long does my Rill Cloud Trial last? A Rill Cloud trial lasts for 30 days. If you have any comments or concerns, please reach out to us on our [various platforms](/contact)! ### What is included in the free trial? The free trial is limited to 2 projects and up to 10 GB of data each. You can invite as many users as required and there are no locked features. ### What happens to my project if I do not upgrade to a Team plan? Your projects will hibernate. Your project metadata will still be available once you've activated your team plan. If you'd like to delete your deployment from Rill Cloud, you can do so via the [CLI commands.](https://docs.rilldata.com/reference/cli/org/delete) ![Expired Project](/img/FAQ/expired-project.png) ### What is project hibernation? When a project is inactive for a specific number of days or your trial has expired, we automatically hibernate the project. What this means is that all of your information and metadata is saved, and resource consumption will be zero. You will need to unhibernate the project to gain access to the dashboard again. If the project is hibernated due to payment issues, the project will stay in this state until payment is confirmed. Once the payment is confirmed, you can re-access the project with the following CLI command: ``` rill project hibernate --redeploy ``` ## Rill Team Plan ### What is a Rill Team Plan? A Rill Team Plan unlocks unlimited projects with a 50 GB data storage limit per project. Pricing starts at $250/month and includes 10 GB of storage. Use the [pricing calculator](https://www.rilldata.com/pricing) on our pricing site for more insight into how much your data might cost! You'll now have access to all of our features on Rill Cloud that you were using during the trial. ### How many seats am I allowed? At Rill, we do not charge per seat! From subscription to a Rill Team Plan, you'll have access to unlimited seats! Invite all of your colleagues or just a few—the choice is yours. ### How are payments calculated? We charge you by the amount of data that you load into Rill when building your sources and models. Use the [pricing calculator](https://www.rilldata.com/pricing) on our pricing site for more insight into how much your data might cost! If you'd like a more detailed inspection of your objects, [contact us](/contact), and we'll set this up for you. ### When am I billed? You'll be billed on the first of each month via our partner at Stripe. You'll need to set up a valid credit card as explained in [our billing documentation](/developers/other/plans#managing-payment-information). If there are any issues with the card, you'll be notified in the UI and be given a few days' grace period to update your information. If you start in the middle of the month, you'll be billed prorated for the number of days you have access to Rill Cloud. ### Why was I billed $XXX? You can check your data usage in your organization settings usage page. The graph will display the data that you have over 10GB. Use the [pricing calculator](https://www.rilldata.com/pricing) to see your cost of your current data usage. If you'd like a more detailed inspection of your objects, [contact us](/contact), and we'll set this up for you. ## Enterprise Plan ### What is an Enterprise Plan? Enterprise plan includes all the features of a Team Plan but also provides further offerings, such as a dedicated Technical Account Manager and fewer restrictions on data storage. For more information, please visit our pricing page, [here](https://www.rilldata.com/pricing), or [contact us](/contact). Transparent usage-based billing means you only pay for what you need. Flexible pricing based on storage, compute, and network units start at the rates below: **Storage:** Storage is the total compressed data in the cluster. It's available in [two performance tiers](/developers/other/FAQ#what-are-the-compute-requirements-for-each-performance-tier), Hot and Cold, which set minimum [compute requirements](/developers/other/FAQ#what-are-the-compute-requirements-for-data-processing). Data can also be offloaded to an archival tier where it does not consume any compute `$0.0005 / GB per hour` **Compute:** [Rill Compute Units (RCU)](/developers/other/FAQ#what-is-a-rill-compute-unit-rcu) are a combination of CPU, memory, and disk used for ingesting and querying data. RCUs scale up elastically for data ingestion & processing with enterprise discounts on RCUs provisioned for querying. `$0.09 RCU per hour` ### What is a Rill Compute Unit? (RCU) A Rill Compute Unit (RCU) is a usage metering unit that tracks, by the minute, the amount of resources consumed by your Rill cloud service, including compute, memory, disk storage, batch or streaming data ingested. 1. For data ingestion & processing, RCU scale up elastically as you load and transform data into our service. 2. For querying, Rill offers enterprise customers a set of dedicated compute units sized to handle their concurrency requirements across all data sources. Provisioned query RCUs are upscaled or downscaled on a daily basis based on the usage of past week to maintain query performance targets. Tasks that consume more resources will see more RCU usage compared to tasks that consume fewer resources. While there is no one-to-one mapping between the various resources your service consumes and an RCU, 1 RCU is comparable to the resources used by a task that runs for one hour on 1 vCPU with 4 GB RAM. ### How is data size calculated in Rill? When you load data into Rill's service, it is stored in compressed columnar format. Rill charges based on the amount of data stored in Rill's service, after compression, which is typically 3-8x. For egress and ingress, actual data transferred over the network is measured. ### What are the compute requirements for each performance tier? For querying, Rill provisions a fixed number of RCUs for each performance tier based on the following factors: - **Size**: Rill will charge a minimum number of compute units per unit of hot storage and cold storage to maintain optimal performance - **Queries**: As the concurrency of data access or complexity of the queries increase Rill will automatically add more RCUs to maintain optimal performance. - **Performance tier**: Hot tier consumes more RCU/TB of stored data than cold tier. This provides a good cost vs performance trade off. Cost can be reduced by moving data to the cold tier. Below are the minimum RCU charged for each performance tier:
| **Storage Tier** | **Performance** | **RCU Consumption** | | ---------------- | ------------------------------- | ---------------------------------------------------------------------- | | Hot Performance | 8 RCU per 25 GB of data stored | High performance for frequently accessed data | | Cold Performance | 8 RCU per 250 GB of data stored | Optimized for less frequently accessed data | | Archival | No RCU consumed | Data is not query-able and can be moved to Hot or Cold tiers as needed |
The number of RCUs for each performance tier will be added in increments depending on the overall provisioned RCUs.
| **Overall RCU Count** | **Available Increment** | | --------------------- | ----------------------- | | Up to 64 RCU | 8 RCU | | Up to 128 RCU | 16 RCU | | Up to 256 RCU | 32 RCU | | Over 256 RCU | 64 RCU |
Provisioned RCUs are upscaled or downscaled on a daily basis based on the usage of past week. ### What are the compute requirements for data processing? For data ingestion & processing, Rill elastically scales up compute slots when you load data into the service. - The number of RCUs consumed depends on the complexity of the ingestion pipeline. - A more complex ingestion involving joins will consume more RCUs as compared to a less complex pipeline. ### How can I estimate my RCU usage? The best way to get an accurate RCU estimate is to load some sample data into Rill service and track the RCU usage. For estimates on larger projects [contact us](/contact) for a pricing calculator that reflects the latest volume incentives and discounts. ### How can I track my RCU usage? RCU usage and utilization can be viewed in the Rill Control Center. Additionally, Administrators can configure to get Daily, Weekly, Monthly summary reports for RCU usage for visibility. --- ##### "Billing Plans Explained" --- title: "Billing Plans Explained" description: How billing works for non-enterprise accounts sidebar_label: Billing Plans Explained sidebar_position: 00 --- Billing cycles begin on the first of every month (12:00 AM UTC). If you start your plan mid-month, your first month will be prorated accordingly. You can subscribe to a Team Plan at any point via your Rill Cloud billing page. ### How does it work? Rill Data does not use a user-based license system. Instead, we calculate your data usage after ingestion and calculate pricing based on that usage. For more information on pricing, see our [pricing page](https://www.rilldata.com/pricing). ## Trial Plan Get started with Rill Cloud with our 30-day free trial! Upon deployment of your first project, your trial will automatically start. On a free trial, you will be allowed one project with up to 10 GB of data storage. Like all plans in Rill Data, this also comes with unlimited seats. As an admin, you'll notice banners at the top of the UI indicating the remaining time left on your trial. Once your time has run out, your projects in Rill Cloud will hibernate. While your project won't be accessible on Rill Cloud, the files will still be available if you choose to upgrade to a Team Plan. ![Deploy Project](/img/manage/billing/deploy-project.png) ### Upgrading to Team Plan Once you are ready to upgrade to a Team Plan, you can do so via the organization billing page, or select `Upgrade` in the top banner. Only organization administrators can upgrade the plan. ![Team Plan](/img/manage/billing/team-plan.png) ### Managing Payment Information Please add a payment method and billing information that is accepted by Stripe. For more information, please visit [Stripe's website](https://docs.stripe.com/payments/payment-methods/overview). ![Stripe](/img/manage/billing/stripe.png) ## Team Plan The Team Plan unlocks unlimited projects with a 50 GB data storage limit per project. Like all plans in Rill Data, this also comes with unlimited seats. As an admin, you will have access to your billing and usage page to monitor your project. If you decide to unsubscribe from your subscription, you will have access to Rill Cloud until the end of the month. Afterward, your project will hibernate. Your project will not be accessible while hibernating. You will need to renew your subscription in order to access your project on Rill Cloud. To calculate your current usage and pricing, see our [pricing page](https://www.rilldata.com/pricing). ![Team Plan2](/img/manage/billing/team-plan2.png) ## Enterprise Plan The Enterprise Plan includes all the features of a Team Plan but also provides further offerings, such as a dedicated Technical Account Manager and fewer restrictions on data storage. For more information, please visit our [pricing page](https://www.rilldata.com/pricing), or [contact us](/contact). ### Enterprise usage-based billing **Storage:** Storage is the total compressed data in the cluster. It's available in [two performance tiers](/developers/other/FAQ#what-are-the-compute-requirements-for-each-performance-tier), Hot and Cold, which set minimum [compute requirements](/developers/other/FAQ#what-are-the-compute-requirements-for-data-processing). Data can also be offloaded to an archival tier where it does not consume any compute. `$0.0005 / GB per hour` **Compute:** [Rill Compute Units (RCU)](/developers/other/FAQ#what-is-a-rill-compute-unit-rcu) are a combination of CPU, memory, and disk used for ingesting and querying data. RCUs scale up elastically for data ingestion and processing, with enterprise discounts on RCUs provisioned for querying. `$0.09 RCU per hour` --- ### Tutorials --- #### Rill Basics --- ###### "1. Source to Dashboard on Rill Cloud in 6 Steps" --- title: "1. Source to Dashboard on Rill Cloud in 6 Steps" sidebar_label: "1. Launch Rill Developer" position: 1 collapsed: false sidebar_position: 1 tags: - Tutorial - OLAP:DuckDB - Rill Developer - Getting Started --- :::note prerequisites You need to [install Rill](https://docs.rilldata.com/developers/get-started/install). ```bash curl https://rill.sh | sh ``` ::: The goal of this six-part tutorial is to get started with Rill and deploy your project to Rill Cloud. Upon deployment, your [30-day trial will start](/developers/other/plans#trial-plan). Each course will build upon the previous one, allowing you to have a fully functioning project with many of our advanced features. This tutorial can be used in tandem with our documentation to ensure you have up-to-date information. ## Start Rill Developer ```yaml rill start my-rill-tutorial ``` :::tip While we support macOS and Linux, you can also get Rill Developer running on a [Windows machine via WSL](https://docs.rilldata.com/developers/get-started/install#rill-on-windows-using-wsl). If you are having any issues installing or starting Rill, please see our [installation page](https://docs.rilldata.com/developers/get-started/install). ::: If running Rill in a new directory, you'll be prompted with the following. Type "Y" and press Enter. ```bash ? Rill will create project files in "~/Desktop/GitHub". Do you want to continue? (Y/n) ``` Rill Developer will automatically open in your default browser. If not, you can access it via the following URL: ``` localhost:9009 ``` Welcome to Rill Developer! :::note What is Rill Developer? Rill Developer is used to develop your Rill project, as editing in Rill Cloud is not yet available. In Rill Developer, you will create connections to your source files, perform last-mile ETL, define metrics in the metrics layer, and finally create a dashboard. For more details on the differences between Rill Developer and Rill Cloud, see our documentation [here](/developers/deploy/cloud-vs-developer) ::: ![New Rill Project](/img/tutorials/rill-basics/new-rill-project.png) Let's go ahead and select `Start with an empty project`. If you want to skip the basics, you can select one of the quick start projects and refer to our Quick Start Guide for the corresponding project. Note that we have many more projects available in our public repo [here](https://github.com/rilldata/rill-examples).
Where am I in the terminal? You can use the `pwd` command to see which directory you are in within the terminal.
If this is not where you'd like to make the directory, use the `cd` command to change directories.
--- ###### "2. Import the Source" --- title: "2. Import the Source" sidebar_label: "2. Import the Source" sidebar_position: 1 hide_table_of_contents: false tags: - Tutorial - OLAP:DuckDB - Rill Developer - Getting Started --- Let's start at the beginning of all data pipelines: **the source**. ### What is a Source? In Rill, a source model represents your raw data. See our list of [connectors](/developers/build/connectors) or select Add -> Add Data from Rill Developer to see the supported sources. ![Add Data](/img/tutorials/rill-basics/add_data.png) :::tip Where is the data being imported? By default, Rill uses DuckDB as the underlying OLAP engine (see Connect OLAP engines). Please see our docs for the supported list of connectors. We support various OLAP engines. If you have a different OLAP engine that you're interested in using, please let us know! Looking for a ClickHouse tutorial? Click [here!](/developers/tutorials/rill-clickhouse/r_ch_launch) ::: ### Add a Data Source Select the `+Add` dropdown and select `Data`. This will open a UI showing supported connectors. ![Adding Data](/img/tutorials/rill-basics/Adding-Data.gif) For our tutorial, let's add two GCS storage sources from our public storage. In Rill, each dataset is added separately as a single source model. Once imported into Rill, you can then transform the data via SQL modeling (we'll cover that on the next page). Follow the steps in the UI and use the following URIs below. ```yaml gs://rilldata-public/github-analytics/Clickhouse/2025/03/modified_files_*.parquet gs://rilldata-public/github-analytics/Clickhouse/2025/03/commits_*.parquet ``` :::tip Data These are datasets derived from the commit history and modified files of our friends at ClickHouse's GitHub repository. In our example, we'll ingest a single month of data. However, Rill supports glob patterns, so you could modify the URL to `gs://rilldata-public/github-analytics/Clickhouse/**/modified_files_*.parquet` to ingest all years and months of data. However, that's a lot of data to ingest! ::: Once imported, you'll see the UI change with several things: 1. A `source_name.yaml` file created in the file explorer. 2. A DuckDB database created in the Connectors explorer. 3. Within the DuckDB database, under main, the source table with a preview when selected. 4. The right panel showing a summary of the data source and column values. ![Add GCS](/img/tutorials/rill-basics/Add-GCS.gif) Now we're ready to create a `model`.
Don't see what you're looking for? We are continually adding new sources and connectors in our releases. For a comprehensive list, you can refer to our connectors page. Please don't hesitate to reach out if there's a connector you'd like us to add! If this is your first time, you may need to refresh the browser for DuckDB to appear in the UI.
:::tip Too much data? By default, all environments running locally are considered `dev` environments. This means that you can use environment variables to filter the input data, as Rill Developer is designed for testing purposes. For example, you can filter the repository data on the `author_date` column or simply use `limit ####`. ``` --- ###### "3. Create a SQL Model" --- title: "3. Create a SQL Model" sidebar_label: "3. Create a SQL Model" sidebar_position: 2 hide_table_of_contents: false tags: - Tutorial - OLAP:DuckDB - Rill Developer - Getting Started --- ### What is a model? A model in Rill is used to perform intermediate processing as well as any last-mile ETL on the source data. This includes joining lookup tables to your table to enrich and transform the data for analysis. We recommend creating [One Big Table](/developers/build/models/models-101#one-big-table-and-dashboarding) to power your metrics views. ### Create a Model from the Source Go ahead and select the `Create Model` button in the top right-hand corner of the commits dataset. ![Add Model](/img/tutorials/rill-basics/Add-Model.gif) You'll be automatically redirected to the model page. On the left panel, you'll see a models folder created with the file `commits__model.sql`. On the right panel, you'll find general information about your model table, the column values, and at the bottom of the page, a preview of the data. ```SQL select * from commits ``` Let's try to make some changes to our SQL and see how the UI reacts. ```SQL select * from commits order by author_date DESC ``` Notice that the preview table is automatically updated as we modify the SQL. This is due to our auto-save feature. In case any errors are encountered, the UI will update accordingly and display the error. ![Model Sql](/img/tutorials/rill-basics/Model-SQL.gif) :::tip Our Autosave feature can be enabled or disabled as needed via the rill.yaml file / project settings, or by simply selecting the toggle in the UI. ::: ### Let's merge two tables! Each dataset independently gives us some interesting information, but we want to view the data from both of these datasets in a single dashboard. - `commits` gives us information about the user who committed the changes. - `modified_files` gives us information on the actual changes to the file and its directory. We will grab all the columns from `commits` and only a few from `modified_files` as seen below. We will join the two datasets on the `commit_hash` column. As this is the SQL view that our dashboard will be based on, we want to materialize it! ```SQL -- Model SQL -- Reference documentation: https://docs.rilldata.com/developers/build/models -- @materialize: true SELECT a.*, b.filename, b.added_lines, b.deleted_lines FROM commits a INNER JOIN modified_files b ON a.commit_hash = b.commit_hash ``` :::tip You can see all referenced source tables in the right panel, as well as the column schema and an overview of the data. Select a model to be redirected to the model. ::: ### Concept: What is materialization? By default, models created in DuckDB will be views. Both views and tables will be shown in under your DuckDB's connector table UI. ```yaml -- @materialize: true ```
Why materialize? You may experience some improved performance materializing SQL views for intermediate models in the case of complex SQL or large data. We generally recommend materializing finals models that power dashboards. However, you might experience some degradation of modeling experience [auto-save feature] for some specific situations including cross joins.
--- ###### "4. Create the Metrics View" --- title: "4. Create the Metrics View" sidebar_label: "4. Create the Metrics View" sidebar_position: 3 hide_table_of_contents: false tags: - Tutorial - OLAP:DuckDB --- ### What is a Metrics View? A metrics view is a layer in which you can create and define your measures and dimensions. Think of it as the layer that takes all of your raw data and makes sense of it. In this layer, you can define, for example, what Net Revenue is using basic to advanced [arithmetic functions available in DuckDB](https://duckdb.org/docs/stable/sql/functions/numeric.html). You also define what dimensions to use to slice and dice your data in the Explore Dashboard. If using Canvas dashboards, you can view multiple metrics views on a single page! ### Let's create a metrics view! Now that the data is ready in your model file, we can create a metrics view. There are two ways to do so: 1. Generate metrics with AI 2. Start simple using the +Add, Metrics
How does Generate metrics with AI work? We send a set of YAML files along with some context to OpenAI to suggest the dimensions, measures, and various other key pairs for your dashboard.
Let's go ahead and create a simple metrics layer via the UI and build on it. ![Create Metrics View UI](/img/tutorials/rill-basics/create-metrics-view-ui.png) As you can see, the default dashboard YAML is as follows: ```yaml # Metrics View YAML # Reference documentation: https://docs.rilldata.com/reference/project-files/metrics_views version: 1 type: metrics_view model: example_model # Choose a model to underpin your metrics timeseries: timestamp_column # Choose a timestamp column (if any) from your table dimensions: - name: category display_name: Category column: category description: Description of the dimension measures: - name: total_revenue display_name: Total Revenue expression: SUM(revenue) description: Total revenue generated ``` For now, you'll see a red box around the UI and the preview button grayed out. This indicates something is wrong with the YAML. ## Fixing the Metrics View From here, we have two options! Starting from version 0.50, we have introduced the [visual-metric-editor](/developers/build/metrics-view/what-are-metrics-views#using-the-visual-metrics-editor), in the top right corner, you can select whether you want to modify the YAML directly or use a UI tool. ## Via the Visual Metrics Editor In the top right of the UI, select the viz button to navigate to the visual metrics editor. Below is an example of a completed visual metrics editor. We will make some modifications to our current file to build something similar. We can go ahead and change the following components as directed in the UI: - `model`: commits_model - `time column`: author_date - `measures`: sum(added_lines) - `dimensions`: author_name ![Basic Viz Editor](/img/tutorials/rill-basics/basic-viz-editor.png) Once finished, the red border will disappear and your explore dashboard is ready to be created. If you need further information on each component, see the next section, via the YAML. ## Via the YAML Let's go over each component and what they are in order to better understand the metrics view and how to fix the dashboard. ### Type ```yaml version: 1 type: metrics_view ``` The type is a Rill-required key pair as it indicates to Rill what type of file this is. Whether a `source`, `metrics_view`, `connector`, etc. We can keep this as is. --- ### Underlying Table ### ```yaml table: commits_model # Note that this has 3 "_"! ``` The underlying table can be defined here. Let's change it to `commits_model`. --- ### Time series ### ```yaml timeseries: author_date # Select an actual timestamp column (if any) from your table ``` The time-series column is a date-type column within your table. Let's set this to `author_date`. --- ### Dimensions ### ```yaml dimensions: - name: author_name # name for this dimension display_name: "The Author's Name" # A display label column: author_name # column in model that this dimension is derived from description: "The name of the author of the commit" #A description, displayed when hovered over dimension ``` Dimensions are used for exploring segments and filtering the dashboard. Each dimension has a set of required key pairs. For the `column`, you will need to set it to a column from your model or table, assign it a label (this is how it's labeled in the dashboard), and provide a description. The description will be displayed when hovering over and can provide more context. Our first dimension will be the author's name. Let's go ahead and make the changes above. --- ### Measures ### ```yaml measures: - name: added_lines expression: SUM(added_lines) display_name: Sum of Added lines description: The aggregate sum of added_lines column ``` Measures are the numeric aggregates of columns from your data model. These functions will use DuckDB SQL aggregation functions and expressions. Similar to dimensions, you will need to create an expression based on the column of your underlying model or table. Our first measure will be: `SUM(added_lines)`. ## Adding more Functionality Let's add further dimensions and measure to the metrics layer and see the changes to the explore dashboard. ### Dimensions From our dataset, we can add more dimensions to allow more filtering and exploration of the measures we will create. Add the following dimensions, with title and description. - author_name - author_timezone - filename ### Measures We can definitely create better aggregations for some more meaningful data based on these commits. - sum(added_lines) - sum(deleted_lines) You may need to reference the metrics view YAML reference guide to figure out the above. Your final output should look something like this!
Working Metrics View YAML ```yaml # Metrics View YAML # Reference documentation: https://docs.rilldata.com/reference/project-files/metrics_views version: 1 type: metrics_view table: commits_model timeseries: author_date # Select an actual timestamp column (if any) from your table dimensions: - column: author_name name: author_name display_name: The Author's Name description: The name of the author of the commit - column: author_timezone display_name: "The Author's TZ" description: "The Author's Timezone" - column: filename display_name: "The filename" description: "The name of the modified filename" measures: - expression: SUM(added_lines) name: added_lines display_name: Sum of Added lines format_preset: humanize description: The aggregate sum of added_lines column. valid_percent_of_total: true - expression: "SUM(deleted_lines)" name: deleted_lines display_name: "Sum of deleted lines" description: "The aggregate sum of deleted_lines column." ```
### Completed visual metrics editor If you decide to build out the metrics view via the UI, it should look something like below! ![New Viz Editor](/img/tutorials/rill-basics/new-viz-editor.png) --- ###### "5. Create an Explore and Canvas Dashboard" --- title: "5. Create an Explore and Canvas Dashboard" sidebar_label: "5. Create a Dashboard" sidebar_position: 3 hide_table_of_contents: false tags: - Tutorial - OLAP:DuckDB --- ## Explore Dashboard We have the choice to start with either an Explore or Canvas dashboard. Let's start with an Explore Dashboard by selecting `Create Explore Dashboard`. This will automatically populate the explore dashboard to select all of the created metrics and dimensions in your metrics view. We can make changes to the view via the YAML or visual dashboard editor. ![Completed 100 Dashboard](/img/tutorials/rill-basics/Completed-100-dashboard.png) ## Making Changes ### Visual Explore Dashboard ![Visual Dashboard Tutorial](/img/tutorials/rill-basics/visual-dashboard-tutorial.png) On the right panel, you are able to select measures, dimensions, time ranges, and various other components that control the view of your dashboard. In the main area, you will see a preview of what your dashboard will look like. You can also select the code view to make any needed changes and/or set more advanced settings as found in our [explore dashboard YAML reference.](https://docs.rilldata.com/reference/project-files/explore-dashboards) ### YAML By default, the page will contain the basic parameters as seen below. You can add more advanced settings as required for your use case. ```YAML # Explore YAML # Reference documentation: https://docs.rilldata.com/reference/project-files/explore-dashboards type: explore title: "commits_model_metrics dashboard" metrics_view: commits_model_metrics dimensions: '*' measures: '*' ``` ### Explore Dashboard Components For a quick summary of the different components of an explore dashboard and their respective parts in the dashboard UI: ![Simple Dashboard](/img/tutorials/rill-basics/simple-dashboard.gif) --- ## Canvas Dashboard Next, let's try to make a Canvas dashboard! ![Canvas Dashboard](/img/tutorials/rill-basics/canvas-dashboard.png) Select `Add widget` to build a component using our metrics view. The right-hand panel will display the different parameters that you can set per component. Try to create the following: - KPI with measures: [Sum of Added lines, Sum of deleted lines] - Bar Chart: Added lines over Author Name. Once completed, select `Preview`. You should see something like the following image: ![Complete Canvas Dashboard](/img/tutorials/rill-basics/complete-canvas-dashboard.png) --- ###### 6. Deploy to Rill Cloud --- title: 6. Deploy to Rill Cloud sidebar_label: '6. Deploy to Rill Cloud' sidebar_position: 9 hide_table_of_contents: false tags: - Tutorial - OLAP:DuckDB --- :::tip Rill Cloud Trial If this is the first time you have deployed a project onto Rill Cloud, you will automatically start your [Rill Cloud Trial](/developers/other/plans#trial-plan) upon deployment of your Rill project. Your trial will last for 30 days. Please refer to our documentation for more information on the details of your trial. ::: ## Deploy via the UI! Depending on your version, you'll notice a `Deploy to share` button in the top right corner. Let's select it! :::note Don't see the button? Deploying via the UI was introduced in v0.48. Please upgrade Rill by running `rill upgrade` in the CLI. The version is on the bottom left of the Rill Developer UI. ::: ![Deploy UI](/img/tutorials/rill-basics/deploy-ui.gif) Steps to deploy to Rill Cloud: 1. Select the `Deploy to share` button. 2. Select `continue` on the [free trial](https://www.rilldata.com/pricing). - If you have already deployed, select an existing organization. 3. Select `continue` on user invites. We will discuss this in a later section. 4. You will be navigated to the /status page of your deployed project. ## Welcome to Rill Cloud! Take note of the following features in the UI. We'll go over them later. ![UI Explained](/img/tutorials/rill-basics/ui-explained.gif) --- ###### Deployment Success! --- title: Deployment Success! sidebar_label: Deployment Success! sidebar_position: 10 hide_table_of_contents: false tags: - OLAP:DuckDB - Tutorial --- ### Congratulations! You have successfully deployed your project from Rill Developer to Rill Cloud in six steps! ![Rill Cloud Success](/img/tutorials/rill-basics/rill-cloud-success.png) ### Key Takeaways 1. With Rill Developer, you can import data, transform data, and create dashboards. 2. You can deploy your dashboard to Rill Cloud in just a few clicks. 3. Rill Developer and Rill Cloud offer different capabilities. While you have completed the Basics Course, there are still many more features in our product, with new ones coming out regularly! Please continue on to our next series, **Rill's Advanced Features**, to dive into our more advanced features and advanced modeling/dashboarding. Or, if you are looking for administration topics, please refer to **Administration Topics**. For guidance on how to use Rill Cloud and its features, please refer to our [**Explore section**](https://docs.rilldata.com/guide/dashboard-101) of our docs for more information! --- #### Rill Clickhouse --- ###### "1. Launch Rill Developer" --- title: "1. Launch Rill Developer" sidebar_label: "1. Launch Rill Developer" sidebar_position: 3 hide_table_of_contents: false tags: - OLAP:ClickHouse - Tutorial --- :::note prerequisites You will need to [install Rill](https://docs.rilldata.com/developers/get-started/install). ```bash curl https://rill.sh | sh ``` You need access to either a [locally running ClickHouse Server](https://clickhouse.com/docs/en/install) or [ClickHouse Cloud](https://docs.rilldata.com/developers/build/connectors/olap/clickhouse#connecting-to-clickhouse-cloud). We recommend using ClickHouse Cloud as this will make deploying to Rill Cloud easier. Please review the documentation, [here](https://docs.rilldata.com/developers/build/connectors/olap/clickhouse). ::: ## Start Rill Developer ```yaml rill start my-rill-clickhouse ``` After running the command, Rill Developer should automatically open in your default browser. If not, you can access it via the following url: ``` localhost:9009 ``` You should see the following webpage appear. ![New Rill Project](/img/tutorials/rill-basics/new-rill-project.png) Let's go ahead and select `Start with an empty project`.
Where am I in the terminal? You can use the `pwd` command to see which directory in the terminal you are.
If this is not where you'd like to make the directory use the `cd` command to change directories.
--- ###### "2. Connect to ClickHouse" --- title: "2. Connect to ClickHouse" sidebar_label: "2. Connect to ClickHouse" sidebar_position: 3 hide_table_of_contents: false tags: - OLAP:ClickHouse - Tutorial --- import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; ## Default OLAP connection and Connect to ClickHouse Within Rill you can set the default OLAP connection on the [project level](https://docs.rilldata.com/reference/project-files/rill-yaml) or the [dashboard level](https://docs.rilldata.com/reference/project-files/explore-dashboards). For this course, we will set it up on the project level so all of our dashboards will be based on our ClickHouse table. :::tip You have two options for your ClickHouse server: 1. Use a [local running ClickHouse server](https://clickhouse.com/docs/en/install) 2. Use [ClickHouse Cloud](https://clickhouse.com/docs/en/cloud/overview) Depending what you choose, the contents of your connection will change and I recommend looking through [our ClickHouse documentation](https://docs.rilldata.com/developers/build/connectors/olap/clickhouse) for further information. ::: ### Connect to ClickHouse We can create the clickhouse connection by selection `+Add Data` > `ClickHouse` and fill in the components on the UI. ![ClickHouse Connectionector](/img/tutorials/ch/clickhouse-connector.png) :::tip You can obtain the credentials from your ClickHouse Cloud account by clicking the `Connect` button in the left panel.: ![ClickHouse Cloud Credential](/img/tutorials/ch/clickhouse-cloud-credential.png) ``` "https://:?username=&password=&secure=true&skip_verify=true" ``` ::: Once this is created, a `clickhouse.yaml` file will appear in the `connectors` directory and the following will be added to your rill.yaml. ```yaml compiler: rillv1 title: "Rill and ClickHouse Project" olap_connector: clickhouse #automatically added ``` Example for a locally running ClickHouse server: ```yaml host: "localhost" port: "9000" ``` or ```yaml dsn: "clickhouse://localhost:9000" ``` You can either add the credentials in plain text or dsn via the yaml file or add the credentials via the CLI. Please see our documentation to find the DSN for [your ClickHouse Cloud instance](https://docs.rilldata.com/developers/build/connectors/olap/clickhouse#connecting-to-clickhouse-cloud). ### How to pass the credentials to Rill There are a few way to define the credentials within Rill. Please create a file called clickhouse.yaml and add the following contents. ```yaml type: connector driver: clickhouse host: "localhost" port: "9000" ``` or ```yaml type: connector driver: clickhouse dsn: "clickhouse://localhost:9000" ``` Navigate back to the Terminal and stop the Rill process. You can run the following to add a variable and use this is within Rill. ``` rill start --env host='localhost' --env port='9000' ``` Afterwards, create a file called clickhouse.yaml and add the following contents: ```yaml type: connector driver: clickhouse host: "{{ .env.host }}" port: "{{ .env.port }}" ``` There's a few way to generate the .env file. Making a source that requires credentials will automatically generate it. Else, you can create it using `touch .env` in the rill directory. ``` CLICKHOUSE_HOST="localhost" CLICKHOUSE_PORT=9000 CLICKHOUSE_USERNAME="" CLICKHOUSE_PASSWORD="" CLICKHOUSE_SSL=false or CLICKHOUSE_DSN="..." ``` :::tip Via the UI If you connect to ClickHouse via the UI, this will automatically create a template with connectors/clickhouse.yaml as well as a reference of your DSN to the .env folder. This will automatically get pushed along with your project to Rill Cloud. ::: You should now be able to see the contents of your ClickHouse database in the left panel of your UI. ![Olap Connectionector](/img/tutorials/ch/olap-connector.png) --- ###### "3. Create Metrics View Dashboard in Rill" --- title: "3. Create Metrics View Dashboard in Rill" sidebar_label: "3. Create Metrics View Dashboard in Rill" sidebar_position: 3 hide_table_of_contents: false tags: - OLAP:ClickHouse - Tutorial --- ## Create the Metrics View. If you noticed in the previous screenshot, we had a table called `uk_price_paid`. This is a dataset that is used in ClickHouse's Learning portal, so we thought it was fitting to continue with this dataset. :::note In the case that you have not already added this table to your local or Cloud database, please follow the steps on [ClickHouse's site](https://clickhouse.com/docs/en/getting-started/example-datasets/uk-price-paid) for the steps to do so! ::: ### Create metrics view Let's create a metrics view based on the table via the `Generate metrics via AI`. ![AI Generate](/img/tutorials/ch/ai-generate.gif) ### What are we looking at? This is our metrics view, where we can define measures and dimensions to be used on dashboards. ```yaml # Metrics view YAML # Reference documentation: https://docs.rilldata.com/reference/project-files/explore-dashboards # This file was generated using AI. version: 1 type: metrics_view title: UK Price Paid Metrics connector: clickhouse table: uk_price_paid timeseries: date dimensions: ... measures: ... ``` While we go into more details in our [Rill Basics course](/developers/tutorials/rill-basics/dashboard) and [our documentation](https://docs.rilldata.com/developers/build/dashboards), let's go over it quickly. --- `timeseries` - This is our time column, which is used as our x-axis for graphs. `connector` - this is our manually defined ClickHouse connector `dimensions` - These are our categorical columns that we can use on the dashboard to filter and slice; `measures` - These are our numerical aggregates defined in the metrics layer. We can see functions such as MAX(), COUNT(), and AVG() used on the underlying table. --- ###### "4. Create Explore Dashboard in Rill" --- title: "4. Create Explore Dashboard in Rill" sidebar_label: "4. Create Explore Dashboard in Rill" sidebar_position: 4 hide_table_of_contents: false tags: - OLAP:ClickHouse - Tutorial --- ### Create the explore dashboard When you're ready, you can create the visualization on top of the metrics layer. Select `Create Explore dashboard`. This will create a simple explore-dashboards/uk_price_paid_metrics_explore.yaml file that reads in all the dimensions and measures. For more information on the available key-value pairs, please refer to the [reference documentation](https://docs.rilldata.com/reference/project-files/explore-dashboards). --- ### What can we do in Rill? In our case, since we generated this with AI, we can look through the description of the populated measures for more information. Based on this, we can find some specific information on the UK properties dataset at a glance, such as: 1. In 2023, what was the minimum/maximum detached property sold in London? [46.5K, 65.0M] 2. In 2023, what was the average price of detached properties sold in London? How many? [2.5M, 981] ![2023 London](/img/tutorials/ch/2023-london.png) If we want to go further into the details, we can even compare detached vs flat vs terraced properties using our compare feature. Using the x-axis, we can drill down further from the 2023 year into a specific month, week, or even day. ![2023 London Compare](/img/tutorials/ch/2023-london-compare.png) You can also compare total transactions between 2022 and 2023. In the screenshot below, we selected the Total Transactions metric and enabled the time-compare feature to see the delta and delta percent change between the two time periods. ![Time Compare](/img/tutorials/ch/time-compare.png) --- ###### "5. Deploy to Rill Cloud" --- title: "5. Deploy to Rill Cloud" sidebar_label: '5. Deploy to Rill Cloud' sidebar_position: 9 hide_table_of_contents: false tags: - OLAP:ClickHouse - Tutorial --- :::tip Rill Cloud Trial If this is the first time you have deployed a project onto Rill Cloud, you will automatically start your [Rill Cloud Trial] () upon deployment of your Rill project. Your trial will last for 30 days. Please refer [here] () for more information on the details of your trial. ::: ## Deploy via the UI! Select the `Deploy to share` button in the top right corner of a dashboard. ![Deploy UI](/img/tutorials/rill-basics/deploy-ui.gif) Steps to deploy to Rill Cloud: 1. Select the `Deploy to share` button. 2. Select `continue` on the free trial [link to article of free trial explanation] - If you have multiple organizations, please select Rill_Learn and `continue`. 3. Select `continue` on user invites. 4. You will be navigated to the /status page of your deployed project. Take note of the following features in the UI: ![UI Explained](/img/tutorials/rill-basics/ui-explained.gif) ## In case of the following error: ```bash connection: dial tcp 127.0.0.1:9000: connect: connection refused ``` This is likely due to using a locally running ClickHouse server. If so, you will not be able to access your locally running server from Rill Cloud. Instead, we suggest using [ClickHouse Cloud](https://clickhouse.com/cloud). For steps to set up ClickHouse Cloud, please refer to [our documentation](https://docs.rilldata.com/developers/build/connectors/olap/clickhouse#connecting-to-clickhouse-cloud). --- ###### Visualize your ClickHouse Tables in Rill! --- title: Visualize your ClickHouse Tables in Rill! description: tutorial for ClickHouse sidebar_position: 3 tags: - Tutorial - OLAP:ClickHouse --- ![ClickHouse Rill](/img/tutorials/ch/clickhouse-rill.png) In this tutorial, we will discuss how to use ClickHouse as the default OLAP database with Rill to visualize your ClickHouse tables in as little as a few clicks. The overall steps of the tutorial are: 1. Launch Rill Developer 2. Connect to your ClickHouse server whether it is locally run or via ClickHouse Cloud 3. Create a metrics view based on a table in ClickHouse 4. Create an explore dashboard via Generative AI 5. Deploy the dashboard to Rill Cloud, and share to colleagues Once completed, you will have a basic understanding of how to use Rill Developer with ClickHouse and can customize the Rill Developer project to create further metrics views based on ClickHouse tables and create more dashboards. ### Let's get started! --- ###### "Ingesting Data into ClickHouse" --- title: "Ingesting Data into ClickHouse" sidebar_label: "Ingesting Data Directly into ClickHouse" sidebar_position: 50 hide_table_of_contents: false tags: - OLAP:ClickHouse - Tutorial --- ## Importing your own Data into ClickHouse from ... Currently, ClickHouse lacks some [direct ingestion](https://clickhouse.com/docs/en/migrations/snowflake) from certain providers. You can navigate to their website for a full list of data sources in which they support [direct ingestion](https://clickhouse.com/docs/en/integrations), via manual import or [ClickPipes](https://clickhouse.com/cloud/clickpipes). ### How does this affect Rill? When switching from DuckDB, you may have noticed some changes to the capabilities of Rill. By default, we disable modeling when ClickHouse is enabled as the default OLAP engine. However, we can change this behavior by enabling the feature flag `clickhouseModeling`. ```yaml features: clickhouseModeling: true ``` Once this is enabled, you'll be able to create model files and add sources in the UI and use these for SQL transformations, as you would with DuckDB. :::note Currently not all the functionality is supported but our team is working on this to add more features! Please reach out to us on our community or via GitHub for any specific missing functionality that you looking for. ::: ## Ingestion directly from Snowflake to ClickHouse In the below example, we are importing data from Snowflake to ClickHouse using S3 as an intermediate stage. ```yaml type: model materialize: true -- the source of data in Snowflake connector: snowflake sql: > select * from CUSTOMER limit 1001 -- the staging table in S3 stage: connector: s3 path: s3://rill-developer.rilldata.io/snow -- the output clickhouse connector output: connector: clickhouse materialize: true ``` In order to use this method you will need to set your credentials in `.env`. If the .env does not already exist (it will be created by default if you have created a source), you can create a .env file in the rill directory by running `touch .env` and this should now be visible in Rill Developer. ``` CLICKHOUSE_HOST="localhost" CLICKHOUSE_PORT=9000 SNOWFLAKE_DSN="" AWS_ACCESS_KEY_ID="" AWS_SECRET_ACCESS_KEY="" ``` :::note If you already set up ClickHouse via the `.env` file, you will just need to add your Snowflake and S3 credentials. ::: ## Ingestion directly from BigQuery to ClickHouse In the below example, we are importing data from Big Query directly to ClickHouse using GCS as an intermediate stage. ```yaml type: model materialize: true connector: bigquery sql: | SELECT * FROM `..` project_id: "" stage: connector: gcs path: 'gs://rill-bq-ch/temp/' output: connector: clickhouse ``` You'll need to ensure that your provided `google_application_credentials` have all the required permissions on both [BigQuery](https://cloud.google.com/bigquery/docs/access-control) and [GCS](https://cloud.google.com/storage/docs/access-control/iam-roles). Ensure that your .env has the following: ``` CLICKHOUSE_HOST="localhost" CLICKHOUSE_PORT=9000 GOOGLE_APPLICATION_CREDENTIALS="" ``` --- ##### "Clone a Project - Quick Start" --- title: "Clone a Project - Quick Start" sidebar_label: "Clone an existing Project" sidebar_position: 3 hide_table_of_contents: false tags: - Getting Started - Quickstart - Tutorial --- # Clone a Project - Quick Start This guide will help you get started with an existing Rill project by cloning it from a repository and setting it up locally. ## Prerequisites Before you begin, make sure you have: - **Rill CLI** installed ([Installation Guide](/developers/get-started/install)) ```bash curl https://rill.sh | sh ``` - **Access to the [Rill Project](https://ui.rilldata.com/)** ## Step 1: Clone the Repository Depending on whether your project is synced to GitHub or not, select the correct clone method. If you are unsure, please see the Settings page in the project. ### From GitHub ![Github Pushed Changes](/img/tutorials/rill-advanced/github-pushed-changes.png) ```bash # Clone the repository git clone https://github.com/username/rill-project.git # Replace 'username' and 'rill-project' with your actual URL cd ``` ### Using Rill CLI ![Status](/img/tutorials/rill-advanced/status.png) ```bash # Clone from Rill rill project clone ``` ## Step 2: Explore the Project Structure A typical Rill project contains: ``` / ├── rill.yaml # Project configuration ├── sources/ # Data source definitions │ ├── database.yaml # Database connections │ ├── api.yaml # API endpoints │ └── files.yaml # File-based sources ├── models/ # SQL transformations │ ├── staging/ # Staging models │ ├── marts/ # Business logic models │ └── metrics/ # Metric definitions ├── dashboards/ # Dashboard configurations │ └── main_dashboard.yaml ├── alerts/ # Alert definitions ├── .env # Environment variables (not in git, need to run rill env pull) └── .gitignore # Git ignore rules ``` ## Step 3: Set Up Environment Variables If you cloned the project via GitHub, you will need to run the following command to bring down the environment variables to your local machine. ```bash rill env pull ``` If you cloned the project via the Rill CLI, you should see the following in the Terminal: ```bash Updated .env file with cloud credentials from project "your-project-here". ``` :::tip Admin of your project? As an admin, when running `rill start`, we'll automatically retrieve your credentials for you. No need for extra steps. ::: ## Step 4: Check your Source YAML before starting Rill We want to check to see if any `{{if dev}} ... {{end}}` parameters have been set in your source ingestion. If not, when you start Rill, this will initiate a full ingestion of your data, which might take some time and, depending on the source location, could incur costs (e.g., Snowflake, BigQuery). However, if your data is not that large, it may be safe to start Rill without these guardrails. ## Step 5: Start Rill Developer ### Start the Development Server ![Clone Project](/img/tutorials/quickstart/clone-project.png) ```bash # Start Rill Developer rill start ``` This will: - Start the web UI at `http://localhost:9009` - Initiate ingestion of data sources - Start building your models and dashboards - Show any errors or warnings ## Step 6: Explore the Project and Make Changes Once your sources and models have built and you are able to explore your dashboards, make the needed changes to the files and get ready to update your Rill project. :::warning Changes to sources and models Changes to sources and models will initiate a full refresh of the source, unless otherwise indicated via `patch_mode`. We highly recommend reviewing the changes to ensure that you do not push unwanted changes to your production environment. ::: ### via git For projects that were cloned via git, you'll need to run the required git commands to add, commit, and push changes. Keep in mind the basic git practices about merging files to main without having an approval process. ### via Rill Update Button For projects cloned via the CLI, the underlying connection to the deployment will also be brought locally so that when the button to "Deploy" is now "Update" the existing deployment. Keep in mind the warning above about changes to sources and models. --- ##### "Cost Monitoring Analytics Demo" --- title: "Cost Monitoring Analytics Demo" sidebar_label: "Cost Monitoring Analytics Demo" sidebar_position: 20 hide_table_of_contents: false tags: - Tutorial - Quickstart - Example Project - Cost Management - Margin Analysis - Business Intelligence --- # Cost Monitoring Analytics Demo This guide walks you through the Cost Monitoring Analytics demo project, which showcases Rill's capabilities for analyzing cost and usage data. You'll learn how to clone the project, understand its structure, and explore the dashboard. ## Overview This dataset is modeled after a similar dashboard we use internally at Rill to both identify opportunities to improve our cloud infrastructure operations and to manage customer implementations. Typical users include engineering, customer success, and finance. In this example, we've combined cloud services, other hosting costs, and revenue metrics. The Cost Monitoring Analytics demo analyzes operational costs and revenue data, providing insights into: - **Margin trends** – Daily, weekly, and monthly profitability patterns - **Customer profitability** – Which customers are driving the highest margins - **Cost efficiency** – Understanding operational cost effectiveness across services - **Revenue optimization** – Monitoring revenue performance against operational costs - **Business intelligence** – Identifying opportunities for improved profitability ## Step 1: Clone the Project ### Clone from GitHub ```bash # Clone the Cost Monitoring Analytics project git clone https://github.com/rilldata/rill-examples.git cd rill-examples/rill-cost-monitoring ``` ## Step 2: Project Structure The project is organized as follows: ``` rill-cost-monitoring/ ├── rill.yaml # Project configuration ├── sources/ # Data source definitions │ └── metrics_margin_monitoring.yaml # Margin source dataset ├── models/ # SQL transformations │ └── metrics_margin_model.sql # Cost trend analysis ├── metrics/ # Defined measures and dimensions │ └── metrics_margin_metrics.yaml # Cost monitoring metrics ├── dashboards/ # Dashboard configurations │ ├── margin_scorecard.yaml # Canvas dashboard │ └── metrics_margin_explore.yaml # Explore Dashboard └── README.md # Project documentation ``` ## Step 3: Data Sources The source connects to our static public dataset containing operational cost and revenue data. You can modify this [source](/developers/build/connectors) to point to your own data export. ```yaml # Visit https://docs.rilldata.com/ to learn more about Rill code artifacts. type: source connector: "https" uri: "https://storage.googleapis.com/rilldata-public/metrics_margin_monitoring.parquet" ``` **What this data contains:** - **Cost data** – Operational expenses broken down by component, environment, and pipeline - **Revenue data** – Customer revenue associated with different billing plans and SKUs - **Customer information** – Company names and associated billing plans - **Operational metadata** – Location, environment, application, and pipeline details - **Time series data** – Daily granular data for trend analysis **What this source does:** - Connects to our public GCS bucket (if modified, will need to be verified via [credentials](/developers/build/connectors/credentials)) - Ingests the data into Rill's OLAP Engine (DuckDB) - Provides the foundation for margin analysis and business intelligence ## Step 4: Data Models :::tip Modeling In our example, we've already processed the data, but if you need to do some last-mile ETL in Rill, this is possible via a [model](/developers/build/models). ::: ## Step 5: Creating your Metrics View Metrics in Rill define the measures and dimensions that power your margin monitoring dashboards: ```yaml # Metrics view YAML # Reference documentation: https://docs.rilldata.com/reference/project-files/metrics-views # This file was generated using AI. version: 1 type: metrics_view display_name: Metrics Margin Model KPIs model: metrics_margin_model timeseries: __time smallest_time_grain: "day" dimensions: - name: customer display_name: Customer column: company description: "The name of the customer" - name: plan_name display_name: Plan Name column: plan_name description: "The name of the billing plan" - name: location display_name: "Cost by Region" column: "location" description: "The region incurring costs" - name: component display_name: Cost by Component column: component description: "The component generating costs" - name: app_name display_name: "Cost by App Name" column: "app_name" description: "The app generating costs" - name: sku_description display_name: "Cost by SKU" column: "sku_description" description: "The sku description for costs" - name: pipeline display_name: "Cost by Data Pipeline" column: "pipeline" description: "The pipeline incurring costs" - name: environment display_name: "Cost by Environment" column: "environment" description: "The environment incurring costs" measures: - display_name: "Total Cost" expression: "SUM(cost)" name: total_cost description: "The sum of cost" format_preset: currency_usd - display_name: "Total Revenue" expression: SUM(revenue) name: total_revenue description: The sum of revenue format_preset: currency_usd - display_name: "Net Revenue" expression: "SUM(revenue) - SUM(cost)" name: net_revenue description: "The sum of revenue minus the sum of cost" format_preset: currency_usd - display_name: "Gross Margin %" expression: "(SUM(revenue) - SUM(cost))/SUM(revenue)" name: gross_margin_percent description: "Net revenue divided by sum of revenue" format_preset: percentage - display_name: "Unique Customers" expression: "COUNT(DISTINCT company)" name: unique_customers description: "The count of unique companies" format_preset: humanize ``` **What this metrics file does:** - **Measures** define the key business calculations you want to perform: - `total_cost` – Sum of all operational costs - `total_revenue` – Sum of all revenue generated - `net_revenue` – Revenue minus costs (profit calculation) - `gross_margin_percent` – Profitability percentage calculation - `unique_customers` – Count of distinct customers - **Dimensions** define how you can slice and dice the margin data: - `customer` – Analysis by individual customer/company - `plan_name` – Breakdown by billing plan types - `location` – Regional cost and revenue analysis - `component` – Analysis by system components - `app_name` – Application-specific margin analysis - `sku_description` – Product SKU-level insights - `pipeline` – Data pipeline cost attribution - `environment` – Environment-based analysis (dev/staging/prod) ## Step 6: Dashboard Exploration #### **Features** - **Margin Overview** – High-level profitability trends and key metrics - **Customer Analysis** – Detailed margin analysis by customer - **Product Breakdown** – Profitability by plan types and SKUs - **Regional Analysis** – Cost and revenue distribution across locations - **Component Analysis** – Margin breakdown by system components - **Environment Tracking** – Cost allocation across environments #### **Selectors** - **Date range selector** – Analyze specific time periods - **Time Comparison Toggle** – Compare with previous periods - **Customer filter** – Focus on specific customers - **Plan filter** – Analyze specific billing plans #### **Filters** - **Region filter** – Analyze specific regions - **Component filter** – Focus on specific system components - **Margin threshold filter** – Focus on high/low margin segments --- ##### "Analyze Your GitHub Repository" --- title: "Analyze Your GitHub Repository" sidebar_label: "GitHub Analytics" sidebar_position: 20 hide_table_of_contents: false tags: - Tutorial - Quickstart - Example Project --- # Analyze Your GitHub Repository This guide shows you how to build a powerful analytics dashboard for your own GitHub repository using Rill. In just a few commands, you'll have a fully interactive dashboard to discover hot zones in your codebase, track contributor activity, analyze code churn, and measure development velocity. :::tip See it live **[Explore the live demo →](https://ui.rilldata.com/demo/rill-github-analytics)** to see interactive dashboards analyzing real repositories (DuckDB, Rill). This is exactly what you'll create for your own repository. ::: ## Step 1: Clone the Project ### Clone from GitHub ```bash # Clone the GitHub Analytics project git clone https://github.com/rilldata/rill-examples.git cd rill-examples/rill-github-analytics ``` ### Install Dependencies The project includes Python scripts for downloading GitHub data and generating Rill project files. The project uses [Poetry](https://python-poetry.org/) for dependency management ([installation guide](https://python-poetry.org/docs/#installation)). ```bash # Using Poetry (recommended) poetry install # Or using pip with venv python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate pip install pandas pydriller ``` ## Step 2: Scrape Git History and Save to Object Storage **Prerequisites:** - [Create a Google Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) - Set up GCS authentication: - [Create a service account key](https://cloud.google.com/iam/docs/keys-create-delete) with Storage Object Admin role - Set `GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json` environment variable The project includes a `download_commits.py` script that clones your target repository, extracts commit metadata and file changes, then uploads the data as parquet files to your GCS bucket. ```bash # Download and upload to GCS python download_commits.py owner/repo --gcs --bucket gs://your-bucket/github-analytics # Or limit to recent commits for faster testing python download_commits.py owner/repo --gcs --bucket gs://your-bucket/github-analytics --limit 1000 ``` **Note:** Files will be saved to `gs://your-bucket/github-analytics/owner/repo/` to keep data organized by repository. :::note Private repositories For private repos, use a fine-grained access token: 1. [Create a fine-grained personal access token](https://github.com/settings/tokens?type=beta) with **read-only** access to the repository 2. Store it as an environment variable: `export GITHUB_TOKEN=your_token_here` 3. Git will automatically use it when cloning (works in local dev and CI) ::: **Note:** For large repositories with 10,000+ commits, the download may take 10-30 minutes. Use `--limit` to test with a smaller dataset first. ## Step 3: Generate Rill Project Files The project includes a `generate_project.py` script that will generate: - Source definitions pointing to your GCS bucket - Data transformation models - Metrics definitions - An explore dashboard Run the script with your repository and bucket: ```bash # Generate Rill files configured for your GCS bucket python generate_project.py owner/repo --gcs --bucket gs://your-bucket/github-analytics # Examples: python generate_project.py duckdb/duckdb --gcs --bucket gs://your-bucket/github-analytics python generate_project.py your-org/your-repo --gcs --bucket gs://your-bucket/github-analytics ``` **Note:** Rill supports both Google Cloud Storage (GCS) and Amazon S3. The download script currently supports GCS. For S3, you'll need to modify the script. :::note Just want to explore locally? Use the `--local` flag instead: `python generate_project.py owner/repo --local` This is great for testing, but you won't be able to deploy to Rill Cloud without migrating to cloud storage later. ::: ## Step 4: Deploy to Rill Cloud Deploy your dashboard to share with your team: ```bash rill deploy ``` This creates a live, shareable link where your team can explore the data together. Since your data is in cloud storage, Rill Cloud can access it directly. :::tip Preview locally first Want to verify everything looks good before deploying? Run `rill start` to preview the dashboard locally, then deploy when ready. ::: ## Next Steps Now that you have GitHub Analytics deployed: 1. **Keep data fresh** – Schedule the download script to run regularly (cron, GitHub Actions, etc.) and keep your dashboards up to date 2. **Customize metrics** – Edit the metrics YAML files to add team-specific calculations 3. **Add alerts** – Use Rill's alerting features to monitor key metrics --- ##### "" --- title: "" collapsed: true tags: - Tutorial - Getting Started --- # Welcome to Rill Guides and Tutorials! ### We're thrilled you've joined us on this th-Rill-ing journey! These tutorials were built to help you get started with Rill and empower you to build your own projects. The journey covers data import, SQL transformations, dashboard creation, and finally deployment to Cloud. From there, you'll explore Rill Cloud-specific features, push local changes to Rill Cloud, and develop custom features. ### Why Rill? Here at Rill, we believe in business-speed visual analytics. Gone are the days of waiting for your dashboard to load! By utilizing OLAP engines, we strive for quick, reactive dashboards that you can drill down into further to make real-time decisions. Not only that, you'll also be able to make traditional dashboards that we call Canvas Dashboards. --- Follow along, refer to the in-line linked documentation, and explore our product to make the most out of these guides! Please reach out to ask any questions or provide feedback on our Community Slack or Discord! - [**Source to Published Dashboard:**](/developers/tutorials/rill-basics/launch) Learn the basics, understand the core concepts of Rill, and deploy to Cloud! - [**Visualize ClickHouse tables in Rill:**](/developers/tutorials/rill-clickhouse) For our friends from **ClickHouse**, we have created a dedicated course using ClickHouse! - [**Clone a Project:**](/developers/tutorials/clone-a-project) Step-by-step guide to clone an already deployed instance from Rill Cloud to your local machine - [**GitHub Analytics:**](/developers/tutorials/github-analytics) Analyze your own GitHub repository by modifying the Python file, download_commits.py - [**Cost Monitoring Analytics:**](/developers/tutorials/cost-monitoring-analytics) Analyze your cost and usage data - [**OpenRTB Analytics:**](/developers/tutorials/openrtb-analytics) Analyze real-time bidding data to track impressions, bids, wins and more - [**Set Up MCP**](/developers/tutorials/setting-up-mcp) Configure Model Context Protocol (MCP) to enable natural language queries with Claude integration Each section is crafted to ensure a smooth and comprehensive learning experience. Happy learning! --- --- ##### "Integrating with Rill's Admin API" --- title: "Integrating with Rill's Admin API" sidebar_label: "Integrating with Rill's Admin API" sidebar_position: 30 hide_table_of_contents: false tags: - Tutorial - Quickstart - Example Project --- # Integrating with Rill's Admin API Rill's Admin API allows you to programmatically manage your Rill organization, including workspaces, users, service accounts, and permissions. This guide will walk you through the steps to integrate with Rill using the Admin API. > In this scenario, we will generate an admin service account and use it to make requests to the Admin API to provision a new 'viewer' user with custom attributes. This can be useful for automating temporary viewer user provisioning in your organization. ## Prerequisites This guide assumes you have completed the following prerequisites: 1. Installed the Rill developer CLI - see [Install Rill Developer](/developers/get-started/install) 2. A Rill Cloud organization and workspace - see [Rill Quickstart Guide](https://docs.rilldata.com/quickstart) 3. Administrative access to your Rill organization :::info This guide can be completed with user tokens. These tokens are tied to your personal user and access permissions. They are useful for local scripting and experimentation. We don't recommend for production use. ::: - [Rill Admin OpenAPI Specification](/api/admin) - [Service Account CLI Reference](/reference/cli/service) - [Custom API Integration Guide](/developers/build/custom-apis/calling) ## Example: Managing Users with a Rill Service Account This example demonstrates how to create a service account, use it to manage users with custom attributes, and then clean up all resources. This is useful for automating user provisioning and management in your Rill organization. ### Step 1: Create a Service Account First, create a service account with organization admin permissions and custom attributes: ```bash # Create a service account with custom attributes rill service create user-management-service \ --org-role admin \ --attributes '{"department": "engineering", "environment": "production"}' Service created: user-management-service Token: rill_svc_A1B2C3D4E5F6G7H8I9J0K1L2M3N4O5P6Q7R8S9T0U1V2W3X4Y5Z6 ``` ### Step 2: Use Service Account to Add a User with Custom Attributes Now use the service account token to add a user to your organization: ```bash # Add a user to the organization using the service account curl -X POST "https://admin.rilldata.com/v1/orgs/example/members" \ -H "Authorization: Bearer rill_svc_A1B2C3D4E5F6G7H8I9J0K1L2M3N4O5P6Q7R8S9T0U1V2W3X4Y5Z6" \ -H "Content-Type: application/json" \ -d '{ "email": "newuser@example.com", "role": "viewer", }' ``` **Expected Response:** ```json { "pendingSignup": true } ``` The user will receive an email invitation to join the Rill organization as a viewer. ### Step 4: Verify User Access and Attributes Check that the user was added successfully: ```bash # List organization members to verify the user was added curl -X GET "https://admin.rilldata.com/v1/orgs/example/members" \ -H "Authorization: Bearer rill_svc_A1B2C3D4E5F6G7H8I9J0K1L2M3N4O5P6Q7R8S9T0U1V2W3X4Y5Z6" ``` ### Step 5: Clean Up Resources Remove the user and service account when done: ```bash # Remove the user from the organization curl -X DELETE "https://admin.rilldata.com/v1/orgs/example/members/newuser@example.com" \ -H "Authorization: Bearer rill_svc_A1B2C3D4E5F6G7H8I9J0K1L2M3N4O5P6Q7R8S9T0U1V2W3X4Y5Z6" # Revoke the service account token curl -X DELETE "https://admin.rilldata.com/v1/services/tokens/rill_svc_A1B2C3D4E5F6G7H8I9J0K1L2M3N4O5P6Q7R8S9T0U1V2W3X4Y5Z6" \ -H "Authorization: Bearer rill_svc_A1B2C3D4E5F6G7H8I9J0K1L2M3N4O5P6Q7R8S9T0U1V2W3X4Y5Z6" # Delete the service account rill service remove user-management-service ``` ## Service Account API Access Service accounts have access to all Admin API endpoints based on their assigned roles and permissions. Key endpoints include: - **Organization Management**: Add/remove users, manage roles and permissions - **Project Management**: Create, update, and delete projects - **User Management**: Provision users, manage attributes, issue tokens - **Service Management**: Create and manage other service accounts - **Deployment Management**: Start, stop, and configure deployments - **Token Management**: Issue ephemeral user tokens with custom attributes ## Service Account Best Practices - _Secure Storage of Credentials_: Store service account credentials securely, using encrypted storage solutions and access controls to prevent unauthorized access. - _Regular Rotation of Credentials_: Regularly update service account passwords and keys to reduce the risk of compromise. - _Minimum Necessary Permissions_: Grant only the permissions necessary for the specific tasks the service account needs to perform, and review permissions regularly to adapt to changes in application functionality. - _Monitoring and Logging_: Depending on your organization's security needs, you can also consider implementing monitoring and logging of all access and actions taken by service accounts to detect and respond to anomalous activities promptly. - _Use Custom Attributes_: Leverage custom attributes on service accounts to implement fine-grained access controls and security policies. - _Token Lifecycle Management_: Implement proper token lifecycle management, including regular rotation and revocation of unused tokens. --- ##### "OpenRTB Analytics Demo" --- title: "OpenRTB Analytics Demo" sidebar_label: "OpenRTB Analytics Demo" hide_table_of_contents: false sidebar_position: 20 tags: - Tutorial - Quickstart --- # OpenRTB Analytics Demo Learn how to analyze real-time bidding (RTB) data with Rill using the OpenRTB Analytics demo project. This guide shows you how to track ad impressions, bids, wins, and revenue across different publishers, advertisers, and campaigns. ## Step 1: Clone the Project ```bash # Clone the OpenRTB Analytics demo git clone https://github.com/rilldata/rill-examples.git cd rill-examples/rill-openrtb-prog-ads # Start Rill Developer rill start ``` Visit [http://localhost:9009](http://localhost:9009) to explore your OpenRTB analytics dashboard. ## Step 2: Project Structure The project is organized as follows: ``` rill-openrtb-prog-ads/ ├── rill.yaml # Project configuration ├── sources/ # Data source definitions │ ├── auction_data_raw.yaml # Ad impression data │ └── bids_data_raw.yaml # Bid request/response data ├── models/ # SQL transformations │ ├── auction_data_model.sql # Workaround SQL to keep data up to date │ └── bids_data_model.sql # Workaround SQL to keep data up to date ├── metrics/ # Defined measures and dimensions │ ├── auction_metrics.yaml # Auction metrics │ └── bid_metrics.yaml # Bids metrics ├── dashboards/ # Dashboard configurations │ ├── auction_explore.yaml # Auction Explore dashboard │ ├── bids_explore.yaml # Bid Explore dashboard │ └── executive_overview.yaml # Canvas dashboard └── README.md # Project documentation ``` ## Step 3: Data Sources ### Bids Source The bids source captures bid request and response data: ```yaml # sources/bids_data_raw.yaml type: source connector: "https" uri: "https://storage.googleapis.com/rilldata-public/auction_data.parquet" ``` **What this does:** - Records all bid requests and responses - Tracks bid prices and win/loss status - Enables analysis of bidding patterns and market dynamics ### Auction Source The auction source tracks aggregated auction metrics and bid outcomes: ```yaml # sources/auction_data_raw.yaml type: source connector: "https" uri: "https://storage.googleapis.com/rilldata-public/auction_data.parquet" ``` **What this does:** - Connects to the `rilldata-public` GCS bucket - Fetches aggregated auction data with key metrics - Tracks bid requests, responses, wins, and bid floors - Data includes volume metrics, performance ratios, and revenue data ## Step 4: Data Models In this case, we are not joining the data models and having two unique explore models and joining the visualization in a [Canvas Dashboard](/developers/build/dashboards/canvas). ## Step 5: Creating your Metrics View Metrics in Rill define the measures and dimensions that power your RTB dashboards: ```yaml # Metrics view YAML # Reference documentation: https://docs.rilldata.com/reference/project-files/metrics-views version: 1 model: "auction_data_model" type: metrics_view timeseries: "__time" smallest_time_grain: "hour" measures: - display_name: Total Requests name: requests expression: sum(bid_request_cnt) description: Total Requests format_preset: humanize - display_name: "Avg Bid Floor" name: avg_bid_floor expression: "sum(bid_floor) / sum(has_bid_floor_cnt) " description: "Average Bid Floor" format_preset: currency_usd - display_name: "1D QPS" name: 1d_qps expression: "sum(bid_request_cnt) / 86400 " description: "1D QPS" format_preset: humanize dimensions: - name: app_site_name display_name: App Site Name column: app_site_name - display_name: App Site Domain column: app_site_domain name: app_site_domain - name: pub_name display_name: Pub Name column: pub_name - name: app_site_cat display_name: App Site Cat column: app_site_cat - name: ad_size display_name: Ad Size column: ad_size - name: device_state display_name: Device State column: device_state - name: device_osv display_name: Device OS Version column: device_osv - name: bid_floor_bucket display_name: Bid Floor Bucket column: bid_floor_bucket - name: platform_browser display_name: Platform Browser column: platform_browser - name: device_os display_name: Device OS column: device_os - name: device_type display_name: Device Type column: device_type - name: ad_position display_name: Ad Position column: ad_position - name: video_max_duration_bucket display_name: Video Max Duration Bucket column: video_max_duration_bucket - name: video_min_duration_bucket display_name: Video Min Duration Bucket column: video_min_duration_bucket - display_name: Placement Type column: placement_type name: placement_type - name: auction_type display_name: Auction Type column: auction_type - name: app_or_site display_name: App or Site column: app_or_site - name: device_country display_name: "Device Country" column: "device_country" ai_instructions: ... ``` **What this metric view does:** - **Measures** define the key RTB performance indicators: - `Total Requests` - Total number of bid requests processed - `Avg Bid Floor` - Average minimum acceptable price across auctions - `1D QPS` - Daily queries per second (bid request volume) - `avg_profit_margin_pct` - Average profit margin percentage - **Dimensions** enable analysis across different segments: - `app_site_name` - Application or website name for publisher analysis - `pub_name` - Publisher name for performance tracking - `ad_size` - Ad unit dimensions (300x250, 728x90, etc.) - `bid_floor_bucket` - Grouped bid floor ranges for optimization analysis - `device_country` - Geographic performance by country - `device_type` - Mobile, desktop, or tablet device analysis - `auction_type` - Type of auction (first-price, second-price, etc.) - `placement_type` - Ad placement context (banner, video, native) ## Step 6: Dashboard Exploration #### **Features** - **Explore Slice-and-Dice** - For data exploration and ad-hoc analysis - **Canvas** - Traditional charts and visualizations - **Pivot/Flat Table** - Tabular data views with sorting and grouping - **Measure's TDD** - Granular Analysis of single measure #### **Selectors** - **Date range selector** - Analyze specific time periods - **Time Comparison Toggle** - Compare previous time periods - **Dimensions Comparison** - Compare unique dimension values #### **Filters** - **Publisher filter** - Focus on specific publishers - **Advertiser filter** - Analyze specific advertisers - **Geographic filter** - Regional performance analysis ### Let's answer some basic questions! For campaign `Leafly_MarketMight`, which site domains are getting the most bids? ![OpenRTB Analytics 1](/img/tutorials/quickstart/openrtb-analytics-1.png) Which US region has the highest activity and what devices are being used? ![OpenRTB Analytics 2](/img/tutorials/quickstart/openrtb-analytics-2.png) Compare Bids and Auctions. ![OpenRTB Analytics 3](/img/tutorials/quickstart/openrtb-analytics-3.png) These are just some of the insights that you can find within your explore dashboard but you'll find more hidden gems in your data as you continue to use Rill. Please let us know if you have any other questions! --- ##### Troubleshooting Performance in Rill --- title: Troubleshooting Performance in Rill description: Dev/Prod Setup sidebar_label: Optimize Performance in Rill sidebar_position: 10 --- On this page, we've gathered a running list of recommendations and general guidelines to ensure your experience of using Rill remains performant and optimized. These best practices will help to ensure your dashboards remain performant, and that things continue to "just work" (for both Rill Developer and Rill Cloud), even as the size of your underlying data and deployment continues to grow. These best practices and guidelines will also continue to evolve but please don't hesitate to [reach out](/contact) if you start facing any bottlenecks or have further questions about ways to improve the Rill experience! If you're looking for connector specific optimization see, [Dev/Prod Connector Environments](/developers/build/connectors/templating). If you're looking for model specific optimization see, [Performance Optimization](/developers/build/models/performance). :::info Working with very large data from the get go? Generally speaking, Rill's [embedded DuckDB OLAP engine](/developers/build/connectors/olap/duckdb) works very well out-of-the-box for datasets _up to around 50GB in size_. If you plan to be working with and ingesting volumes of data larger than 50GB, please [**get in touch**](/contact) and we can explore using one of our other enterprise-grade [OLAP engine](/developers/build/connectors/olap) options. ::: ## Dashboard and Model Performance Depending on the complexity of your underlying models and the size of the data models, there are things that you can do to improve performance. ### Consider which models to materialize By default, models will be materialized as views (in DuckDB). This allows for a dynamic and highly interactive experience when modeling, such as keystroke-by-keystroke profiling. However, since views are logical in nature, as the complexity and size of your data models continue grow (especially if the underlying data is very large), this can start to significantly impact performance as these complex queries will need to be continuously re-executed along with a number of profiling queries that the Rill runtime will send in the backend. In such scenarios, we recommend [materializing these models as tables.](/developers/build/models/performance#model-performance) However, there are some tradeoffs to consider. - **Pros:** Materializing a model will generally ensure significantly improved performance for downstream dependent models and dashboards. - **Cons:** Enabling materialization for a model can severely impact or break the "keystroke-by-keystroke" experience and these models may also take longer to update (because the results are being written to a table vs remaining a view). It can also lead to _degraded_ performance for very specific operations, such as when you need to perform cross joins. :::tip Materialize models powering dashboards We strongly recommend materializing final models that are being used directly in dashboards to ensure this data is served more quickly. ::: ## Refreshing Source Models Another area to review when your data source starts getting larger is the ingestion performance. By default, when refreshing a [source model](/developers/build/models/source-models) in Rill, it drops and re-ingests the entire table/file. When your data is small, this isn't an issue, but it's not appropriate for larger datasets. In these cases, we recommend using [partitions and incremental models](/developers/build/models/incremental-partitioned-models). ### Partitioned Models Partitioned models divide your data into logical segments based on specific criteria, typically time-based columns like dates. This approach allows you to selectively refresh a partition where you know data has been altered. Example partition configuration: ```yaml partitions: glob: path: 'gs://my-bucket/**/*.parquet' ``` ### Incremental Models Incremental models only process new or changed data since the last refresh, rather than reprocessing the entire dataset. This dramatically improves performance for large datasets: - **Faster Refresh Times**: Process only delta changes instead of full datasets - **Reduced Resource Usage**: Lower CPU, memory, and storage requirements - **Frequent Updates**: Enable near real-time data updates without performance degradation - **Cost Efficiency**: Minimize compute costs for large-scale data processing ```yaml type: model incremental: true partitions: glob: path: gs://rilldata-public/github-analytics/Clickhouse/2024/*/* partition: directory sql: | SELECT * FROM read_parquet('{{ .partition.uri }}/commits_*.parquet') WHERE '{{ .partition.uri }}' IS NOT NULL ``` By combining partitioning and incremental processing, you'll significantly reduce model refresh times and ensure your dashboards display the most current information. ## Local Development / Rill Developer When used in conjunction, Rill Developer and Rill Cloud are meant to serve two different but complementary purposes. For larger and distributed teams, Rill Developer is meant to primarily be used for local development purposes, which allow developers to quickly model their data and validate logic. Then, Rill Cloud enables shared collaboration at scale and where production consumption of dashboards should be happening (against your full data). ### Work with a subset of your source data for local development and modeling As a general rule of thumb, we strongly recommend working with a segment of the data for modeling purposes as part of your local development workflow. This becomes increasingly important as the size of your source data grows in size, which will help ensure that your developer experience remains optimal in Rill Developer. With Rill Developer, it's best to work with a "dev partition" to help validate that the model logic is correct and producing results as expected. Then, once finalized, you can push to Rill Cloud for primary dashboard consumption (including analysis and sharing as necessary). There are a few ways to achieve this: - Defining an [**environment-specific database/cluster**](/developers/build/connectors/templating#environment-specific-connectors) to connect to between development and production - Pointing to [**different source data endpoints/databases**](/developers/build/connectors/templating#environment-specific-data-source-location) between your development and production environments - Working with a [**sample or subset of data**](/developers/build/models/templating#inline-sql-templating) during local development (but making sure the full dataset is being used in production dashboards) - Applying [**filters or other if/else predefined logic**](/developers/build/models/templating#inline-sql-templating) to run different SQL whether a model is being run locally or in production #### Environment-Specific Connectors The most common use case for connector templating is defining separate databases for your development and production operations. This approach gives you the freedom to experiment, test, and iterate on your models without the risk of accidentally modifying or corrupting your production data. Example: Here's how you can configure a ClickHouse connector to use different environments: ```yaml type: connector driver: clickhouse dev: dsn: "clickhouse://user:password@localhost:9000/dev_database" # ClickHouse connection DSN # Production environment configuration prod: host: "{{ .env.CLICKHOUSE_HOST }}" port: "{{ .env.CLICKHOUSE_PORT }}" database: "{{ .env.CLICKHOUSE_DATABASE }}" username: "{{ .env.CLICKHOUSE_USERNAME }}" password: "{{ .env.CLICKHOUSE_PASSWORD }}" ssl: true cluster: "{{ .env.CLICKHOUSE_CLUSTER }}" ``` #### Limiting the source data to a smaller time range There are different ways this can be achieved and the method also depends heavily on the data source being used. For example, assuming we had a [S3 source](/developers/build/connectors/data-source/s3) that was well partitioned by year and month (and written into a partitioned bucket), the recommended pattern would be to leverage the `path` [source property](/reference/project-files/sources) and a glob pattern to limit the size of the ingestion in your development environment. Something like (as your `source.yaml`): ```yaml type: source connector: s3 path: s3://bucket/path/**/*.parquet dev: path: s3://bucket/path/year=2023/month=12/**/*.parquet ``` By leveraging the [environment YAML syntax](/developers/build/models/templating), this ensures that only data from December 2023 will be read in from this S3 source when using Rill Developer locally while the full range of data will still be used in production (on Rill Cloud). However, if this data was **not** partitioned, then we could simply leverage DuckDB's ability to read from S3 files directly and _apply a filter post-download_ on the source. Taking this same example and using some [templating](/developers/build/connectors/templating), the `source.yaml` could be rewritten to something like the following: ```yaml type: source connector: "duckdb" sql: SELECT * FROM read_parquet('s3://bucket/path/*.parquet') {{ if dev }} where timestamp_column >= '2023-12-01' AND timestamp_column < '2024-01-01' {{ end }} ``` #### Creating intermediate staging models Another option would be to create intermediate staging models from your sources, either through [statistical sampling](https://duckdb.org/docs/sql/samples.html) or by applying a [raw limit](https://duckdb.org/docs/sql/query_syntax/limit.html), to reduce the size of your models in development. For example, with [templating](/developers/build/connectors/templating) and [environments](/developers/build/models/templating), this `model.sql` applies a five percent sample to a source: ```sql -- @materialize SELECT * FROM {{ ref "source_name" }} {{ if dev }} USING SAMPLE 5% {{ end }} ``` Similarly, if we were less concerned about skewing our sample data, we could apply a simple `LIMIT` to our source data: ```sql -- @materialize SELECT * FROM {{ ref "source_name" }} {{ if dev }} LIMIT 1000 {{ end }} ``` :::info Why are we materializing the intermediate model? As mentioned in an [above section](#consider-which-models-to-materialize), models by default will be materialized as _views_ in DuckDB unless otherwise specified. When working with intermediate models, we'll want them to be stored as **tables** to ensure downstream models and queries remain performant. ::: :::warning When applying templated logic to model SQL, make sure to leverage the `ref` function If you use templating in SQL models, you must replace references to tables / models created by other sources or models with `ref` tags. See this section on ["Referencing other tables or models in SQL when using templating"](/developers/build/connectors/templating#environment-specific-data-source-location). This ensures that the native Go templating engine used by Rill is able to resolve and correctly compile the SQL syntax during runtime (to avoid any potential downstream errors). ::: #### Use data from a dev / staging environment Some organizations might have both a development and production version of source data. In these cases, your sources should be configured to use the "dev" bucket or database for local development (in Rill Developer) and pointed to the "prod" bucket or database when in production (when deployed to Rill Cloud). Please refer to [this example](/developers/build/connectors/templating#example-clickhouse-connector-with-environment-separation) and [this example](/developers/build/connectors/templating#environment-specific-data-source-location) for a complete walkthrough of how this can be configured. ## Query Optimization Query optimization is crucial for maintaining high performance and efficiency, especially when working with data-intensive applications. As Rill dashboards are powered by [OLAP engines](/developers/build/connectors/olap), designed for analytical queries, ensuring that our queries are well-optimized can help maximize the responsiveness and speed of our dashboards. There are also additional potential second-order benefits to optimizing queries in Rill, such as improving ingestion times, how long it takes to build models, how resource-intensive it is to build models, how fast profiling queries run, and more. ### Use appropriate data types and avoid casting when possible Casting can be expensive, especially when the underlying models are views and not [materialized](#consider-which-models-to-materialize) as a table. For example, if a timestamp column is actually incorrectly typed as a string, then for timeseries charts, Rill ends up having to iterate across each row to try to infer the timestamp and a lot of time parsing has to occur. Similarly, for incorrectly typed or cast columns that are used in calculations, the calculations will have to be constantly looped through, which can be both inefficient and expensive over time (and simply make everything slower). Similarly, choosing the right data type for each column is also important. Smaller data types, when applicable, consume less memory and can improve query performance. For example, use `INT` instead of `BIGINT` if your data range permits. ### Select the columns you need and avoid `SELECT *` when possible Because most [OLAP databases](/developers/build/connectors/olap) store data in a columnar format, including [DuckDB](/developers/build/connectors/olap/duckdb), selecting only the columns that you need during the modeling phase ensures that DuckDB will only ingest and store the data _it actually needs_ (speeding up model build times and reducing footprint). Furthermore, columnar formats are optimized for analytical queries so by selecting only the columns that you need (instead of a blanket `SELECT *`), this will help to minimize data processing times and improve the query execution speed. ### Consider sorting your data by an appropriate timestamp column Generally speaking, if possible, it is recommended to make sure that your upstream data is relatively well organized and/or sorted by timestamp before being ingested into Rill. This helps to ensure that timeseries queries are efficient when they execute against resulting models and can result in an order of magnitude difference in query performance. This can also help improve the effectiveness of filters by reducing I/O. :::info When to sort vs not to sort? Sorting, especially in DuckDB, _can also be computationally intensive_ and most input data is generally sorted enough (by time). If the data ingested is completely unsorted or sorted by a different, non-timestamp column, it could be worth the computational overhead to sort by a timestamp column (especially if used in a dashboard). If you're unsure, please feel free to [reach out](/contact) and we'd be happy to help you assess the best path forward! ::: ### Use joins efficiently Plan your joins carefully, especially when working with large datasets. Most [OLAP engines](/developers/build/connectors/olap), DuckDB included, will optimize join operations, but ensuring the join keys are well chosen and considering the size of the datasets being joined can reduce processing time. For example, if you're looking to perform a cross or cartesian join across a very wide table, be sure it's necessary as it can otherwise explode the size of your result set. ### Apply filters early and use WHERE clauses wisely When possible, it can be good practice to apply filtering early in your queries with `WHERE` clauses to reduce the amount of data being processed in subsequent steps (or downstream models / queries). This can both help to reduce the amount of data being scanned and, given the columnar nature of most [OLAP engines](/developers/build/connectors/olap), significantly speed up queries. ### Optimize your subqueries to leverage joins or CTEs when possible Subqueries can very often prove to be inefficient and result in suboptimal query execution plans. When possible, it is generally better practice to rewrite subqueries as joins or use Common Table Expressions (CTEs) to make them more readable and potentially more efficient. ### Rather than UNION, consider using UNION ALL when possible Depending on the [OLAP engine](/developers/build/connectors/olap), `UNION` can be a very expensive operation and much more computationally intensive than `UNION ALL`. For example, when using [DuckDB](/developers/build/connectors/olap/duckdb), a `UNION` will require performing full duplicate eliminations _across all columns_ while a `UNION ALL` will simply concatenate the tables together. If a concatenation is sufficient (for the query), this will be both much quicker and significantly less resource intensive for the query to complete. ## OLAP Engines Often, data in external [OLAP engines](/developers/build/connectors/olap) is quite large in size and requires different considerations to improve performance. At Rill, we manage clusters in the 100's of TB so included some tips below based on our experience. ### Data Lifecycle Management One common way to decrease overall data size and improve query performance (by scanning less data) is to rollup your data to higher time grains historically. Typically, this means taking hourly data and rollup up to daily data when the additional level of granularity is no longer necessary for business needs. Databases like Apache Druid have these lifecycle tools built in or reach out to Rill with questions. A couple of considerations when rolling data from lower to higher time grains: - Daily data loses time zone querying as everything is rolled up to a single time zone (usually UTC) - Consider hashed compaction when going from hourly to daily to reduce data size even further - Watch out for rolling up metrics. Some metrics should be summed - but others (like a bid floor or campaign budget) should stay unique and be rolled up as a max ### Dimension Stripping Dimension stripping is another tool to reduce data size by removing high cardinality fields that are not required for analysis. While this can be done upfront in the data set, another practice would be to drop these fields at certain intervals when they no longer add business value. Most frequently, we see a couple decision points where these fields are dropped: - After a day to first week, dropping user level details no longer needed for monitoring - After a week to multiple weeks, dropping "double click" level details that aren't needed for reporting (e.g. the minor release number on an Operating System field) - After a month to months, dropping fields no longer interesting for analysis ### Sampling & Datasketches There are times where you may look at sampling data feeds to trade data accuracy for lower costs and faster query speeds. Sampling involves sending only a percentage of your data, then extrapolating the values to get an estimate. Rill does not recommend sampling your primary KPIs, any records that require a join or are tied to revenue. This filtered data should be decided in random fashion to not skew or bias the results. Please note, tracking uniques is not recommended if you choose to sample. If looking to track uniques, but with smaller datasets and significantly improved performance, you can load unique values (ip addresses, user ids, URLs, etc) with [datasketches](https://datasketches.apache.org). There are multiple types of datasketches supported depending on your engine. At a high level, datasketches use algorithms to approximate unique values. Common use cases for datasketches include count distincts (campaign reach, unique visitors) and quantiles (time spent, frequency). Check out the [Apache Datasketches](https://datasketches.apache.org/docs/Architecture/MajorSketchFamilies.html) site for more details on methodology and use cases. ### Lookups While joins can kill the performance of [OLAP engines](/developers/build/connectors/olap), lookups (key-value pairs) are common to reduce data size and improve query speeds. Lookups can be done during ingestion time (a static lookup to enrich the source data) or at query time (dynamic lookups). **Static Lookups** Static Lookups are lookups that are ingested at processing time. When a record is being processed, if a match is found between the record and lookup's key, the lookup's corresponding value at that moment in time is extracted and carbon-copied into Druid for the records it processed. Static lookups are best suited for: - Dimensions with values that require a historical record for how it has changed over time - Values are never expected to change (leverage Dynamic Lookups if the values are expected to change) - Extremely large lookups (hundreds of thousands or records or >50MB lookup file) to improve query performance Customers typically store lookup values in s3 or GCS and the lookup file is then updated by customers as needed and consumed by ETL logic. **Dynamic Lookups** Since static lookups transform and store the data permanently, any changes to the mapping would require reprocessing the entire data set to ensure consistency. To address the case when values in a lookup are expected to change with time we, developed dynamic lookups. Dynamic Lookups, also known as Query Time Lookups, are lookups that retrieved at query time, as apposed to being used at ingestion time. Benefits of dynamic lookups include: - Historical continuity for dimensions that change frequently without reprocessing the entire data set - Time savings, because there is no data set reprocessing required to complete the update - Dynamic lookups are kept separate from the data set. Thus, any human errors introduced in the lookup do not impact the underlying data set - Ability for users to create new dimension tables from metadata associated with a dimension table. For example, account ownership can change during the course of a quarter. In such cases, a dynamic lookup can be updated on the fly to reflect the most current changes --- ##### "Set up MCP and more!" --- title: "Set up MCP and more!" sidebar_label: "MCP Setup and Customization" hide_table_of_contents: false sidebar_position: 20 tags: - Quickstart - Tutorial --- Connecting your Rill project to your AI Agent has never been easier! ## Prerequisites Before you begin, make sure you have: - An MCP Client (Claude Desktop, ChatGPT Desktop, etc.) - **Access to a [Rill Project](https://ui.rilldata.com/)** For detailed steps, please refer to our documentation on [MCP Servers](/guide/ai/mcp) ## Step 1: Access your Rill Project's AI tab. ![Project AI](/img/explore/mcp/project-ai.png) In your project's AI tab, you'll be able to create a user token (personal access token) and copy the MCP config.json directly in the UI. Simply paste this into your application of choice. :::tip Learn more about user tokens For details on creating, managing, and using personal access tokens, see [User Tokens](/guide/administration/access-tokens/user-tokens. ::: ## Step 2: Add config.json into Client Depending on your client application, paste the config from step 1. - [Anthropic's Claude Desktop Documentation](https://modelcontextprotocol.io/quickstart/user) - [OpenAI's ChatGPT Documentation](https://platform.openai.com/docs/guides/tools-remote-mcp#page-top) - [Cursor Documentation](https://docs.cursor.com/context/model-context-protocol) - [Windsurf Documentation](https://docs.windsurf.com/windsurf/cascade/mcp) ### Working with OpenAI's ChatGPT To use Rill with OpenAI's ChatGPT, you'll need to follow the same steps as above to set up your MCP connection. Once that's done, you can start querying your Rill metrics directly from ChatGPT interfaces. - Import your remote MCP servers directly in [ChatGPT settings](https://chatgpt.com/#settings) or in the desktop application navigate to settings by clicking on your profile picture in the bottom left corner. - Connect your server in the Connectors tab. You may have to add the server as a source with the create button. > Note: You may need at least the `Plus` plan to access connectors. ## Step 3: Start Querying your Agent about your Rill metrics. A below is an example chat with Claude using our MCP server to get information on the commit history for one of our demo projects. ![MCP Main](/img/explore/mcp/mcp-main.gif) ## Step 4: Agent Customization If you watched the whole clip above, you'll see that Claude tries to code a web app to visualize the information it found, but you'll notice it takes time (fast-forwarded in the clip). Instead, you can provide the Agent with context and instructions on how you'd like the Agent to behave. I highly recommend taking a look at our series of videos on [Conversation BI with Claude and Rill](https://www.youtube.com/playlist?list=PL_ZoDsg2yFKjSeetRNHbdI4GzmVn-XbBT).
--- ## Guide --- ### Administration --- #### Access Tokens --- ###### "Access Tokens" --- title: "Access Tokens" description: "Manage access tokens in Rill Cloud" sidebar_label: "Access Tokens" sidebar_position: 0 --- ## Access Tokens - **[User Tokens](/guide/administration/access-tokens/user-tokens)** - Learn how to create and manage personal access tokens - **[Service Tokens](/guide/administration/access-tokens/service-tokens)** - Learn how to create and manage service accounts for production use --- ###### Service Tokens --- title: Service Tokens description: Create and manage service tokens for programmatic access to Rill sidebar_label: Service Tokens sidebar_position: 27 --- Service tokens (also called service accounts) provide programmatic access to Rill Cloud for production systems, scheduled jobs, backend APIs, and other automated workflows. Unlike [user tokens](user-tokens), service tokens persist even if the creating user is removed from the organization. ## Overview Service tokens are designed for: - **Production integrations** - Backend services that need to access Rill APIs - **Scheduled jobs** - Automated reports, data syncs, or ETL processes - **CI/CD pipelines** - Automated testing and deployment workflows - **Custom applications** - Applications that integrate with Rill APIs ## Creating Service Tokens ### Basic Creation Create a service token with an organization-level role: ```bash rill service create my-service --org-role admin ``` Or with a project-level role: ```bash rill service create my-service --project my-project --project-role viewer ``` ### With Custom Attributes Custom attributes allow you to pass metadata that can be used in [security policies](/developers/build/metrics-view/security). This is particularly useful for multi-tenant applications or when you need fine-grained access control. ```bash rill service create my-service \ --org-role admin \ --attributes '{"department":"engineering","region":"us-west","tier":"premium"}' ``` **Example attributes:** - `department` - Organizational department (engineering, sales, finance) - `region` - Geographic region (us-west, eu-central, ap-south) - `customer_id` - Customer identifier for multi-tenant systems - `tier` - Service tier (free, premium, enterprise) - `environment` - Deployment environment (production, staging) ### Complete Example ```bash # Create a service for automated reporting rill service create reporting-service \ --org my-org \ --project analytics-dashboard \ --project-role viewer \ --attributes '{"purpose":"reporting","schedule":"daily","region":"us-east"}' ``` The command will output: ``` Created service "reporting-service" in org "my-org". Access token: rill_svc_[TOKEN_HERE] ``` :::warning Store tokens securely Service tokens have powerful permissions. Store them securely in a secrets manager (AWS Secrets Manager, HashiCorp Vault, etc.) and never commit them to version control. ::: ## Roles and Permissions Service tokens can be assigned roles at both the organization and project levels. For more details on using attributes with security policies, see the [security policies](/developers/build/metrics-view/security) documentation. ## Managing Service Tokens ### Listing Service Tokens View all service tokens in your organization: ```bash rill service list --org my-org ``` Output: ``` NAME ORG ROLE PROJECT ROLES ATTRIBUTES reporting-service - analytics-dashboard {"purpose":"reporting"} global-admin admin - {} dashboard-viewer - sales-analytics {} ``` ### Editing Service Tokens Update a service token's name or attributes: ```bash # Change the name rill service edit my-service --new-name renamed-service # Update attributes rill service edit my-service \ --attributes '{"department":"finance","region":"eu-west"}' # Clear attributes rill service edit my-service --attributes '{}' ``` ### Managing Roles Update organization role: ```bash rill service set-role my-service --org-role editor ``` Add or update project role: ```bash rill service set-role my-service \ --project analytics-dashboard \ --project-role admin ``` Remove project role: ```bash rill service set-role my-service \ --project analytics-dashboard \ --remove ``` ### Viewing Service Details Get detailed information about a service: ```bash rill service show my-service --org my-org ``` ### Revoking Access Delete a service token to immediately revoke its access: ```bash rill service delete my-service --org my-org ``` ## Issuing Ephemeral Tokens Service tokens can issue short-lived ephemeral tokens for end users. This is useful for: - Embedded dashboards - Temporary API access - User-specific data access ### With User Email For simple cases, you can just provide the end user's email: ```bash curl -X POST https://api.rilldata.com/v1/orgs//projects//credentials \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "user_email": "user@example.com", "ttl_seconds": 3600 }' ``` The response contains a short-lived JWT token that can be used to access Rill APIs on behalf of the user. ### With Custom User Attributes For advanced cases, you can pass custom user attributes: ```bash curl -X POST https://api.rilldata.com/v1/orgs//projects//credentials \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "attributes": { "email": "user@example.com", "department": "sales", "region": "us-west", "customer_id": "acme-corp" }, "ttl_seconds": 3600 }' ``` The custom attributes can be referenced in the project's [security policies](/developers/build/metrics-view/security). When using custom attributes, make sure that you pass a value for every attribute referenced in the project's security policies. ### User identity The `user_email`, `attributes`, and `external_user_id` parameters serve two purposes in ephemeral tokens: - **User attributes for security policies:** `user_email` and `attributes` determine which attributes (such as `email`, `domain`, and `admin`) are available for use in [security policies](/developers/build/metrics-view/security). - **Per-user state:** `external_user_id` establishes a stable user identity that isolates per-user features such as AI chat history. Without a user identity, these features are not available. Only one of `user_email` or `attributes` can be provided for a given token. The `external_user_id` parameter can optionally be combined with either of them. Here is how each parameter works: - `user_email`: Looks up the user in Rill Cloud by email and populates their standard attributes. If no matching user is found, it generates limited attributes with only the fields `email`, `domain` and `admin` (set to `false`). Does not enable per-user state on its own; combine with `external_user_id` to enable per-user state. - `attributes`: Passes the provided attributes through directly. Make sure to include all attributes referenced in your security policies (e.g. `email`, `domain`, `admin`, or custom attributes like `tenant_id`). Does not enable per-user state on its own; combine with `external_user_id` to enable per-user state. - `external_user_id`: Any stable identifier for an external end user. This is usually the user's ID in your own database. Setting it enables per-user state such as AI chat history. Can be combined with `user_email` or `attributes`. --- ###### User Tokens --- title: User Tokens description: Create and manage personal access tokens for development and scripting sidebar_label: User Tokens sidebar_position: 26 --- User tokens (also called personal access tokens or PATs) provide programmatic access to Rill Cloud tied to your personal user account. They inherit your user permissions and are ideal for local development, scripting, and integrations like MCP (Model Context Protocol). ## Overview User tokens are designed for: - **Local development** - Testing and developing with Rill APIs from your machine - **Personal scripts** - Automating personal workflows and data analysis - **AI integrations** - Connecting AI assistants (Claude Desktop, ChatGPT) via MCP - **Experimentation** - Trying out Rill APIs without production concerns - **CLI authentication** - Authenticating Rill CLI commands ## Creating User Tokens ### Basic Creation Create a user token with the CLI: ```bash rill token issue ``` You'll be prompted to provide a display name and optionally set an expiration time. ### With Display Name Give your token a descriptive name: ```bash rill token issue --display-name "Local Development" ``` ### With Expiration Set an expiration time in minutes: ```bash # Expires in 24 hours (1440 minutes) rill token issue --display-name "MCP Token" --ttl-minutes 1440 # Expires in 7 days (10080 minutes) rill token issue --display-name "Testing Token" --ttl-minutes 10080 ``` :::warning Store tokens securely User tokens provide access to your data with your permissions. Store them securely and never commit them to version control. Treat them like passwords. ::: ## Token Permissions User tokens inherit your personal permissions from your user account: - **Organization permissions** - Your role in the organization (admin, editor, viewer, guest) - **Project permissions** - Your role in specific projects (admin, editor, viewer) - **Security policies** - Applied based on your user attributes (email, domain, groups) For more details on roles, see [Roles and Permissions](/guide/administration/users-and-access/roles-permissions). ## Managing User Tokens ### Listing Your Tokens View all your active user tokens: ```bash rill token list ``` Output: ``` ID DISPLAY NAME CREATED EXPIRES rill_usr_abc123... Local Development 2024-01-15 10:30 Never rill_usr_def456... MCP Token 2024-01-16 14:20 2024-01-17 14:20 rill_usr_ghi789... Testing Token 2024-01-14 09:00 2024-01-21 09:00 ``` ### Revoking Tokens Revoke a token by its ID: ```bash rill token revoke ``` Or revoke by display name: ```bash rill token revoke --display-name "Local Development" ``` :::tip Token rotation For security, periodically rotate your tokens by creating new ones and revoking old ones. This is especially important for long-lived tokens. ::: ## Using User Tokens ### With Custom APIs User tokens can be used to authenticate requests to Rill's custom APIs: ```bash curl https://api.rilldata.com/v1/organizations//projects//runtime/api/ \ -H "Authorization: Bearer " ``` For more details, see [Calling Custom APIs](/developers/build/custom-apis/calling). --- #### Project Settings --- ###### "GitHub Integration" --- title: "GitHub Integration" description: "Connect and manage GitHub repository integration for your Rill Cloud projects" sidebar_label: "GitHub Integration" sidebar_position: 21 --- # GitHub Integration Each Rill Cloud project can be connected to a single GitHub repository. This enables continuous deployment—your project automatically redeploys whenever you push changes to the connected repository. ## Connecting to a GitHub Repository On first deployment, if you've deployed via the UI, your project will not be connected to a GitHub repository. You will need to manually connect it: 1. Navigate to your project's **Settings** page 2. Select **Connect to GitHub** 3. Follow the steps to connect your repository ![Select Repo](/img/deploy/existing-project/select-repo.png) ## Modifying GitHub Repository Connection In some cases, you will need to change the repository that your project is synced to: 1. Navigate to your project's **Settings** page 2. Select the dropdown next to the repository name 3. Choose **Disconnect** to remove the current connection ![Disconnect GitHub](/img/manage/project-management/disconnect-github.png) This action has no effect on your current deployment and will not require a source re-ingest. After disconnecting, you can follow the same steps as [connecting to a GitHub repository](#connecting-to-a-github-repository) to reconnect your project to a new repository. ## Deploying from a Branch Other Than `main` By default, Rill Cloud deploys from the `main` branch of your connected repository. You can change this to deploy from a different branch. ### Via Rill Cloud UI If you have already [set up your connection to GitHub](/developers/deploy/deploy-dashboard/#syncing-your-github-repository), you can edit the branch from the project settings: ![Main Branch](/img/manage/project-management/main-branch.png) ### Via CLI To change the branch via the CLI, run: ```bash rill project edit ``` This will open an interactive prompt where you can update the branch name and other project properties. ## Automatic Deployment Your project on Rill Cloud will automatically redeploy every time you push changes to the connected GitHub repository. This ensures your dashboards always reflect the latest version of your project code. :::tip Manual Refresh To manually refresh data sources without pushing code changes (or redeploying your project), use: ```bash rill project refresh [--source/model] (source_name or model_name) ``` ::: --- ###### "Managing Projects in Rill Cloud" --- title: "Managing Projects in Rill Cloud" description: Basic management for projects sidebar_position: 20 --- Once an organization is created, you can populate it with projects. Each project has its own set of sources, models, metrics views, and explore dashboards. A defined object in a project cannot be shared to another project. :::info GitHub Integration Projects can be connected to a GitHub repository for continuous deployment. See [GitHub Integration](/guide/administration/project-settings/github-integration) for details on connecting and managing repository connections. ::: ## Project ![Project Home]() A project is a single deployed instance from Rill Developer (or what we refer to as a Rill project). Once you have deployed a project to Rill Cloud, you can make changes to it via the CLI or via Rill Cloud. ## Checking deployment status After deploying to Rill Cloud, you can navigate to the status page to monitor your project's health. The status page provides an overview of your deployment details, resource statuses, tables, and errors, with dedicated tabs for deeper inspection. ![Status]() ### Resources The Resources tab lists all project resources (sources, models, metrics views, explores, etc.) with their current reconciliation status. You can search, filter by resource type or status (OK, Error, Warn), and trigger a full refresh of all sources and models. Parse errors are also surfaced here. You can filter by resource type and status to get quick information about the status of your project. ![Resources]() ### Tables The Tables tab provides visibility into the tables in your OLAP database. Tables are split into two sections: **Models** (tables managed by Rill) and **External Tables** (tables that exist in the OLAP engine but are not managed by Rill). You can search and filter by type (table or view), view partition details for incremental models, and trigger model refreshes. ![Tables]() ### Logs The Logs tab streams live runtime logs from your deployment via a real-time connection. You can search log messages and filter by level (Debug, Info, Warn, Error) to quickly diagnose issues. ![Logs]() ### Managing Project settings You can also manage project objects on the settings page, including public URLs (created in an Explore dashboard) and environment variables. For more information on managing variables, see [variables and credentials](/guide/administration/project-settings/variables-and-credentials). ![Project Settings]() ## Managing a Rill project from the CLI Managing a project includes the project itself and all components or resources that belong to the project. Via the CLI, you can make changes to the project's properties such as description, public access, etc. Run `rill project -h` for an overview of available commands. ### Refreshing the deployment If your project is connected to a GitHub repository, it will automatically redeploy every time you push changes. To manually refresh data sources without pushing code changes (or redeploying your project), run the following command: ``` rill project refresh [--source/model] (source_name or model_name) [--local] ``` ### Checking deployment status In case you need to check the project status via the CLI, you can run the following: ``` rill project status ``` ## Make a project public Projects on Rill Cloud are private by default. To make a project's dashboards publicly accessible without authentication, run: ``` rill project edit --public=true ``` :::caution Avoid Sharing Private Data **Warning**: If you make a project public, make sure it does not expose any confidential data. ::: --- ###### Environment Variables and Credentials in Rill Cloud --- title: Environment Variables and Credentials in Rill Cloud sidebar_label: "Variables and Credentials" sidebar_position: 50 --- The credentials in deployed Rill Cloud projects can be managed on the Settings page or via the CLI. If you have yet to deploy your credentials, please follow the steps in our [deploy credentials page](/developers/deploy/deploy-credentials). ## Modifying Variables and Credentials via the Settings Page Upon deployment via Rill Developer, if you have populated your .env file, the contents will be visible as seen below. If there are no environment variables defined, please run `rill env push` from your local CLI and Rill will automatically push the credentials in your project's `.env` file to Rill Cloud. If you'd like to manually add the credentials, please see [our naming convention](#credentials-naming-schema) to get started. ![Environment Variable UI](/img/tutorials/admin/env-var-ui.png) ### Adding and Editing Environment Variables / Importing a `.env` file Once your environment variables are added to Rill Cloud, they can be modified as needed. ![Add Variable](/img/manage/var-and-creds/add-variable.png) :::tip Can't find the .env file? By default, the hidden files will not be visible in the finder window. In order to view hidden files, you will need to enable "show hidden files". Keyboard shortcut: Command + Shift + . ::: ## Pushing and pulling credentials to / from Rill Cloud via the CLI If you'd prefer to use the CLI to manage your credentials, this can be done by running `rill env pull` to pull your deployed Rill Cloud project's variables locally, or `rill env push` to update your Rill Cloud project's variables. :::tip Avoid committing sensitive information to Git It's never a good idea to commit sensitive information to Git and goes against security best practices. Similar to credentials, if there are sensitive variables that you don't want to commit publicly to your `rill.yaml` configuration file (and thus potentially accessible by others), it's recommended to set them in your `.env` file directly and/or use `rill env set` via the CLI (and then optionally push / pull them as necessary). ::: ### `rill env push` As a project admin, you can run `rill env push` from your local CLI, which will update your Rill project with the contents of your *`/.env`* file. - Rill Cloud will use the specified credentials and variables in this `.env` file for the deployed project. - Other users will also be able to use `rill env pull` to retrieve these defined credentials for local use (with Rill Developer). :::warning Overriding Cloud credentials If a credential and/or variable has already been configured in Rill Cloud, Rill will warn you about overriding if you attempt to push a new value in your `.env` file. This is because overriding credentials can impact your deployed project and/or other users (if they pull these credentials locally). ::: ### `rill env pull` For projects that have been deployed to Rill Cloud, an added benefit of our Rill Developer-Cloud architecture is that credentials that have been configured can be pulled locally for easier reuse (instead of having to manually reconfigure these credentials in Rill Developer). To do this, you can run `rill env pull` from your project's root directory to retrieve the latest credentials (after cloning the project's git repository to your local environment). ```bash rill env pull Updated .env file with cloud credentials from project "". ``` :::info Overriding local credentials Please note when you run `rill env pull`, Rill will *automatically override any existing credentials or variables* that have been configured in your project's `.env` file if there is a match in the key name. This may result in unexpected behavior if you are using different credentials locally. ::: ### Credentials Naming Schema Connector credentials use a standardized naming convention. Generic credentials (shared across connectors like cloud providers) use standard names without a driver prefix, while driver-specific credentials use the `DRIVER_PROPERTY` format. Please see below for each source and its required properties. If you have any questions or need specifics, [contact us](/contact)! :::note Legacy Naming Convention Older projects may use the `connector..` syntax (e.g., `connector.druid.dsn`, `connector.clickhouse.dsn`). This format is still supported for backwards compatibility. :::
| **Source Name** | Property | Environment Variable | | :-------------: | :------------------------------: | :---------------------------------- | | **GCS** | `google_application_credentials` | `GOOGLE_APPLICATION_CREDENTIALS` | | | `key_id` | `GCP_ACCESS_KEY_ID` | | | `secret` | `GCP_SECRET_ACCESS_KEY` | | **AWS S3** | `aws_access_key_id` | `AWS_ACCESS_KEY_ID` | | | `aws_secret_access_key` | `AWS_SECRET_ACCESS_KEY` | | **Azure** | `azure_storage_account` | `AZURE_STORAGE_ACCOUNT` | | | `azure_storage_key` | `AZURE_STORAGE_KEY` | | | `azure_storage_connection_string`| `AZURE_STORAGE_CONNECTION_STRING` | | | `azure_storage_sas_token` | `AZURE_STORAGE_SAS_TOKEN` | | **Big Query** | `google_application_credentials` | `GOOGLE_APPLICATION_CREDENTIALS` | | **Snowflake** | `dsn` | `SNOWFLAKE_DSN` | | | `password` | `SNOWFLAKE_PASSWORD` | | **ClickHouse** | `host` | `CLICKHOUSE_HOST` | | | `port` | `CLICKHOUSE_PORT` | | | `username` | `CLICKHOUSE_USERNAME` | | | `password` | `CLICKHOUSE_PASSWORD` | | | `ssl` | `CLICKHOUSE_SSL` | | | `database` | `CLICKHOUSE_DATABASE` | | | `dsn` | `CLICKHOUSE_DSN` |
--- #### Users And Access --- ###### "Users and Access" --- title: "Users and Access" description: "Manage users, groups, and access controls in Rill Cloud" sidebar_label: "Users" sidebar_position: 0 --- # Users and Access Manage users, user groups, and access controls for your Rill Cloud projects. ## User Management - **[User Management](/guide/administration/users-and-access/user-management)** - Invite team members, assign roles, and manage user access to your projects - **[User Group Management](/guide/administration/users-and-access/usergroup-management)** - Create and manage user groups for easier permission management - **[Roles and Permissions](/guide/administration/users-and-access/roles-permissions)** - Understand and configure user roles and permission levels --- ###### Roles and Permissions --- title: Roles and Permissions description: Learn more about roles and permissions for organizations and projects in Rill Cloud sidebar_label: Roles and Permissions sidebar_position: 30 --- Access permissions in Rill Cloud are organized into roles at the organization and project level. ## Role inheritance Organization and project level roles are managed separately, but are connected in several ways: 1. By default, all organization members (but not guests) are added to new projects through a user group membership with the **viewer** role. You can manually remove this relationship in the project's member settings. 2. If you grant a project level role to someone who is not a member of the parent organization, they will automatically be added to the organization with the **guest** role. 3. Removing an organization member or guest automatically also removes them from all projects in the organization. 4. Organization admins implicitly have admin privileges on all projects in the organization. ## Organization-level permissions There are four roles available at the organization-level: **Admin**, **Editor**, **Viewer** and **Guest**. | Permission | Description | Admin | Editor | Viewer | Guest | | :------------------- | :--------------------------------------------------- | ----: | -----: | -----: | ----: | | `read_org` | View basic info about the organization | ✔ | ✔ | ✔ | ✔ | | `manage_org` | Change organization settings, manage service accounts and service tokens | ✔ | | | | | `read_projects` | View projects you're a member of in the organization | ✔ | ✔ | ✔ | ✔ | | `create_projects` | Create new projects in the organization | ✔ | ✔ | | | | `manage_projects` | Act as an admin on all projects in the organization | ✔ | | | | | `read_org_members` | View members of the organization | ✔ | ✔ | | | | `manage_org_members` | Add and edit non-admin organization members | ✔ | ✔ | | | | `manage_org_admins` | Add and edit organization admins | ✔ | | | | ## Project-level permissions There are three roles available at the project-level: **Admin**, **Editor**, and **Viewer**. | Permission | Description | Admin | Editor | Viewer | | :----------------------------- | :--------------------------------------------------------- | ----: | -----: | -----: | | `read_project` | View basic info about the project | ✔ | ✔ | ✔ | | `manage_project` | Change project settings | ✔ | | | | `read_prod` | View dashboards deployed from the production (main) branch | ✔ | ✔ | ✔ | | `read_prod_status` | View logs for the production deployment | ✔ | ✔ | | | `manage_prod` | Trigger actions on the production deployment | ✔ | | | | `read_dev` | View dev deployments | ✔ | ✔ | ✔ | | `read_dev_status` | View logs for dev deployments | ✔ | ✔ | | | `manage_dev` | Trigger actions on dev deployments | ✔ | ✔ | | | `read_provisioner_resources` | View managed resources for the project | ✔ | ✔ | | | `manage_provisioner_resources` | Add or remove managed resources for the project | ✔ | | | | `read_project_members` | View members of the project | ✔ | ✔ | | | `manage_project_members` | Add and edit non-admin project members | ✔ | ✔ | | | `manage_project_admins` | Add and edit project admins | ✔ | | | | `create_magic_auth_tokens` | Create shareable URLs | ✔ | ✔ | | | `manage_magic_auth_tokens` | Remove shareable URLs created by others | ✔ | ✔ | | | `create_reports` | Create and edit new scheduled reports | ✔ | ✔ | ✔ | | `manage_reports` | Edit and change scheduled reports created by others | ✔ | | | | `create_alerts` | Create and edit new alerts | ✔ | ✔ | ✔ | | `manage_alerts` | Edit and change alerts created by others | ✔ | | | | `create_bookmarks` | Create and edit new bookmarks | ✔ | ✔ | ✔ | | `manage_bookmarks` | Edit and change bookmarks created by others | ✔ | | | --- ###### User Management --- title: User Management sidebar_label: User Management sidebar_position: 21 --- In Rill Cloud, there are several levels of user management: 1. **Organization members and guests**: a user must have an organization-level role to access anything in an organization. 2. **Project roles**: organization members can have direct roles on a project. For example, the creator of a project automatically becomes an *admin* on the project. 3. **User group members and roles**: organization members can belong to user groups. When a user group has a role on a project, the role automatically propagates to all members of the user group. By default, a system-managed group consisting of all organization members (but not guests) is added to projects with the *viewer* role. :::info For a detailed breakdown of access permissions at different levels, see the [Roles and Permissions](roles-permissions) page! ::: :::tip More Rill Cloud workflows coming We have begun releasing new features around user management via Rill Cloud. If you'd like to learn more, please feel free to [reach out](/contact)! ::: ## Organization Users At the organization-level, users can have one of the following roles: - **Admins** can deploy and manage projects and manage billing. - **Editors** can manage non-admin members of the organization by granting and revoking access. - **Viewers** can browse the projects they have been given direct access to or indirect access to through user group memberships. - **Guests** are similar to viewers, but not part of the "all members" user group, which is added to projects by default. For a detailed list of permissions, please refer to the [Roles and Permissions](roles-permissions). ### How to Add an Organization User Admins can be invited to an organization from the *users* page, or via the CLI. #### From Rill Cloud User page From the organization page, you can manage users under the *Users* tab. Adding users from this page will add the user to the organization. ![Add User Cloud](/img/manage/user-management/add-user-cloud.png) #### Via the CLI ``` rill user add ? Select role [Use arrows to move, type to filter] > admin editor viewer guest ``` You will then be prompted for details about the user. :::tip Check your inbox (or spam) If you add a user who has not yet signed up for Rill, they will receive an email inviting them to sign up and join. ::: ### How to add a user to a user group If you have already set up a user group, instead of setting up users individually, you can add them to a user group. ``` rill user add --group ? Enter email User "" added to the user group "" ``` ### Automatically add members by email domain You can automatically add users to your organization by their email domain during the deployment process and in the Organization settings page. This is limited to the same domain as your user email. If you want to whitelist other domains, contact us! ![Rill Org Settings](/img/manage/user-management/rill-org-settings.png) For example, if you whitelist `yourdomain.com`, new and existing users with an email address ending in `@yourdomain.com` will automatically be added to your organization. :::info Interested in whitelisting a different domain? The feature currently requires manual action by a support representative at Rill. Just [reach out here](https://www.rilldata.com/contact) and ask us to whitelist your domain. ::: ## Project Users Access to projects is managed at the individual project level, subject to some notable rules: 1. By default, all organization members (but not guests) are added to new projects through a user group membership with the **viewer** role. You can manually remove this relationship in the project's member settings. 2. If you grant a project level role to someone who is not a member of the parent organization, they will automatically be added to the organization with the **guest** role. 3. Removing an organization member or guest automatically also removes them from all projects in the organization. 4. Organization admins implicitly have admin privileges on all projects in the organization. ### Project roles Project users can have one of three roles: - **Viewers** can browse the specific project, view dashboards, and setup alerts and reports - **Editors** can add and remove non-admin project members - **Admins** can manage the project by updating the project files, configuring environment variables, and accessing the status page For a detailed list of permissions, please refer to the [Roles and Permissions](roles-permissions). ### How to add a Project User There are a few ways to add a project user to Rill Cloud. 1. Admin invites user to the project using `Share`. 2. User requests access via the project URL: `https://ui.rilldata.com/` 3. Administrator invites user via the CLI with `--project ` flag. ### Admin invites user from Rill Cloud From the project's splash screen, select Share and type the email address(es) along with the type of permissions. ![Share Project](/img/manage/user-management/share-project.png) Once sent, your invited users will receive this email and will need to accept it to view the project. ![Email Invite](/img/manage/user-management/email-invite.png) ### User requests access via URL Alternatively, if you provide the project URL to your users, they can request access to the group admin. Users can request access via the page below: ![Request Access](/img/manage/user-management/request-access.png) The admin will receive an email to allow access, and can set the permission after accepting the request via the UI. ![Admin Reply](/img/manage/user-management/admin-reply.png) --- ### Admin invites user via the CLI To add a member to a project, run the following command: ``` rill user add --project [PROJECT NAME] ``` You will then be prompted for details about the user. HINT: Run `rill project list` to show available projects. If you add a user who has not yet signed up for Rill, they will receive an email inviting them to join. #### Other actions Run `rill user --help` to show commands for listing members or changing access. ## User Groups Another way to manage access is via user groups. You use the Rill CLI to create user groups and add members to them. Once you have created a user group, you can assign roles to it at the organization or project level, similar to how you assign roles to individual users. User groups are scoped to an organization. They cannot be created only for a single project. Only organization members can be added to user groups, and user groups can only be added to projects within the organization where they were created. For more information on setting up user groups, see [user group permissions](usergroup-management). ### How to add a user to a user group There are two ways to add a user to a user group. 1. Admin adds them via Rill Cloud (Coming soon!) 2. Admin adds them via the CLI #### Add a user to a user group in Rill Cloud ### Managing Users via Rill Cloud There are two ways that a user can get access to Rill Cloud. **Organization invites from Admin** From the Users page on the Organization page, you can invite a user to the organization. Please note that organization viewers have access to view all projects. ![Org User Management](/img/tutorials/admin/org-user-management.png) **Project level access requests** Please refer to the documentation on how a user can request access to a project, or how an admin can invite a user to the project. #### Add a user to a user group with the Rill CLI ``` rill user add --group ``` You will then be prompted for details about the user. :::note If the user you are trying to add is not part of the organization yet, the CLI will prompt you to add them to the organization first then proceed to adding them to a group. ::: To see the current members of a group: ``` rill user list --group ``` To find the current user group roles, with project flag if looking for specific project's role: ``` rill usergroup list [--project my_project_name] ``` ## Which privilege wins? Rill uses a logical **OR** operand to define the winning privilege. In other words, if any direct role or indirect role through a user group allows a user to take an action, the action will succeed. ## Logging into Rill Cloud In order to access a deployed project and/or view a shared dashboard, users will need to first log in to [Rill Cloud](https://ui.rilldata.com/). When you first navigate to https://ui.rilldata.com/, you will see a few different options to log in, including: - Google SSO - Microsoft SSO - Email _(basic auth)_ :::info SAML Authentication Rill Cloud **does** support SAML authentication for our enterprise customers. If this is a requirement, [please get in contact](/contact) with us and we can discuss appropriate next steps to help you with your setup. ::: If this is the first time you are accessing Rill Cloud, you will want to sign up instead. ![Sign Up](/img/manage/user-management/sign-up.png) :::tip Signing up with basic auth If you are unsure which option to select, select `Continue with Email` and set up basic authentication (email address / password). ::: Afterwards, you should receive an email verification to complete the sign-up process. ![Verification Email](/img/manage/user-management/verification-email.png) You should now be authenticated with Rill Cloud and be able to sign in directly going forward! --- ###### User Group Permissions --- title: User Group Permissions sidebar_label: User Groups sidebar_position: 24 --- Creating user groups in Rill allows administrators to easily grant permissions to multiple projects at different access levels. It is possible to mix and match viewer and administrator permissions in a single group, and users can be part of multiple groups. However, please keep in mind that the higher permission will be applied. ## Managing User Group Permissions There are two ways to set up user groups in Rill. 1. Administer them via Rill Cloud 2. Administer them via CLI ### How to Manage User Groups in Rill Cloud From the organization page, you can manage user groups under the Users tab. Adding user groups from this page will add the user group to the organization. You can then add users to a user group to inherit the group [permissions](/guide/administration/users-and-access/roles-permissions). ![User Group Management](/img/manage/user-management/usergroup-management.png) ### How to Manage User Groups via the CLI ``` rill usergroup Manage user groups Usage: rill usergroup [command] Available Commands: list List groups show Show group create Create a group rename Rename a group edit Edit a group delete Delete a group add Add a group to a project or organization set-role Change a group's role on a project or organization remove Remove a group's role on a project or organization ``` ## Creating the User Group You can create a new user group by running the following and following the CLI instructions: ``` rill usergroup create ``` You will be prompted for the new user group name. ### Adding permissions to the group Next, you will need to add the roles and access to the user group. ``` rill usergroup add --project --group string User group --org string Organization (default "Rill_Learn") --project string Project --role string Role of the user group (options: admin, editor, viewer) ``` You will be prompted for the role and the name of the group you are editing. If you want to specify a specific project, please use the --project flag. If no project flag is defined, you will be setting permissions at the organization level. If you have any questions on permission levels, please review the [Roles and Permissions page](roles-permissions). ### Add a member to the group To add a member to the user group, run the following command: ``` rill user add --group ``` You will be prompted for the email address for the user. Once added, you can confirm the user group by running the following command: ``` rill user list --group ``` --- ##### "Settings" --- title: "Settings" description: "Configure organization and project settings in Rill Cloud" sidebar_label: "Settings" sidebar_position: 0 --- # Settings Configure your organization and project settings in Rill Cloud, including organization management, project configuration, and service tokens. ## Organization Settings **[Organization Settings](/guide/administration/organization-settings)** provide centralized management for your organization. You can: - Manage your organization settings, billing, branding, and configuration - View and manage projects within your organization - Configure organization-wide settings and preferences ## Project Settings **[Project Settings](/guide/administration/project-settings)** offer comprehensive project management capabilities. Manage: - Project configuration and deployment settings - Project-level preferences and defaults - Project resources and configurations ## GitHub Integration **[GitHub Integration](/guide/administration/project-settings/github-integration)** connects your projects to GitHub repositories. You can: - Connect and manage GitHub repository integration - Enable automatic deployments from your repositories - Configure branch-based deployment workflows ## Variables & Credentials **[Variables & Credentials](/guide/administration/project-settings/variables-and-credentials)** help you manage sensitive information securely. Store: - Sensitive credentials and API keys securely - Environment-specific variables for different deployments - Configuration values that vary across environments ## User Management **[User Management](/guide/administration/users-and-access/user-management)** enables you to control access to your projects. You can: - Invite team members and manage user access to your projects - Assign roles and permissions to users - Control who can view, edit, or manage your projects ## User Group Management **[User Group Management](/guide/administration/users-and-access/usergroup-management)** simplifies permission management through groups. Create: - User groups for easier permission management - Group-based access controls - Streamlined user organization ## Roles and Permissions **[Roles and Permissions](/guide/administration/users-and-access/roles-permissions)** define what users can do in your organization. Understand: - Available user roles and their capabilities - How to configure user roles and permission levels - Best practices for access control ## Access Tokens **[User Tokens](/guide/administration/access-tokens)** provide programmatic access to Rill Cloud. Manage: - Personal access tokens for API access - Token permissions and scopes - Token lifecycle and security ## Service Tokens **[Service Tokens](/guide/administration/access-tokens/service-tokens)** enable automated access for services and applications. Create: - Service tokens for programmatic access to Rill Cloud - Long-lived tokens for automated workflows - Secure authentication for integrations --- ##### "Managing Organizations in Rill Cloud" --- title: "Managing Organizations in Rill Cloud" description: Basic management for organizations sidebar_label: "Organization Settings" sidebar_position: 00 --- Before a project can be deployed onto Rill Cloud, an organization must be created. If you are deploying via the UI, this will automatically be done for you. As an administrator, you can also create, edit, and delete organizations from the CLI. From the organization page, you will be able to view your projects, users, and overall settings. ## Organization ![Rill Org](/img/manage/project-management/rill-org.png) An organization in Rill is the parent management object and encompasses how your team interfaces with Rill Cloud. Organizations are designed to hold the different components of your Rill project. Organizations consist of projects that each have their own sources, models, metrics views, dashboards, user management, and general settings. ### Organization Hierarchy ``` Organization ├── Users & Groups │ ├── Admins │ ├── Editors │ ├── Members │ └── Guests ├── Projects │ ├── Project A │ └── Resources │ ├── Data Sources │ ├── Models │ ├── Metrics Views │ ├── Dashboards │ └── APIs │ └── Project B │ ... ``` ### User Management From the Users page, you can view and manage users within your organization. Note that users with specific project access will not appear on this page and can be managed via each individual project. For more information, please review our [User Management documentation](/guide/administration/users-and-access/user-management). ### Org Settings via Rill Cloud On the organization settings page, depending on your plan type, you can view the general information, billing, and current usage. The Billing tab is only available for those on a `Team Plan`. You can use this page to add or modify your current payment type. For more information, please review our [Billing Information documentation](/developers/other/plans). ![Rill Org Settings](/img/manage/project-management/rill-org-settings.png) ### Logo and Favicon Along with general organization settings, you are also able to modify the Logo in the top right corner as well as the Favicon in the browser. Simply upload a supported file into the project and see the icon change! ## Managing an Organization from the CLI Similar to the UI, if you want to make any changes to the organization via the CLI, this is possible using the following: ``` rill org Manage organisations Usage: rill org [command] Available Commands: create Create organization edit Edit organization details switch Switch to other organization list List all organizations delete Delete organization rename Rename organization Global Flags: --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` :::tip Access to Rill can be granted on the [organization level](/guide/administration/users-and-access/user-management#how-to-add-an-organization-user), [project level](/guide/administration/users-and-access/user-management#how-to-add-a-project-user), and [user group level](/guide/administration/users-and-access/user-management#how-to-add-a-user-to-a-user-group). ::: --- ### Ai --- ##### "Contextual AI Chat" --- title: "Contextual AI Chat" description: Chat with your data using natural language in Rill Cloud sidebar_label: "AI Chat" sidebar_position: 01 --- ## Overview AI Chat in Rill Cloud allows you to have natural language conversations with your data directly in your browser. Instead of building queries or navigating through dashboards, simply ask questions using everyday conversational language and get instant insights backed by your metrics views—complete with **interactive charts and visualizations** that render right in the chat interface, plus **direct links** to your existing dashboards for deeper exploration. AI Chat is powered by [Rill's Model Context Protocol (MCP)](/guide/ai/mcp) integration, which ensures that responses are accurate, governed, and consistent with the metrics displayed in your dashboards. By querying data with **predefined measures and dimensions**, you can trust that the answers you receive are as reliable as the data in your Rill dashboards. **What makes AI Chat different?** Every response includes direct links to your Explore dashboards with filters pre-applied, so you can always **verify where the numbers came from**. No black box—just transparent, trustworthy analytics. ![Project Chat](/img/explore/chat/project-chat.png) ## How It Works AI Chat uses the same [Rill MCP Server](/guide/ai/mcp) technology that powers external AI integrations with tools like Claude Desktop. This means: - **Fast!** - Get instant answers powered by Rill's optimized query engine - **Accurate Responses** - The Agent only queries [metrics views](/developers/build/metrics-view) you've already defined, ensuring accuracy and consistency - **Secure Data Access** - Respects your [project's access](/developers/build/metrics-view/security) controls and user permissions ## Accessing AI Chat ### Access AI Chat from Project Home 1. Navigate to your [Rill Cloud](https://ui.rilldata.com) project home page 2. Click on the **AI** tab in the project navigation 3. Start typing your question in the chat interface ### Access AI Chat from a Dashboard You can also access AI Chat directly while exploring a dashboard, making it easy to ask questions about what you're currently viewing: 1. While viewing any [Explore dashboard](/guide/dashboards/explore), look for the **AI Chat icon** in the top navigation bar 2. Click the AI Chat icon to open the chat panel alongside your dashboard 3. Ask questions about the data you're currently viewing ![Dashboard Chat](/img/explore/chat/dashboard-chat.png) When you open AI Chat from a dashboard, the AI is automatically aware of: - **Current dashboard context** - The metrics view you're viewing - **Applied filters** - Any dimension or measure filters you've set - **Time range** - The time period currently selected - **Comparison settings** - Any active time comparisons This context-aware functionality means you can ask questions like: - "Why did this measure spike?" (referring to what's visible on screen) - "What's driving this change?" (analyzing the current time period) - "Show me more details about these results" (diving deeper into filtered data) :::tip Context-Aware Queries Opening AI Chat from within a dashboard allows for more natural, context-aware questions. The AI understands what you're looking at, so you don't need to repeat filters or time ranges in your questions. ::: ## Understanding Responses AI Chat provides rich, multi-layered responses to help you understand your data quickly while maintaining easy access to deeper exploration: ### What's Included in Responses 1. **Summary** - A concise answer to your question with key findings and insights 2. **Interactive Visualizations** - Charts and graphs that help you see patterns at a glance. The AI automatically chooses the most appropriate visualization based on your data and question, including: - **Line charts** - Show trends and changes over time - **Area charts** - Highlight cumulative trends and patterns - **Bar charts** - Compare values across categories or dimensions - **Stacked bar charts** - Show part-to-whole relationships across categories - **Donut charts** - Display proportional breakdowns of a total - **Combo charts** - Combine multiple measures with different scales - **Heatmaps** - Visualize distribution across two dimensions 3. **Detailed Results** - Tables or lists with specific numbers and breakdowns 4. **Dashboard Links** - Direct links to your existing [Explore dashboards](/guide/dashboards/explore) with filters and time ranges pre-applied based on your question 5. **Suggested Follow-ups** - Related questions or areas to investigate further ### Visual Components AI Chat automatically generates interactive visualizations to complement answers when appropriate. The AI intelligently selects chart types based on your data structure, question context, and visualization best practices—while skipping charts when a table or text-only response is more suitable. ### Linking Back to Dashboards **Trust the numbers.** The most powerful feature of AI Chat is its transparency. When the AI answers your question, it automatically generates links to your Explore dashboards, allowing you to **see exactly where the numbers came from**. Every AI response includes dashboard links that: - **Pre-apply relevant filters** - The dashboard opens with filters matching your question context - **Set appropriate time ranges** - Time periods from your question are automatically selected - **Enable comparison periods** - When you ask about changes, comparison views are activated This means you can **verify every answer** by clicking through to the full dashboard. No black box—just transparent, trustworthy analytics. Start with a quick AI summary, then explore the underlying data with full confidence in its accuracy. ## Improving AI Chat with Instructions To get the most accurate and contextual responses from AI Chat, you can add custom `ai_instructions` to your project files. These instructions provide the AI with additional context about your data, business logic, and preferred response formats. ### Why Add AI Instructions? AI instructions help the AI: - Understand your specific business context and terminology - Format responses in ways that match your team's preferences - Focus on the metrics and dimensions most relevant to your use case ### Where to Add Instructions You can add `ai_instructions` in two places: 1. **`rill.yaml`** - Project-wide instructions that apply to all queries across your entire project 2. **`.yaml`** - Metrics view-specific instructions for individual dashboards For detailed examples and best practices on writing effective AI instructions, see the [AI Configuration guide](/developers/build/ai-configuration). ## Use in Your Favorite AI Client Prefer to chat with your data in Claude Desktop, ChatGPT, or another AI assistant? You can connect your Rill projects to external AI clients using the **[Rill MCP Server](/guide/ai/mcp)**. This gives you the same governed, accurate analytics experience—powered by your predefined metrics—but integrated into your preferred AI workflow. Perfect for data teams who want deep analysis sessions, local development access, or integration with other tools. See the **[Rill MCP Server documentation](/guide/ai/mcp)** to learn more. ## Need Help? [Contact our team](/contact) if you have questions or feedback about AI Chat! --- ##### "AI Features" --- title: "AI Features" description: "AI-powered tools for exploring and analyzing your data" sidebar_label: "AI Features" sidebar_position: 0 --- # AI Features Rill includes AI capabilities across both development and data exploration. ## Build Projects with AI Rill projects are defined as YAML and SQL files, which makes them a natural fit for AI coding agents. Run `rill init` to generate agent instructions for Claude Code, Cursor, or other tools, then use your AI agent to build connectors, models, metrics views, and dashboards. See the **[Agentic Quickstart](/developers/get-started/ai-quickstart)** to get started. ## AI Chat **[AI Chat](/guide/ai/ai-chat)** is built directly into Rill Cloud, allowing you to: - Ask questions about your data in natural language - Get instant insights without writing SQL queries - Explore metrics and dimensions conversationally - Receive explanations and recommendations AI Chat understands your project's metrics views, measures, and dimensions, providing context-aware answers based on your data structure. ## Rill MCP Server The **[Rill MCP Server](/guide/ai/mcp)** connects your Rill projects to external AI assistants like Claude Desktop using the Model Context Protocol. This enables you to: - Query your Rill data from Claude Desktop or other MCP-compatible tools - Get governed, accurate analytics through AI assistants - Maintain security and access controls while using external AI tools - Leverage advanced AI capabilities for data analysis --- ##### "Rill MCP Server" --- title: "Rill MCP Server" description: How to connect to Rill MCP and query your metrics views sidebar_label: "Rill MCP Server" sidebar_position: 05 ---

The Rill Model Context Protocol (MCP) server exposes Rill's most essential APIs to LLMs. It is currently designed primarily for data analysts, not data engineers, and focuses on consuming Rill metrics views—not creating them. :::tip Looking for AI Chat in Rill Cloud? If you want to chat with your data directly in your browser without any setup, check out [AI Chat](/guide/ai/ai-chat), which uses the same MCP technology but is built right into Rill Cloud. ::: ## Why use MCP with Rill? Instead of blindly exposing your entire data warehouse to external platforms in hopes of uncovering trends, Rill's MCP integration provides a **structured and governed** alternative. By querying data that already has **predefined measures and dimensions**, the responses you get are guaranteed to be as **accurate and consistent** as the metrics displayed in your Rill dashboards. Rill offers two ways to use MCP: - **Rill MCP Server** (this guide) - Connect external AI assistants like Claude Desktop to your Rill projects - **[AI Chat](/guide/ai/ai-chat)** - Built-in chat interface in Rill Cloud with zero setup required You can also add `ai_instructions` to your project file and metrics views, which will give your LLM additional context on how to use the Rill MCP Server for best results. :::tip Configure AI instructions Set project-wide AI instructions to provide context unique to your project and improve MCP responses. [Learn more about AI configuration →](/developers/build/ai-configuration) ::: Users can then ask questions like: - What are my *week-on-week* __increases or decreases in sales__ of `XYZ service`? - During the *current year*, do I have any __outliers in website views__? What might this correlate to? - In the *previous quarter*, compared to the current ongoing quarter, what are the __trends for customer access__? - In the *last 7 days*, how many __auction requests were there from mobile vs desktop__? This ensures **trustworthy, governed analytics** while empowering users to **self-serve answers** to everyday business questions—without delays caused by email chains or ticket requests. The result: greater team productivity, clearer data ownership, and faster, more confident decision-making across your organization. ## Installation ### Prerequisites To use the Rill MCP server, you'll need: - An **MCP client** - A **running Rill project** (locally or hosted on Rill Cloud) ## Connect using OAuth (Recommended) The easiest way to connect your Rill app to Claude Desktop or ChatGPT is through their custom connector interfaces, which handle authentication automatically via OAuth. This eliminates the need to manually create access tokens or edit configuration files. ### Claude Desktop (Paid Plan) :::info Paid Claude Desktop Required Custom connectors are only available in the paid plan of Claude Desktop. [Learn more about Claude Desktop custom connectors →](https://support.claude.com/en/articles/11175166-getting-started-with-custom-connectors-using-remote-mcp) ::: 1. Open Claude Desktop and navigate to **Settings → Connectors** 2. Click **Add custom connector** 3. Enter the Rill MCP URL for your project: ``` https://api.rilldata.com/v1/orgs/{org_name}/projects/{project_name}/runtime/mcp ``` Replace `{org_name}` and `{project_name}` with your organization and project names. 4. The OAuth flow will automatically start in your browser 5. Log in to Rill and authorize the connection 6. Claude Desktop will receive an access token and your Rill app will be connected ### Claude Code (Paid Plan) 1. In your terminal, run the following command to add an MCP server with Claude Code: ```bash claude mcp add --transport http https://api.rilldata.com/v1/orgs/{org_name}/projects/{project_name}/runtime/mcp ``` Replace `{org_name}` and `{project_name}` with your organization and project names. `` will be the name you assign to this MCP server. 2. Open Claude Code using `claude` cmd. 3. In Claude Code, use `/mcp` command to see the list of MCP servers. 4. Choose the Rill MCP server you just added. 5. Select `Authenticate` to start the OAuth flow in your browser. 6. Log in to Rill and authorize the connection. 7. Claude Code will receive an access token, and your Rill app will be connected. ### ChatGPT Web Interface (Paid Plan) :::info Paid ChatGPT Required Custom apps with Developer mode are only available in the paid plans of ChatGPT. [Learn more about ChatGPT Developer mode →](https://platform.openai.com/docs/guides/developer-mode) ::: 1. Open ChatGPT and navigate to **Settings → Apps & Connectors → Advanced Settings** 2. Enable **Developer mode** 3. Go back to **Apps & Connectors** and click **Create** in the Apps section 4. Enter the Rill MCP URL for your project: ``` https://api.rilldata.com/v1/orgs/{org_name}/projects/{project_name}/runtime/mcp ``` Replace `{org_name}` and `{project_name}` with your organization and project names. 5. The OAuth flow will automatically start in your browser 6. Log in to Rill and authorize the connection 7. ChatGPT will receive an access token and your Rill app will be connected ### Connecting to a specific branch The normal MCP URL targets your project's production deployment. To connect to the deployment for a specific branch (e.g. a dev/preview deployment), insert `/branch/{branch_name}` after the project name: ``` https://api.rilldata.com/v1/orgs/{org_name}/projects/{project_name}/branch/{branch_name}/runtime/mcp ``` ## Manual Configuration (Alternative Method) If you prefer to manually configure the connection or need to connect to a local Rill instance, you can edit configuration files directly and provide your own access token. Note: If you select this option, you must have Node.js installed on your system. It can be downloaded from [nodejs.org](https://nodejs.org/en). ### Create a Rill Personal Access Token (if your project is on Rill Cloud) **Via UI (recommended):** Navigate to the AI tab in your project to retrieve both the JSON config and create a personal access token automatically: ![Project AI](/img/explore/mcp/project-ai.png) **Via CLI:** ```bash # Install the Rill CLI if you haven't already curl https://rill.sh | sh # Create a token rill token issue ``` :::tip Learn more about user tokens For comprehensive documentation on creating, managing, and using personal access tokens, see [User Tokens](/guide/administration/access-tokens/user-tokens). ::: ### Configure Claude Desktop Edit your `claude_desktop_config.json` file. By default, the JSON file is found in the following directories: - macOS: `/Users/{USER}/Library/Application Support/Claude/claude_desktop_config.json` - Windows: `C:\Users\{USER}\AppData\Roaming\Claude\claude_desktop_config.json` ### config.json Depending on which Rill instance you are trying to connect to (locally running Rill Developer, public Rill project on Rill Cloud, or private Rill project on Rill Cloud (default)), the configuration will vary. For Rill Cloud deployed projects, you can navigate to the AI page to retrieve the `config.json`. __*Private Rill Project on Rill Cloud*__ Replace `org` and `project` with the ID of your organization and project. ```json { "mcpServers": { "rill": { "command": "npx", "args": [ "mcp-remote", "https://api.rilldata.com/v1/organizations/{org}/projects/{project}/runtime/mcp", "--header", "Authorization:${AUTH_HEADER}" ], "env": { "AUTH_HEADER": "Bearer " } } } } ``` __*Public Rill Project on Rill Cloud*__ See [our demo page](https://ui.rilldata.com/demo) for public projects to test. ```json { "mcpServers": { "rill": { "command": "npx", "args": [ "mcp-remote", "https://api.rilldata.com/v1/organizations/demo/projects/rill-github-analytics/runtime/mcp" ] } } } ``` __*Locally Running Rill Developer*__ ```json { "mcpServers": { "rill": { "command": "npx", "args": [ "mcp-remote", "http://localhost:9009/mcp" ] } } } ``` :::tip Restart Claude! Restart Claude Desktop for any changes to your JSON file to take effect. ::: ### Troubleshooting If Claude Desktop cannot connect to the MCP server, check that Rill is running (locally) or that you are able to connect to your [Rill project](https://ui.rilldata.com) from your browser. If your project is private, check that the token is valid via the CLI or create a new one in the UI and edit the `config.json` file. If you're still experiencing issues, check the logs in Claude Desktop. Click on Developer → Open MCP Log File and check the logs for any errors. ## Adding AI instructions to your metrics view or project YAML LLMs give their best results when they have good context. For a conversation with Rill Data, this means things like clarifying project-specific terms, routing questions to the correct metrics view, or defining business rules. Rather than expecting the user to provide this context every time, you can add `ai_instructions` to your model. This adds the context automatically for every conversation. There are two places to add `ai_instructions`: 1. `rill.yaml` for project-wide context, such as instructions on how to use Rill MCP Server 2. Every metrics view YAML (`.yaml`), with examples of Explore URLs for that metrics view For detailed examples and best practices on writing effective AI instructions, see the [AI Configuration guide](/developers/build/ai-configuration). You can look at one of our [example projects](https://github.com/rilldata/rill-examples/tree/main/rill-openrtb-prog-ads) to see how these are used. Experiment with the instructions and see what works best for your requirements. ## Using Rill MCP Server in Claude ![MCP Main](/img/explore/mcp/mcp-main.gif) ### Supported Actions - __*List metrics views*__ – Use `list_metrics_views` to discover available metrics views in the project. - __*Get metrics view spec*__ – Use `get_metrics_view` to fetch a metrics view's specification. This is important to understand all the dimensions and measures in a metrics view. - __*Query the time range*__ – Use `query_metrics_view_summary` to obtain the available time range for a metrics view. This is important to understand what time range the data spans. - __*Query the metrics*__ – Use `query_metrics_view` to run queries to get aggregated results. ### Usage Examples Using all the above concepts, you can ask the Rill MCP server questions like: - What are my *week-on-week* __increases or decreases in sales__ of `XYZ service`? - During the *current year*, do I have any __outliers in website views__? What might this correlate to? - In the *previous quarter*, compared to the current ongoing quarter, what are the __trends for customer access__? - In the *last 7 days*, how many __auction requests were there from mobile vs desktop__? ## Conclusion While [Explore dashboards](/guide/dashboards/explore) are a great way to slice and dice to find insights, sometimes you just need a quick, overall summary of your data via a text conversation. The Rill MCP server enables this through external AI assistants like Claude Desktop. Since Rill MCP is built on top of your existing metrics, you can be confident that the returned data will be correct. **Want AI chat directly in Rill Cloud?** Check out [AI Chat](/guide/ai/ai-chat) for a browser-based experience that uses the same MCP technology with zero setup required. ## Need help? [Contact our team](/contact) if you have any questions, comments, or concerns! --- ### Alerts --- ##### "Alerts" --- title: "Alerts" description: Setting up Alerts in Rill sidebar_label: "Alerts" sidebar_position: 50 slug: /guide/alerts ---

Prefer video? Check out our [YouTube playlist](https://www.youtube.com/watch?v=wTP46eOzoCk&list=PL_ZoDsg2yFKgi7ud_fOOD33AH8ONWQS7I&index=1) for a quick start! ## Overview Alerting is a key element for any BI or analytics workflow. Because Rill's dashboards are typically built off of raw or near-raw data, we expose alerting on a wide range of filters and depth, including high cardinality fields. Alerts are accessible from any dashboard via the upper-right alarm bell icon and can be used to create context-based triggers or alerts to bring you back to an analysis if an alert is triggered. This allows the analyst or end user to then dive deeper and use Rill dashboards to interactively explore their data as needed. ![Alerts](/img/explore/alerts/alerts.gif) ## Setting and managing alerts To set an alert, click on the alarm bell icon in the upper-right nav bar of Rill Cloud (next to your profile and bookmarks). This will trigger the Alert modal to open up and walk you through the alert creation process. ![Alerts Icon](/img/explore/alerts/alerts_icon.png) ### Data Selection When creating an alert, it is important to note that any existing filters will _automatically_ be carried over into the alert but can be adjusted or removed as needed. In the first tab of creating your alert, you will want to set the following: 1. Add a name for the alert **(required)** 2. Set what measure to alert on **(required)**, including any dimension splits for the metric _(optional)_ and time grains for analysis _(optional)_ Before selecting **Next**, you will have the option to preview the data for which your alert will be created. :::tip Maximize Your Alert Filters To avoid getting over alerted, consider adding a metric filter to avoid long tail changes on small values. Some examples include: - Creating a filter on customers with less than 100 logins but filtered to more than 100 users to alert on active user drops within active accounts. - Creating a filter against campaigns with a greater than 20% decrease in spend but filtered to spend greater than $1000 to avoid large percentage changes on small campaigns. For more information, refer to our documentation on [metric filters](/guide/dashboards/filters#filter-by-measures). ::: ### Criteria On the second tab, you will have the opportunity to specify the criteria for which your alert will be triggered once certain conditions are met. 1. For any metric in your data selection (previous tab), set your alert criteria to include an operator (e.g. less than, greater than), the value or percentage, and a comparison period and/or threshold amount. 2. You will also have the ability to set dependencies or add multiple criteria using boolean conditions (AND/OR) to combine conditions across measures. ### Delivery On the final tab, you will choose how and where your alert is delivered. By default, the alert will be checked whenever the source data is [refreshed](/developers/build/models/data-refresh). There are a few additional things worth noting: 1. To limit the number of alerts, you can set an optional **Snooze** period after an alert is triggered. 2. Depending on the available notification targets (see next section), choose which targets and/or users to subscribe to the alert. Afterwards, click **Create** to finish creating the alert. ## Available alert notification targets Rill Cloud currently supports the following notification targets: - Email (default) - Slack (can be enabled) When creating an alert, all available notification targets that can be configured for an alert will be presented in the **Delivery** tab. :::note Interested in other alerting notification targets? If there is a potential alerting destination that you'd like to use with Rill but don't currently see available as a target, please don't hesitate to [contact us](/contact). We're always iterating and would love to learn more about your use case! ::: ### Configuring email targets Email is the default notification target for alerts and is automatically enabled. When creating an alert, simply specify the email addresses to include for a particular alert and an email will be sent with a link to the alert when the alert is triggered. ![Email Notifications](/img/explore/alerts/email-notifications.png) ### Configuring Slack targets Slack is also an available target for alert notifications and Rill can be configured to send alerts to your workspace, either in specified Slack channels (public / private) or as private messages via a configured bot. However, Slack will first need to be enabled to show up as an available notification target for alerts. For more information, refer to our [Configuring Slack integration](/developers/build/connectors/services/slack) documentation. ![Slack Notifications](/img/explore/alerts/slack-notifications.png) :::warning Adding your Slack app to the correct channels After having the Slack admin create the app / bot with appropriate permissions, please make sure to first add it to your target channels (using the */invite* command). Otherwise, the Slack alert will trigger an error that the channel can't be found when trying to post a notification! ::: ## Managing & Editing Alerts To view or make changes to existing alerts, navigate to the project home page and select the `Alerts` tab. Selecting an alert will give details on the configured alert criteria, including frequency and filters. You will also have the option to edit the alert settings. ![Project Home Alerts](/img/explore/alerts/project_home_alerts.png) ## Common use cases ### Troubleshooting Alerts for troubleshooting purposes are useful for making sure that applications are running as expected, campaigns are set up correctly, or any use case where the outcome is binary. For these alerts, the criteria is often: is the amount > 0 or is the amount below a threshold. These alerts are best mixed with dimension filters to be alerted on any instance or split (e.g. Impressions > 0 for all Campaign_ID). ### Pacing Alerts for pacing purposes are good for budgeting and threshold use cases - where a pre-defined range can be applied to evaluate progress towards a goal. These alerts tend to be more specific (setting up filters and criteria for specific values) and mark progress towards that goal. Consider setting up multiple threshold alerts like 50%, 75% attainment. ### Monitoring & Comparison Alerts for monitoring purposes are probably the most common alerting use case, i.e. wanting to be alerted based on relative values to prior periods. For these alerts, rather than absolutes, create criteria for % change of values. --- ### Dashboards --- #### Explore --- ###### "Explore Dashboard Quickstart" --- title: "Explore Dashboard Quickstart" description: Dashboard Quickstart sidebar_label: "Explore Dashboards" sidebar_position: 15 ---

Prefer video? Check out our [YouTube playlist](https://www.youtube.com/watch?v=wTP46eOzoCk&list=PL_ZoDsg2yFKgi7ud_fOOD33AH8ONWQS7I&index=1) for a quick start! ## Navigating the Dashboard ![Quickstart](/img/explore/dashboard101/quickstart.png) **Explore** The main screen of any Rill dashboard is called the _Explore_ page. As seen above, this is divided into three sections. - Navigation Bar - Measures panel (Left) - Dimensions Leaderboard (Right) ### Navigation Bar - _**Time Selector and Time Selector Comparison:**_ You can change the period of analysis to different ranges of time (see `red` box), either by selecting from a pre-defined period (such as last week) or choosing a custom date range. Along with this, you can enable a comparison filter to compare a range of dates with one click. - _**Filtering:**_ Underneath the time selector, you'll also be able to find your filter bar (see `orange` box) where you can [add filters](/guide/dashboards/filters) for metrics (e.g. `campaigns>1000`) or for dimensions (e.g. `campaign_name = Instacart`). - _**Explore or Pivot:**_ You can switch the view from _explore_ to [_pivot_](/guide/dashboards/explore/pivot) by selecting either option from the UI (see `pink` box). - _**Alerts, Bookmarks and Sharing:**_ You can create an [alert](/guide/alerts) by selecting the bell, customize the default view of the dashboard (see `purple` box) to a predefined set of metrics, dimensions, and filters by selecting the [bookmark](/guide/dashboards/bookmarks), or share the dashboard ([internally by clicking the `Share` button](/guide/administration/users-and-access/user-management#admin-invites-user-from-rill-cloud) or [externally via Public URLs](/guide/dashboards/public-urls)). ### KPI Widget (Measures) Panel

- _**Measures:**_ All _**metrics**_ that are available in the underlying model are viewable on the left-hand side, broken out with summary numbers (e.g. eCPM) and timeseries visualizations (based on your configured `timeseries` column in your [dashboard YAML](/reference/project-files/explore-dashboards)). You can add, remove, or reorder your metrics from the page by simply selecting them from the dropdown above the charts (see `yellow` box). If you select any specific measure, you will navigate to the [Time Dimension Detail](/guide/dashboards/explore/tdd). - _**Time Dimension Detail:**_ A detailed view of a single specific measure that can be further drilled down to understand minute details in your data. As with the Explore page, you can add comparison dimensions to visualize the value for several specific dimension values. For more information see: [Time Dimension Detail](/guide/dashboards/explore/tdd). :::note Big Number Formatting [Formatting of your measures](/developers/build/metrics-view#measures) will not change the granularity of the Big Number, but you'll see the formatting being applied to the TDD, Dimension Leaderboard, and Pivot tables. ::: ### Dimensions Leaderboard Panel

- _**Dimensions:**_ All _**dimensions**_ available in the underlying model on the right-hand side via leaderboard / toplist charts. You can add, remove or re-order any dimension from the page by simply selecting them from the dropdown above the charts (see `green` boxes). You can also drill into leaderboards further (see `blue` box) to see all corresponding metrics for a specific dimension. Within that drilldown, you can also then sort by metric, search your dimensions, and/or [export data](/guide/reports/exports). It is also possible to display [multiple measures in the dimension leaderboard](/guide/dashboards/explore/multi-metrics). :::info Search for individual attributes After drilling into a leaderboard (or what we sometimes refer to as a _toplist_ chart), rather than scrolling and finding an individual attribute (especially if the list is very long), you can also quickly search for a value and select / apply it to your dashboard by using the upper-right search box. ![Search Box](/img/explore/dashboard101/search-box.png) ::: :::tip Don't have a Rill project or dashboard deployed yet? If you want to get hands on and see what interacting with a Rill dashboard feels like, we have a set of [demo projects](https://ui.rilldata.com/demo) already deployed on Rill Cloud and publicly available for everyone to try out. These [same projects](/developers/tutorials/) are also available on GitHub and can be deployed locally using Rill Developer. ::: For more details about additional capabilities and/or how to utilize more advanced functionality within Rill dashboards, please see the [reference](#reference) section. ### Keyboard shortcuts Whether you need to see the full value of a long JSON, or copy a value, there are some available keyboard shortcuts in the Rill Cloud Dashboards. More coming soon! List of commands: - __*Copy values*__ ( ``shift + click`` ) - Copy the value of the row value. - __*Value previewer*__ ( ``space`` ) - See the full text value of the row value. - __*Lock Inspector*__ ( ``L`` ) - Lock the inspector (allows scrolling through long values) ![Preview Value](/img/explore/dashboard101/preview-value.png) ## Display Settings ### Dark Mode Toggle between light and dark themes by clicking your profile icon in the top right and selecting your preferred theme. ![Dark Mode Toggle](/img/guide/theme/dark-mode.png) ## Banners! Another additional feature that you can add to an Explore dashboard are banners. Whether it is to inform your end-users about specific guidelines on how to use Rill, or an informational post about the datasets being used, you can design the banner to whatever text you'd like. Simply add the following to your explore-dashboard.yaml ```yaml banner: Your custom message here! ``` ![Banner](/img/explore/dashboard101/banner.png) ## Reference - [Filters & Comparisons](/guide/dashboards/filters) - [Bookmarks & Sharing](/guide/dashboards/bookmarks) - [Exports & Scheduled Reports](/guide/reports/exports) - [Public URL](/guide/dashboards/public-urls) - [Alerts](/guide/alerts) --- ###### "Multiple Measure Leaderboard" --- title: "Multiple Measure Leaderboard" description: Expanding your leaderboard sidebar_label: "Multiple Measure Leaderboard" sidebar_position: 03 --- By default, the Explore dashboard displays a single measure for each dimension in the leaderboard. However, there are many cases where comparing multiple measures side-by-side can provide deeper insights.

Prefer video? Check out our [YouTube playlist](https://www.youtube.com/watch?v=wTP46eOzoCk&list=PL_ZoDsg2yFKgi7ud_fOOD33AH8ONWQS7I&index=1) for a quick start! ## Benefits of Multiple Measure Dashboards With the **Multiple Measure Leaderboard** feature, you can easily select and compare several measures for any dimension. Simply open the dropdown in the dimension leaderboard, enable `multi-select`, and choose any combination of visible measures to display them together. ![Multi Measure Select](/img/explore/dashboard101/multi-measure-select.png) ### Why Compare Multiple Measures? Comparing multiple measures at once allows you to: - **Spot correlations** between different metrics for the same dimension. - **Quickly identify outliers** or trends that may not be obvious when viewing a single measure. - **Make more informed decisions** by seeing the full context of your data. --- ###### "Pivot and Flat Tables" --- title: "Pivot and Flat Tables" description: Pivot sidebar_label: "Pivot and Flat Tables" sidebar_position: 30 --- Rill's multi-dimensional pivot provides lightning-fast, Excel-style pivot tables across multiple dimensions, time granularities, and metrics. The interface allows users to drag and drop any combination of fields for quick comparison and report generation. While the Explore view is great for data discovery, the Pivot view serves other use cases like: - Regular reporting: Monthly financial reports, campaign reporting, client reports - Time analysis: Dimensions by time columns for daily or weekly comparisons - Data export: Scheduled exports for generating CSVs for external analysis (_coming soon_)

Prefer video? Check out our [YouTube playlist](https://www.youtube.com/watch?v=wTP46eOzoCk&list=PL_ZoDsg2yFKgi7ud_fOOD33AH8ONWQS7I&index=1) for a quick start! ## Using Rill's Pivot View Pivots can be split by time, measures, and dimensions. You can drag and drop any of these options to rows and columns to create your pivot table (and can drag between columns/rows as well). Dimensions are also searchable in the top left, or select the ```+``` to see the dimension list. As each dimension is added, you can track the kernels on the top left to see which rows have been added to the table. Each column dimension will also be visualized in the pivot. Like Time Dimension Detail, the same filter bar can be used to filter the pivot table overall. ![Pivot Overview](/img/explore/pivot/pivot-overview.png) :::note Measures as splits Measures can only be used as column splits to define which metric is to be used in the table and cannot be added to the rows filter ::: As new dimensions are added to the table, the comparison will be aggregated by default and can be expanded as you navigate the table. Pivot tables can then be scrolled through to quickly navigate the row/column combinations. If a given split does not have any valid data, ```no data``` will be displayed in the cell. Select the window button on the top left of the pivot table to expand the view to full screen. To quickly reset all the expanded rows in the pivot table, you can select collapse all in the top left. ![Pivot](/img/explore/pivot/pivot.gif) ## Flat Tables While Pivot tables are a powerful tool to slice and dice your data into different configurations, sometimes a simple flat table is all you need. By selecting the toggle next to `Columns`, you can swap between pivot and flat tables. Unlike Pivot, which gives you options to add both row and column dimensions, a flat table will only allow you to add columns. ![Flat Table](/img/explore/pivot/flat-table.png) --- ###### "Time Dimension Detail" --- title: "Time Dimension Detail" description: Time Dimension Detail sidebar_label: "Time Dimension Detail" sidebar_position: 25 --- The Time Dimension Detail is a separate visualization accessed by clicking on any time series chart in Explore. Expanding the time series allows for more surface area to compare a larger number of dimensions trended over time. Additionally, you can create rapid pivot tables based on time across any dimension.

Prefer video? Check out our [YouTube playlist](https://www.youtube.com/watch?v=wTP46eOzoCk&list=PL_ZoDsg2yFKgi7ud_fOOD33AH8ONWQS7I&index=1) for a quick start! ## Time Dimension Detail (TDD) Within the TDD screen, you can apply all of the same filters and comparisons as Explore. Any filters applied will be carried into the TDD or will be carried out to Explore if you return to the main page on the top left. Underneath the expanded time series chart, you will see two sets of dropdowns - one for Rows to change the comparison (this defaults to Time but can be changed to any dimension) and one for Columns (which can be used to cycle through all of your metrics). On the top right, you can also export the TDD pivot view for quick time series reporting. Similar to the filters, the Search and Exclude options from the expanded Leaderboards are available on the top right of the TDD table. Lastly, you can also take the TDD table directly to the Pivot view for multi-dimensional analysis - more details on [Pivot here](/guide/dashboards/explore/pivot). ![TDD](/img/explore/tdd/tdd.gif) :::tip Adjusting Time Grains The TDD screen will carry the time dimension from your previous filters, which may be more granular than needed (displaying too many columns). Change the ```Metric Trends by``` filter on the top right to switch to alternate ranges like days, weeks, etc. ::: --- ##### "Bookmarks & Sharing" --- title: "Bookmarks & Sharing" description: Creating & Sharing Saved Views in Rill sidebar_label: "Bookmarks & Sharing" sidebar_position: 35 --- Bookmarks are useful to return to regular analyses and filter sets commonly used for reporting or deep dives. If you have a regular view of the data, a bookmark is also a good alternative to creating an entirely new dashboard, as it contains a subset of the view while retaining all of the fields available for analysis. Common use cases for Bookmarks include: - Weekly/Monthly reporting - Setting filters for specific users/use cases (eg. teams or executives with a narrower view, an account manager's book of clients) - Answering common troubleshooting questions by starting with a subset of problem dimensions

Prefer video? Check out our [YouTube playlist](https://www.youtube.com/watch?v=wTP46eOzoCk&list=PL_ZoDsg2yFKgi7ud_fOOD33AH8ONWQS7I&index=1) for a quick start! ## How to set and use Bookmarks On the top right portion of the screen, you will see the Bookmark icon to bring up the option to save a bookmark. Clicking that icon brings up options to save the current page as your home screen, create a new bookmark page, or to see your list of bookmarks (and shared bookmarks). On the bookmark screen, you'll then be able to set the options related to the saved view: - **Name:** Set a name and description (_optional_) - **Filters:** All filters from your previous analysis will be carried into the bookmark. Add/remove any additional filters as necessary - **Shared or Local:** Select a category (_for admins who wish to make a public bookmark_) - **Save Filters Only:** this option will save only the filter combination to be re-used later without restricting the measures and dimensions - **Absolute Dates:** use this option if you have a saved time period and return exactly to that period versus using the time period filter ![Setbookmark](/img/explore/bookmarks/setbookmark.png) --- ##### Canvas Dashboards --- title: Canvas Dashboards sidebar_label: "Canvas Dashboards" sidebar_position: 20 --- After logging into [Rill Cloud](https://ui.rilldata.com), you should see all projects within your [organization](/guide/administration/organization-settings#organization) that are available and/or have been granted permissions to your user profile. Within each project, you'll then be able to access the corresponding individual dashboards that belong to a particular Rill project.

Prefer video? Check out our [YouTube playlist](https://www.youtube.com/watch?v=wTP46eOzoCk&list=PL_ZoDsg2yFKgi7ud_fOOD33AH8ONWQS7I&index=1) for a quick start! ## Navigating the Dashboard ![Canvas Dashboard](/img/explore/canvas/canvas-dashboard.png) Similar to our [Explore dashboards](/guide/dashboards/explore), Canvas Dashboards also include a similar navigation bar to control the dashboard components. ### Navigation Bar - _**Time Selector and Time Selector Comparison:**_ You can change the period of analysis to different ranges of time (see `red` box), either by selecting from a pre-defined period (such as last week) or choosing a custom date range. Along with this, you can enable a comparison filter to compare a range of dates with one click. - _**Filtering:**_ Underneath the time selector, you'll also be able to find your filter bar (see `orange` box) where you can [add filters](/guide/dashboards/filters) for metrics (e.g. `campaigns>1000`) or for dimensions (e.g. `campaign_name = Instacart`). :::tip identical names in metrics views If your dimensions or measures have the same name in your metrics view, filters will apply to all components, regardless if it's in a different metrics view. ::: ## Component Navigation ![Canvas Navigation](/img/explore/canvas/canvas-navigaton.png) If you want to further drill into a component's data, select the top right button to take you to the equivalent Explore dashboard. :::tip no button? If no Explore dashboard exists, and/or you don't have [permissions to view it](/developers/build/dashboards/customization#define-dashboard-access), no button will appear and is as designed. ::: --- ##### "Filters & Comparisons" --- title: "Filters & Comparisons" description: Filters & Comparisons sidebar_label: "Filters & Comparisons" sidebar_position: 25 ---

If videos are more your jam, take a look at [our series of YouTube videos](https://www.youtube.com/watch?v=wTP46eOzoCk&list=PL_ZoDsg2yFKgi7ud_fOOD33AH8ONWQS7I&index=1) to get started! ## Overview Rill is particularly suited for exploratory analysis - to be able to slice & dice data quickly. As such, there are a variety of filter types and filter mechanisms throughout the app. The goal for each Rill dashboard is to provide users with all metrics and dimensions required for each use case and create an interactive experience to cut data in any form. :::tip Human readable URL With the release of v0.52, we have introduced an easy way to craft specific views by modifying the URL directly. As you select filters, time ranges, and default dimension and measures, the URL will change accordingly. ../explore/explore_dashboard?tr=P3M&compare_tr=rill-PP&f=dimension in ('exampleA', 'exampleB') ::: ## Add / Hide Dimensions and Metrics Users can add or hide dimensions and metrics to create a subset of fields they wish to see at any given time. At the top left, above the time series and above the top left leaderboard, you'll find the Measures & Dimensions selectors to add or hide fields from the page. In the example below, `network` and `country` are deselected, so they would be hidden from view. ![Hide](/img/explore/filters/hide.png) :::tip hiding metrics and dimensions by default You can also change settings in the dashboard configuration to hide certain fields by default. You may want to do this to make dashboards easier to use (less complicated, narrowed to most commonly used) and to improve performance (hide high cardinality dimensions or complicated expressions in metrics). For more details, check out [dashboard customizations](/developers/build/dashboards/customization#setting-default-views-for-dashboards). Or, an administrator can set the default view of a dashboard by [bookmarking the view](bookmarks) as Home. ::: ## Dimensions ### Filter by Dimensions The primary/easiest way to filter data is by selecting values in the dimension tables. Leaderboards within Rill are fully interactive. Selecting any dimension in the table will automatically filter the remaining leaderboards and metrics by that selection. To add or remove dimensions on the page - select the All Dimensions picker above the Leaderboards. Next to the All Dimensions picker, you can also change which Metric is being highlighted to be able to update the entire page and cycle through each dimension table sorted by each metric. You can also expand each dimension table to see all metrics and full list of those dimensions. In the expanded Leaderboard, you can search for dimension values, select all values returned, or exclude values from the result. Any filter applied in the Leaderboard will also show up in the filter bar next to the time picker. You can apply the same search capabilities and select features in the filter bar as well. ![Filter](/img/explore/filters/filter.gif) ### Dimension Comparisons In addition to time comparisons, you can select multiple dimension values to compare trends of those specific data points. Select the comparison option on the top left of any leaderboard and select multiple dimensions Deselect the comparison option or clear the filter bar to remove your comparisons. ![Comparison](/img/explore/filters/comparison.gif) :::note For more advanced time and dimension comparisons, visit the [Time Dimension Detail](/guide/dashboards/explore/tdd) page. ::: ## Measures ### Filter by Measures There are also use cases where you want to filter by the metric values returned. As an example - all customers over $1000 in revenue, all campaigns with at least 1 million impressions, all delivery locations with late times over 4%, etc. To add or remove metrics on the page - select the All Measures picker above the Time Series charts. These metric filters can be applied from the filter bar. To apply a metric filter: - Select the metric you wish to filter by (e.g. Total Cost) - Select which dimension to sort/key the metric by (e.g. Cost by Region) - Select your Threshold Type (e.g. Greater Than) - Input your Threshold amount and Click Enter ![Image](/img/explore/filters/image.png) :::tip Metric filters are a good way to "sort" by two different metrics. First, apply your metric threshold. Then, sort by your metrics within the Leaderboard to do multi-metric sorting. As an example - to see the most active enterprise customers - filter all customers with revenue greater than $1000, then sort by number of users in descending order. ::: --- ##### "Dashboards" --- title: "Dashboards" description: "Explore and interact with your data through Rill dashboards" sidebar_label: "Dashboards" sidebar_position: 0 --- import ThemedImage from '@theme/ThemedImage'; # Dashboards Rill Cloud provides powerful interactive dashboards for exploring, analyzing, and sharing your data. Dashboards make it easy to slice, dice, and drill down into your metrics to discover insights. ## Explore Dashboards **[Explore Dashboard Quickstart](/guide/dashboards/explore)** provides an introduction to navigating and using Rill dashboards. Learn how to: - Navigate the dashboard interface with measures, dimensions, and time series - Understand the different components and their interactions - Get started with your first dashboard exploration ### Multiple Measures **[Multiple Measures](/guide/dashboards/explore/multi-metrics)** allows you to compare multiple metrics simultaneously. You can: - Compare multiple metrics side-by-side in the dimension leaderboard - Analyze relationships between different measures - Gain comprehensive insights across your metrics ### Time Dimension Details **[Time Dimension Details](/guide/dashboards/explore/tdd)** provides deep-dive analysis into specific time periods. You can: - Drill down into specific time periods for granular analysis - Compare dimensions across time - Export time-based data for reporting ### Pivot Tables **[Pivot Tables](/guide/dashboards/explore/pivot)** offer Excel-style cross-tabulations for data analysis. Create: - Multi-dimensional pivot tables with drag-and-drop functionality - Cross-tabulations across dimensions, measures, and time - Flat tables for simple column-based views ## Canvas Dashboards **[Canvas Dashboards](/guide/dashboards/canvas)** provide custom dashboard layouts for personalized views. Create: - Custom dashboard layouts with multiple visualizations - Personalized views tailored to your needs - Flexible arrangements of charts and components ## Time Series Analysis **[Time Series Analysis](/guide/dashboards/time-series)** enables you to analyze trends and patterns over time. Explore: - Trends and patterns in your data across different time periods - Time-based comparisons to understand changes - Granular time analysis for detailed insights ## Filters & Comparisons **[Filters & Comparisons](/guide/dashboards/filters)** help you focus your analysis on specific data subsets. You can: - Apply powerful filters to metrics and dimensions - Use time comparisons to analyze changes over different periods - Combine multiple filters for precise data exploration ## Bookmarks **[Bookmarks](/guide/dashboards/bookmarks)** let you save and share specific dashboard states. You can: - Save specific dashboard states with filters, metrics, and dimensions - Share bookmarks with your team members - Set bookmarks as default views for quick access ## Public URLs **[Public URLs](/guide/dashboards/public-urls)** enable external sharing of your dashboards. Share: - Dashboards externally with customers and partners - Views without requiring Rill accounts - Secure, shareable links to specific dashboard configurations --- ##### "Sharing Dashboards with a Public URL" --- title: "Sharing Dashboards with a Public URL" description: Share dashboards with a few clicks using a public link sidebar_label: "Public Shareable URLs" sidebar_position: 36 --- ## Overview Sharing your dashboard is a key way to promote collaboration with other users and allows for quick access to your dashboards. As an admin, you also have the ability to create public shareable URLs, meaning that you can generate an expirable public link to a dashboard with specific filters pre-applied. The receiver of the link will then be able to engage and interact with this dashboard as if they were logged in (with the parent filters locked). :::tip Recipients are not required to log in to view the dashboard and will not be able to change your pre-defined filters! ::: ### How to create a public URL from the UI After applying the filters and modifying the dashboard to your liking, please select the `Share` button. If not already selected, select `Create public URL`. If you want to set an expiration, please select the toggle and set the expiration. ### What the recipient sees As expected, when opening the public URL, the user can view the dashboard. Like a logged-in user, they are able to navigate within the dashboard and drill or slice-and-dice as needed. Unlike a logged-in user, they are not able to make any changes to the filters that you've set. :::info Did you know? In fact, users who click on a public shareable URL cannot see the parent filters that have been applied! ::: ## How to manage public URLs ### Via the UI You can now manage public URLs via the UI. You will find a new "settings" tab in the Rill Cloud UI as an administrator. ![Public URL Settings](/img/explore/publicurl/public-url-settings.png) ### Via the CLI ``` rill public-url Manage public URLs Usage: rill public-url [command] Available Commands: list List all public URLs create Create a public URL delete Delete a public URL Flags: --org string Organization Name (default "Rill_Learning") Global Flags: --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) Use "rill public-url [command] --help" for more information about a command. ``` Using the Rill CLI, you can list, create, or delete public URLs. #### Deleting a public URL To delete a public URL, you will need an `id` parameter. In order to retrieve the appropriate `id`, you will need to first list out the public URLs. You can do so using the command below with any flags that help you. ``` rill public-url list ``` Once you have obtained the `id` you can run the following: ``` rill public-url delete ``` If you are interested in creating a public URL directly from the CLI, you can do so by passing the required parameters. (You can use the --help flag to see what additional flags are required.) --- ##### "Time Series Filter" --- title: "Time Series Filter" sidebar_label: "Time Series Filter" hide_table_of_contents: false sidebar_position: 20 tags: - Quickstart - Tutorial ---

Prefer video? Check out our [YouTube playlist](https://www.youtube.com/watch?v=wTP46eOzoCk&list=PL_ZoDsg2yFKgi7ud_fOOD33AH8ONWQS7I&index=1) for a quick start! ## Overview Once you've [built your metrics view](/developers/build/metrics-view) and assigned a time series column, you'll be able to start visualizing your measures and dimensions in either an Explore dashboard or Canvas dashboard. This guide will discuss all the features in the time navigator that exists at the top of all dashboards and how to customize it to your needs. ## Time Series Filter Component The time series navigation is divided into four main sections: ### 1. Forward/Backward Navigation Allows you to step forward or backward through your selected time range. Use the arrow buttons to navigate through time periods while maintaining your current selection. ### 2. Time Selector The dropdown menu containing default time ranges (like "Last 7 days", "Last 30 days") or custom time ranges you've created. This is where you choose your primary time period. It also has a selector for custom ranges in a calendar view as well as time zone selector. ### 3. As Of... Controls the reference point for your time calculations. This determines whether your time range is relative to the current moment (wallclock), latest data based on grain, or "completed" data. ### 4. Comparing Enables time comparison functionality, allowing you to compare current data against previous periods (e.g., "vs. previous period" or "vs. same period last year"). ![Time Pill](/img/build/metrics-view/time-series/time-pill.png) ## Key Concepts Before diving into this complicated topic, it's important to understand these fundamental concepts: - **Reference Points**: The anchor point from which durations are calculated - **Snapping**: Aligning time boundaries to specific grains (day, hour, etc.) - **Duration vs Fixed Points**: The difference between "4 days ago" and "4 days ending at a specific point" - **Wallclock Time**: The actual current time vs. data-relative time ### Time Selection Types There are several ways to specify time ranges: 1. **Duration-based ranges**: Specify a length of time from a reference point (e.g., `7d as of latest/d`) 2. **Fixed-point ranges**: Specify exact start and end points (e.g., `-4d to now/d`) 3. **ISO 8601 ranges**: Use standard date format (e.g., `2024-01-01 to 2024-01-05`) The following sections explain how each type works and when to use them. ## Time Ranges Explained Understanding the difference between duration-based and fixed-point ranges is crucial for avoiding confusion. ### Duration-Based Ranges These specify a length of time from a reference point: - `4d` = 4 days duration - `4d as of now/d` = 4 days ending at start of current day ### Fixed-Point Ranges These specify exact start and end points: - `-4d to now/d` = from 4 days ago (wallclock) to start of current day - `2024-01-01 to 2024-01-05` = exact date range :::tip Not the same Many users expect `-4d to now/d` and `4d as of now/d` to be equivalent, but they're not: - **`-4d to now/d`**: Starts 4 days ago from current wallclock time, ends at start of current day - If it's 12:30 PM on Sept 3, this starts at 12:30 PM on Aug 30 - Duration: ~3.5 days (partial Aug 30 + full Aug 31, Sep 1, Sep 2) - **`4d as of now/d`**: Exactly 4 days ending at start of current day - Starts at start of Aug 30, ends at start of Sept 3 - Duration: exactly 4 full days ::: ### ISO 8601 Ranges ISO 8601 ranges use the standard date format for specifying exact time periods: - `2024-01-01 to 2024-01-05` - From January 1st to January 5th, 2024 - `2024-01-01T00:00:00Z to 2024-01-01T23:59:59Z` - Full day with explicit timestamps - `2024-01-01T09:00:00 to 2024-01-01T17:00:00` - Business hours on a specific day These ranges are useful when you need precise control over the exact start and end times, especially for reporting on specific events or periods. ### Snapping Snapping aligns time boundaries to specific grains (day, hour, etc.) rather than using exact wallclock times. This allows your dashboards to show complete summaries of data rather than incomplete sets. - `/s` = snap to second boundaries - `/m` = snap to minute boundaries - `/h` = snap to hour boundaries - `/d` = snap to day boundaries (start of day) - `/w` = snap to week boundaries (start of week) - `/M` = snap to month boundaries (start of month) - `/y` = snap to year boundaries (start of year) ## As of ### Reference - **Complete data `watermark`**: Uses the [watermark timestamp](/developers/build/metrics-view/time-series#watermark) from your metrics view. This ensures you only see data that has been fully processed and is considered "complete" according to your data pipeline's watermark settings. - **Latest data `latest`**: Uses the most recent data point available in your dataset, regardless of completeness. This is useful when you want to see the freshest data even if it might be incomplete. - **Current time `now`**: Uses the current wallclock time as the reference point. This means your time ranges will always be relative to the present moment, which can include future time periods if your data extends beyond the current time. :::tip Unsure which one? If you hover on any of the three options, it will give you the actual time that will be considered in the time filter. ::: ### Grain The grain determines the time granularity for your reference point. This affects how your time ranges can be rolled up and displayed. Depending on the `smallest_time_grain` in your metrics view, the options will be limited. You can also configure grain settings directly in the Time Selector using the [snapping](#snapping) options. ### Anchor The anchor determines whether to include incomplete time periods in your data. After snapping to a specific grain, you can choose to: - **Include incomplete periods**: Show data for the current partial time grain (e.g., today's data even if the day isn't finished) - **Exclude incomplete periods**: Only show data for complete time grains (e.g., exclude today if it's not finished) This is similar to the `watermark` concept but operates at the grain level you've selected, providing more granular control over data completeness. ## Bringing it all together The following are the most common sources of confusion and how to avoid them: ### 1. "I want the last 4 calendar days" **Confusing approaches:** - `-4d to now/d` - This gives you ~3.5 days if it's not midnight - `4d` - This gives you exactly 4 days from current wallclock time **Correct approaches:** - `4d as of now/d` - Exactly 4 days ending at start of current day - `-4d/d to now/d` - From start of day 4 days ago to start of current day - `-4d to ref as of now/d` - 4 days ending at start of current day ### 2. "Why does my range show 3 days when I asked for 4?" This happens when you mix wallclock time with snapped boundaries: - `-4d to now/d` at 12:30 PM = Aug 30 12:30 PM to Sept 3 00:00 AM - Duration: Partial Aug 30 + Aug 31 + Sep 1 + Sep 2 = ~3.5 days **Solution:** Use consistent snapping: `-4d/d to now/d` ### 3. "The end date keeps shifting when I change the 'as of'" This happens because the end date uses `now` (wallclock time) instead of a snapped reference: - `-4d to now/d` - End date is always start of current day - `-4d to now/d as of now/d+1d` - End date becomes start of tomorrow **Solution:** Use duration-based ranges: `4d as of now/d` ### 4. "I want to look at data relative to the latest data point" **Problem:** Using `now` when data might be stale - `-7d as of latest/d` - 7 days ending at latest data day - `-7d` - 7 days ending at current wallclock time (might include future) **Solution:** Always specify the reference point explicitly ## Time Comparisons Along with setting time ranges, you have the ability to set a comparison period in the "Comparing" section of the time pill. This allows you to analyze trends and changes by comparing your current time period against a previous period. ### Comparison Types 1. **Previous Period**: Compares against the immediately preceding period of the same duration 2. **Previous ...**: Compares current selected period vs a set period (day, week, month, year) 3. **Custom Comparison**: Set a specific comparison period using the same time range syntax ### Understanding Comparison Results Once comparison is enabled, you'll see slightly different information in your dashboard. Along with the current value, you'll see both change in value as well as % change over periods. This gives you a quick glance at how your metrics are performing compared to the previous period. ![Comparison](/img/explore/filters/comparison.png) ## Filter by Scrubbing For a specific view into your time series graph, you can interactively scrub directly on the time series graph. This feature allows you to zoom into specific time periods by clicking and dragging across the chart. ![Scrub](/img/explore/filters/scrub.png) ## Multiple Time Dimensions Your dashboard may have multiple time dimensions available for filtering, beyond the primary time series. These appear as additional filter options and allow you to analyze data across different temporal perspectives. ![Multiple Time Dimensions](/img/build/metrics-view/time-series/mulitple-time-series.png) ### Common Use Cases - **Order vs. Ship Date**: Filter by when orders were placed or when they were fulfilled - **Created vs. Updated**: Analyze records by original creation or last modification - **Event vs. Processing Time**: Distinguish between when something happened and when it was recorded ### Using Time Dimension Filters Time dimensions appear in the filter panel alongside your other dimensions. When you select a time dimension filter: 1. **Choose the time dimension** from the filter panel 2. **Select a time range** using the same duration syntax as the main time series 3. **Combine with other filters** to create complex queries :::tip Time dimension filters work independently from the main time series selector. This means you can view your time series chart by order date while filtering to only show records with a ship date in the last 7 days. ::: ### Example: Analyzing Fulfillment Delays To find orders placed last month that haven't shipped yet: 1. Set the main time series to "Last 30 days" (viewing by order date) 2. Add a filter on Ship Date dimension 3. Set the Ship Date filter to exclude records (or filter to null values) This combination lets you identify orders that are still pending fulfillment. For more details on configuring time dimensions, see the [Time Dimensions](/developers/build/metrics-view/dimensions/time-dimensions) developer documentation. --- ### Guide --- ##### "Getting Started with Rill Cloud" --- title: "Getting Started with Rill Cloud" description: "Introduction to Rill Cloud, AI features, and management" sidebar_label: "Rill Cloud" sidebar_position: 0 slug: "/guide" --- import TileIcon from '@site/src/components/TileIcon'; import ThemedImage from '@theme/ThemedImage'; # Getting Started with Rill Cloud Rill Cloud is a fully-managed platform that brings powerful, interactive analytics directly to your fingertips. Explore your data through instant dashboards, ask questions in plain English with AI Chat, and share insights with your team—all without writing SQL or waiting for custom reports. Your data team has already set up the metrics and dashboards you need; you simply explore, analyze, and collaborate. This guide is for **anyone who consumes dashboards and insights in Rill Cloud** — business analysts, marketing and finance teams, operations leads, executives, and anyone else on your team who needs to explore data, set up alerts, schedule reports, or share findings. No coding or SQL experience required. Need to **build or maintain Rill projects** — connect data sources, write models, define metrics, or deploy dashboards? See the [Developer Docs](/).
} /> } /> } /> } /> } /> } /> } /> } /> } /> } />
--- ### Reports --- ##### "Exports & Scheduled Reports" --- title: "Exports & Scheduled Reports" description: Downloading and exporting data from Rill sidebar_label: "Exports & Scheduled Reports" sidebar_position: 38 --- Rill provides the ability to export data from multiple locations - leaderboards, Time Dimension detail, and Pivot Tables (_coming soon_). In addition to direct downloads, you can also set up recurring reports to your inbox. Scheduled Reports allow you to effortlessly generate and manage email reports for any exportable content within your Rill Cloud dashboard. - [Exporting from Rill](#exporting-from-rill) - [How to Schedule an Email Report](#how-to-schedule-an-email-report) - [Managing Scheduled Reports](#managing-scheduled-reports)

Prefer video? Check out our [YouTube playlist](https://www.youtube.com/watch?v=wTP46eOzoCk&list=PL_ZoDsg2yFKgi7ud_fOOD33AH8ONWQS7I&index=1) for a quick start! ## Exporting from Rill There are several places to export your data from Rill. In each case, you will see options for exporting to CSV, XLSX, or Parquet formats. Exports are available from: - Leaderboards: expand a leaderboard in Explore and select Export on the top right - Time Dimension Detail: after choosing a comparison, select Export on the time series chart - Pivot Tables: _coming soon_ ## How to Schedule an Email Report

Follow these simple steps to schedule an email report: 1. **Navigate to Content:** Expand a dimension table by clicking on the dimension name. 2. **Export Options:** Click on the Export button and select "Create scheduled report..." Filters, comparisons, and sort orders will be preserved for your report. 3. **Configure Report Settings:** Choose a frequency and set a time to receive the output. Choose a report format (CSV/Excel/Parquet) and specify a list of recipients. 4. **Complete and Enjoy:** Click "Done." User-created reports will be delivered directly to your inbox 🎉. ![Scheduled](/img/explore/exports/scheduled.png) ## Managing Scheduled Reports Reports are managed from your home screen. To navigate to the report admin page, select the Home button on the top left and the Reports tab from your homepage. - **Accessing Reports:** When browsing projects, find related assets for each project, including dashboards, reports, and logs. - **Permissions:** Viewers can see their own reports, while administrators can view both their own and other users' reports. - **Report Details:** Click into a report to view schedule details, next run information, recipients, and execution history. - **Deletion:** To delete a report, click the three-dot menu next to the report name and select "Delete Report." - **Unsubscribe Option:** Recipients have the option to unsubscribe by clicking the provided link within the report delivery email. --- ## Reference --- ### Api --- ### Cli --- #### Billing --- ##### Plan --- ####### rill billing plan list --- note: GENERATED. DO NOT EDIT. title: rill billing plan list --- ## rill billing plan list List plans ``` rill billing plan list [flags] ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill billing plan](plan.md) - Get billing plans --- ####### rill billing plan --- note: GENERATED. DO NOT EDIT. title: rill billing plan --- ## rill billing plan Get billing plans ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill billing](../billing.md) - Billing related commands for org * [rill billing plan list](list.md) - List plans --- ##### Subscription --- ####### rill billing subscription cancel --- note: GENERATED. DO NOT EDIT. title: rill billing subscription cancel --- ## rill billing subscription cancel Cancel subscription for an organization ``` rill billing subscription cancel [flags] ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill billing subscription](subscription.md) - Manage organization subscriptions --- ####### rill billing subscription edit --- note: GENERATED. DO NOT EDIT. title: rill billing subscription edit --- ## rill billing subscription edit Edit organization subscription ``` rill billing subscription edit [flags] ``` ### Flags ``` --plan string Plan Name to change subscription to ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill billing subscription](subscription.md) - Manage organization subscriptions --- ####### rill billing subscription list --- note: GENERATED. DO NOT EDIT. title: rill billing subscription list --- ## rill billing subscription list List subscriptions for an organization ``` rill billing subscription list [flags] ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill billing subscription](subscription.md) - Manage organization subscriptions --- ####### rill billing subscription renew --- note: GENERATED. DO NOT EDIT. title: rill billing subscription renew --- ## rill billing subscription renew Renew cancelled organization subscription ``` rill billing subscription renew [flags] ``` ### Flags ``` --plan string Plan name to renew subscription to ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill billing subscription](subscription.md) - Manage organization subscriptions --- ####### rill billing subscription --- note: GENERATED. DO NOT EDIT. title: rill billing subscription --- ## rill billing subscription Manage organization subscriptions ### Flags ``` --org string Organization Name ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill billing](../billing.md) - Billing related commands for org * [rill billing subscription cancel](cancel.md) - Cancel subscription for an organization * [rill billing subscription edit](edit.md) - Edit organization subscription * [rill billing subscription list](list.md) - List subscriptions for an organization * [rill billing subscription renew](renew.md) - Renew cancelled organization subscription --- ###### rill billing --- note: GENERATED. DO NOT EDIT. title: rill billing --- ## rill billing Billing related commands for org ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill](../cli.md) - A CLI for Rill * [rill billing list-issues](list-issues.md) - List billing issues for an organization * [rill billing plan](plan/plan.md) - Get billing plans * [rill billing subscription](subscription/subscription.md) - Manage organization subscriptions --- ###### rill billing list-issues --- note: GENERATED. DO NOT EDIT. title: rill billing list-issues --- ## rill billing list-issues List billing issues for an organization ``` rill billing list-issues [flags] ``` ### Flags ``` --org string Organization Name ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill billing](billing.md) - Billing related commands for org --- #### Docs --- ###### rill docs --- note: GENERATED. DO NOT EDIT. title: rill docs --- ## rill docs Open docs.rilldata.com ``` rill docs [flags] ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill](../cli.md) - A CLI for Rill --- #### Env --- ###### rill env configure --- note: GENERATED. DO NOT EDIT. title: rill env configure --- ## rill env configure Configures connector variables for all sources ``` rill env configure [] [flags] ``` ### Flags ``` --path string Project directory (default ".") --project string --environment string Optional environment to resolve for (options: dev, prod) ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill env](env.md) - Manage variables for a project --- ###### rill env --- note: GENERATED. DO NOT EDIT. title: rill env --- ## rill env Manage variables for a project ### Flags ``` --org string Organization Name ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill](../cli.md) - A CLI for Rill * [rill env pull](pull.md) - Pull cloud credentials into local .env file * [rill env push](push.md) - Push local .env contents to cloud * [rill env rm](rm.md) - Remove an env variable * [rill env set](set.md) - Set variable * [rill env show](show.md) - Show credentials and other variables --- ###### rill env pull --- note: GENERATED. DO NOT EDIT. title: rill env pull --- ## rill env pull Pull cloud credentials into local .env file ``` rill env pull [] [flags] ``` ### Flags ``` --environment string Environment to resolve for (options: dev, prod) (default "dev") --path string Project directory (default ".") --project string Cloud project name (will attempt to infer from Git remote if not provided) ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill env](env.md) - Manage variables for a project --- ###### rill env push --- note: GENERATED. DO NOT EDIT. title: rill env push --- ## rill env push Push local .env contents to cloud ``` rill env push [] [flags] ``` ### Flags ``` --environment string Environment to push for (options: dev, prod) (default: all environments found in .env files) --path string Project directory (default ".") --project string Cloud project name (will attempt to infer from Git remote if not provided) ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill env](env.md) - Manage variables for a project --- ###### rill env rm --- note: GENERATED. DO NOT EDIT. title: rill env rm --- ## rill env rm Remove an env variable ``` rill env rm [] [flags] ``` ### Flags ``` --environment string Optional environment to resolve for (options: dev, prod) --path string Project directory (default ".") --project string Cloud project name (will attempt to infer from Git remote if not provided) ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill env](env.md) - Manage variables for a project --- ###### rill env set --- note: GENERATED. DO NOT EDIT. title: rill env set --- ## rill env set Set variable ``` rill env set [] [flags] ``` ### Flags ``` --environment string Optional environment to set for (options: dev, prod) --path string Project directory (default ".") --project string Cloud project name (will attempt to infer from Git remote if not provided) ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill env](env.md) - Manage variables for a project --- ###### rill env show --- note: GENERATED. DO NOT EDIT. title: rill env show --- ## rill env show Show credentials and other variables ``` rill env show [] [flags] ``` ### Flags ``` --environment string Optional environment to resolve for (options: dev, prod) --path string Project directory (default ".") --project string Cloud project name (will attempt to infer from Git remote if not provided) ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill env](env.md) - Manage variables for a project --- #### Org --- ###### rill org create --- note: GENERATED. DO NOT EDIT. title: rill org create --- ## rill org create Create organization ``` rill org create [flags] ``` ### Flags ``` --display-name string Display name --description string Description ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill org](org.md) - Manage organizations --- ###### rill org delete --- note: GENERATED. DO NOT EDIT. title: rill org delete --- ## rill org delete Delete organization ### Synopsis Delete an organization and all its associated projects. This operation cannot be undone. Use --force to skip confirmation. ``` rill org delete [] [flags] ``` ### Examples ``` rill org delete myorg rill org delete myorg --force ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill org](org.md) - Manage organizations --- ###### rill org edit --- note: GENERATED. DO NOT EDIT. title: rill org edit --- ## rill org edit Edit organization details ``` rill org edit [] [flags] ``` ### Flags ``` --org string Organization name --display-name string Display name --description string Description --default-project-role string Default role for members on new projects (options: admin, editor, viewer, none) --billing-email string Billing email ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill org](org.md) - Manage organizations --- ###### rill org list --- note: GENERATED. DO NOT EDIT. title: rill org list --- ## rill org list List all organizations ``` rill org list [flags] ``` ### Flags ``` --page-size uint32 Number of orgs to return per page (default 50) --page-token string Pagination token ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill org](org.md) - Manage organizations --- ###### rill org --- note: GENERATED. DO NOT EDIT. title: rill org --- ## rill org Manage organizations ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill](../cli.md) - A CLI for Rill * [rill org create](create.md) - Create organization * [rill org delete](delete.md) - Delete organization * [rill org edit](edit.md) - Edit organization details * [rill org list](list.md) - List all organizations * [rill org rename](rename.md) - Rename organization * [rill org show](show.md) - Show org details * [rill org switch](switch.md) - Switch to other organization * [rill org upload-favicon](upload-favicon.md) - Upload a custom favicon * [rill org upload-logo](upload-logo.md) - Upload a custom logo --- ###### rill org rename --- note: GENERATED. DO NOT EDIT. title: rill org rename --- ## rill org rename Rename organization ``` rill org rename [flags] ``` ### Flags ``` --org string Current org name --new-name string New org name --display-name string New display name ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill org](org.md) - Manage organizations --- ###### rill org show --- note: GENERATED. DO NOT EDIT. title: rill org show --- ## rill org show Show org details ``` rill org show [] [flags] ``` ### Flags ``` --org string Organization name ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill org](org.md) - Manage organizations --- ###### rill org switch --- note: GENERATED. DO NOT EDIT. title: rill org switch --- ## rill org switch Switch to other organization ``` rill org switch [] [flags] ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill org](org.md) - Manage organizations --- ###### rill org upload-favicon --- note: GENERATED. DO NOT EDIT. title: rill org upload-favicon --- ## rill org upload-favicon Upload a custom favicon ``` rill org upload-favicon [ []] [flags] ``` ### Flags ``` --org string Organization name --path string Path to image file (PNG or JPEG) --remove Remove the current favicon ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill org](org.md) - Manage organizations --- ###### rill org upload-logo --- note: GENERATED. DO NOT EDIT. title: rill org upload-logo --- ## rill org upload-logo Upload a custom logo ``` rill org upload-logo [ []] [flags] ``` ### Flags ``` --org string Organization name --path string Path to image file (PNG or JPEG) --remove Remove the current logo --dark Target the dark-mode logo variant ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill org](org.md) - Manage organizations --- #### Project --- ##### Deployment --- ####### rill project deployment create --- note: GENERATED. DO NOT EDIT. title: rill project deployment create --- ## rill project deployment create Create a deployment for a specific branch ``` rill project deployment create [] [flags] ``` ### Flags ``` --path string Project directory (default ".") --project string Project name ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill project deployment](deployment.md) - Manage project deployments --- ####### rill project deployment delete --- note: GENERATED. DO NOT EDIT. title: rill project deployment delete --- ## rill project deployment delete Delete a deployment by branch ``` rill project deployment delete [] [flags] ``` ### Flags ``` --path string Project directory (default ".") --project string Project name ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill project deployment](deployment.md) - Manage project deployments --- ####### rill project deployment --- note: GENERATED. DO NOT EDIT. title: rill project deployment --- ## rill project deployment Manage project deployments ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill project](../project.md) - Manage projects * [rill project deployment create](create.md) - Create a deployment for a specific branch * [rill project deployment delete](delete.md) - Delete a deployment by branch * [rill project deployment list](list.md) - List all deployments for a project * [rill project deployment show](show.md) - Show details of a deployment * [rill project deployment start](start.md) - Start a deployment by branch * [rill project deployment stop](stop.md) - Stop a deployment by branch --- ####### rill project deployment list --- note: GENERATED. DO NOT EDIT. title: rill project deployment list --- ## rill project deployment list List all deployments for a project ``` rill project deployment list [] [flags] ``` ### Flags ``` --environment string Filter deployments by environment (prod/dev) --path string Project directory (default ".") --project string Project name ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill project deployment](deployment.md) - Manage project deployments --- ####### rill project deployment show --- note: GENERATED. DO NOT EDIT. title: rill project deployment show --- ## rill project deployment show Show details of a deployment ``` rill project deployment show [] [flags] ``` ### Flags ``` --path string Project directory (default ".") --project string Project name ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill project deployment](deployment.md) - Manage project deployments --- ####### rill project deployment start --- note: GENERATED. DO NOT EDIT. title: rill project deployment start --- ## rill project deployment start Start a deployment by branch ``` rill project deployment start [] [flags] ``` ### Flags ``` --path string Project directory (default ".") --project string Project name ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill project deployment](deployment.md) - Manage project deployments --- ####### rill project deployment stop --- note: GENERATED. DO NOT EDIT. title: rill project deployment stop --- ## rill project deployment stop Stop a deployment by branch ``` rill project deployment stop [] [flags] ``` ### Flags ``` --path string Project directory (default ".") --project string Project name ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill project deployment](deployment.md) - Manage project deployments --- ###### rill project clone --- note: GENERATED. DO NOT EDIT. title: rill project clone --- ## rill project clone Clone Project ``` rill project clone [flags] ``` ### Flags ``` --path string Project path to clone to ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill project](project.md) - Manage projects --- ###### rill project connect-github --- note: GENERATED. DO NOT EDIT. title: rill project connect-github --- ## rill project connect-github Deploy project to Rill Cloud by pulling project files from a git repository ``` rill project connect-github [] [flags] ``` ### Flags ``` --path string Path to project repository (default: current directory) (default ".") --subpath string Relative path to project in the repository (for monorepos) --remote string Remote name (default: origin) (default "origin") --org string Org to deploy project in --name string Project name (default: Git repo name) --description string Project description --public Make dashboards publicly accessible --provisioner string Project provisioner --primary-branch string Git branch to deploy from (default: the default Git branch) --push-env Push local .env file to Rill Cloud (default true) ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill project](project.md) - Manage projects --- ###### rill project delete --- note: GENERATED. DO NOT EDIT. title: rill project delete --- ## rill project delete Delete the project ``` rill project delete [] [flags] ``` ### Flags ``` --force Delete forcefully, skips the confirmation --project string Project Name --path string Project directory (default ".") ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill project](project.md) - Manage projects --- ###### rill project deploy --- note: GENERATED. DO NOT EDIT. title: rill project deploy --- ## rill project deploy Deploy project to Rill Cloud by using a Rill Managed Git repo ``` rill project deploy [] [flags] ``` ### Flags ``` --path string Path to project repository (default: current directory) (default ".") --subpath string Relative path to project in the repository (for monorepos) --org string Org to deploy project in --project string Project name (default: Git repo name) --description string Project description --public Make dashboards publicly accessible --provisioner string Project provisioner --primary-branch string Git branch to deploy from (default: the default Git branch) --push-env Push local .env file to Rill Cloud (default true) --force-push Force push local changes ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill project](project.md) - Manage projects --- ###### rill project describe --- note: GENERATED. DO NOT EDIT. title: rill project describe --- ## rill project describe Retrieve detailed state for a resource ### Synopsis Retrieve detailed state for a specific resource (source, model, dashboard, ...) ``` rill project describe [] [flags] ``` ### Flags ``` --branch string Target deployment by Git branch (default: primary deployment) --path string Project directory (default ".") --project string Project name ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill project](project.md) - Manage projects --- ###### rill project edit --- note: GENERATED. DO NOT EDIT. title: rill project edit --- ## rill project edit Edit the project details ``` rill project edit [] [flags] ``` ### Flags ``` --project string Project Name --description string Project Description --primary-branch string Primary branch name --public Make dashboards publicly accessible --path string Project directory (default ".") --remote-url string Github remote URL --subpath string Relative path to project in the repository (for monorepos) --provisioner string Project provisioner (default: current provisioner) --prod-ttl-seconds int Time-to-live in seconds for production deployment (0 means no expiration) --dev-ttl-seconds int Time-to-live in seconds for dev deployment (must be greater than 0) --prod-slots int Slots to allocate for production deployments --dev-slots int Slots to allocate for dev deployments ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill project](project.md) - Manage projects --- ###### rill project hibernate --- note: GENERATED. DO NOT EDIT. title: rill project hibernate --- ## rill project hibernate Hibernate project ``` rill project hibernate [] [flags] ``` ### Flags ``` --project string Name --path string Project directory (default ".") --redeploy Restore a previously hibernated project --force Force a redeploy for a non-hibernated project ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill project](project.md) - Manage projects --- ###### rill project list --- note: GENERATED. DO NOT EDIT. title: rill project list --- ## rill project list List all the projects ``` rill project list [flags] ``` ### Flags ``` --page-size uint32 Number of projects to return per page (default 50) --page-token string Pagination token ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill project](project.md) - Manage projects --- ###### rill project logs --- note: GENERATED. DO NOT EDIT. title: rill project logs --- ## rill project logs Show project logs ``` rill project logs [] [flags] ``` ### Flags ``` --project string Project Name --path string Project directory (default ".") --branch string Target deployment by Git branch (default: primary deployment) -f, --follow Follow logs -t, --tail int Number of lines to show from the end of the logs, use -1 for all logs (default -1) --level string Minimum log level to show (DEBUG, INFO, WARN, ERROR, FATAL) (default "INFO") ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill project](project.md) - Manage projects --- ###### rill project partitions --- note: GENERATED. DO NOT EDIT. title: rill project partitions --- ## rill project partitions List partitions for a model ``` rill project partitions [] [flags] ``` ### Flags ``` --project string Project Name --path string Project directory (default ".") --branch string Target deployment by Git branch (default: primary deployment) --model string Model Name --pending Only fetch pending partitions --errored Only fetch errored partitions --skipped Only fetch skipped partitions --local Target locally running Rill --page-size uint32 Number of partitions to return per page (default 50) --page-token string Pagination token ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill project](project.md) - Manage projects --- ###### rill project --- note: GENERATED. DO NOT EDIT. title: rill project --- ## rill project Manage projects ### Flags ``` --org string Organization Name ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill](../cli.md) - A CLI for Rill * [rill project clone](clone.md) - Clone Project * [rill project connect-github](connect-github.md) - Deploy project to Rill Cloud by pulling project files from a git repository * [rill project delete](delete.md) - Delete the project * [rill project deploy](deploy.md) - Deploy project to Rill Cloud by using a Rill Managed Git repo * [rill project deployment](deployment/deployment.md) - Manage project deployments * [rill project describe](describe.md) - Retrieve detailed state for a resource * [rill project edit](edit.md) - Edit the project details * [rill project hibernate](hibernate.md) - Hibernate project * [rill project list](list.md) - List all the projects * [rill project logs](logs.md) - Show project logs * [rill project partitions](partitions.md) - List partitions for a model * [rill project refresh](refresh.md) - Refresh one or more resources * [rill project rename](rename.md) - Rename project * [rill project show](show.md) - Show project details * [rill project skip-partition](skip-partition.md) - Skip partitions for a model * [rill project status](status.md) - Project deployment status * [rill project tables](tables.md) - Get information about tables in a project --- ###### rill project refresh --- note: GENERATED. DO NOT EDIT. title: rill project refresh --- ## rill project refresh Refresh one or more resources ``` rill project refresh [] [flags] ``` ### Flags ``` --project string Project name --path string Project directory (default ".") --branch string Target deployment by Git branch (default: primary deployment) --local Target locally running Rill --all Refresh all resources except alerts and reports (default) --full Fully reload the targeted models (use with --all or --model) --model strings Refresh a model --partition strings Refresh a model partition (must set --model) --errored-partitions Refresh all model partitions with errors (must set --model) --skipped-partitions Refresh all skipped model partitions (must set --model) --source strings Refresh a source --metrics-view strings Refresh a metrics view --alert strings Refresh an alert --report strings Refresh a report --connector strings Re-validate a connector --parser Refresh the parser (forces a pull from Github) ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill project](project.md) - Manage projects --- ###### rill project rename --- note: GENERATED. DO NOT EDIT. title: rill project rename --- ## rill project rename Rename project ### Synopsis Rename project Warning: Renaming a project will invalidate all dashboard URLs. ``` rill project rename [flags] ``` ### Flags ``` --project string Current Project Name --new-name string New Project Name ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill project](project.md) - Manage projects --- ###### rill project show --- note: GENERATED. DO NOT EDIT. title: rill project show --- ## rill project show Show project details ``` rill project show [] [flags] ``` ### Flags ``` --project string Name --path string Project directory (default ".") ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill project](project.md) - Manage projects --- ###### rill project skip-partition --- note: GENERATED. DO NOT EDIT. title: rill project skip-partition --- ## rill project skip-partition Skip partitions for a model ### Synopsis Mark partitions as skipped so they are excluded from execution and from the model's error state. Skipped partitions remain skipped until they are explicitly triggered (e.g. via 'rill project refresh --partition'). ``` rill project skip-partition [] [flags] ``` ### Flags ``` --project string Project Name --path string Project directory (default ".") --branch string Target deployment by Git branch (default: primary deployment) --model string Model Name --partition strings Skip specific partitions by key --pending Skip all pending partitions --errored Skip all errored partitions --local Target locally running Rill ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill project](project.md) - Manage projects --- ###### rill project status --- note: GENERATED. DO NOT EDIT. title: rill project status --- ## rill project status Project deployment status ``` rill project status [] [flags] ``` ### Flags ``` --branch string Target deployment by Git branch (default: primary deployment) --local Target locally running Rill --path string Project directory (default ".") --project string Project Name ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill project](project.md) - Manage projects --- ###### rill project tables --- note: GENERATED. DO NOT EDIT. title: rill project tables --- ## rill project tables Get information about tables in a project ``` rill project tables [] [flags] ``` ### Flags ``` --branch string Target deployment by Git branch (default: primary deployment) --local Target local runtime instead of Rill Cloud --path string Project directory (default ".") --project string Project name ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill project](project.md) - Manage projects --- #### Public Url --- ###### rill public-url create --- note: GENERATED. DO NOT EDIT. title: rill public-url create --- ## rill public-url create Create a public URL ``` rill public-url create [] [flags] ``` ### Flags ``` --project string Project name --path string Project directory (default ".") --ttl-minutes int Duration until the token expires (use 0 for no expiry) --filter string Limit access to the provided filter (json) --fields strings Limit access to the provided fields ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill public-url](public-url.md) - Manage public URLs --- ###### rill public-url delete --- note: GENERATED. DO NOT EDIT. title: rill public-url delete --- ## rill public-url delete Delete a public URL ``` rill public-url delete [flags] ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill public-url](public-url.md) - Manage public URLs --- ###### rill public-url list --- note: GENERATED. DO NOT EDIT. title: rill public-url list --- ## rill public-url list List all public URLs ``` rill public-url list [] [flags] ``` ### Flags ``` --project string Project name --path string Project directory (default ".") --page-size uint32 Number of projects to return per page (default 50) --page-token string Pagination token ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill public-url](public-url.md) - Manage public URLs --- ###### rill public-url --- note: GENERATED. DO NOT EDIT. title: rill public-url --- ## rill public-url Manage public URLs ### Flags ``` --org string Organization Name ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill](../cli.md) - A CLI for Rill * [rill public-url create](create.md) - Create a public URL * [rill public-url delete](delete.md) - Delete a public URL * [rill public-url list](list.md) - List all public URLs --- #### Service --- ##### Token --- ####### rill service token issue --- note: GENERATED. DO NOT EDIT. title: rill service token issue --- ## rill service token issue Issue service token ``` rill service token issue [] [flags] ``` ### Flags ``` --service string Service Name ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill service token](token.md) - Manage service tokens --- ####### rill service token list --- note: GENERATED. DO NOT EDIT. title: rill service token list --- ## rill service token list List tokens ``` rill service token list [] [flags] ``` ### Flags ``` --service string Service Name ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill service token](token.md) - Manage service tokens --- ####### rill service token revoke --- note: GENERATED. DO NOT EDIT. title: rill service token revoke --- ## rill service token revoke Revoke token ``` rill service token revoke [flags] ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill service token](token.md) - Manage service tokens --- ####### rill service token --- note: GENERATED. DO NOT EDIT. title: rill service token --- ## rill service token Manage service tokens ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill service](../service.md) - Manage service accounts * [rill service token issue](issue.md) - Issue service token * [rill service token list](list.md) - List tokens * [rill service token revoke](revoke.md) - Revoke token --- ###### rill service create --- note: GENERATED. DO NOT EDIT. title: rill service create --- ## rill service create Create service ``` rill service create [flags] ``` ### Flags ``` --attributes string JSON object of key-value pairs for service attributes --org-role string Organization role to assign to the service (admin, editor, viewer, guest) --project string Project to assign the role to (required if project-role is set) --project-role string Project role to assign to the service (admin, editor, viewer) ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill service](service.md) - Manage service accounts --- ###### rill service delete --- note: GENERATED. DO NOT EDIT. title: rill service delete --- ## rill service delete Delete service ``` rill service delete [flags] ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill service](service.md) - Manage service accounts --- ###### rill service edit --- note: GENERATED. DO NOT EDIT. title: rill service edit --- ## rill service edit edit service properties ``` rill service edit [flags] ``` ### Flags ``` --new-name string New service name --attributes string JSON object of key-value pairs for service attributes ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill service](service.md) - Manage service accounts --- ###### rill service list --- note: GENERATED. DO NOT EDIT. title: rill service list --- ## rill service list List service ``` rill service list [flags] ``` ### Flags ``` --project string Project name to filter services ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill service](service.md) - Manage service accounts --- ###### rill service remove --- note: GENERATED. DO NOT EDIT. title: rill service remove --- ## rill service remove remove service from org or project ``` rill service remove [flags] ``` ### Flags ``` --project string Project to remove service from ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill service](service.md) - Manage service accounts --- ###### rill service --- note: GENERATED. DO NOT EDIT. title: rill service --- ## rill service Manage service accounts ### Flags ``` --org string Organization Name ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill](../cli.md) - A CLI for Rill * [rill service create](create.md) - Create service * [rill service delete](delete.md) - Delete service * [rill service edit](edit.md) - edit service properties * [rill service list](list.md) - List service * [rill service remove](remove.md) - remove service from org or project * [rill service set-role](set-role.md) - Set role for service * [rill service show](show.md) - Show service * [rill service token](token/token.md) - Manage service tokens --- ###### rill service set-role --- note: GENERATED. DO NOT EDIT. title: rill service set-role --- ## rill service set-role Set role for service ``` rill service set-role [flags] ``` ### Flags ``` --project string Project --role string Role of the service (options: admin, editor, viewer, guest) ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill service](service.md) - Manage service accounts --- ###### rill service show --- note: GENERATED. DO NOT EDIT. title: rill service show --- ## rill service show Show service ``` rill service show [flags] ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) --org string Organization Name ``` ### SEE ALSO * [rill service](service.md) - Manage service accounts --- #### Token --- ###### rill token issue --- note: GENERATED. DO NOT EDIT. title: rill token issue --- ## rill token issue Issue personal access token ``` rill token issue [flags] ``` ### Flags ``` --display-name string Display name for the token --ttl-minutes int Optional minutes until the token should expire ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill token](token.md) - Manage personal access tokens --- ###### rill token list --- note: GENERATED. DO NOT EDIT. title: rill token list --- ## rill token list List personal access tokens ``` rill token list [flags] ``` ### Flags ``` --page-size uint32 Number of tokens to return per page (default 1000) --page-token string Pagination token --refresh List refresh tokens only ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill token](token.md) - Manage personal access tokens --- ###### rill token revoke --- note: GENERATED. DO NOT EDIT. title: rill token revoke --- ## rill token revoke Revoke personal access token(s) ``` rill token revoke [token-id] [flags] ``` ### Flags ``` --all Revoke all access and refresh tokens for the current user ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill token](token.md) - Manage personal access tokens --- ###### rill token --- note: GENERATED. DO NOT EDIT. title: rill token --- ## rill token Manage personal access tokens ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill](../cli.md) - A CLI for Rill * [rill token issue](issue.md) - Issue personal access token * [rill token list](list.md) - List personal access tokens * [rill token revoke](revoke.md) - Revoke personal access token(s) --- #### User --- ##### Whitelist --- ####### rill user whitelist list --- note: GENERATED. DO NOT EDIT. title: rill user whitelist list --- ## rill user whitelist list List whitelisted email domains for the org or project ``` rill user whitelist list [flags] ``` ### Flags ``` --org string Organization --project string Project ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill user whitelist](whitelist.md) - Whitelist access by email domain --- ####### rill user whitelist remove --- note: GENERATED. DO NOT EDIT. title: rill user whitelist remove --- ## rill user whitelist remove Remove whitelisted email domain for the org or project ``` rill user whitelist remove [flags] ``` ### Flags ``` --org string Organization --project string Project ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill user whitelist](whitelist.md) - Whitelist access by email domain --- ####### rill user whitelist setup --- note: GENERATED. DO NOT EDIT. title: rill user whitelist setup --- ## rill user whitelist setup Whitelist an email domain for the org or project ``` rill user whitelist setup [flags] ``` ### Flags ``` --org string Organization --project string Project name --role string Role of the user (default "viewer") ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill user whitelist](whitelist.md) - Whitelist access by email domain --- ####### rill user whitelist --- note: GENERATED. DO NOT EDIT. title: rill user whitelist --- ## rill user whitelist Whitelist access by email domain ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill user](../user.md) - Manage users * [rill user whitelist list](list.md) - List whitelisted email domains for the org or project * [rill user whitelist remove](remove.md) - Remove whitelisted email domain for the org or project * [rill user whitelist setup](setup.md) - Whitelist an email domain for the org or project --- ###### rill user add --- note: GENERATED. DO NOT EDIT. title: rill user add --- ## rill user add Add user to a project, organization or group ``` rill user add [flags] ``` ### Flags ``` --canvas stringArray Canvas resource to restrict to (repeat for multiple) --email string Email of the user --explore stringArray Explore resource to restrict to (repeat for multiple) --group string User group --org string Organization --project string Project --restrict-resources Restrict the user to the provided resources (defaults to true when resources are provided) --role string Role of the user (options: admin, editor, viewer, guest) ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill user](user.md) - Manage users --- ###### rill user list --- note: GENERATED. DO NOT EDIT. title: rill user list --- ## rill user list List users ``` rill user list [flags] ``` ### Flags ``` --group string User group --org string Organization --page-size uint32 Number of users to return per page (default 50) --page-token string Pagination token --project string Project ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill user](user.md) - Manage users --- ###### rill user remove --- note: GENERATED. DO NOT EDIT. title: rill user remove --- ## rill user remove Remove a user ``` rill user remove [flags] ``` ### Flags ``` --email string Email of the user --group string User group --org string Organization --project string Project ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill user](user.md) - Manage users --- ###### rill user set-attributes --- note: GENERATED. DO NOT EDIT. title: rill user set-attributes --- ## rill user set-attributes Set custom attributes for a user ``` rill user set-attributes [flags] ``` ### Flags ``` --attribute stringToString Attributes in key=value format (--attribute app=foo --attribute dept=bar) (default []) --email string Email of the user (required) --force Skip confirmation prompt when overwriting existing attributes --json string Attributes as JSON object (--json '{"app":"foo","dept":"bar"}') --org string Organization ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill user](user.md) - Manage users --- ###### rill user set-resources --- note: GENERATED. DO NOT EDIT. title: rill user set-resources --- ## rill user set-resources Set a user's project resources and restriction flag (overwrites existing list) ``` rill user set-resources [flags] ``` ### Flags ``` --canvas stringArray Canvas Resource to set (repeat for multiple) --email string Email of the user (required) --explore stringArray Explore Resource to set (repeat for multiple) --org string Organization --project string Project (required) --restrict-resources Whether to restrict the user to the provided resources (defaults to true when resources are provided) ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill user](user.md) - Manage users --- ###### rill user set-role --- note: GENERATED. DO NOT EDIT. title: rill user set-role --- ## rill user set-role Change a user's role ``` rill user set-role [flags] ``` ### Flags ``` --email string Email of the user --org string Organization --project string Project --role string Role of the user (options: admin, editor, viewer, guest) ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill user](user.md) - Manage users --- ###### rill user show --- note: GENERATED. DO NOT EDIT. title: rill user show --- ## rill user show Show detailed information about a user ``` rill user show [flags] ``` ### Flags ``` --email string Email of the user (required) --org string Organization --project string Project name to include project membership details (optional) ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill user](user.md) - Manage users --- ###### rill user --- note: GENERATED. DO NOT EDIT. title: rill user --- ## rill user Manage users ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill](../cli.md) - A CLI for Rill * [rill user add](add.md) - Add user to a project, organization or group * [rill user list](list.md) - List users * [rill user remove](remove.md) - Remove a user * [rill user set-attributes](set-attributes.md) - Set custom attributes for a user * [rill user set-resources](set-resources.md) - Set a user's project resources and restriction flag (overwrites existing list) * [rill user set-role](set-role.md) - Change a user's role * [rill user show](show.md) - Show detailed information about a user * [rill user whitelist](whitelist/whitelist.md) - Whitelist access by email domain --- #### Usergroup --- ###### rill usergroup add --- note: GENERATED. DO NOT EDIT. title: rill usergroup add --- ## rill usergroup add Add a group to a project or organization ``` rill usergroup add [flags] ``` ### Flags ``` --canvas stringArray Canvas resource to restrict to (repeat for multiple) --explore stringArray Explore resource to restrict to (repeat for multiple) --group string User group --org string Organization --project string Project --restrict-resources Restrict the user group to provided resources (defaults to true when resources are provided) --role string Role of the user group (options: admin, editor, viewer) ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill usergroup](usergroup.md) - Manage user groups --- ###### rill usergroup create --- note: GENERATED. DO NOT EDIT. title: rill usergroup create --- ## rill usergroup create Create a group ``` rill usergroup create [] [flags] ``` ### Flags ``` --org string Organization ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill usergroup](usergroup.md) - Manage user groups --- ###### rill usergroup delete --- note: GENERATED. DO NOT EDIT. title: rill usergroup delete --- ## rill usergroup delete Delete a group ``` rill usergroup delete [flags] ``` ### Flags ``` --org string Organization ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill usergroup](usergroup.md) - Manage user groups --- ###### rill usergroup edit --- note: GENERATED. DO NOT EDIT. title: rill usergroup edit --- ## rill usergroup edit Edit a group ``` rill usergroup edit [flags] ``` ### Flags ``` --description string Description --new-name string New user group name --org string Organization ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill usergroup](usergroup.md) - Manage user groups --- ###### rill usergroup list --- note: GENERATED. DO NOT EDIT. title: rill usergroup list --- ## rill usergroup list List groups ``` rill usergroup list [flags] ``` ### Flags ``` --org string Organization --page-size uint32 Number of user groups to return per page (default 50) --page-token string Pagination token --project string Project ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill usergroup](usergroup.md) - Manage user groups --- ###### rill usergroup remove --- note: GENERATED. DO NOT EDIT. title: rill usergroup remove --- ## rill usergroup remove Remove a group's role on a project or organization ``` rill usergroup remove [flags] ``` ### Flags ``` --group string User group --org string Organization --project string Project ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill usergroup](usergroup.md) - Manage user groups --- ###### rill usergroup rename --- note: GENERATED. DO NOT EDIT. title: rill usergroup rename --- ## rill usergroup rename Rename a group ``` rill usergroup rename [] [flags] ``` ### Flags ``` --new-name string New user group name --org string Organization ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill usergroup](usergroup.md) - Manage user groups --- ###### rill usergroup set-resources --- note: GENERATED. DO NOT EDIT. title: rill usergroup set-resources --- ## rill usergroup set-resources Set a user group's project resources and restriction flag (overwrites existing list) ``` rill usergroup set-resources [flags] ``` ### Flags ``` --canvas stringArray Canvas resource to restrict to (repeat for multiple) --explore stringArray Explore resource to restrict to (repeat for multiple) --group string User group (required) --org string Organization --project string Project (required) --restrict-resources Whether to restrict the group to the provided resources (defaults to true when resources are provided) ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill usergroup](usergroup.md) - Manage user groups --- ###### rill usergroup set-role --- note: GENERATED. DO NOT EDIT. title: rill usergroup set-role --- ## rill usergroup set-role Change a group's role on a project or organization ``` rill usergroup set-role [flags] ``` ### Flags ``` --group string User group --org string Organization --project string Project --role string Role of the user group (options: admin, editor, viewer) ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill usergroup](usergroup.md) - Manage user groups --- ###### rill usergroup show --- note: GENERATED. DO NOT EDIT. title: rill usergroup show --- ## rill usergroup show Show group ``` rill usergroup show [flags] ``` ### Flags ``` --org string Organization ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill usergroup](usergroup.md) - Manage user groups --- ###### rill usergroup --- note: GENERATED. DO NOT EDIT. title: rill usergroup --- ## rill usergroup Manage user groups ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill](../cli.md) - A CLI for Rill * [rill usergroup add](add.md) - Add a group to a project or organization * [rill usergroup create](create.md) - Create a group * [rill usergroup delete](delete.md) - Delete a group * [rill usergroup edit](edit.md) - Edit a group * [rill usergroup list](list.md) - List groups * [rill usergroup remove](remove.md) - Remove a group's role on a project or organization * [rill usergroup set-resources](set-resources.md) - Set a user group's project resources and restriction flag (overwrites existing list) * [rill usergroup set-role](set-role.md) - Change a group's role on a project or organization * [rill usergroup show](show.md) - Show group --- ##### rill chat --- note: GENERATED. DO NOT EDIT. title: rill chat --- ## rill chat Chat with the Rill AI ``` rill chat [] [flags] ``` ### Flags ``` --project string Project name --path string Project directory (default ".") --branch string Target deployment by Git branch (default: primary deployment) --local Target locally running Rill ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill](cli.md) - A CLI for Rill --- ##### CLI usage --- note: GENERATED. DO NOT EDIT. title: CLI usage sidebar_position: 15 --- ## rill A CLI for Rill ### Synopsis Work with Rill projects from the command line. ### Flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) -v, --version Show rill version ``` ### SEE ALSO * [rill billing](billing/billing.md) - Billing related commands for org * [rill chat](chat.md) - Chat with the Rill AI * [rill deploy](deploy.md) - Deploy project to Rill Cloud * [rill docs](docs/docs.md) - Open docs.rilldata.com * [rill env](env/env.md) - Manage variables for a project * [rill init](init.md) - Initialize a new Rill project * [rill login](login.md) - Authenticate with the Rill API * [rill logout](logout.md) - Logout of the Rill API * [rill org](org/org.md) - Manage organizations * [rill project](project/project.md) - Manage projects * [rill public-url](public-url/public-url.md) - Manage public URLs * [rill query](query.md) - Query data in a project * [rill service](service/service.md) - Manage service accounts * [rill start](start.md) - Build project and start web app * [rill token](token/token.md) - Manage personal access tokens * [rill uninstall](uninstall.md) - Uninstall the Rill binary * [rill upgrade](upgrade.md) - Upgrade Rill to the latest version * [rill user](user/user.md) - Manage users * [rill usergroup](usergroup/usergroup.md) - Manage user groups * [rill validate](validate.md) - Validate project resources * [rill version](version.md) - Show Rill version * [rill whoami](whoami.md) - Show current user --- ##### rill deploy --- note: GENERATED. DO NOT EDIT. title: rill deploy --- ## rill deploy Deploy project to Rill Cloud ``` rill deploy [] [flags] ``` ### Flags ``` --path string Path to project repository (default: current directory) (default ".") --subpath string Relative path to project in the repository (for monorepos) --remote string Remote name (default: origin) (default "origin") --org string Org to deploy project in --project string Project name (default: Git repo name) --description string Project description --public Make dashboards publicly accessible --provisioner string Project provisioner --primary-branch string Git branch to deploy from (default: the default Git branch) --push-env Push local .env file to Rill Cloud (default true) --force-push Force push local changes in case of Rill managed repos --managed Create project using rill managed repo --github Use github repo to create the project ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill](cli.md) - A CLI for Rill --- ##### rill init --- note: GENERATED. DO NOT EDIT. title: rill init --- ## rill init Initialize a new Rill project ### Synopsis Initialize a new Rill project. Use flags to customize the project or run interactively to be prompted for each option. Available example projects: - rill-cost-monitoring (duckdb) - rill-github-analytics (duckdb) - rill-openrtb-prog-ads (duckdb) ``` rill init [] [flags] ``` ### Examples ``` # Interactive initialization (prompts for all options) rill init # Create an empty DuckDB project with Claude agent instructions rill init my-project --olap duckdb --agent claude # Add Claude agent instructions to an existing Rill project rill init ./existing-project --agent claude ``` ### Flags ``` --agent string Agent instructions (options: claude, cursor, agentsmd, all, none) (default "claude") --example string Example project name (default: empty project) --olap string OLAP engine (options: duckdb, clickhouse) (default "duckdb") ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill](cli.md) - A CLI for Rill --- ##### rill login --- note: GENERATED. DO NOT EDIT. title: rill login --- ## rill login Authenticate with the Rill API ``` rill login [flags] ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill](cli.md) - A CLI for Rill --- ##### rill logout --- note: GENERATED. DO NOT EDIT. title: rill logout --- ## rill logout Logout of the Rill API ``` rill logout [flags] ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill](cli.md) - A CLI for Rill --- ##### rill query --- note: GENERATED. DO NOT EDIT. title: rill query --- ## rill query Query data in a project ### Synopsis Query data in a project. You can query data by providing a SQL query and optional connector name. As an advanced option, you can also query other resolvers such as metrics_sql. Note that large results are automatically truncated (use --limit to override). ``` rill query [] [flags] ``` ### Examples ``` # SQL query against a Rill Cloud project rill query my-project --sql "SELECT * FROM my-table" # SQL query against a local Rill project running with 'rill start' rill query --local --sql "SELECT * FROM my-table" ``` ### Flags ``` --args stringToString Explicit resolver args (only with --resolver) (default []) --branch string Target deployment by Git branch (default: primary deployment) --connector string Connector to execute against. Defaults to the OLAP connector. --limit int The maximum number of rows to print (default 100) --local Target local runtime instead of Rill Cloud --org string Organization Name --path string Project directory (default ".") --project string Project name --properties stringToString Explicit resolver properties (only with --resolver) (default []) --resolver string Explicit resolver (cannot be combined with --sql) --sql string A SELECT query to execute ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill](cli.md) - A CLI for Rill --- ##### rill start --- note: GENERATED. DO NOT EDIT. title: rill start --- ## rill start Build project and start web app ``` rill start [] [flags] ``` ### Flags ``` -e, --env strings Set environment variables --environment string Environment name (default "dev") --reset Clear and re-ingest source data --pull-env Pull environment variables from Rill Cloud before starting the project (default true) --no-open Do not open browser --verbose Sets the log level to debug --preview Start in dashboard-only view (no code editor) --port int Port for HTTP (default 9009) --port-grpc int Port for gRPC (internal) (default 49009) --no-ui Serve only the backend --debug Collect additional debug info --log-format string Log format (options: "console", "json") (default "console") --tls-cert string Path to TLS certificate --tls-key string Path to TLS key file --allowed-origins strings Override allowed origins for CORS ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill](cli.md) - A CLI for Rill --- ##### rill uninstall --- note: GENERATED. DO NOT EDIT. title: rill uninstall --- ## rill uninstall Uninstall the Rill binary ``` rill uninstall [flags] ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill](cli.md) - A CLI for Rill --- ##### rill upgrade --- note: GENERATED. DO NOT EDIT. title: rill upgrade --- ## rill upgrade Upgrade Rill to the latest version ``` rill upgrade [flags] ``` ### Flags ``` --nightly Install the latest nightly build --version string Install a specific version of Rill ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill](cli.md) - A CLI for Rill --- ##### rill validate --- note: GENERATED. DO NOT EDIT. title: rill validate --- ## rill validate Validate project resources ``` rill validate [] [flags] ``` ### Flags ``` -e, --env strings Set environment variables --reset Clear and re-ingest source data --pull-env Pull environment variables from Rill Cloud before starting the project (default true) --environment string Environment name (default "dev") --verbose Sets the log level to debug --silent Suppress all log output by setting log level to panic, overrides verbose flag --debug Collect additional debug info --log-format string Log format (options: "console", "json") (default "console") --model-timeout-seconds uint32 Timeout for reconciliation of models, set 0 for no timeout (default 60) -o, --output-file string Output file for validation results (JSON format) ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill](cli.md) - A CLI for Rill --- ##### rill version --- note: GENERATED. DO NOT EDIT. title: rill version --- ## rill version Show Rill version ``` rill version [flags] ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill](cli.md) - A CLI for Rill --- ##### rill whoami --- note: GENERATED. DO NOT EDIT. title: rill whoami --- ## rill whoami Show current user ``` rill whoami [flags] ``` ### Global flags ``` --api-token string Token for authenticating with the cloud API --format string Output format (options: "human", "json", "csv") (default "human") -h, --help Print usage --interactive Prompt for missing required parameters (default true) ``` ### SEE ALSO * [rill](cli.md) - A CLI for Rill --- ### Project Files --- ##### Alert YAML --- note: GENERATED. DO NOT EDIT. title: Alert YAML sidebar_position: 37 --- Along with dashboard-level alerts that can be created via the UI, you can develop more extensive alerting in an alert YAML file. When creating an alert via a YAML file, you'll see this denoted in the UI as `Created through code`. ## Properties ### `type` _[string]_ - Refers to the resource type and must be `alert` _(required)_ ### `refresh` _[object]_ - Refresh schedule for the alert ```yaml refresh: cron: "* * * * *" #every: "24h" ``` _(required)_ - **`cron`** - _[string]_ - A cron expression that defines the execution schedule - **`time_zone`** - _[string]_ - Time zone to interpret the schedule in (e.g., 'UTC', 'America/Los_Angeles'). - **`disable`** - _[boolean]_ - If true, disables the resource without deleting it. - **`ref_update`** - _[boolean]_ - If true, allows the resource to run when a dependency updates. - **`run_in_dev`** - _[boolean]_ - If true, allows the schedule to run in development mode. ### `display_name` _[string]_ - Display name for the alert ### `description` _[string]_ - Description for the alert ### `intervals` _[object]_ - Defines the alert interval to check. - **`duration`** - _[string]_ - An ISO 8601 duration to define the interval duration. - **`limit`** - _[integer]_ - Maximum number of intervals to check on invocation. - **`check_unclosed`** - _[boolean]_ - Whether unclosed intervals should be checked. ### `watermark` _[string]_ - Specifies how the watermark is determined for incremental processing. Use 'trigger_time' to set it at runtime or 'inherit' to use the upstream model's watermark. ### `timeout` _[string]_ - Defines the timeout of the alert in seconds. (optional) ### `data` _[oneOf]_ - Data source for the alert _(required)_ - **option 1** - _[object]_ - Executes a raw SQL query against the project's data models. - **`sql`** - _[string]_ - Raw SQL query to run against existing models in the project. _(required)_ - **`connector`** - _[string]_ - Specifies the connector to use when running SQL or glob queries. - **option 2** - _[object]_ - Executes a SQL query that targets a defined metrics view. - **`metrics_sql`** - _[string]_ - SQL query that targets a metrics view in the project _(required)_ - **option 3** - _[object]_ - Calls a custom API defined in the project to compute data. - **`api`** - _[string]_ - Name of a custom API defined in the project. _(required)_ - **`args`** - _[object]_ - Arguments to pass to the custom API. - **option 4** - _[object]_ - Uses a file-matching pattern (glob) to query data from a connector. - **`glob`** - _[oneOf]_ - Simple path/glob pattern or path/glob pattern with advanced options. _(required)_ - **option 1** - _[string]_ - Glob pattern used to match files or directories in the object store. - **option 2** - _[object]_ - Configuration for specifying a file path/glob pattern with advanced options. - **`connector`** - _[string]_ - Specifies the object store connector to use (e.g. "s3", "gcs"). If not provided, it is inferred from the scheme of the path. - **`path`** - _[string]_ - Glob pattern used to match files or directories in the object store. _(required)_ - **`start`** - _[string]_ - Defines the lower bound (inclusive) for partition filtering. Only partitions with paths greater than or equal to this value are considered. - **`end`** - _[string]_ - Defines the upper bound (exclusive) for partition filtering. Only partitions with paths less than this value are considered. - **`last`** - _[integer]_ - Limits the result to the last N partitions (the N highest paths in lexicographic order). This hard limit always applies, including on the first run when there is no existing data. Additionally, when previously processed partitions exist, it raises the lower bound to the Nth partition from the end of those successfully processed partitions, forming a rolling window that prevents full re-listings each time. - **`partition`** - _[string]_ - Controls how matched files are grouped: - "file" (default) : Each matched path is returned as a row. Use the glob pattern to match files or directories at the level you want (for example, file-level or directory-level). - "directory": This mode is deprecated. Instead, use "file" with a glob that directly matches the directory level you want. - "hive": groups files by directory and extracts Hive-style partition values from the path as columns. - **`rollup_files`** - _[boolean]_ - If true, includes a "files" array listing all files in each partition. Only applicable when using "directory" or "hive" partitioning. - **`transform_sql`** - _[string]_ - Optional DuckDB SQL query used to transform the results. The resolved data is available as a table referenced using `{{ .table }}`. - **option 5** - _[object]_ - Uses the status of a resource as data. - **`resource_status`** - _[object]_ - Based on resource status _(required)_ - **`where_error`** - _[boolean]_ - Indicates whether the condition should trigger when the resource is in an error state. - **option 6** - _[object]_ - Invokes multiple resolvers and returns the union of their results. Each entry in the list is a resolver definition (e.g. sql, glob, metrics_sql, api). - **`union`** - _[array of object]_ - List of resolver definitions whose results are combined into a single result set. _(required)_ - **option 7** - _[object]_ - Uses AI to generate insights and analysis from metrics data. Only available for reports. - **`ai`** - _[object]_ - AI resolver configuration for generating automated insights _(required)_ - **`prompt`** - _[string]_ - Custom prompt to guide the AI analysis. If not provided, a default analysis prompt is used. - **`time_range`** - _[object]_ - Time range for the analysis period - **`iso_duration`** - _[string]_ - ISO 8601 duration (e.g., P7D for 7 days, P1M for 1 month) - **`iso_offset`** - _[string]_ - ISO 8601 offset from current time (e.g., P1D to start from yesterday) - **`start`** - _[string]_ - Start timestamp in ISO 8601 format - **`end`** - _[string]_ - End timestamp in ISO 8601 format - **`expression`** - _[string]_ - Rill time expression (e.g., 'last 7 days', 'this month') - **`comparison_time_range`** - _[object]_ - Optional comparison time range for period-over-period analysis - **`iso_duration`** - _[string]_ - ISO 8601 duration for comparison period - **`iso_offset`** - _[string]_ - ISO 8601 offset for comparison period (e.g., P7D to compare with previous week) - **`start`** - _[string]_ - Start timestamp in ISO 8601 format - **`end`** - _[string]_ - End timestamp in ISO 8601 format - **`expression`** - _[string]_ - Rill time expression for comparison period - **`context`** - _[object]_ - Context to constrain the AI analysis - **`explore`** - _[string]_ - Name of the explore dashboard to analyze - **`dimensions`** - _[array of string]_ - List of dimensions to include in analysis - **`measures`** - _[array of string]_ - List of measures to include in analysis ### `for` _[oneOf]_ - Specifies how user identity or attributes should be evaluated for security policy enforcement. - **option 1** - _[object]_ - Specifies a unique user identifier for applying security policies. - **`user_id`** - _[string]_ - The unique user ID used to evaluate security policies. _(required)_ - **option 2** - _[object]_ - Specifies a user's email address for applying security policies. - **`user_email`** - _[string]_ - The user's email address used to evaluate security policies. _(required)_ - **option 3** - _[object]_ - Specifies a set of arbitrary user attributes for applying security policies. - **`attributes`** - _[object]_ - A dictionary of user attributes used to evaluate security policies. _(required)_ ### `on_recover` _[boolean]_ - Send an alert when a previously failing alert recovers. Defaults to false. ### `on_fail` _[boolean]_ - Send an alert when a failure occurs. Defaults to true. ### `on_error` _[boolean]_ - Send an alert when an error occurs during evaluation. Defaults to false. ### `renotify` _[boolean]_ - Enable repeated notifications for unresolved alerts. Defaults to false. ### `renotify_after` _[string]_ - Defines the re-notification interval for the alert (e.g., '10m','24h'), equivalent to snooze duration in UI, defaults to 'Off' ### `notify` _[object]_ - Notification configuration for email and Slack delivery _(required)_ - **`email`** - _[object]_ - Send notifications via email. - **`recipients`** - _[array of string]_ - An array of email addresses to notify. _(required)_ - **`slack`** - _[object]_ - Send notifications via Slack. - **`users`** - _[array of string]_ - An array of Slack user IDs to notify. - **`channels`** - _[array of string]_ - An array of Slack channel names to notify. - **`webhooks`** - _[array of string]_ - An array of Slack webhook URLs to send notifications to. ### `annotations` _[object]_ - Key-value pairs used for annotations. ## Common Properties ### `name` _[string]_ - Name is usually inferred from the filename, but can be specified manually. ### `refs` _[array of string]_ - List of resource references ### `tags` _[array of string]_ - Tags for organizing and filtering the resource (e.g. on the project dashboards list). ### `dev` _[object]_ - Overrides any properties in development environment. ### `prod` _[object]_ - Overrides any properties in production environment. ## Examples ```yaml # Example: To send alert when data lags by more than 1 day to slack channel #rill-cloud-alerts type: alert display_name: Data lags by more than 1 day # Check the alert every hour. refresh: cron: 0 * * * * # Query that returns non-empty results if the measures lag by more than 1 day. data: sql: |- SELECT * FROM ( SELECT MAX(event_time) AS max_time FROM rill_metrics_model ) WHERE max_time < NOW() - INTERVAL '1 day' # Send notifications in Slack. notify: slack: channels: - '#rill-cloud-alerts' ``` --- ##### API YAML --- note: GENERATED. DO NOT EDIT. title: API YAML sidebar_position: 39 --- Custom APIs allow you to create endpoints that can be called to retrieve or manipulate data. ## Properties ### `type` _[string]_ - Refers to the resource type and must be `api` _(required)_ ### `openapi` _[object]_ - OpenAPI specification for the API endpoint - **`summary`** - _[string]_ - A brief description of what the API endpoint does - **`parameters`** - _[array of object]_ - List of parameters that the API endpoint accepts - **`request_schema`** - _[object]_ - JSON schema for the request body (use nested YAML instead of a JSON string) - **`response_schema`** - _[object]_ - JSON schema for the response body (use nested YAML instead of a JSON string) ### `security` _[object]_ - Defines [security rules and access control policies](/developers/build/metrics-view/security) for resources - **`access`** - _[oneOf]_ - Expression indicating if the user should be granted access to the dashboard. If not defined, it will resolve to false and the dashboard won't be accessible to anyone. Needs to be a valid SQL expression that evaluates to a boolean. - **option 1** - _[string]_ - SQL expression that evaluates to a boolean to determine access - **option 2** - _[boolean]_ - Direct boolean value to allow or deny access - **`row_filter`** - _[string]_ - SQL expression to filter the underlying model by. Can leverage templated user attributes to customize the filter for the requesting user. Needs to be a valid SQL expression that can be injected into a WHERE clause - **`include`** - _[array of object]_ - List of dimension or measure names to include in the dashboard. If include is defined all other dimensions and measures are excluded - **`if`** - _[string]_ - Expression to decide if the column should be included or not. It can leverage templated user attributes. Needs to be a valid SQL expression that evaluates to a boolean _(required)_ - **`names`** - _[anyOf]_ - List of fields to include. Should match the name of one of the dashboard's dimensions or measures _(required)_ - **option 1** - _[array of string]_ - List of specific field names to include - **option 2** - _[string]_ - Wildcard '*' to include all fields - **`exclude`** - _[array of object]_ - List of dimension or measure names to exclude from the dashboard. If exclude is defined all other dimensions and measures are included - **`if`** - _[string]_ - Expression to decide if the column should be excluded or not. It can leverage templated user attributes. Needs to be a valid SQL expression that evaluates to a boolean _(required)_ - **`names`** - _[anyOf]_ - List of fields to exclude. Should match the name of one of the dashboard's dimensions or measures _(required)_ - **option 1** - _[array of string]_ - List of specific field names to exclude - **option 2** - _[string]_ - Wildcard '*' to exclude all fields - **`rules`** - _[array of object]_ - List of detailed security rules that can be used to define complex access control policies - **`type`** - _[string]_ - Type of security rule - access (overall access), field_access (field-level access), or row_filter (row-level filtering) _(required)_ - **`action`** - _[string]_ - Whether to allow or deny access for this rule - **`if`** - _[string]_ - Conditional expression that determines when this rule applies. Must be a valid SQL expression that evaluates to a boolean - **`names`** - _[array of string]_ - List of field names this rule applies to (for field_access type rules) - **`all`** - _[boolean]_ - When true, applies the rule to all fields (for field_access type rules) - **`sql`** - _[string]_ - SQL expression for row filtering (for row_filter type rules) ### `skip_nested_security` _[boolean]_ - Flag to control security inheritance ## One of Properties Options - [SQL Query](#sql-query) - [Metrics View Query](#metrics-view-query) - [Custom API Call](#custom-api-call) - [File Glob Query](#file-glob-query) - [Resource Status Check](#resource-status-check) - [Union](#union) ## SQL Query Executes a raw SQL query against the project's data models. ### `sql` _[string]_ - Raw SQL query to run against existing models in the project. _(required)_ ### `connector` _[string]_ - Specifies the connector to use when running SQL or glob queries. ```yaml type: api sql: "SELECT * FROM table_name WHERE date >= '2024-01-01'" ``` ## Metrics View Query Executes a SQL query that targets a defined metrics view. ### `metrics_sql` _[string]_ - SQL query that targets a metrics view in the project _(required)_ ```yaml type: api metrics_sql: "SELECT * FROM user_metrics WHERE date >= '2024-01-01'" ``` ## Custom API Call Calls a custom API defined in the project to compute data. ### `api` _[string]_ - Name of a custom API defined in the project. _(required)_ ### `args` _[object]_ - Arguments to pass to the custom API. ```yaml type: api api: "user_analytics_api" args: start_date: "2024-01-01" limit: 10 ``` ## File Glob Query Uses a file-matching pattern (glob) to query data from a connector. ### `glob` _[oneOf]_ - Simple path/glob pattern or path/glob pattern with advanced options. _(required)_ - **option 1** - _[string]_ - Glob pattern used to match files or directories in the object store. - **option 2** - _[object]_ - Configuration for specifying a file path/glob pattern with advanced options. - **`connector`** - _[string]_ - Specifies the object store connector to use (e.g. "s3", "gcs"). If not provided, it is inferred from the scheme of the path. - **`path`** - _[string]_ - Glob pattern used to match files or directories in the object store. _(required)_ - **`start`** - _[string]_ - Defines the lower bound (inclusive) for partition filtering. Only partitions with paths greater than or equal to this value are considered. - **`end`** - _[string]_ - Defines the upper bound (exclusive) for partition filtering. Only partitions with paths less than this value are considered. - **`last`** - _[integer]_ - Limits the result to the last N partitions (the N highest paths in lexicographic order). This hard limit always applies, including on the first run when there is no existing data. Additionally, when previously processed partitions exist, it raises the lower bound to the Nth partition from the end of those successfully processed partitions, forming a rolling window that prevents full re-listings each time. - **`partition`** - _[string]_ - Controls how matched files are grouped: - "file" (default) : Each matched path is returned as a row. Use the glob pattern to match files or directories at the level you want (for example, file-level or directory-level). - "directory": This mode is deprecated. Instead, use "file" with a glob that directly matches the directory level you want. - "hive": groups files by directory and extracts Hive-style partition values from the path as columns. - **`rollup_files`** - _[boolean]_ - If true, includes a "files" array listing all files in each partition. Only applicable when using "directory" or "hive" partitioning. - **`transform_sql`** - _[string]_ - Optional DuckDB SQL query used to transform the results. The resolved data is available as a table referenced using `{{ .table }}`. ```yaml type: api glob: "data/*.csv" ``` ## Resource Status Check Uses the status of a resource as data. ### `resource_status` _[object]_ - Based on resource status _(required)_ - **`where_error`** - _[boolean]_ - Indicates whether the condition should trigger when the resource is in an error state. ```yaml type: api resource_status: where_error: true ``` ## Union Invokes multiple resolvers and returns the union of their results. Each entry in the list is a resolver definition (e.g. sql, glob, metrics_sql, api). ### `union` _[array of object]_ - List of resolver definitions whose results are combined into a single result set. _(required)_ ```yaml # Example for union resolvers type: api union: - connector: duckdb sql: SELECT 1 - connector: clickhouse sql: SELECT 2 ``` --- ##### Canvas Dashboard YAML --- note: GENERATED. DO NOT EDIT. title: Canvas Dashboard YAML sidebar_position: 35 --- Canvas dashboards provide a flexible way to create custom dashboards with drag-and-drop components. ## Properties ### `type` _[string]_ - Refers to the resource type and must be `canvas` _(required)_ ### `display_name` _[string]_ - Refers to the display name for the canvas ### `title` _[string]_ - Deprecated: use display_name instead. Refers to the display name for the canvas ### `description` _[string]_ - Description for the canvas dashboard ### `banner` _[string]_ - Refers to the custom banner displayed in the header of a Canvas dashboard. ### `rows` _[array of object]_ - Refers to all of the rows displayed on the Canvas. Each entry is either a plain row (with `items`) or a tab group (with `tabs`), but not both. - **`height`** - _[string]_ - Height of the row in px - **`items`** - _[array of object]_ - List of components to display in the row - **`component`** - _[string]_ - Name of the component to display. Each component type has its own set of properties. Available component types: - **markdown** - Text component, uses markdown formatting - **kpi_grid** - KPI component, similar to TDD in Rill Explore, display quick KPI charts - **stacked_bar_normalized** - Bar chart normalized to 100% values - **line_chart** - Normal Line chart - **bar_chart** - Normal Bar chart - **stacked_bar** - Stacked Bar chart - **area_chart** - Line chart with area - **image** - Provide a URL to embed into the canvas dashboard - **table** - Similar to Pivot table, add dimensions and measures to visualize your data - **heatmap** - Heat Map chart to visualize distribution of data - **donut_chart** - Donut or Pie chart to display sums of total - **`width`** - _[string, integer]_ - Width of the component (can be a number or string with unit) - **`name`** - _[string]_ - Stable identifier for a tab group, used as its deep-link URL key. Defaults to `group-` if omitted. Only used for tab-group entries. - **`tabs`** - _[array of object]_ - Makes this entry a tab group instead of a plain row. Only the active tab's rows render; tabs cannot be nested. - **`label`** - _[string]_ - User-facing tab label shown on the tab. - **`name`** - _[string]_ - Stable identifier used as the tab's deep-link URL key. Defaults to a slug of the label if omitted. - **`rows`** - _[array of object]_ - Plain rows (with `items`) shown when this tab is active. Tab rows cannot themselves contain `tabs`. - **`height`** - _[string]_ - Height of the row in px - **`items`** - _[array]_ - List of components to display in the row ### `max_width` _[integer]_ - Max width in pixels of the canvas ### `gap_x` _[integer]_ - Horizontal gap in pixels of the canvas ### `gap_y` _[integer]_ - Vertical gap in pixels of the canvas ### `filters` _[object]_ - Indicates if filters should be enabled for the canvas. - **`enable`** - _[boolean]_ - Toggles filtering functionality for the canvas dashboard. ### `allow_custom_time_range` _[boolean]_ - Defaults to true. When set to false, hides the ability to set a custom time range for the user. ### `allow_filter_add` _[boolean]_ - Whether users can add new filters to the canvas dashboard. ### `time_ranges` _[array of oneOf]_ - Overrides the list of default time range selections available in the dropdown. It can be a string or an object with a 'range' and optional 'comparison_offsets'. ```yaml time_ranges: - PT15M // Simplified syntax to specify only the range - PT1H - PT6H - P7D - range: P5D // Advanced syntax to specify comparison_offsets as well - P4W - rill-TD // Today - rill-WTD // Week-To-date ``` - **option 1** - _[string]_ - An [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601#Durations) duration or one of the [Rill ISO 8601 extensions](/reference/time-syntax/rill-iso-extensions#extensions) extensions for the selection. - **option 2** - _[object]_ - Object containing time range and comparison configuration - **`range`** - _[string]_ - An [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601#Durations) duration or one of the [Rill ISO 8601 extensions](/reference/time-syntax/rill-iso-extensions#extensions) extensions for the selection. _(required)_ - **`comparison_offsets`** - _[array of oneOf]_ - List of time comparison options for this time range selection (optional). Must be one of the [Rill ISO 8601 extensions](https://docs.rilldata.com/reference/rill-iso-extensions#extensions). - **option 1** - _[string]_ - Offset string only (range is inferred) - **option 2** - _[object]_ - Object containing offset and range configuration for time comparison - **`offset`** - _[string]_ - Time offset for comparison (e.g., 'P1D' for one day ago) - **`range`** - _[string]_ - Custom time range for comparison period ### `time_zones` _[array of string]_ - Refers to the time zones that should be pinned to the top of the time zone selector. It should be a list of [IANA time zone identifiers](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) ### `defaults` _[object]_ - Defines the defaults YAML struct. ```yaml defaults: time_range: P1M comparison_mode: dimension # time, none comparison_dimension: filename filters: metrics_view_name: "dimension_name = 'value' AND measure_name > 1000" ``` - **`time_range`** - _[string]_ - Refers to the default time range shown when a user initially loads the dashboard. The value must be either an [ISO 8601 duration](https://en.wikipedia.org/wiki/ISO_8601#Durations) (for example, PT12H for 12 hours, P1M for 1 month, or P26W for 26 weeks) or one of the [Rill ISO 8601 extensions](https://docs.rilldata.com/reference/rill-iso-extensions#extensions). - **`comparison_mode`** - _[string]_ - Controls how to compare current data with historical or categorical baselines. Options: `none` (no comparison), `time` (compares with past based on default_time_range), `dimension` (compares based on comparison_dimension values) - **`comparison_dimension`** - _[string]_ - For dimension mode, specify the comparison dimension by name. - **`filters`** - _[object]_ - Default filter expressions keyed by metrics view name. Each value is a Metrics SQL WHERE expression. ### `theme` _[oneOf]_ - Name of the theme to use. Only one of theme and embedded_theme can be set. - **option 1** - _[string]_ - Name of an existing theme to apply to the dashboard - **option 2** - _[object]_ - Inline theme configuration. - **`colors`** - _[object]_ - Used to override the dashboard colors. Either primary or secondary color must be provided. - **`primary`** - _[string]_ - Overrides the primary blue color in the dashboard. Can have any hex, [named colors](https://www.w3.org/TR/css-color-4/#named-colors) or hsl() formats. Note that the hue of the input colors is used for variants but the saturation and lightness is copied over from the [blue color palette](https://tailwindcss.com/docs/customizing-colors). - **`secondary`** - _[string]_ - Overrides the secondary color in the dashboard. Applies to the loading spinner only as of now. Can have any hex, [named colors](https://www.w3.org/TR/css-color-4/#named-colors) or hsl() formats. - **`light`** - _[object]_ - Light theme color configuration - **`primary`** - _[string]_ - Primary color for light theme. Can have any hex, [named colors](https://www.w3.org/TR/css-color-4/#named-colors) or hsl() formats. - **`secondary`** - _[string]_ - Secondary color for light theme. Can have any hex, [named colors](https://www.w3.org/TR/css-color-4/#named-colors) or hsl() formats. - **`kpi-positive`** - _[string]_ - Color for positive KPI delta values in light theme. Defaults to fg-secondary (gray). - **`kpi-negative`** - _[string]_ - Color for negative KPI delta values in light theme. Defaults to red. - **`variables`** - _[object]_ - Custom CSS variables for light theme - **`dark`** - _[object]_ - Dark theme color configuration - **`primary`** - _[string]_ - Primary color for dark theme. Can have any hex, [named colors](https://www.w3.org/TR/css-color-4/#named-colors) or hsl() formats. - **`secondary`** - _[string]_ - Secondary color for dark theme. Can have any hex, [named colors](https://www.w3.org/TR/css-color-4/#named-colors) or hsl() formats. - **`kpi-positive`** - _[string]_ - Color for positive KPI delta values in dark theme. Defaults to fg-secondary (gray). - **`kpi-negative`** - _[string]_ - Color for negative KPI delta values in dark theme. Defaults to red. - **`variables`** - _[object]_ - Custom CSS variables for dark theme ### `security` _[object]_ - Defines [security rules and access control policies](/developers/build/metrics-view/security) for dashboards (without row filtering) - **`access`** - _[oneOf]_ - Expression indicating if the user should be granted access to the dashboard. If not defined, it will resolve to false and the dashboard won't be accessible to anyone. Needs to be a valid SQL expression that evaluates to a boolean. - **option 1** - _[string]_ - SQL expression that evaluates to a boolean to determine access - **option 2** - _[boolean]_ - Direct boolean value to allow or deny access ### `variables` _[array of object]_ - Variables that can be used in the canvas - **`name`** - _[string]_ - Unique identifier for the variable _(required)_ - **`type`** - _[string]_ - Data type of the variable (e.g., string, number, boolean) _(required)_ - **`value`** - _[string, number, boolean, object, array]_ - Default value for the variable. Can be any valid JSON value type ## Common Properties ### `name` _[string]_ - Name is usually inferred from the filename, but can be specified manually. ### `refs` _[array of string]_ - List of resource references ### `tags` _[array of string]_ - Tags for organizing and filtering the resource (e.g. on the project dashboards list). ### `dev` _[object]_ - Overrides any properties in development environment. ### `prod` _[object]_ - Overrides any properties in production environment. --- ##### Component YAML --- note: GENERATED. DO NOT EDIT. title: Component YAML sidebar_position: 41 --- Defines a reusable dashboard component that can be embedded in canvas dashboards ## Properties ### `type` _[string]_ - Refers to the resource type and must be `component` _(required)_ ### `display_name` _[string]_ - Refers to the display name for the component ### `description` _[string]_ - Detailed description of the component's purpose and functionality ### `input` _[array of object]_ - List of input variables that can be passed to the component - **`name`** - _[string]_ - Unique identifier for the variable _(required)_ - **`type`** - _[string]_ - Data type of the variable (e.g., string, number, boolean) _(required)_ - **`value`** - _[string, number, boolean, object, array]_ - Default value for the variable. Can be any valid JSON value type ### `output` _[object]_ - Output variable that the component produces - **`name`** - _[string]_ - Unique identifier for the variable _(required)_ - **`type`** - _[string]_ - Data type of the variable (e.g., string, number, boolean) _(required)_ - **`value`** - _[string, number, boolean, object, array]_ - Default value for the variable. Can be any valid JSON value type ### `line_chart` _[object]_ - (no description) - **`config`** - _[object]_ - (no description) _(required)_ - **`metrics_view`** - _[string]_ - Reference to the metrics view to use _(required)_ - **`x`** - _[object]_ - (no description) - **`field`** - _[string]_ - Field name from the metrics view _(required)_ - **`title`** - _[string]_ - Display title for the field - **`format`** - _[string]_ - Format string for the field - **`type`** - _[string]_ - Data type of the field _(required)_ - **`timeUnit`** - _[string]_ - Time unit for temporal fields - **`y`** - _[object]_ - (no description) - **`field`** - _[string]_ - Field name from the metrics view _(required)_ - **`title`** - _[string]_ - Display title for the field - **`format`** - _[string]_ - Format string for the field - **`type`** - _[string]_ - Data type of the field _(required)_ - **`timeUnit`** - _[string]_ - Time unit for temporal fields - **`color`** - _[oneOf]_ - (no description) - **option 1** - _[object]_ - (no description) - **`field`** - _[string]_ - Field name from the metrics view _(required)_ - **`title`** - _[string]_ - Display title for the field - **`format`** - _[string]_ - Format string for the field - **`type`** - _[string]_ - Data type of the field _(required)_ - **`timeUnit`** - _[string]_ - Time unit for temporal fields - **option 2** - _[string]_ - (no description) - **`tooltip`** - _[object]_ - (no description) - **`field`** - _[string]_ - Field name from the metrics view _(required)_ - **`title`** - _[string]_ - Display title for the field - **`format`** - _[string]_ - Format string for the field - **`type`** - _[string]_ - Data type of the field _(required)_ - **`timeUnit`** - _[string]_ - Time unit for temporal fields - **`title`** - _[string]_ - Chart title - **`description`** - _[string]_ - Chart description ### `bar_chart` _[object]_ - (no description) - **`config`** - _[object]_ - (no description) _(required)_ - **`metrics_view`** - _[string]_ - Reference to the metrics view to use _(required)_ - **`x`** - _[object]_ - (no description) - **`field`** - _[string]_ - Field name from the metrics view _(required)_ - **`title`** - _[string]_ - Display title for the field - **`format`** - _[string]_ - Format string for the field - **`type`** - _[string]_ - Data type of the field _(required)_ - **`timeUnit`** - _[string]_ - Time unit for temporal fields - **`y`** - _[object]_ - (no description) - **`field`** - _[string]_ - Field name from the metrics view _(required)_ - **`title`** - _[string]_ - Display title for the field - **`format`** - _[string]_ - Format string for the field - **`type`** - _[string]_ - Data type of the field _(required)_ - **`timeUnit`** - _[string]_ - Time unit for temporal fields - **`color`** - _[oneOf]_ - (no description) - **option 1** - _[object]_ - (no description) - **`field`** - _[string]_ - Field name from the metrics view _(required)_ - **`title`** - _[string]_ - Display title for the field - **`format`** - _[string]_ - Format string for the field - **`type`** - _[string]_ - Data type of the field _(required)_ - **`timeUnit`** - _[string]_ - Time unit for temporal fields - **option 2** - _[string]_ - (no description) - **`tooltip`** - _[object]_ - (no description) - **`field`** - _[string]_ - Field name from the metrics view _(required)_ - **`title`** - _[string]_ - Display title for the field - **`format`** - _[string]_ - Format string for the field - **`type`** - _[string]_ - Data type of the field _(required)_ - **`timeUnit`** - _[string]_ - Time unit for temporal fields - **`title`** - _[string]_ - Chart title - **`description`** - _[string]_ - Chart description ### `stacked_bar_chart` _[object]_ - (no description) - **`config`** - _[object]_ - (no description) _(required)_ - **`metrics_view`** - _[string]_ - Reference to the metrics view to use _(required)_ - **`x`** - _[object]_ - (no description) - **`field`** - _[string]_ - Field name from the metrics view _(required)_ - **`title`** - _[string]_ - Display title for the field - **`format`** - _[string]_ - Format string for the field - **`type`** - _[string]_ - Data type of the field _(required)_ - **`timeUnit`** - _[string]_ - Time unit for temporal fields - **`y`** - _[object]_ - (no description) - **`field`** - _[string]_ - Field name from the metrics view _(required)_ - **`title`** - _[string]_ - Display title for the field - **`format`** - _[string]_ - Format string for the field - **`type`** - _[string]_ - Data type of the field _(required)_ - **`timeUnit`** - _[string]_ - Time unit for temporal fields - **`color`** - _[oneOf]_ - (no description) - **option 1** - _[object]_ - (no description) - **`field`** - _[string]_ - Field name from the metrics view _(required)_ - **`title`** - _[string]_ - Display title for the field - **`format`** - _[string]_ - Format string for the field - **`type`** - _[string]_ - Data type of the field _(required)_ - **`timeUnit`** - _[string]_ - Time unit for temporal fields - **option 2** - _[string]_ - (no description) - **`tooltip`** - _[object]_ - (no description) - **`field`** - _[string]_ - Field name from the metrics view _(required)_ - **`title`** - _[string]_ - Display title for the field - **`format`** - _[string]_ - Format string for the field - **`type`** - _[string]_ - Data type of the field _(required)_ - **`timeUnit`** - _[string]_ - Time unit for temporal fields - **`title`** - _[string]_ - Chart title - **`description`** - _[string]_ - Chart description ### `kpi` _[object]_ - (no description) - **`metrics_view`** - _[string]_ - Reference to the metrics view to use _(required)_ - **`measure`** - _[string]_ - Measure to display _(required)_ - **`time_range`** - _[string]_ - Time range for the KPI _(required)_ - **`comparison_range`** - _[string]_ - Comparison time range - **`filter`** - _[string]_ - Filter expression - **`title`** - _[string]_ - KPI title - **`description`** - _[string]_ - KPI description ### `table` _[object]_ - (no description) - **`metrics_view`** - _[string]_ - Reference to the metrics view to use _(required)_ - **`measures`** - _[array of string]_ - List of measures to display _(required)_ - **`time_range`** - _[string]_ - Time range for the table _(required)_ - **`row_dimensions`** - _[array of string]_ - Dimensions for table rows - **`col_dimensions`** - _[array of string]_ - Dimensions for table columns - **`hide_totals_row`** - _[boolean]_ - Whether to hide the totals row. Defaults to false. - **`hide_totals_col`** - _[boolean]_ - Whether to hide the totals column. Defaults to false. - **`row_limit`** - _[string]_ - Maximum number of rows to display in a pivot table (one of "5", "10", "25", "50", "100"). Omit or set to "all" for all rows. - **`comparison_range`** - _[string]_ - Comparison time range - **`filter`** - _[string]_ - Filter expression - **`title`** - _[string]_ - Table title - **`description`** - _[string]_ - Table description ### `markdown` _[object]_ - (no description) - **`content`** - _[string]_ - Markdown content _(required)_ - **`css`** - _[object]_ - CSS styles - **`title`** - _[string]_ - Markdown title - **`description`** - _[string]_ - Markdown description ### `image` _[object]_ - (no description) - **`url`** - _[string]_ - Image URL _(required)_ - **`css`** - _[object]_ - CSS styles - **`title`** - _[string]_ - Image title - **`description`** - _[string]_ - Image description --- ##### Connector YAML --- note: GENERATED. DO NOT EDIT. title: Connector YAML sidebar_position: 31 --- Connector YAML files define how Rill connects to external data sources and OLAP engines. Each connector specifies a driver type and its required connection parameters. ## Available Connector Types ### _OLAP Engines_ - [**ClickHouse**](#clickhouse) - ClickHouse analytical database - [**Databricks**](#databricks) - Databricks SQL warehouse - [**Druid**](#druid) - Apache Druid - [**DuckDB**](#duckdb) - Embedded DuckDB engine (default) - [**External DuckDB**](#external-duckdb) - External DuckDB database - [**MotherDuck**](#motherduck) - MotherDuck cloud database - [**Pinot**](#pinot) - Apache Pinot - [**StarRocks**](#starrocks) - StarRocks analytical database ### _Data Warehouses_ - [**Athena**](#athena) - Amazon Athena - [**BigQuery**](#bigquery) - Google BigQuery - [**Databricks**](#databricks) - Databricks SQL warehouse - [**Redshift**](#redshift) - Amazon Redshift - [**Snowflake**](#snowflake) - Snowflake data warehouse ### _Databases_ - [**MySQL**](#mysql) - MySQL databases - [**PostgreSQL**](#postgres) - PostgreSQL databases - [**Supabase**](#supabase) - Supabase (managed PostgreSQL) ### _Object Storage_ - [**Azure**](#azure) - Azure Blob Storage - [**GCS**](#gcs) - Google Cloud Storage - [**S3**](#s3) - Amazon S3 storage ### Service Integrations - [**Claude**](#claude) - Claude connector for chat with your own API key - [**OpenAI**](#openai) - OpenAI connector for chat with your own API key - [**Gemini**](#gemini) - Gemini connector for chat with your own API key - [**Slack**](#slack) - Slack data ### _Other_ - [**HTTPS**](#https) - Public files via HTTP/HTTPS :::warning Security Recommendation For all credential parameters (passwords, tokens, keys), use environment variables with the syntax `{{ .env.KEY_NAME }}`. This keeps sensitive data out of your YAML files and version control. See our [credentials documentation](/developers/build/connectors/credentials/) for complete setup instructions. ::: ## Properties ### `type` _[string]_ - Refers to the resource type and must be `connector` _(required)_ ## Common Properties ### `name` _[string]_ - Name is usually inferred from the filename, but can be specified manually. ### `refs` _[array of string]_ - List of resource references ### `tags` _[array of string]_ - Tags for organizing and filtering the resource (e.g. on the project dashboards list). ### `dev` _[object]_ - Overrides any properties in development environment. ### `prod` _[object]_ - Overrides any properties in production environment. ## Athena ### `driver` _[string]_ - Refers to the driver type and must be driver `athena` _(required)_ ### `aws_access_key_id` _[string]_ - AWS Access Key ID used for authentication. Required when using static credentials directly or as base credentials for assuming a role. ### `aws_secret_access_key` _[string]_ - AWS Secret Access Key paired with the Access Key ID. Required when using static credentials directly or as base credentials for assuming a role. ### `aws_access_token` _[string]_ - AWS session token used with temporary credentials. Required only if the Access Key and Secret Key are part of a temporary session credentials. ### `role_arn` _[string]_ - ARN of the IAM role to assume. When specified, the SDK uses the base credentials to call STS AssumeRole and obtain temporary credentials scoped to this role. ### `role_session_name` _[string]_ - Session name to associate with the STS AssumeRole session. Used only if 'role_arn' is specified. Useful for identifying and auditing the session. ### `external_id` _[string]_ - External ID required by some roles when assuming them, typically for cross-account access. Used only if 'role_arn' is specified and the role's trust policy requires it. ### `workgroup` _[string]_ - Athena workgroup to use for query execution. Defaults to 'primary' if not specified. ### `output_location` _[string]_ - S3 URI where Athena query results should be stored (e.g., s3://your-bucket/athena/results/). Optional if the selected workgroup has a default result configuration. ### `region` _[string]_ - AWS region where Athena and the result S3 bucket are located (e.g., us-east-1). Defaults to 'us-east-1' if not specified. ### `allow_host_access` _[boolean]_ - Allow the Athena client to access host environment configurations such as environment variables or local AWS credential files. Defaults to true, enabling use of credentials and settings from the host environment unless explicitly disabled. ```yaml # Example: Athena connector configuration type: connector # Must be `connector` (required) driver: athena # Must be `athena` _(required)_ aws_access_key_id: "{{ .env.AWS_ACCESS_KEY_ID }}" # AWS Access Key ID for authentication aws_secret_access_key: "{{ .env.AWS_SECRET_ACCESS_KEY }}" # AWS Secret Access Key for authentication aws_access_token: "{{ .env.AWS_ACCESS_TOKEN }}" # AWS session token for temporary credentials role_arn: "arn:aws:iam::123456789012:role/MyRole" # ARN of the IAM role to assume role_session_name: "MySession" # Session name for STS AssumeRole external_id: "MyExternalID" # External ID for cross-account access workgroup: "primary" # Athena workgroup (defaults to 'primary') output_location: "s3://my-bucket/athena-output/" # S3 URI for query results region: "us-east-1" # AWS region (defaults to 'us-east-1') allow_host_access: true # Allow host environment access _(default: true)_ ``` ## Azure ### `driver` _[string]_ - Refers to the driver type and must be driver `azure` _(required)_ ### `azure_storage_account` _[string]_ - Azure storage account name _(required)_ ### `azure_storage_key` _[string]_ - Azure storage access key _(required)_ ### `azure_storage_sas_token` _[string]_ - Optional azure SAS token for authentication ### `azure_storage_connection_string` _[string]_ - Optional azure connection string for storage account ### `path_prefixes` _[string, array]_ - A list of container or virtual directory prefixes that this connector is allowed to access. Useful when different containers or paths use different credentials, allowing the system to route access through the appropriate connector based on the blob path. Example: `azure://my-bucket/`, ` azure://my-bucket/path/` ,`azure://my-bucket/path/prefix` ### `allow_host_access` _[boolean]_ - Allow access to host environment configuration ```yaml # Example: Azure connector configuration type: connector # Must be `connector` (required) driver: azure # Must be `azure` _(required)_ azure_storage_account: "mystorageaccount" # Azure storage account name _(required)_ azure_storage_key: "{{ .env.AZURE_STORAGE_KEY }}" # Azure storage access key _(required)_ ``` ## BigQuery ### `driver` _[string]_ - Refers to the driver type and must be driver `bigquery` _(required)_ ### `google_application_credentials` _[string]_ - Raw contents of the Google Cloud service account key (in JSON format) used for authentication. ### `project_id` _[string]_ - Google Cloud project ID ### `allow_host_access` _[boolean]_ - Enable the BigQuery client to use credentials from the host environment when no service account JSON is provided. This includes Application Default Credentials from environment variables, local credential files, or the Google Compute Engine metadata server. Defaults to true, allowing seamless authentication in GCP environments. ### `log_queries` _[boolean]_ - Controls whether to log raw SQL queries ### `max_bytes_billed` _[integer]_ - Maximum number of bytes billed for a query. Queries that exceed this limit will fail with an error. This can help prevent unexpectedly high costs from large queries. It is highly recommended to set this when running on `on-demand pricing` model. The default value is 0 i.e. no limits are enforced in Rill. ### `allow_standard_api` _[boolean]_ - Allow querying BigQuery using the standard API instead of the Storage Read API. This is less efficient and may lead to higher latency, but can be used as a fallback if the Storage Read API is not available due to insufficient permissions or other issues. ```yaml # Example: BigQuery connector configuration type: connector # Must be `connector` (required) driver: bigquery # Must be `bigquery` _(required)_ google_application_credentials: "{{ .env.GOOGLE_APPLICATION_CREDENTIALS }}" # Google Cloud service account JSON project_id: "my-project-id" # Google Cloud project ID allow_host_access: true # Allow host environment access _(default: true)_ ``` ## ClickHouse ### `driver` _[string]_ - Refers to the driver type and must be driver `clickhouse` _(required)_ ### `managed` _[boolean]_ - `true` means Rill will provision the connector using the default provisioner. `false` disables automatic provisioning. ### `mode` _[string]_ - `read` - Controls the operation mode for the ClickHouse connection. Defaults to 'read' for safe operation with external databases. Set to 'readwrite' to enable model creation and table mutations. Note: When 'managed: true', this is automatically set to 'readwrite'. ### `dsn` _[string]_ - DSN(Data Source Name) for the ClickHouse connection ### `username` _[string]_ - Username for authentication ### `password` _[string]_ - Password for authentication ### `host` _[string]_ - Host where the ClickHouse instance is running ### `port` _[integer]_ - Port where the ClickHouse instance is accessible ### `database` _[string]_ - Name of the ClickHouse database within the cluster ### `ssl` _[boolean]_ - Indicates whether a secured SSL connection is required ### `cluster` _[string]_ - Cluster name, required for running distributed queries ### `sync_replicas` _[boolean]_ - Controls whether to run `SYSTEM SYNC REPLICA` before replacing partitions on a replicated table in a cluster, ensuring all inserted parts are visible across replicas before the partition swap. Defaults to true ### `write_dsn` _[string]_ - Separate connection string for write operations ### `database_whitelist` _[string]_ - Comma-separated list of databases to show ### `log_queries` _[boolean]_ - Controls whether to log raw SQL queries ### `query_settings_override` _[string]_ - override the default settings used in queries. Changing the default settings can lead to incorrect query results and is generally not recommended. If you need to add settings, use `query_settings` ### `query_settings` _[string]_ - query settings to be set on dashboard queries. `query_settings_override` takes precedence over these settings and if set these are ignored. Each setting must be separated by a comma. Example `max_threads = 8, max_memory_usage = 10000000000` ### `embed_port` _[integer]_ - Port to run ClickHouse locally (0 for random port) ### `can_scale_to_zero` _[boolean]_ - Indicates if the database can scale to zero ### `max_open_conns` _[integer]_ - Maximum number of open connections to the database ### `max_idle_conns` _[integer]_ - Maximum number of idle connections in the pool ### `dial_timeout` _[string]_ - Timeout for dialing the ClickHouse server ### `conn_max_lifetime` _[string]_ - Maximum time a connection may be reused ### `read_timeout` _[string]_ - Maximum time for a connection to read data ```yaml # Example: ClickHouse connector configuration type: connector # Must be `connector` (required) driver: clickhouse # Must be `clickhouse` _(required)_ managed: false # Provision the connector using the default provisioner mode: "readwrite" # Enable model creation and table mutations username: "myusername" # Username for authentication password: "{{ .env.CLICKHOUSE_PASSWORD }}" # Password for authentication host: "localhost" # Hostname of the ClickHouse server port: 9000 # Port number of the ClickHouse server database: "mydatabase" # Name of the ClickHouse database ssl: true # Enable SSL for secure connection cluster: "mycluster" # Cluster name ``` ## databricks ### `driver` _[string]_ - Refers to the driver type and must be driver `databricks` _(required)_ ### `host` _[string]_ - Host where the Databricks instance is running ### `http_path` _[string]_ - HTTP path sets up the endpoint to the warehouse ### `token` _[string]_ - Token sets up the Personal Access Token ### `catalog` _[string]_ - Default catalog name. Optional. ### `schema` _[string]_ - Default schema name. Optional. ### `dsn` _[string]_ - DSN (Data Source Name) for the Databricks connection. This is intended for **advanced configuration** where you want to specify properties that are not explicitly defined above. It can only be used when the other connection fields (host, http_path, token, catalog, schema) are **not used**. Refer to https://github.com/databricks/databricks-sql-go for the full list of supported DSN parameters and their formats. ### `log_queries` _[boolean]_ - Controls whether to log raw SQL queries ```yaml # Example: Databricks connector configuration type: connector # Must be `connector` (required) driver: databricks # Must be `databricks` _(required)_ host: "my-databricks-instance.cloud.databricks.com" # Hostname of the Databricks instance http_path: "/sql/1.0/endpoints/1234567890abcdef" # HTTP path for the Databricks SQL warehouse endpoint token: "{{ .env.DATABRICKS_TOKEN }}" # Personal Access Token for authentication catalog: "my_catalog" # Default catalog name (optional) schema: "my_schema" # Default schema name (optional) ``` ## Druid ### `driver` _[string]_ - Refers to the driver type and must be driver `druid` _(required)_ ### `dsn` _[string]_ - Data Source Name (DSN) for connecting to Druid ### `username` _[string]_ - Username for authenticating with Druid ### `password` _[string]_ - Password for authenticating with Druid ### `host` _[string]_ - Hostname of the Druid coordinator or broker ### `port` _[integer]_ - Port number of the Druid service ### `ssl` _[boolean]_ - Enable SSL for secure connection ### `log_queries` _[boolean]_ - Log raw SQL queries sent to Druid ### `max_open_conns` _[integer]_ - Maximum number of open database connections (0 = default, -1 = unlimited) ### `skip_version_check` _[boolean]_ - Skip checking Druid version compatibility ```yaml # Example: Druid connector configuration type: connector # Must be `connector` (required) driver: druid # Must be `druid` _(required)_ username: "myusername" # Username for authentication password: "{{ .env.DRUID_PASSWORD }}" # Password for authentication host: "localhost" # Hostname of the Druid coordinator or broker port: 8082 # Port number of the Druid service ssl: true # Enable SSL for secure connection ``` ## DuckDB ### `driver` _[string]_ - Must be "duckdb" _(required)_ ### `mode` _[string]_ - Set the mode for the DuckDB connection. ### `path` _[string]_ - Path to external DuckDB database ### `attach` _[string]_ - Full ATTACH statement to attach a DuckDB database ### `pool_size` _[integer]_ - Number of concurrent connections and queries allowed ### `cpu` _[integer]_ - Number of CPU cores available to the database ### `memory_limit_gb` _[integer]_ - Amount of memory in GB available to the database ### `read_write_ratio` _[number]_ - Ratio of resources allocated to read vs write operations ### `allow_host_access` _[boolean]_ - Whether access to local environment and file system is allowed ### `init_sql` _[string]_ - SQL executed during database initialization ### `conn_init_sql` _[string]_ - SQL executed when a new connection is initialized ### `boot_queries` _[string]_ - Deprecated - Use init_sql instead ### `log_queries` _[boolean]_ - Whether to log raw SQL queries executed through OLAP ### `create_secrets_from_connectors` _[string, array]_ - List of connector names for which temporary secrets should be created before executing the SQL. ### `database_name` _[string]_ - Name of the attached DuckDB database (auto-detected if not set) ### `schema_name` _[string]_ - Default schema used by the DuckDB database ```yaml # Example: DuckDB connector configuration type: connector # Must be `connector` (required) driver: duckdb # Must be `duckdb` _(required)_ mode: "readwrite" # Set the mode for the DuckDB connection. allow_host_access: true # Whether access to the local environment and file system is allowed cpu: 4 # Number of CPU cores available to the database memory_limit_gb: 16 # Amount of memory in GB available to the database pool_size: 5 # Number of concurrent connections and queries allowed read_write_ratio: 0.7 # Ratio of resources allocated to read vs write operations init_sql: "INSTALL httpfs; LOAD httpfs;" # SQL executed during database initialization log_queries: true # Whether to log raw SQL queries executed through OLAP ``` ## External DuckDB ### `driver` _[string]_ - Refers to the driver type and must be driver `duckdb` _(required)_ ### `path` _[string]_ - Path to the DuckDB database ### `mode` _[string]_ - Set the mode for the DuckDB connection. ```yaml # Example: DuckDB as a source connector configuration type: connector # Must be `connector` (required) driver: duckdb # Must be `duckdb` _(required)_ path: "/path/to/my-duckdb-database.db" # Name of the DuckDB database mode: "read" # Set the mode for the DuckDB connection. ``` ## GCS ### `driver` _[string]_ - Refers to the driver type and must be driver `gcs` _(required)_ ### `google_application_credentials` _[string]_ - Google Cloud credentials JSON string ### `key_id` _[string]_ - Optional S3-compatible Key ID when used in compatibility mode ### `secret` _[string]_ - Optional S3-compatible Secret when used in compatibility mode ### `path_prefixes` _[string, array]_ - A list of bucket path prefixes that this connector is allowed to access. Useful when different buckets or bucket prefixes use different credentials, allowing the system to select the appropriate connector based on the bucket path. Example: `gs://my-bucket/`, ` gs://my-bucket/path/` ,`gs://my-bucket/path/prefix` ### `allow_host_access` _[boolean]_ - Allow access to host environment configuration ```yaml # Example: GCS connector configuration type: connector # Must be `connector` (required) driver: gcs # Must be `gcs` _(required)_ google_application_credentials: "{{ .env.GOOGLE_APPLICATION_CREDENTIALS }}" # Google Cloud credentials JSON string ``` ## HTTPS ### `driver` _[string]_ - Refers to the driver type and must be driver `https` _(required)_ ### `headers` _[object]_ - HTTP headers to include in the request ### `path_prefixes` _[string, array]_ - A list of HTTP/HTTPS URL prefixes that this connector is allowed to access. Useful when different URL namespaces use different credentials, enabling the system to choose the appropriate connector based on the URL path. Example: `https://example.com/`, ` https://example.com/path/` ,`https://example.com/path/prefix` ```yaml # Example: HTTPS connector configuration type: connector # Must be `connector` (required) driver: https # Must be `https` _(required)_ headers: "Authorization": 'Bearer {{ .env.HTTPS_TOKEN }}' # HTTP headers to include in the request ``` ## MotherDuck ### `driver` _[string]_ - Refers to the driver type and must be driver `duckdb`. _(required)_ ### `path` _[string]_ - Path to your MD database _(required)_ ### `schema_name` _[string]_ - Define your schema if not main, uses main by default ### `token` _[string]_ - MotherDuck token _(required)_ ### `init_sql` _[string]_ - SQL executed during database initialization. ### `mode` _[string]_ - Set the mode for the MotherDuck connection. By default, it is set to 'read' which allows only read operations. Set to 'readwrite' to enable model creation and table mutations. ### `create_secrets_from_connectors` _[string, array]_ - List of connector names for which temporary secrets should be created before executing the SQL. ```yaml # Example: MotherDuck connector configuration type: connector # Must be `connector` (required) driver: duckdb # Must be `duckdb` _(required)_ token: "{{ .env.MOTHERDUCK_TOKEN }}" # Set the MotherDuck token from your .env file _(required)_ path: "md:my_database" # Path to your MD database schema_name: "my_schema" # Define your schema if not main, uses main by default ``` ## MySQL ### `driver` _[string]_ - Refers to the driver type and must be driver `mysql` _(required)_ ### `dsn` _[string]_ - **Data Source Name (DSN)** for the MySQL connection, provided in [MySQL URI format](https://dev.mysql.com/doc/refman/8.4/en/connecting-using-uri-or-key-value-pairs.html#connecting-using-uri). The DSN must follow the standard MySQL URI scheme: ```text mysql://user:password@host:3306/my-db ``` Rules for special characters in password: - The following characters are allowed [unescaped in the URI](https://datatracker.ietf.org/doc/html/rfc3986#section-2.3): `~` `.` `_` `-` - All other special characters must be percent-encoded (`%XX` format). ```text mysql://user:pa%40ss@localhost:3306/my-db # password contains '@' mysql://user:pa%3Ass@localhost:3306/my-db # password contains ':' ``` ### `host` _[string]_ - Hostname of the MySQL server ### `port` _[integer]_ - Port number for the MySQL server ### `database` _[string]_ - Name of the MySQL database ### `user` _[string]_ - Username for authentication ### `password` _[string]_ - Password for authentication ### `ssl-mode` _[string]_ - ssl mode options: `disabled`, `preferred`, or `required`. ### `log_queries` _[boolean]_ - Controls whether to log raw SQL queries ```yaml # Example: MySQL connector configured using individual properties type: connector driver: mysql host: localhost port: 3306 database: mydb user: user password: "{{ .env.MYSQL_PASSWORD }}" ssl-mode: preferred ``` ```yaml # Example: MySQL connector configured using dsn type: connector driver: mysql dsn: "{{ .env.MYSQL_DSN }}" # Define DSN in .env file ``` ## OpenAI ### `driver` _[string]_ - The driver type, must be set to "openai" ### `api_key` _[string]_ - API key for connecting to OpenAI _(required)_ ### `model` _[string]_ - The OpenAI model to use (e.g., 'gpt-4o') ### `max_output_tokens` _[number]_ - Maximum number of tokens to generate in the completion (default: 8192) ### `reasoning_effort` _[string]_ - Constrains effort on reasoning for reasoning models (e.g., 'low', 'medium', 'high') ### `base_url` _[string]_ - The base URL for the OpenAI API (e.g., 'https://api.openai.com/v1') ### `api_type` _[string]_ - The type of OpenAI API to use ### `api_version` _[string]_ - The version of the OpenAI API to use (e.g., '2023-05-15'). Required when API Type is AZURE or AZURE_AD ```yaml # Example: OpenAI connector configuration type: connector # Must be `connector` (required) driver: openai # Must be `openai` _(required)_ api_key: "{{ .env.OPENAI_API_KEY }}" # API key for connecting to OpenAI model: "gpt-4o" # The OpenAI model to use (e.g., 'gpt-4o') max_output_tokens: 8192 # Maximum number of tokens to generate in the completion (default: 8192) reasoning_effort: "medium" # Constrains effort on reasoning for reasoning models (e.g., 'low', 'medium', 'high') base_url: "https://api.openai.com/v1" # The base URL for the OpenAI API (e.g., 'https://api.openai.com/v1') api_type: "openai" # The type of OpenAI API to use api_version: "2023-05-15" # The version of the OpenAI API to use (e.g., '2023-05-15'). Required when API Type is AZURE or AZURE_AD ``` ## Claude ### `driver` _[string]_ - The driver type, must be set to "claude" ### `api_key` _[string]_ - API key for connecting to Claude _(required)_ ### `model` _[string]_ - The Claude model to use (e.g., 'claude-opus-4-5') ### `max_tokens` _[number]_ - Maximum number of tokens in the response (e.g., 8192) ### `temperature` _[number]_ - Sampling temperature to use (e.g., 0.0) ### `base_url` _[string]_ - The base URL for the Claude API ```yaml # Example: Claude connector configuration type: connector driver: claude api_key: "{{ .env.claude_api_key }}" model: claude-opus-4-5 ``` ## Gemini ### `driver` _[string]_ - The driver type, must be set to "gemini" ### `api_key` _[string]_ - API key for connecting to Gemini _(required)_ ### `model` _[string]_ - The Gemini model to use (e.g., 'gemini-2.5-pro-preview-05-06') ### `include_thoughts` _[boolean]_ - Whether to include thinking/reasoning in the response ### `thinking_level` _[string]_ - Level of 'thinking' for the model's response (e.g., 'MINIMAL', 'LOW', 'MEDIUM', 'HIGH'). Default is 'LOW'. ### `max_output_tokens` _[number]_ - Maximum number of tokens in the response (e.g., 8192) ### `temperature` _[number]_ - Sampling temperature to use (0.0-2.0) ### `top_p` _[number]_ - Nucleus sampling parameter ### `top_k` _[number]_ - Top-K sampling parameter ```yaml # Example: Gemini connector configuration type: connector driver: gemini api_key: "{{ .env.gemini_api_key }}" model: gemini-2.5-pro-preview-05-06 ``` ## Pinot ### `driver` _[string]_ - Refers to the driver type and must be driver `pinot` _(required)_ ### `dsn` _[string]_ - DSN(Data Source Name) for the Pinot connection ### `username` _[string]_ - Username for authenticating with Pinot ### `password` _[string]_ - Password for authenticating with Pinot ### `broker_host` _[string]_ - Hostname of the Pinot broker ### `broker_port` _[integer]_ - Port number for the Pinot broker ### `controller_host` _[string]_ - Hostname of the Pinot controller ### `controller_port` _[integer]_ - Port number for the Pinot controller ### `ssl` _[boolean]_ - Enable SSL connection to Pinot ### `log_queries` _[boolean]_ - Log raw SQL queries executed through Pinot ### `max_open_conns` _[integer]_ - Maximum number of open connections to the Pinot database ### `timeout_ms` _[integer]_ - Query timeout in milliseconds ```yaml # Example: Pinot connector configuration type: connector # Must be `connector` (required) driver: pinot # Must be `pinot` _(required)_ username: "myusername" # Username for authentication password: "{{ .env.PINOT_PASSWORD }}" # Password for authentication broker_host: "localhost" # Hostname of the Pinot broker broker_port: 9000 # Port number for the Pinot broker controller_host: "localhost" # Hostname of the Pinot controller controller_port: 9000 # Port number for the Pinot controller ssl: true # Enable SSL connection to Pinot log_queries: true # Log raw SQL queries executed through Pinot max_open_conns: 100 # Maximum number of open connections to the Pinot database timeout_ms: 30000 # Query timeout in milliseconds ``` ## StarRocks ### `driver` _[string]_ - Refers to the driver type and must be driver `starrocks` _(required)_ ### `dsn` _[string]_ - DSN (Data Source Name) for the StarRocks connection. Follows MySQL protocol format. ### `host` _[string]_ - StarRocks FE (Frontend) server hostname ### `port` _[integer]_ - MySQL protocol port of StarRocks FE ### `username` _[string]_ - Username for authentication ### `password` _[string]_ - Password for authentication ### `catalog` _[string]_ - StarRocks catalog name (for external catalogs like Iceberg, Hive) ### `database` _[string]_ - StarRocks database name ### `ssl` _[boolean]_ - Enable SSL/TLS encryption ### `log_queries` _[boolean]_ - Controls whether to log raw SQL queries ```yaml # Example: StarRocks connector configuration type: connector # Must be `connector` (required) driver: starrocks # Must be `starrocks` _(required)_ host: "starrocks-fe.example.com" # Hostname of the StarRocks FE server port: 9030 # MySQL protocol port of StarRocks FE username: "analyst" # Username for authentication password: "{{ .env.STARROCKS_PASSWORD }}" # Password for authentication catalog: "default_catalog" # StarRocks catalog name database: "my_database" # StarRocks database name ssl: false # Enable SSL/TLS encryption ``` ## Postgres ### `driver` _[string]_ - Refers to the driver type and must be driver `postgres` _(required)_ ### `dsn` _[string]_ - **Data Source Name (DSN)** for the PostgreSQL connection, provided in [PostgreSQL connection string format](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING). PostgreSQL supports both **key=value format** and **URI format**. key=value format example: ```text user=user password=password host=host port=5432 dbname=mydb ``` Rules for key=value format for special characters: - To write an empty value, or a value containing spaces, `=`, single quotes, or backslashes, surround it with single quotes. - Single quotes and backslashes inside a value must be escaped with a backslash (`\'` and `\\`). URI format example: ```text postgres://user:password@host:5432/mydb ``` Rules for URI format: - The following characters are allowed [unescaped in the URI](https://datatracker.ietf.org/doc/html/rfc3986#section-2.3): `~` `.` `_` `-` - All other special characters must be percent-encoded (`%XX` format). Examples (URI format with encoded characters): ```text postgres://user:pa%40ss@localhost:5432/my-db # '@' is encoded as %40 postgres://user:pa%3Ass@localhost:5432/my-db # ':' is encoded as %3A ``` ### `host` _[string]_ - Hostname of the Postgres server ### `port` _[string]_ - Port number for the Postgres server ### `dbname` _[string]_ - Name of the Postgres database ### `user` _[string]_ - Username for authentication ### `password` _[string]_ - Password for authentication ### `sslmode` _[string]_ - ssl mode options: `disable`, `allow`, `prefer` or `require`. ### `max_open_conns` _[integer]_ - Maximum number of open connections to the database (defaults to 1) ### `conn_max_lifetime` _[string]_ - Maximum time a connection may be reused, as a Go duration string (defaults to 1m) ### `log_queries` _[boolean]_ - Controls whether to log raw SQL queries ```yaml # Example: Postgres connector configured using individual properties type: connector driver: postgres host: localhost port: 5432 dbname: mydatabase user: myusername password: "{{ .env.POSTGRES_PASSWORD }}" sslmode: prefer ``` ```yaml # Example: Postgres connector configured using dsn type: connector driver: postgres dsn: "{{ .env.POSTGRES_DSN }}" # Define DSN in .env file ``` ## Supabase ### `driver` _[string]_ - Refers to the driver type and must be driver `postgres` _(required)_ ### `dsn` _[string]_ - **Data Source Name (DSN)** for the Supabase connection, provided in [PostgreSQL connection string format](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING). Supabase uses PostgreSQL under the hood, so all PostgreSQL connection formats are supported. URI format example: ```text postgresql://postgres.[ref]:[password]@aws-0-[region].pooler.supabase.com:5432/postgres ``` ### `host` _[string]_ - Hostname of the Supabase database (e.g. aws-0-us-east-1.pooler.supabase.com) ### `port` _[string]_ - Port number for the Supabase database ### `dbname` _[string]_ - Name of the Supabase database ### `user` _[string]_ - Username for authentication (e.g. postgres.[your-project-ref]) ### `password` _[string]_ - Password for authentication ### `sslmode` _[string]_ - ssl mode options: `disable`, `allow`, `prefer` or `require`. ### `max_open_conns` _[integer]_ - Maximum number of open connections to the database (defaults to 1) ### `conn_max_lifetime` _[string]_ - Maximum time a connection may be reused, as a Go duration string (defaults to 1m) ### `log_queries` _[boolean]_ - Controls whether to log raw SQL queries ```yaml # Example: Supabase connector configured using individual properties type: connector driver: postgres host: aws-0-us-east-1.pooler.supabase.com port: 5432 dbname: postgres user: "postgres.[your-project-ref]" password: "{{ .env.SUPABASE_PASSWORD }}" sslmode: require ``` ```yaml # Example: Supabase connector configured using dsn type: connector driver: postgres dsn: "{{ .env.SUPABASE_DSN }}" # Define DSN in .env file ``` ## Redshift ### `driver` _[string]_ - Refers to the driver type and must be driver `redshift` _(required)_ ### `aws_access_key_id` _[string]_ - AWS Access Key ID used for authenticating with Redshift. _(required)_ ### `aws_secret_access_key` _[string]_ - AWS Secret Access Key used for authenticating with Redshift. _(required)_ ### `aws_access_token` _[string]_ - AWS Session Token for temporary credentials (optional). ### `region` _[string]_ - AWS region where the Redshift cluster or workgroup is hosted (e.g., 'us-east-1'). ### `database` _[string]_ - Name of the Redshift database to query. _(required)_ ### `workgroup` _[string]_ - Workgroup name for Redshift Serverless, in case of provisioned Redshift clusters use 'cluster_identifier'. ### `cluster_identifier` _[string]_ - Cluster identifier for provisioned Redshift clusters, in case of Redshift Serverless use 'workgroup' . ### `allow_host_access` _[boolean]_ - Allow access to host environment configuration ### `log_queries` _[boolean]_ - Controls whether to log raw SQL queries ```yaml # Example: Redshift connector configuration type: connector # Must be `connector` (required) driver: redshift # Must be `redshift` _(required)_ aws_access_key_id: "{{ .env.AWS_ACCESS_KEY_ID }}" # AWS Access Key ID used for authenticating with Redshift. aws_secret_access_key: "{{ .env.AWS_SECRET_ACCESS_KEY }}" # AWS Secret Access Key used for authenticating with Redshift. aws_access_token: "{{ .env.AWS_ACCESS_TOKEN }}" # AWS Session Token for temporary credentials (optional). region: "us-east-1" # AWS region where the Redshift cluster or workgroup is hosted (e.g., 'us-east-1'). database: "mydatabase" # Name of the Redshift database to query. workgroup: "my-workgroup" # Workgroup name for Redshift Serverless, in case of provisioned Redshift clusters use 'cluster_identifier'. cluster_identifier: "my-cluster-identifier" # Cluster identifier for provisioned Redshift clusters, in case of Redshift Serverless use 'workgroup' . ``` ## S3 ### `driver` _[string]_ - Refers to the driver type and must be driver `s3` _(required)_ ### `aws_access_key_id` _[string]_ - AWS Access Key ID used for authentication ### `aws_secret_access_key` _[string]_ - AWS Secret Access Key used for authentication ### `aws_access_token` _[string]_ - Optional AWS session token for temporary credentials ### `endpoint` _[string]_ - Optional custom endpoint URL for S3-compatible storage ### `region` _[string]_ - AWS region of the S3 bucket ### `aws_role_arn` _[string]_ - ARN of the IAM role to assume for accessing S3 resources ### `aws_role_session_name` _[string]_ - Session name to use when assuming the IAM role ### `aws_external_id` _[string]_ - External ID for cross-account role assumption ### `path_prefixes` _[string, array]_ - A list of bucket path prefixes that this connector is allowed to access. Useful when different buckets or bucket prefixes use different credentials, allowing the system to select the appropriate connector based on the bucket path. Example: `s3://my-bucket/`, ` s3://my-bucket/path/` ,`s3://my-bucket/path/prefix` ### `allow_host_access` _[boolean]_ - Allow access to host environment configuration ```yaml # Example: S3 connector configuration type: connector # Must be `connector` (required) driver: s3 # Must be `s3` _(required)_ aws_access_key_id: "{{ .env.AWS_ACCESS_KEY_ID }}" # AWS Access Key ID used for authentication aws_secret_access_key: "{{ .env.AWS_SECRET_ACCESS_KEY }}" # AWS Secret Access Key used for authentication aws_access_token: "{{ .env.AWS_ACCESS_TOKEN }}" # Optional AWS session token for temporary credentials endpoint: "https://my-s3-endpoint.com" # Optional custom endpoint URL for S3-compatible storage region: "us-east-1" # AWS region of the S3 bucket ``` ## Slack ### `driver` _[string]_ - Refers to the driver type and must be driver `slack` _(required)_ ### `bot_token` _[string]_ - Bot token used for authenticating Slack API requests _(required)_ ```yaml # Example: Slack connector configuration type: connector # Must be `connector` (required) driver: slack # Must be `slack` _(required)_ bot_token: "{{ .env.SLACK_BOT_TOKEN }}" # Bot token used for authenticating Slack API requests ``` ## Snowflake ### `driver` _[string]_ - Refers to the driver type and must be driver `snowflake` _(required)_ ### `account` _[string]_ - Snowflake account identifier. To find your Snowflake account identifier, look at your Snowflake account URL. The account identifier is everything before .snowflakecomputing.com ### `user` _[string]_ - Username for the Snowflake connection. ### `password` _[string]_ - Password for the Snowflake connection. _(deprecated, use privateKey instead)_ ### `privateKey` _[string]_ - Private key for JWT authentication. :::tip Private key must be generated as a **PKCS#8 (nocrypt) key**, since the Snowflake Go driver only supports unencrypted private keys. After generating, it must be **base64 URL encoded**. Example commands to generate and encode: ```bash # Generate a 2048-bit unencrypted PKCS#8 private key openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 -nocrypt # Convert URL safe format for Snowflake cat rsa_key.p8 | grep -v "\----" | tr -d '\n' | tr '+/' '-_' ``` See: https://docs.snowflake.com/en/guide/key-pair-auth ::: ### `authenticator` _[string]_ - Optional authenticator type (e.g., SNOWFLAKE_JWT). ### `database` _[string]_ - Name of the Snowflake database. ### `schema` _[string]_ - Schema within the database to use. ### `warehouse` _[string]_ - Compute warehouse to use for queries. ### `role` _[string]_ - Snowflake role to use. ### `dsn` _[string]_ - DSN (Data Source Name) for the Snowflake connection. This is intended for **advanced configuration** where you want to specify properties that are not explicitly defined above. It can only be used when the other connection fields (account, user, password, database, schema, warehouse, role, authenticator, privateKey) are **not used**. For details on private key generation and encoding, see the `privateKey` property. ### `parallel_fetch_limit` _[integer]_ - Maximum number of concurrent fetches during query execution. ### `log_queries` _[boolean]_ - Controls whether to log raw SQL queries ```yaml # Example: Snowflake connector basic configuration type: connector driver: snowflake account: my_account_identifier user: my_user privateKey: "{{ .env.SNOWFLAKE_PRIVATE_KEY }}" # define SNOWFLAKE_PRIVATE_KEY in .env file database: my_db schema: my_schema warehouse: my_wh role: my_role parallel_fetch_limit: 2 ``` ```yaml # Example: Snowflake connector advance configuration type: connector driver: snowflake dsn: "{{ .env.SNOWFLAKE_DSN }}" # define SNOWFLAKE_DSN in .env file parallel_fetch_limit: 2 ``` --- ##### Explore Dashboard YAML --- note: GENERATED. DO NOT EDIT. title: Explore Dashboard YAML sidebar_position: 36 --- Explore dashboards provide an interactive way to explore data with predefined measures and dimensions. ## Properties ### `type` _[string]_ - Refers to the resource type and must be `explore` _(required)_ ### `display_name` _[string]_ - Refers to the display name for the explore dashboard ### `metrics_view` _[string]_ - Refers to the metrics view resource _(required)_ ### `description` _[string]_ - Refers to the description of the explore dashboard ### `banner` _[string]_ - Refers to the custom banner displayed at the header of an explore dashboard ### `dimensions` _[oneOf]_ - List of dimension names. Use '*' to select all dimensions (default) - **option 1** - _[string]_ - Wildcard(*) selector that includes all available fields in the selection - **option 2** - _[array of string]_ - Explicit list of fields to include in the selection - **option 3** - _[object]_ - Advanced matching using regex, DuckDB expression, or exclusion - **`regex`** - _[string]_ - Select fields using a regular expression - **`expr`** - _[string]_ - DuckDB SQL expression to select fields based on custom logic - **`exclude`** - _[object]_ - Select all fields except those listed here ```yaml # Example: Select a dimension dimensions: - country ``` ```yaml # Example: Select all dimensions except one dimensions: exclude: - country ``` ```yaml # Example: Select all dimensions that match a regex dimensions: expr: "^public_.*$" ``` ### `measures` _[oneOf]_ - List of measure names. Use '*' to select all measures (default) - **option 1** - _[string]_ - Wildcard(*) selector that includes all available fields in the selection - **option 2** - _[array of string]_ - Explicit list of fields to include in the selection - **option 3** - _[object]_ - Advanced matching using regex, DuckDB expression, or exclusion - **`regex`** - _[string]_ - Select fields using a regular expression - **`expr`** - _[string]_ - DuckDB SQL expression to select fields based on custom logic - **`exclude`** - _[object]_ - Select all fields except those listed here ```yaml # Example: Select a measure measures: - sum_of_total ``` ```yaml # Example: Select all measures except one measures: exclude: - sum_of_total ``` ```yaml # Example: Select all measures that match a regex measures: expr: "^public_.*$" ``` ### `theme` _[oneOf]_ - Name of the theme to use. Only one of theme and embedded_theme can be set. - **option 1** - _[string]_ - Name of an existing theme to apply to the dashboard - **option 2** - _[object]_ - Inline theme configuration. - **`colors`** - _[object]_ - Used to override the dashboard colors. Either primary or secondary color must be provided. - **`primary`** - _[string]_ - Overrides the primary blue color in the dashboard. Can have any hex, [named colors](https://www.w3.org/TR/css-color-4/#named-colors) or hsl() formats. Note that the hue of the input colors is used for variants but the saturation and lightness is copied over from the [blue color palette](https://tailwindcss.com/docs/customizing-colors). - **`secondary`** - _[string]_ - Overrides the secondary color in the dashboard. Applies to the loading spinner only as of now. Can have any hex, [named colors](https://www.w3.org/TR/css-color-4/#named-colors) or hsl() formats. - **`light`** - _[object]_ - Light theme color configuration - **`primary`** - _[string]_ - Primary color for light theme. Can have any hex, [named colors](https://www.w3.org/TR/css-color-4/#named-colors) or hsl() formats. - **`secondary`** - _[string]_ - Secondary color for light theme. Can have any hex, [named colors](https://www.w3.org/TR/css-color-4/#named-colors) or hsl() formats. - **`kpi-positive`** - _[string]_ - Color for positive KPI delta values in light theme. Defaults to fg-secondary (gray). - **`kpi-negative`** - _[string]_ - Color for negative KPI delta values in light theme. Defaults to red. - **`variables`** - _[object]_ - Custom CSS variables for light theme - **`dark`** - _[object]_ - Dark theme color configuration - **`primary`** - _[string]_ - Primary color for dark theme. Can have any hex, [named colors](https://www.w3.org/TR/css-color-4/#named-colors) or hsl() formats. - **`secondary`** - _[string]_ - Secondary color for dark theme. Can have any hex, [named colors](https://www.w3.org/TR/css-color-4/#named-colors) or hsl() formats. - **`kpi-positive`** - _[string]_ - Color for positive KPI delta values in dark theme. Defaults to fg-secondary (gray). - **`kpi-negative`** - _[string]_ - Color for negative KPI delta values in dark theme. Defaults to red. - **`variables`** - _[object]_ - Custom CSS variables for dark theme ### `time_ranges` _[array of oneOf]_ - Overrides the list of default time range selections available in the dropdown. It can be a string or an object with a 'range' and optional 'comparison_offsets'. ```yaml time_ranges: - PT15M // Simplified syntax to specify only the range - PT1H - PT6H - P7D - range: P5D // Advanced syntax to specify comparison_offsets as well - P4W - rill-TD // Today - rill-WTD // Week-To-date ``` - **option 1** - _[string]_ - An [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601#Durations) duration or one of the [Rill ISO 8601 extensions](/reference/time-syntax/rill-iso-extensions#extensions) extensions for the selection. - **option 2** - _[object]_ - Object containing time range and comparison configuration - **`range`** - _[string]_ - An [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601#Durations) duration or one of the [Rill ISO 8601 extensions](/reference/time-syntax/rill-iso-extensions#extensions) extensions for the selection. _(required)_ - **`comparison_offsets`** - _[array of oneOf]_ - List of time comparison options for this time range selection (optional). Must be one of the [Rill ISO 8601 extensions](https://docs.rilldata.com/reference/rill-iso-extensions#extensions). - **option 1** - _[string]_ - Offset string only (range is inferred) - **option 2** - _[object]_ - Object containing offset and range configuration for time comparison - **`offset`** - _[string]_ - Time offset for comparison (e.g., 'P1D' for one day ago) - **`range`** - _[string]_ - Custom time range for comparison period ### `time_zones` _[array of string]_ - Refers to the time zones that should be pinned to the top of the time zone selector. It should be a list of [IANA time zone identifiers](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) ### `lock_time_zone` _[boolean]_ - When true, the dashboard will be locked to the first time provided in the time_zones list. When no time_zones are provided, the dashboard will be locked to UTC ### `allow_custom_time_range` _[boolean]_ - Defaults to true, when set to false it will hide the ability to set a custom time range for the user. ### `defaults` _[object]_ - Defines the defaults YAML struct. ```yaml defaults: #define all the defaults within here dimensions: - dim_1 - dim_2 measures: - measure_1 - measure_2 time_range: P1M comparison_mode: dimension #time, none comparison_dimension: filename ``` - **`dimensions`** - _[oneOf]_ - Provides the default dimensions to load on viewing the dashboard - **option 1** - _[string]_ - Wildcard(*) selector that includes all available fields in the selection - **option 2** - _[array of string]_ - Explicit list of fields to include in the selection - **option 3** - _[object]_ - Advanced matching using regex, DuckDB expression, or exclusion - **`regex`** - _[string]_ - Select fields using a regular expression - **`expr`** - _[string]_ - DuckDB SQL expression to select fields based on custom logic - **`exclude`** - _[object]_ - Select all fields except those listed here - **`measures`** - _[oneOf]_ - Provides the default measures to load on viewing the dashboard - **option 1** - _[string]_ - Wildcard(*) selector that includes all available fields in the selection - **option 2** - _[array of string]_ - Explicit list of fields to include in the selection - **option 3** - _[object]_ - Advanced matching using regex, DuckDB expression, or exclusion - **`regex`** - _[string]_ - Select fields using a regular expression - **`expr`** - _[string]_ - DuckDB SQL expression to select fields based on custom logic - **`exclude`** - _[object]_ - Select all fields except those listed here - **`time_range`** - _[string]_ - Refers to the default time range shown when a user initially loads the dashboard. The value must be either an [ISO 8601 duration](https://en.wikipedia.org/wiki/ISO_8601#Durations) (for example, PT12H for 12 hours, P1M for 1 month, or P26W for 26 weeks) or one of the [Rill ISO 8601 extensions](https://docs.rilldata.com/reference/rill-iso-extensions#extensions). - **`comparison_mode`** - _[string]_ - Controls how to compare current data with historical or categorical baselines. Options: `none` (no comparison), `time` (compares with past based on default_time_range), `dimension` (compares based on comparison_dimension values) - **`comparison_dimension`** - _[string]_ - For dimension mode, specify the comparison dimension by name. ### `embeds` _[object]_ - Configuration options for embedded dashboard views - **`hide_pivot`** - _[boolean]_ - When true, hides the pivot table view in embedded mode ### `security` _[object]_ - Defines [security rules and access control policies](/developers/build/metrics-view/security) for dashboards (without row filtering) - **`access`** - _[oneOf]_ - Expression indicating if the user should be granted access to the dashboard. If not defined, it will resolve to false and the dashboard won't be accessible to anyone. Needs to be a valid SQL expression that evaluates to a boolean. - **option 1** - _[string]_ - SQL expression that evaluates to a boolean to determine access - **option 2** - _[boolean]_ - Direct boolean value to allow or deny access ## Common Properties ### `name` _[string]_ - Name is usually inferred from the filename, but can be specified manually. ### `refs` _[array of string]_ - List of resource references ### `tags` _[array of string]_ - Tags for organizing and filtering the resource (e.g. on the project dashboards list). ### `dev` _[object]_ - Overrides any properties in development environment. ### `prod` _[object]_ - Overrides any properties in production environment. --- ##### YAML Syntax --- note: GENERATED. DO NOT EDIT. title: YAML Syntax sidebar_position: 30 --- ## Overview When you create models and dashboards, these objects are represented as object files on the file system. You can find these files in your `models` and `dashboards` folders in your project by default. :::info Working with resources outside their native folders It is possible to define resources (such as [models](models.md), [metrics-views](metrics-views.md), [dashboards](explore-dashboards.md), [custom APIs](apis.md), or [themes](themes.md)) within any nested folder within your Rill project directory. However, for any YAML configuration file, it is imperative that the `type` property is appropriately defined within the underlying resource configuration, or Rill will not be able to resolve the resource type correctly! ::: Projects can simply be rehydrated from Rill project files into an explorable data application as long as there is sufficient access and credentials to the source data - figuring out the dependencies, pulling down data, & validating your model queries and metrics view configurations. The result is a set of functioning exploratory dashboards. You can see a few different example projects by visiting our [example GitHub repository](https://github.com/rilldata/rill-examples). :::tip For more information about using Git or cloning projects locally, please see our page on [GitHub Basics](/developers/deploy/deploy-dashboard/github-101). ::: ## Project files types - [Connector YAML](connectors.md) - [Source YAML](sources.md) - [Models YAML](models.md) - [Metrics View YAML](metrics-views.md) - [Canvas Dashboard YAML](canvas-dashboards.md) - [Explore Dashboard YAML](explore-dashboards.md) - [Alert YAML](alerts.md) - [Report YAML](reports.md) - [API YAML](apis.md) - [Theme YAML](themes.md) - [Component YAML](component.md) - [Project YAML](rill-yaml.md) --- ##### Metrics View YAML --- note: GENERATED. DO NOT EDIT. title: Metrics View YAML sidebar_position: 34 --- In your Rill project directory, create a metrics view, `.yaml`, file in the `metrics` directory. Rill will ingest the metric view definition next time you run `rill start`. ## Properties ### `version` _[string]_ - The version of the metrics view schema ### `type` _[string]_ - Refers to the resource type and must be `metrics_view` ### `connector` _[string]_ - Refers to the connector type for the metrics view, see [OLAP engines](/developers/build/connectors/olap) for more information ### `display_name` _[string]_ - Refers to the display name for the metrics view ### `description` _[string]_ - Refers to the description for the metrics view ### `ai_instructions` _[string]_ - Extra instructions for [AI agents](/guide/ai/mcp). Used to guide natural language question answering and routing. ### `parent` _[string]_ - Refers to the parent metrics view from which this metrics view is derived. If specified, this will inherit properties from the parent metrics view ### `model` _[string]_ - Refers to the model powering the dashboard (either model or table is required) ### `database` _[string]_ - Refers to the database to use in the OLAP engine (to be used in conjunction with table). Otherwise, will use the default database or schema if not specified ### `database_schema` _[string]_ - Refers to the schema to use in the OLAP engine (to be used in conjunction with table). Otherwise, will use the default database or schema if not specified ### `table` _[string]_ - Refers to the table powering the dashboard, should be used instead of model for dashboards create from external OLAP tables (either table or model is required) ### `timeseries` _[string]_ - Refers to the timestamp column from your model that will underlie x-axis data in the line charts. If not specified, the line charts will not appear ### `watermark` _[string]_ - A SQL expression that tells us the max timestamp that the measures are considered valid for. Usually does not need to be overwritten ### `data_time_range` _[string]_ - Optional [rilltime](https://docs.rilldata.com/reference/time-syntax) expression describing the base table's time coverage (e.g. `-5Y to now`). When set, Rill skips the `min`/`max` OLAP probe for the base table and uses the declared bounds for coverage checks. The start must be bounded; `inf` and `earliest` are rejected. To declare full history, use a concrete early bound such as `-100Y to now` or omit this field to probe the table. ### `smallest_time_grain` _[string]_ - Refers to the smallest time granularity the user is allowed to view. The valid values are: millisecond, second, minute, hour, day, week, month, quarter, year ### `first_day_of_week` _[integer]_ - Refers to the first day of the week for time grain aggregation (for example, Sunday instead of Monday). The valid values are 1 through 7 where Monday=1 and Sunday=7 ### `first_month_of_year` _[integer]_ - Refers to the first month of the year for time grain aggregation. The valid values are 1 through 12 where January=1 and December=12 ### `max_query_time_range` _[string]_ - The maximum time span any single query against this metrics view may cover, expressed as an ISO 8601 duration with day-or-larger granularity (e.g. `P90D`, `P3M`, `P1Y`). Sub-day durations such as `PT12H` are not supported. Applies independently to the primary and comparison time ranges. If unset, no limit is enforced. ### `dimensions` _[array of object]_ - Relates to exploring segments or dimensions of your data and filtering the dashboard - **`name`** - _[string]_ - a stable identifier for the dimension - **`display_name`** - _[string]_ - a display name for your dimension - **`description`** - _[string]_ - a freeform text description of the dimension - **`tags`** - _[array of string]_ - optional list of tags for categorizing the dimension (defaults to empty) - **`type`** - _[string]_ - Dimension type: "geo" for geospatial dimensions, "time" for time dimensions, or "categorical" for categorical dimensions. Default is undefined and the type will be inferred instead. - **`column`** - _[string]_ - A categorical column. - **`expression`** - _[string]_ - A non-aggregate expression such as string_split(domain, '.'). One of column and expression is required, but cannot have both at the same time. - **`unnest`** - _[boolean]_ - If true, allows multi-valued dimensions to be unnested (such as lists), and filters will automatically switch to "contains" instead of exact match. - **`uri`** - _[string, boolean]_ - Enable if your dimension is a clickable URL to enable single-click navigation (boolean or valid SQL expression). - **`lookup_table`** - _[string]_ - The name of a ClickHouse dictionary to use for query-time lookups. Use `database.dictionary_name` for dictionaries in a non-default database. All three `lookup_*` fields (`lookup_table`, `lookup_key_column`, `lookup_value_column`) must be specified together. See [Query-Time Joins](/developers/build/metrics-view/dimensions/lookup) for details. - **`lookup_key_column`** - _[string]_ - the primary key column in the lookup dictionary that corresponds to the dimension's `column` in the fact table - **`lookup_value_column`** - _[string]_ - the attribute column in the lookup dictionary whose values will be displayed for this dimension - **`lookup_default_expression`** - _[string]_ - an optional SQL expression used as a fallback value when no match is found in the dictionary (maps to `dictGetOrDefault`) ### `measures` _[array of object]_ - Used to define the numeric aggregates of columns from your data model - **`name`** - _[string]_ - a stable identifier for the measure _(required)_ - **`display_name`** - _[string]_ - the display name of your measure. - **`label`** - _[string]_ - a label for your measure, deprecated use display_name - **`description`** - _[string]_ - a freeform text description of the measure - **`tags`** - _[array of string]_ - optional list of tags for categorizing the measure (defaults to empty) - **`type`** - _[string]_ - Measure calculation type: "simple" for basic aggregations, "derived" for calculations using other measures, or "time_comparison" for period-over-period analysis. Defaults to "simple" unless dependencies exist. - **`expression`** - _[string]_ - a combination of operators and functions for aggregations - **`window`** - _[anyOf]_ - A measure window can be defined as a keyword string (e.g. 'time' or 'all') or an object with detailed window configuration. For more information, see the [window functions](/developers/build/metrics-view/measures/windows) documentation. - **option 1** - _[string]_ - Shorthand: `time` or `true` means time-partitioned, `all` means non-partitioned. - **option 2** - _[object]_ - Detailed window configuration for measure calculations, allowing control over partitioning, ordering, and frame definition. - **`partition`** - _[boolean]_ - Controls whether the window is partitioned. When true, calculations are performed within each partition separately. - **`order`** - _[string]_ - Specifies the fields to order the window by, determining the sequence of rows within each partition. - **option 1** - _[string]_ - Simple field name as a string. - **option 2** - _[array of oneOf]_ - List of field selectors, each can be a string or an object with detailed configuration. - **option 1** - _[string]_ - Shorthand field selector, interpreted as the name. - **option 2** - _[object]_ - Detailed field selector configuration with name and optional time grain. - **`name`** - _[string]_ - Name of the field to select. _(required)_ - **`time_grain`** - _[string]_ - Time grain for time-based dimensions. - **`frame`** - _[string]_ - Defines the window frame boundaries for calculations, specifying which rows are included in the window relative to the current row. - **`per`** - _[oneOf]_ - Dimensions to partition the measure by. - **option 1** - _[string]_ - Simple field name as a string. - **option 2** - _[array of oneOf]_ - List of field selectors, each can be a string or an object with detailed configuration. - **option 1** - _[string]_ - Shorthand field selector, interpreted as the name. - **option 2** - _[object]_ - Detailed field selector configuration with name and optional time grain. - **`name`** - _[string]_ - Name of the field to select. _(required)_ - **`time_grain`** - _[string]_ - Time grain for time-based dimensions. - **`requires`** - _[oneOf]_ - Uses an available measure or dimension in your metrics view to set a required parameter. Cannot be used with simple measures. See [referencing measures](/developers/build/metrics-view/measures/referencing) for more information. - **option 1** - _[string]_ - Simple field name as a string. - **option 2** - _[array of oneOf]_ - List of field selectors, each can be a string or an object with detailed configuration. - **option 1** - _[string]_ - Shorthand field selector, interpreted as the name. - **option 2** - _[object]_ - Detailed field selector configuration with name and optional time grain. - **`name`** - _[string]_ - Name of the field to select. _(required)_ - **`time_grain`** - _[string]_ - Time grain for time-based dimensions. - **`valid_percent_of_total`** - _[boolean]_ - Indicates whether percent-of-total values should be rendered for this measure. - **`format_preset`** - _[string]_ - Controls the formatting of this measure using a predefined preset. Measures cannot have both `format_preset` and `format_d3`. If neither is supplied, the measure will be formatted using the `humanize` preset by default. Available options: - `humanize`: Round numbers into thousands (K), millions(M), billions (B), etc. - `none`: Raw output. - `currency_usd`: Round to 2 decimal points with a dollar sign ($). - `currency_eur`: Round to 2 decimal points with a euro sign (€). - `percentage`: Convert a rate into a percentage with a % sign. - `interval_ms`: Convert milliseconds into human-readable durations like hours (h), days (d), years (y), etc. (optional) - **`format_d3`** - _[string]_ - Controls the formatting of this measure using a [d3-format](https://d3js.org/d3-format) string. If an invalid format string is supplied, the measure will fall back to `format_preset: humanize`. A measure cannot have both `format_preset` and `format_d3`. If neither is provided, the humanize preset is used by default. Example: `format_d3: ".2f"` formats using fixed-point notation with two decimal places. Example: `format_d3: ",.2r"` formats using grouped thousands with two significant digits. (optional) - **`format_d3_locale`** - _[object]_ - Locale configuration passed through to D3, enabling changes to the currency symbol and other formatting options. For details, see the docs for D3's formatLocale. ```yaml format_d3: "$," format_d3_locale: grouping: [3, 2] currency: ["₹", ""] ``` - **`grouping`** - _[array]_ - Grouping for the currency symbol. - **`currency`** - _[array]_ - the currency symbol - **`treat_nulls_as`** - _[string]_ - Configures the value to fill in for missing time buckets. This also works generally as COALESCE over non-empty time buckets. - **`lower_is_better`** - _[boolean]_ - When true, decreases in this measure are favorable (e.g. bounce rate, latency, error count). UI surfaces that render comparison deltas (KPIs, big numbers, leaderboards, pivot tables, time-series tooltips) swap their positive/negative coloring accordingly. ### `parent_dimensions` _[oneOf]_ - Optional field selectors for dimensions to inherit from the parent metrics view. - **option 1** - _[string]_ - Wildcard(*) selector that includes all available fields in the selection - **option 2** - _[array of string]_ - Explicit list of fields to include in the selection - **option 3** - _[object]_ - Advanced matching using regex, DuckDB expression, or exclusion - **`regex`** - _[string]_ - Select fields using a regular expression - **`expr`** - _[string]_ - DuckDB SQL expression to select fields based on custom logic - **`exclude`** - _[object]_ - Select all fields except those listed here ### `parent_measures` _[oneOf]_ - Optional field selectors for measures to inherit from the parent metrics view. - **option 1** - _[string]_ - Wildcard(*) selector that includes all available fields in the selection - **option 2** - _[array of string]_ - Explicit list of fields to include in the selection - **option 3** - _[object]_ - Advanced matching using regex, DuckDB expression, or exclusion - **`regex`** - _[string]_ - Select fields using a regular expression - **`expr`** - _[string]_ - DuckDB SQL expression to select fields based on custom logic - **`exclude`** - _[object]_ - Select all fields except those listed here ### `annotations` _[array of object]_ - Used to define annotations that can be displayed on charts - **`name`** - _[string]_ - A stable identifier for the annotation. Defaults to model or table names when not specified - **`model`** - _[string]_ - Refers to the model powering the annotation (either table or model is required). The model must have 'time' and 'description' columns. Optional columns include 'time_end' for range annotations and 'grain' to specify when the annotation should appear based on dashboard grain level. - **`database`** - _[string]_ - Refers to the database to use in the OLAP engine (to be used in conjunction with table). Otherwise, will use the default database or schema if not specified - **`database_schema`** - _[string]_ - Refers to the schema to use in the OLAP engine (to be used in conjunction with table). Otherwise, will use the default database or schema if not specified - **`table`** - _[string]_ - Refers to the table powering the annotation, should be used instead of model for annotations from external OLAP tables (either table or model is required) - **`connector`** - _[string]_ - Refers to the connector to use for the annotation - **`measures`** - _[oneOf]_ - Specifies which measures to apply the annotation to. Applies to all measures if not specified - **option 1** - _[string]_ - Wildcard(*) selector that includes all available fields in the selection - **option 2** - _[array of string]_ - Explicit list of fields to include in the selection - **option 3** - _[object]_ - Advanced matching using regex, DuckDB expression, or exclusion - **`regex`** - _[string]_ - Select fields using a regular expression - **`expr`** - _[string]_ - DuckDB SQL expression to select fields based on custom logic - **`exclude`** - _[object]_ - Select all fields except those listed here ### `rollups` _[array of object]_ - Pre-aggregated rollup tables that can be used to accelerate queries. When a query's dimensions, measures, time grain, and time range match a rollup, the query is automatically routed to the rollup table instead of the base table. - **`model`** - _[string]_ - Refers to the model or table powering the rollup (required) _(required)_ - **`database`** - _[string]_ - Refers to the database to use in the OLAP engine - **`database_schema`** - _[string]_ - Refers to the schema to use in the OLAP engine - **`time_grain`** - _[string]_ - The time grain of the rollup (required). Valid values are: millisecond, second, minute, hour, day, week, month, quarter, year _(required)_ - **`time_zone`** - _[string]_ - IANA timezone of the rollup table (e.g. America/New_York). For day+ grains, queries are only routed to the rollup if the query timezone matches. - **`dimensions`** - _[oneOf]_ - Optional field selectors for dimensions to include in the rollup from the base metrics view. If not specified, all dimensions are included. - **option 1** - _[string]_ - Wildcard(*) selector that includes all available fields in the selection - **option 2** - _[array of string]_ - Explicit list of fields to include in the selection - **option 3** - _[object]_ - Advanced matching using regex, DuckDB expression, or exclusion - **`regex`** - _[string]_ - Select fields using a regular expression - **`expr`** - _[string]_ - DuckDB SQL expression to select fields based on custom logic - **`exclude`** - _[object]_ - Select all fields except those listed here - **`measures`** - _[oneOf]_ - Optional field selectors for measures to include in the rollup from the base metrics view. If not specified, all measures are included. - **option 1** - _[string]_ - Wildcard(*) selector that includes all available fields in the selection - **option 2** - _[array of string]_ - Explicit list of fields to include in the selection - **option 3** - _[object]_ - Advanced matching using regex, DuckDB expression, or exclusion - **`regex`** - _[string]_ - Select fields using a regular expression - **`expr`** - _[string]_ - DuckDB SQL expression to select fields based on custom logic - **`exclude`** - _[object]_ - Select all fields except those listed here - **`data_time_range`** - _[string]_ - Optional [rilltime](https://docs.rilldata.com/reference/time-syntax) expression describing the rollup's time coverage (e.g. `-1Y to now`, `-5Y to -1Y`). When set, Rill skips the `min`/`max` OLAP probe for this rollup and uses the declared bounds for coverage checks. The start must be bounded; `inf` and `earliest` are rejected. To declare full history, use a concrete early bound such as `-100Y to now` or omit this field to probe the table. ### `security` _[object]_ - Defines [security rules and access control policies](/developers/build/metrics-view/security) for resources - **`access`** - _[oneOf]_ - Expression indicating if the user should be granted access to the dashboard. If not defined, it will resolve to false and the dashboard won't be accessible to anyone. Needs to be a valid SQL expression that evaluates to a boolean. - **option 1** - _[string]_ - SQL expression that evaluates to a boolean to determine access - **option 2** - _[boolean]_ - Direct boolean value to allow or deny access - **`row_filter`** - _[string]_ - SQL expression to filter the underlying model by. Can leverage templated user attributes to customize the filter for the requesting user. Needs to be a valid SQL expression that can be injected into a WHERE clause - **`include`** - _[array of object]_ - List of dimension or measure names to include in the dashboard. If include is defined all other dimensions and measures are excluded - **`if`** - _[string]_ - Expression to decide if the column should be included or not. It can leverage templated user attributes. Needs to be a valid SQL expression that evaluates to a boolean _(required)_ - **`names`** - _[anyOf]_ - List of fields to include. Should match the name of one of the dashboard's dimensions or measures _(required)_ - **option 1** - _[array of string]_ - List of specific field names to include - **option 2** - _[string]_ - Wildcard '*' to include all fields - **`exclude`** - _[array of object]_ - List of dimension or measure names to exclude from the dashboard. If exclude is defined all other dimensions and measures are included - **`if`** - _[string]_ - Expression to decide if the column should be excluded or not. It can leverage templated user attributes. Needs to be a valid SQL expression that evaluates to a boolean _(required)_ - **`names`** - _[anyOf]_ - List of fields to exclude. Should match the name of one of the dashboard's dimensions or measures _(required)_ - **option 1** - _[array of string]_ - List of specific field names to exclude - **option 2** - _[string]_ - Wildcard '*' to exclude all fields - **`rules`** - _[array of object]_ - List of detailed security rules that can be used to define complex access control policies - **`type`** - _[string]_ - Type of security rule - access (overall access), field_access (field-level access), or row_filter (row-level filtering) _(required)_ - **`action`** - _[string]_ - Whether to allow or deny access for this rule - **`if`** - _[string]_ - Conditional expression that determines when this rule applies. Must be a valid SQL expression that evaluates to a boolean - **`names`** - _[array of string]_ - List of field names this rule applies to (for field_access type rules) - **`all`** - _[boolean]_ - When true, applies the rule to all fields (for field_access type rules) - **`sql`** - _[string]_ - SQL expression for row filtering (for row_filter type rules) ### `cache` _[object]_ - Enable caching of query results for metrics views backed by externally-managed tables (e.g. in Snowflake, BigQuery). These settings have no effect for metrics views backed by Rill models (where queries are automatically cached and invalidated when the model is refreshed). Each cache entry is keyed by a hash of the query combined with the latest result of `key_sql`. Cached results stay valid as long as `key_sql` returns the same value; when its result changes, prior results become unreachable. `key_sql` itself runs at most once per `key_ttl`, decoupling freshness checks from query traffic. Example: a `key_sql` of `SELECT MAX(updated_at) FROM orders` with `key_ttl: 5m` checks for new data every 5 minutes but only invalidates cached results when new data has actually landed. - **`enabled`** - _[boolean]_ - Whether to cache query results for this metrics view. Defaults to false for metrics views backed by externally-managed tables and to true for metrics views backed by a Rill model. - **`key_sql`** - _[string]_ - SQL returning a single value used in the cache key, typically a max timestamp, version, or row count. Cached results are invalidated when this value changes. Optional; defaults to the metrics view's watermark expression (which itself defaults to `MAX(