Configure Source Refresh
When creating or updating a source in Rill Cloud, you also have the option to configure how often the underlying source is refreshed (and thus ingested into the underlying OLAP layer powering Rill models and dashboards). By default, sources are refreshed manually but this can also be automated to a predefined schedule. This is handled through the underlying source and/or project YAML using standard cron or Go duration syntax.
It is generally strongly recommended to configure source refreshes when deploying a project to Rill Cloud to ensure that your production data (and dashboards) remains up-to-date. The interval that you should set really depends on how often your own source data is refreshed. Furthermore, while it is technically possible to configure source refreshes for Rill Developer as well, Rill Developer is primarily used for local development and thus typically does not require working with the most up-to-date data (local source refreshes that occur too often as well could also lead to resource constraints on your local machine). For more details, please see our pages on environments, templating, and performance optimization.
Configuring source refresh individually
To specify a source refresh schedule for a particular source, this can be handled using the refresh
property in the underlying YAML file. For example, to set a daily refresh for a source, you can do the following:
refresh:
every: 24h
Similarly, if you would like to utilize cron syntax, the following example would update a source every 15 minutes:
refresh:
cron: '*/15 * * * *'
For more details about available source configurations and properties, check our Source YAML reference page.
Configuring a project-wide default
You can also specify a project-wide refresh schedule that will apply to all sources by default. This can be done through the rill.yaml
file. More details can be found here.
Using the same example as above, the following sets a project-wide default of refreshing sources every 24 hours:
sources:
refresh:
every: 24h
Similarly, the following would use cron syntax to set a project-wide configuration of refreshing sources by default every 15 minutes (unless overridden at the individual source level):
sources:
refresh:
cron: '*/15 * * * *'
If you have both a project-wide default and source specific refresh schedule configured in the same project, the source specific refresh will override the project default based on how inheritance works in Rill. Otherwise, if not specified, the project-wide default will be used instead!
Running scheduled refreshes in development
By default, scheduled refreshes are not applied in Rill Developer. If you want to run or test scheduled refreshes in local development, you can override this behavior using the run_in_dev
property:
refresh:
cron: 0 * * * *
run_in_dev: true