Skip to main content

Configure Source Refresh

When creating or updating a source in Rill Cloud, you also have the option to configure how often the underlying source is refreshed (and thus ingested into the underlying OLAP layer powering Rill models and dashboards). By default, sources are refreshed manually but this can also be automated to a predefined schedule. This is handled through the underlying source and/or project YAML using standard cron or Go duration syntax.

Configuring source refreshes for Cloud deployments

It is generally strongly recommended to configure source refreshes when deploying a project to Rill Cloud to ensure that your production data (and dashboards) remains up-to-date. The interval that you should set really depends on how often your own source data is refreshed. Furthermore, while it is technically possible to configure source refreshes for Rill Developer as well, Rill Developer is primarily used for local development and thus typically does not require working with the most up-to-date data (local source refreshes that occur too often as well could also lead to resource constraints on your local machine). For more details, please see our pages on environments, templating, and performance optimization.

Configuring source refresh individually​

To specify a source refresh schedule for a particular source, this can be handled using the refresh property in the underlying YAML file. For example, to set a daily refresh for a source, you can do the following:

refresh:
  every: 24h

Similarly, if you would like to utilize cron syntax, the following example would update a source every 15 minutes:

refresh:
  cron: '*/15 * * * *'
Source settings

For more details about available source configurations and properties, check our Source YAML reference page.

Configuring a project-wide default​

You can also specify a project-wide refresh schedule that will apply to all sources by default. This can be done through the rill.yaml file. More details can be found here.

Using the same example as above, the following sets a project-wide default of refreshing sources every 24 hours:

sources:
  refresh:
    every: 24h

Similarly, the following would use cron syntax to set a project-wide configuration of refreshing sources by default every 15 minutes (unless overridden at the individual source level):

sources:
  refresh:
    cron: '*/15 * * * *'
Did you know?

If you have both a project-wide default and source specific refresh schedule configured in the same project, the source specific refresh will override the project default based on how inheritance works in Rill. Otherwise, if not specified, the project-wide default will be used instead!

Running scheduled refreshes in development​

By default, scheduled refreshes are not applied in Rill Developer. If you want to run or test scheduled refreshes in local development, you can override this behavior using the run_in_dev property:

refresh:
  cron: 0 * * * *
  run_in_dev: true