Skip to main content

Partitions and Increments

In order to help with data ingestion into Rill, we will introduce the concepts of partitions and incremental models Before diving into our ClickHouse project, let's understand what each of these are used for and do.

Incremental Model​

An incremental model is defined using the following key pair.

incremental: true

Once this is enabled, this allows Rill to configure the model YAML as an incrementing model. In some following examples, we will use both a time based incremental and glob based increments.

Partitions​

Partitions in models are enabled by defining the partition parameter as seen below:

partitions:
  sql/glob: some partitions definition

Depending on your data, this can be defined as a SQL: statement or a glob: pattern. Once configured, Rill will try to partition your existing data into smaller subcategories which allows you to refresh specific sections only instead of reingesting the whole dataset. (only when incremental is enabled)

By running the following command, you can see all the available partitions and run a refresh command on a specific key or keys.

rill project partition <model_name>
rill project refresh --model <model_name> --partition <partition_key>

Let's look at a few simple examples before diving into our ClickHouse project.

Was this content helpful?