Skip to main content

Source YAML

Deprecated Feature

Sources have been deprecated and are now considered "source models." While sources remain backward compatible, we recommend migrating to the new source model format for access to the latest features and improvements.

Next steps:

  • Continue using sources if needed (backward compatible)
  • Migrate to source models via the type:model parameter for existing projects
  • See our model YAML reference for current documentation and best practices

Properties​

type​

[string] - Refers to the resource type and must be connector (required)

connector​

[string] - Refers to the connector type for the source, see connectors for more information (required)

uri​

[string] - Refers to the URI of the remote connector you are using for the source. Rill also supports glob patterns as part of the URI for S3 and GCS (required for type: http, s3, gcs).

  • s3://your-org/bucket/file.parquet — the s3 URI of your file
  • gs://your-org/bucket/file.parquet — the gsutil URI of your file
  • https://data.example.org/path/to/file.parquet — the web address of your file

path​

[string] - Refers to the local path of the connector you are using for the source

sql​

[string] - Sets the SQL query to extract data from a SQL source

region​

[string] - Sets the cloud region of the S3 bucket or Athena

endpoint​

[string] - Overrides the S3 endpoint to connect to

output_location​

[string] - Sets the query output location and result files in Athena

workgroup​

[string] - Sets a workgroup for Athena connector

project_id​

[string] - Sets a project id to be used to run BigQuery jobs

timeout​

[string] - The maximum time to wait for source ingestion

refresh​

[object] - Specifies the refresh schedule that Rill should follow to re-ingest and update the underlying source data (optional).

refresh:
  cron: "* * * * *"
  every: "24h"

  • cron - [string] - A cron schedule expression, which should be encapsulated in single quotes, e.g. * * * * *

  • every - [string] - A Go duration string, such as 24h

db​

[string] - Sets the database for motherduck connections and/or the path to the DuckDB/SQLite db file

database_url​

[string] - Postgres connection string that should be used

duckdb​

[object] - Specifies the raw parameters to inject into the DuckDB read_csv, read_json or read_parquet statement

dsn​

[string] - Used to set the Snowflake connection string

Common Properties​

name​

[string] - Name is usually inferred from the filename, but can be specified manually.

refs​

[array of string] - List of resource references

dev​

[object] - Overrides any properties in development environment.

prod​

[object] - Overrides any properties in production environment.