Skip to main content

HTTPS

Overview​

The HTTPS connector allows you to import data from remote sources accessible via HTTP or HTTPS URLs into your Rill project. This includes publicly accessible files hosted on web servers, CDNs, or cloud storage services—perfect for working with datasets that are regularly updated or shared publicly.

Authentication Methods​

To connect to data via HTTPS, you need to choose an authentication approach. Rill supports two methods:

  1. Use Authenticated Endpoints (for private APIs or protected resources)
  2. Use Public URLs (for publicly accessible files without authentication)

When you add data from HTTPS through the Rill UI, the process follows two steps:

  1. Configure Authentication - Set up your HTTPS connector with authentication headers (if required)
  2. Configure Data Model - Define the URL path to your data file

This two-step flow ensures your credentials are securely stored in the connector configuration, while your data model references remain clean and portable.

If your endpoint requires authentication, create an HTTPS connector with your credentials. This method works for both local development and Rill Cloud deployments.

Using the UI​

  1. Click Add Data in your Rill project
  2. Select HTTPS as the data source type
  3. In the authentication step:
    • Configure authentication headers (e.g., Bearer token)
  4. In the data model configuration step, enter the full URL to your data
  5. Click Create to finalize

After the model YAML is generated, you can add additional model settings directly to the file.

Manual Configuration​

If you prefer to configure manually:

Step 1: Create connector configuration

Create connectors/https.yaml:

type: connector
driver: https

headers:
  Authorization: "Bearer {{ .env.HTTPS_TOKEN }}"

Step 2: Add credentials to .env

HTTPS_TOKEN=your_api_token_here
Did you know?

If this project has already been deployed to Rill Cloud and credentials have been set for this connector, you can use rill env pull to pull these cloud credentials locally (into your local .env file). Please note that this may override any credentials you have set locally for this source.

Then, create your first model.

Method 2: Public URLs​

For publicly accessible files that don't require authentication, you can use the HTTPS connector directly without credentials.

Using the UI​

  1. Click Add Data in your Rill project
  2. Select HTTPS as the data source type
  3. In the authentication step, skip or leave authentication blank
  4. In the data model configuration step, enter the full public URL
  5. Click Create to finalize

Manual Configuration​

For public URLs, you can create a simple connector or use the built-in https connector:

Create connectors/public_https.yaml:

type: connector
driver: https

Then, create your first model.

Create Your First Model​

Once your connector is configured using any method above, create a model to define what data to pull.

Create models/https_data.yaml:

type: model
connector: https

path: https://api.endpoint.com/v1/data/export.csv

After creating the model, you can add additional model settings directly to the file.

Supported URL Types​

The HTTPS connector supports various remote data sources:

  • Public HTTP/HTTPS URLs: Direct links to CSV, JSON, Parquet, and other data files
  • Cloud Storage URLs: Public links to files in Google Cloud Storage, Amazon S3, or other cloud providers
  • API Endpoints: REST APIs that return data in supported formats

Separating Dev and Prod Environments​

When ingesting data locally, consider setting parameters in your connector file to limit how much data is retrieved, since costs can scale with the data source. This also helps other developers clone the project and iterate quickly by reducing ingestion time.

For more details, see our Dev/Prod setup docs.

Deploy to Rill Cloud​

When deploying a project to Rill Cloud, Rill requires you to explicitly provide authentication credentials for protected HTTPS endpoints with access to HTTPS used in your project. Please refer to our connector YAML reference docs for more information.

If you subsequently add sources that require new credentials (or if you simply entered the wrong credentials during the initial deploy), you can update the credentials by pushing the Deploy button to update your project or by running the following command in the CLI:

rill env push
Did you know?

If you've already configured credentials locally (in your <RILL_PROJECT_DIRECTORY>/.env file), you can use rill env push to push these credentials to your Rill Cloud project. This will allow other users to retrieve and reuse the same credentials automatically by running rill env pull.

Best Practices​

  • Use HTTPS URLs: Prefer secure HTTPS connections over HTTP when possible
  • Check File Accessibility: Ensure your URLs are publicly accessible or properly authenticated
  • Monitor File Size: Large files may take longer to download and process
  • Regular Updates: Set up automated refreshes for frequently updated datasets

Reference​

For advanced configuration options and properties, see the Connector YAML Reference.