Skip to main content

Athena

Overview​

Amazon Athena is an interactive query service that makes it easy to analyze data directly in Amazon S3 using standard SQL. It is serverless, so there is no infrastructure to manage, and you pay only for the queries you run, making it cost-effective for a wide range of data analysis tasks. Athena is designed for quick, ad-hoc querying of large datasets, enabling businesses to easily integrate it into their analytics and business intelligence tools for immediate insights from their data stored in S3. Rill supports natively connecting to and reading from Athena as a source by leveraging the AWS SDK for Go.

Authentication Methods​

To connect to Amazon Athena, you need to provide authentication credentials. Rill supports two methods:

  1. Use Access Key/Secret Key (recommended for production)
  2. Use Local AWS credentials (local development only - not recommended for production)

When you add data from Athena through the Rill UI, the process follows two steps:

  1. Configure Authentication - Set up your Athena connector with AWS credentials (Access Key/Secret Key)
  2. Configure Data Model - Define which database, table, or query to execute

This two-step flow ensures your credentials are securely stored in the connector configuration, while your data model references remain clean and portable.

Access Key and Secret Key credentials provide the most reliable authentication for Athena. This method works for both local development and Rill Cloud deployments.

Using the UI​

  1. Click Add Data in your Rill project
  2. Select Amazon Athena as the data source type
  3. In the authentication step:
    • Enter your AWS Access Key ID
    • Enter your AWS Secret Access Key
    • Specify the output location (S3 bucket for query results)
    • Specify the AWS region
  4. In the data model configuration step, enter your SQL query
  5. Click Create to finalize

After the model YAML is generated, you can add additional model settings directly to the file.

Manual Configuration​

If you prefer to configure manually:

Step 1: Create connector configuration

Create connectors/athena.yaml:

type: connector
driver: athena

aws_access_key_id: "{{ .env.AWS_ACCESS_KEY_ID }}"
aws_secret_access_key: "{{ .env.AWS_SECRET_ACCESS_KEY }}"
output_location: "s3://my-bucket/athena-results/"
region: "us-east-1"

Step 2: Add credentials to .env

AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Did you know?

If this project has already been deployed to Rill Cloud and credentials have been set for this connector, you can use rill env pull to pull these cloud credentials locally (into your local .env file). Please note that this may override any credentials you have set locally for this source.

Then, create your first model.

Method 2: Local AWS Credentials​

For local development, you can use credentials from the AWS CLI. This method is not suitable for production or Rill Cloud deployments.

Not recommended for production

Local AWS credentials only work for local development. If you deploy to Rill Cloud using this method, your dashboards will fail. Always use Access Key/Secret Key for production deployments.

Setup​

  1. Install the AWS CLI if not already installed
  2. Authenticate with your AWS account:
    • If your organization has SSO configured, reach out to your admin for instructions on how to authenticate using aws sso login
    • Otherwise, run aws configure and provide your access key, secret, and default region
  3. Verify your authentication: 
    aws iam get-user --no-cli-pager

Manual Configuration​

Create connectors/athena.yaml:

type: connector
driver: athena

output_location: "s3://my-bucket/athena-results/"
region: "us-east-1"

When no explicit credentials are provided in the connector, Rill will automatically use your local AWS CLI credentials. Then, create your first model.

Create Your First Model​

Once your connector is configured using any method above, create a model to define what data to pull.

Create models/athena_data.yaml:

type: model
connector: athena
dev:
  sql: SELECT * FROM my_database.my_table limit 10000

sql: SELECT * FROM my_database.my_table

After creating the model, you can add additional model settings directly to the file.

Separating Dev and Prod Environments​

When ingesting data locally, consider setting parameters in your connector file to limit how much data is retrieved, since costs can scale with the data source. This also helps other developers clone the project and iterate quickly by reducing ingestion time.

For more details, see our Dev/Prod setup docs.

Deploy to Rill Cloud​

When deploying a project to Rill Cloud, Rill requires you to explicitly provide an access key and secret for an AWS service account with access to Athena used in your project. Please refer to our connector YAML reference docs for more information.

If you subsequently add sources that require new credentials (or if you simply entered the wrong credentials during the initial deploy), you can update the credentials by pushing the Deploy button to update your project or by running the following command in the CLI:

rill env push
Did you know?

If you've already configured credentials locally (in your <RILL_PROJECT_DIRECTORY>/.env file), you can use rill env push to push these credentials to your Rill Cloud project. This will allow other users to retrieve and reuse the same credentials automatically by running rill env pull.

Appendix​

Athena/S3 Permissions​

The Athena connector performs the following AWS queries while ingesting data from Athena:

  1. Athena: GetWorkGroup to determine an output location if not specified explicitly.
  2. S3: ListObjects to identify files unloaded by Athena.
  3. S3: GetObject to ingest files unloaded by Athena.

Make sure your account or service account has the corresponding permissions to perform these requests.

How to Create an AWS Service Account​

For detailed instructions on creating an AWS service account with the appropriate permissions, see the S3 connector documentation.