Learn how to use AWS IAM roles to allow Firebolt to access your data lake in Amazon S3.
Firebolt uses AWS Identity and Access Management (IAM) permissions to load data from an Amazon S3 bucket into Firebolt. This requires you to set up permissions using the AWS Management Console. Specify credentials when you create an external table using one of the following options:
This guide explains how to create an AWS IAM permissions policy and an IAM role to grant Firebolt the necessary permissions to access and read data from an Amazon S3 bucket.
Log in to the AWS Identity and Access Management (IAM) Console.
From the left navigation panel, under Access management, choose Account settings.
Under Security Token Service (STS), in the Endpoints list, find the Region name where your account is located. If the status is Inactive, choose Activate.
Choose Policies from the left navigation panel.
Select Create Policy.
Select the JSON tab.
Add a policy document that grants Firebolt access to the Amazon S3 bucket and folder.
The following policy in JSON format provides Firebolt with the required permissions to unload data using a single bucket and folder path. Copy and paste the text into the policy editor. Replace <bucket>
and <prefix>
with the actual bucket name and path prefix.
Select Next in the bottom-right corner of the workspace.
In the Review and create pane, under Policy details, enter the Policy name. For example, _firebolt-s3-access_
.
Enter an optional Description.
Select the Create policy button in the bottom-right corner of the workspace.
Setting the s3:prefix condition key to * grants access to all prefixes in the specified bucket for the associated action.
To integrate Firebolt with AWS, you must create an IAM role and associate it with the permission policy that you created in the previous Create an IAM permissions policy in AWS section. The following steps guide you through creating an IAM role, configuring the required trust policy from the Firebolt Workspace, and associating it with your IAM permissions policy. Once completed, you can use the role’s Amazon Resource Name (ARN) in Firebolt’s CREDENTIALS
clause to enable secure data ingestion.
arn:aws:iam::123456789012:role/your_role_name
. Use the ARN value in the Firebolt CREDENTIALS
clause as the AWS_ROLE_ARN
, as shown in the following sections.Once you’ve created your IAM policy and associated it with your IAM role, you’re ready to load data into Firebolt using IAM roles. Firebolt assumes the IAM role to securely access and read data from your Amazon S3 bucket.
Firebolt supports AWS IAM roles for secure access to Amazon S3 when loading data. You can specify an IAM role in different ways, including in the COPY FROM
statement, the Firebolt Load Data wizard, or an external table definition. The following sections explain how to configure IAM roles for each method.
COPY FROM
Use the IAM role ARN from the previous step in the CREDENTIALS of the COPY FROM
statement. If you specified an external ID, make sure to specify it in addition to the role ARN. When you use the COPY FROM
statement to load data from your source, Firebolt assumes the IAM role to obtain permissions to read from the location specified in the COPY FROM
statement.
For a step-by-step guide, see The simplest COPY FROM workflow.
Example
The following code example loads data from a CSV file in an Amazon S3 bucket into the tutorial
table in Firebolt, using an AWS IAM role for authentication, treating the first row as a header, and automatically creating the table if it does not exist:
You can use the role ARN from the previous step when loading data using the Load data wizard in the Firebolt Workspace. For a step-by-step guide, see Load data using a wizard.
Specify the IAM role ARN and the optional external_id
in the CREDENTIALS
of the CREATE EXTERNAL TABLE
statement. Firebolt assumes this IAM role when using an INSERT INTO
statement to load data into a fact or dimension table.
Example
The following code example creates an external table which maps to Parquet files stored in an Amazon S3 bucket, using an AWS IAM role for access, and extracts partition values for c_type
from the file path based on a specified regex pattern:
If you see the error: “Ensure that the provided credentials have the necessary permissions to access this resource”, it may be due to the prefix condition in the IAM policy.
Firebolt’s ingestion queries (COPY FROM
, external tables, read TVFs) include both URL
and PATTERN
components. COPY FROM
and external tables manage them through separate URL
and PATTERN
parameters. Read TVFs use URL
parameters with optional glob patterns. For example, if passing url = 's3://foo/bar/*parquet'
, then foo/bar/
becomes the URL and *parquet
becomes the pattern.
When listing files from S3, only the URL
portion is sent in the AWS request. The pattern is later applied by Firebolt to filter the results.
So, if your IAM policy restricts access using a s3:prefix
condition, ensure that your IAM policy’s prefix condition matches the URL
parameter of your COPY FROM
query, external table definition, or read TVF. If you require broader access permissions, you may need to remove the condition entirely.