Learn how to connect dbt to Firebolt.
pip
package manager, but you can use the following ways to install DBT: Homebrew, Docker, and from source.
You will need the following:
venv
, as shown in the following script example:
dbt-core
and dbt-firebolt
and their respective versions.profiles.yml
file to store the connection information. This file generally lives outside of your dbt project to avoid checking in sensitive information in to version control.
The usual place to create this file on Mac and Linux is ~/.dbt/profiles.yml
.
~/.dbt/profiles.yml
with your preferred text editor.
<client-id>
and <client-secret>
are key and secret of your service account. If you don’t have one, follow the steps in the Manage service accounts page to learn how to set one up.
<database-name>
and <engine-name>
are the Firebolt’s database and engine that you want your queries to run.
<account-name>
is a Firebolt account that you’re connected to. Learn more here.
<table-prefix>
is a prefix prepended to your table names. Since Firebolt does not support custom schemas, this prefix serves as a workaround to prevent table name conflicts during concurrent development.
jaffle_shop
is a fictional ecommerce store. This dbt project transforms raw data from an app database into a customers and orders model ready for analytics. This version is designed to showcase Firebolt’s integration with DBT.
jaffle-shop-firebolt
repository and change to the newly created directory, as follows:
profiles.yml
is set up correctly, is in the right directory on your system, and that the engine. is running.
Also check that you’re still in dbt-env
virtual Python environment that we’ve setup earlier and that both packages are present.
us-east-1
AWS region then refer to the Readme on how to copy the files.
customers
and orders
tables in your database, created using dbt models. From here you can explore more of DBT’s capabilities, including incremental models, documentation generation, and more, by following the official guides in the section below.
dbt_project.yml
, you can specify the credentials for your external table in fields aws_key_id
and aws_secret_key
, as shown in the following code example:
dbt_project.yml
file. Every external table must contain the fields: url
, type
, and object_pattern
. The Firebolt external table specification requires fewer fields than those specified in the dbt documentation.
COPY_FROM
does not create an intermediate table and writes your data straight into Firebolt so you can start working with it right away.
The copy syntax in dbt closely adheres to the syntax in Firebolt’s COPY_FROM
.
To use COPY FROM
instead of creating an external table, set strategy: copy
in your external source definition. For backwards compatibility, if no strategy is specified, the external table strategy is used by default.
csv_options
are indented. For detailed descriptions of these options and their allowed values, refer to the parameter specification.