Overview

DBT, or Data Build Tool, is a framework designed for managing and executing data transformations within modern data warehousing architectures. It facilitates the development and deployment of SQL-based transformations in a version-controlled environment, enabling collaboration and ensuring reproducibility of data pipelines. DBT streamlines the process of transforming raw data into analytics-ready datasets, accelerating the delivery of insights.

The Firebolt adapter for dbt brings together dbt’s state-of-the-art development tools and Firebolt’s next-generation analytics performance. On top of dbt’s core features, the adapter offers native support for all of Firebolt’s index types and has been specifically enhanced to support ingestion from S3 using Firebolt’s external tables mechanics.

Prerequisites

There are two ways to deploy DBT: self-hosted DBT Core and managed DBT Cloud.

In this guide we’ll setup a local installation of DBT Core. There are several ways to do so, we will use Python’s package manager pip, but there are ways to install DBT via Homebrew, Docker and from source.

You will need:

GitHub account
Python 3.8+

Quickstart

Following this guide you will setup DBT with Firebolt and run your first DBT model.

Setup DBT Core

Create a new Python virual environment
```
 python3 -m venv dbt-env
```
Activate your venv
```
 source dbt-env/bin/activate
```
Install Firebolt’s adapter for DBT
```
 python -m pip install dbt-firebolt
```
(Optional) Check that both dbt packages are installed
```
 python -m pip list | grep dbt
```
This command should return dbt-core and dbt-firebolt and their versions.

Setup connection to Firebolt

DBT uses a profiles.yml file to store the connection information. This file generally lives outside of your dbt project to avoid checking in sensitive information in to version control.

The usual place to create this file on Mac and Linux is ~/.dbt/profiles.yml.

Open ~/.dbt/profiles.yml with your preferred text editor.

Paste the following sample configuration

 jaffle-shop:
 target: dev
 outputs:
     dev:
     type: firebolt
     client_id: "<client-id>"
     client_secret: "<client-secret>"
     database: "<database-name>"
     engine_name: "<engine-name>"
     account_name: "<account-name>"

Replace the placeholders with your account’s information

<client-id> and <client-secret> are key and secret of your service account. If you don’t have one, follow the steps in documentation on how to set one up.

<database-name> and <engine-name> are the Firebolt’s database and engine that you want your queries to run.

<account-name> is a Firebolt account that you’re connected to. This page explains what account is in details.

Setup Jaffle Shop, a sample dbt project

jaffle_shop is a fictional ecommerce store. This dbt project transforms raw data from an app database into a customers and orders model ready for analytics. Our version is designed to showcase Firebolt’s integration with DBT.

Clone jaffle-shop-firebolt repository and change to the newly created directory

 git clone https://github.com/firebolt-db/jaffle_shop_firebolt.git
 cd jaffle_shop_firebolt

Ensure your profile is setup correctly
```
 dbt debug
```
If you’re seeing an error here, check that your profile.yml is setup correctly is in the right directory on your system and that the engine is running. Also check that you’re still in dbt-env virtual Python environment that we’ve setup earlier and that both packages are present.
Install dependent packages
```
 dbt deps
```
Run the external table model. If your database is not in us-east-1 AWS region then refer to the Readme on how to copy the files.
```
 dbt run-operation stage_external_sources
```
Load sample CSV in your database
```
 dbt seed
```
Run the models
```
 dbt run
```

Now in your database you should see the tables customers and orders generated with the help of the dbt models! From here you can explore more capabilities of DBT, like incremental models, documentation generation and more by following the official guides from the section below.

Limitations

Not every feature of DBT is supported in Firebolt. You can find an up-to-date list of features in the adapter documentation.