> ## Documentation Index
> Fetch the complete documentation index at: https://docs.firebolt.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Amazon S3

Configuring object storage is the recommended setup for all production deployments of Firebolt Core. With object storage as the backing store for table data, durability does not depend on the persistent volumes mounted to each node — even a complete loss of all volumes does not cause data loss. See [Data durability](/firebolt-core/firebolt-core-operation/firebolt-core-fault-tolerance#data-durability) for a full explanation.

To enable object storage, configure the storage scheme via `managed_table_storage_api_scheme` and set the bucket name via `managed_table_bucket_name_override`.

## Prerequisites

Before you begin, ensure that you have the following installed and configured:

* A [Kubernetes](https://kubernetes.io/) cluster (v1.19+)
* `kubectl` command-line tool configured to access your cluster
* `helm` (v3+) installed on your local machine
* An AWS account with permissions to create S3 buckets, IAM roles, and IAM policies

## Use Amazon S3

The following examples use an S3 bucket named `firebolt-core-demo-data` but you can choose any name you like.

### Create an S3 bucket

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export AWS_DEFAULT_REGION=us-east-1
export FIREBOLT_CORE_MANAGED_BUCKET_NAME=firebolt-core-demo-data

aws s3api create-bucket \
  --bucket "${FIREBOLT_CORE_MANAGED_BUCKET_NAME}"

aws s3api put-public-access-block \
  --bucket "${FIREBOLT_CORE_MANAGED_BUCKET_NAME}" \
  --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

aws s3api put-bucket-encryption \
  --bucket "${FIREBOLT_CORE_MANAGED_BUCKET_NAME}" \
  --server-side-encryption-configuration '{
    "Rules": [
      {
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "AES256"
        }
      }
    ]
  }'

aws s3api put-bucket-lifecycle-configuration \
  --bucket "${FIREBOLT_CORE_MANAGED_BUCKET_NAME}" \
  --lifecycle-configuration '{
    "Rules": [
      {
        "ID": "expire_soft_deleted_objects",
        "Status": "Enabled",
        "Filter": {
          "Tag": {
            "Key": "IsDeleted",
            "Value": "true"
          }
        },
        "Expiration": {
          "Date": "2016-01-12T00:00:00+00:00"
        }
      },
      {
        "ID": "abort_incomplete_multipart_upload",
        "Status": "Enabled",
        "Filter": {
          "Prefix": ""
        },
        "AbortIncompleteMultipartUpload": {
          "DaysAfterInitiation": 1
        }
      }
    ]
  }'
```

### IAM role and IAM policy

Create an IAM role with the following IAM policy that grants Firebolt Core permissions to manage objects in this bucket. Use a mechanism like [AWS IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) or [AWS Pod Identity](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html) to assign an IAM role with these permissions to your Firebolt Core workload.

```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::{{FIREBOLT_CORE_MANAGED_BUCKET_NAME}}"
            ],
            "Sid": "StorageBuckets"
        },
        {
            "Action": [
                "s3:GetObject*",
                "s3:PutObject*"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::{{FIREBOLT_CORE_MANAGED_BUCKET_NAME}}/*"
            ],
            "Sid": "ObjectAccess"
        }
    ]
}
```

### Configure Core to use S3

Set `aws_region`, `managed_table_storage_api_scheme` and `managed_table_bucket_name_override` in your Firebolt Core node configuration.

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
cat <<EOF > firebolt-core-config.json
{
  "config": {
    "aws_region": "${AWS_DEFAULT_REGION}"
    "managed_table_storage_api_scheme": "s3://",
    "managed_table_bucket_name_override": "${FIREBOLT_CORE_MANAGED_BUCKET_NAME}"
  }
}
EOF

helm install core-demo ../firebolt-core/helm/ -n firebolt-core \
  --set-json customNodeConfig="$(cat firebolt-core-config.json)"
```

#### Custom Kubernetes ServiceAccount

If you use [AWS IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) and created your own Kubernetes ServiceAccount, you can set it in the Helm chart via:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
helm install core-demo ../firebolt-core/helm/ -n firebolt-core \
  --set-json customNodeConfig="$(cat firebolt-core-config.json)" \
  --set serviceAccount=core-demo
```

### Confirm that object storage is working

To confirm that your managed storage works, you can create a table and see if new prefixes are created in your bucket. If the queries hang, make sure to check the logs of Firebolt Core and look out for any AWS IAM access denied error logs.

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
kubectl port-forward pod/core-demo-firebolt-core-0 3473:3473 -n firebolt-core

curl -s "http://localhost:3473" --data-binary "create table test (val int);";
curl -s "http://localhost:3473" --data-binary "insert into test values (1);";

aws s3 ls firebolt-core-demo-data
# expected output similar to:
#    PRE SRd8FBoIadUX_Jd-pxV9qQ~31~all~0/
#    PRE drU1S3fjduVWesJyToDXDQ~33~all~0/
```