READ_ICEBERG

A table-valued function (TVF) that reads data from Apache Iceberg tables. The function can use either a location object or individual TVF parameters to access the data, and can read from file-based catalogs as well as REST catalogs. READ_ICEBERG returns a table with data from the specified Iceberg table.

Topics:

• Syntax
• Parameters
• Return Type
• Best Practices
• Examples
• About Metadata Versions in File-Based Catalogs
• Limitations

​

Syntax

-- Using a LOCATION object (credentials stored in LOCATION)
READ_ICEBERG (
  LOCATION => <location_name>
  [, NAMESPACE => '<namespace_name>' ]
  [, TABLE => '<table_name>' ]
  [, MAX_STALENESS => <max_staleness_allowed> ]
)

-- Using individual TVF parameters
READ_ICEBERG (
  URL => '<s3_or_rest_api_url>'
  -- Parameters for file-based (S3-hosted) Iceberg tables
  [, AWS_ACCESS_KEY_ID = '<aws_access_key_id>' ]
  [, AWS_SECRET_ACCESS_KEY = '<aws_secret_access_key>' ]
  [, AWS_SESSION_TOKEN = '<aws_session_token>' ]
  [, AWS_ROLE_ARN = '<aws_role_arn>' ]
  [, AWS_ROLE_EXTERNAL_ID = '<aws_role_external_id>' ]
  -- Parameters for Iceberg REST catalogs
  [, WAREHOUSE => '<warehouse_name>' ]
  [, NAMESPACE => '<namespace_name>' ]
  [, TABLE => '<table_name>' ]
  [, OAUTH_CLIENT_ID => '<oauth_client_id>' ]
  [, OAUTH_CLIENT_SECRET = '<oauth_client_secret>' ]
  [, OAUTH_SCOPE => '<oauth_scope>' ]
  [, OAUTH_SERVER_URL => '<oauth_server_url>' ]
  -- Other common parameters
  [, MAX_STALENESS => <max_staleness_allowed> ]
)

​

Parameters

​

Using a LOCATION object

For a comprehensive guide to LOCATION objects, see LOCATION objects. For syntax details, see CREATE LOCATION and CREATE LOCATION (Iceberg).

​

Using individual TVF parameters

​

Common Parameters

​

Parameters for file-based Iceberg tables (Iceberg tables hosted in S3)

​

Parameters for Iceberg REST catalogs

​

Return Type

The result is a table with data from the Iceberg files. Columns are read and parsed using their inferred data types.

​

Best practices

• Use a LOCATION object to store credentials for authentication. This approach centralizes credential management and eliminates the need to specify individual credential parameters in each query. See CREATE LOCATION and CREATE LOCATION (Iceberg).
• Specifying a value for MAX_STALENESS can help improve performance in tight loops and infrequently-updated tables.
• It is recommended to specify all parameters using the named-parameter syntax rather than relying on parameter positions. For example: use URL => 'http://example.com' rather than omitting the URL => parameter name specifier.

​

Examples

​

Reading using a LOCATION

The following code example reads the first 5 rows from an Iceberg table using a LOCATION object:

CREATE LOCATION my_location
WITH 
  SOURCE = ICEBERG
  CATALOG = FILE_BASED
  CATALOG_OPTIONS = (
    URL = 's3://my-bucket/path/to/iceberg/table'
  )
  CREDENTIALS = ( AWS_ACCESS_KEY_ID = '1231' AWS_SECRET_ACCESS_KEY = '567' );

SELECT *
FROM READ_ICEBERG(
  LOCATION => my_location,
  MAX_STALENESS => INTERVAL '30 seconds'
)
LIMIT 5;

LOCATION objects are supported for file-based (S3-hosted) catalogs as well as REST catalogs. For more examples of LOCATION, see CREATE LOCATION and CREATE LOCATION (Iceberg).

​

Using TVF parameters, from a public location in S3

The following code example reads the first 5 rows from an Iceberg table in a file-based catalog stored on S3:

SELECT * 
FROM READ_ICEBERG(
  URL => 's3://firebolt-publishing-public/help_center_assets/firebolt_sample_iceberg/tpch/iceberg/lineitem',
  MAX_STALENESS => INTERVAL '30 seconds'
) 
LIMIT 5;

Returns

​

Using a direct path to a metadata.json file

A URL pointing to an Iceberg metadata.json file will also return the same result:

SELECT * 
FROM READ_ICEBERG(
  URL => 's3://firebolt-publishing-public/help_center_assets/firebolt_sample_iceberg/tpch/iceberg/lineitem/metadata/00001-2b7ef7c8-a1d8-4239-abf5-41df838d20ba.metadata.json',
  MAX_STALENESS => INTERVAL '30 seconds'
) 
LIMIT 5;

Returns

​

Authenticated read from S3

The following code examples use various valid combinations of AWS secrets to read from S3:

Using access key + secret:

SELECT * 
FROM READ_ICEBERG(
  URL => 's3://my-bucket/path/to/iceberg/table',
  AWS_ACCESS_KEY_ID => '1231',
  AWS_SECRET_ACCESS_KEY => '567',
  MAX_STALENESS => INTERVAL '30 seconds'
) 
LIMIT 5;

Using access key, secret, and session token:

SELECT * 
FROM READ_ICEBERG(
  URL => 's3://my-bucket/path/to/iceberg/table',
  AWS_ACCESS_KEY_ID => '1231',
  AWS_SECRET_ACCESS_KEY => '567',
  SESSION_TOKEN => 'session-token',
  MAX_STALENESS => INTERVAL '30 seconds'
) 
LIMIT 5;

Using role:

SELECT * 
FROM READ_ICEBERG(
  URL => 's3://my-bucket/path/to/iceberg/table',
  AWS_ROLE_ARN => 'arn:aws:iam::123456789012:role/S3Access',
  MAX_STALENESS => INTERVAL '30 seconds'
) 
LIMIT 5;

Using role + external id:

SELECT * 
FROM READ_ICEBERG(
  URL => 's3://my-bucket/path/to/iceberg/table',
  AWS_ROLE_ARN => 'arn:aws:iam::123456789012:role/S3Access',
  AWS_ROLE_EXTERNAL_ID => 'my-role-external-id',
  MAX_STALENESS => INTERVAL '30 seconds'
) 
LIMIT 5;

​

Reading from REST

The following code example reads the first 5 rows from an Iceberg table in a REST catalog:

SELECT *
FROM READ_ICEBERG(
  URL => 'https://my-iceberg-rest-catalog/api',
  WAREHOUSE  => 'my_warehouse',
  NAMESPACE => 'my_namespace',
  TABLE => 'my_table_name',
  OAUTH_CLIENT_ID => '00000000-0000-0000-0000-000000000000',
  OAUTH_CLIENT_SECRET => '1234',
  OAUTH_SCOPE => 'example_permission:all',
  MAX_STALENESS => INTERVAL '30 seconds'
)
LIMIT 5;

For more on Iceberg REST catalogs, see the Iceberg REST API spec.

​

Reading from REST, with a custom OAuth token URL

Some Iceberg REST catalogs do not support the /v1/oauth/tokens API, and serve OAuth tokens from a different API. For these catalogs, specify OAUTH_SERVER_URL.

SELECT *
FROM READ_ICEBERG(
  URL => 'https://my-iceberg-rest-catalog/api',
  WAREHOUSE  => 'my_warehouse',
  NAMESPACE => 'my_namespace',
  TABLE => 'my_table_name',
  OAUTH_CLIENT_ID => '00000000-0000-0000-0000-000000000000',
  OAUTH_CLIENT_SECRET => '1234',
  OAUTH_SCOPE => 'example_permission:all',
  OAUTH_SERVER_URL => 'https://my-iceberg-rest-catalog/example/token',
  MAX_STALENESS => INTERVAL '30 seconds'
)
LIMIT 5;

​

Reading from Databricks Unity Catalog

One example of an Iceberg REST catalog that requires a custom OAUTH_SERVER_URL is the Databricks Unity Catalog. The following code example reads the first 5 rows from a table in a Databricks Unity Catalog, showing how Databricks concepts map to READ_ICEBERG parameters:

SELECT *
FROM READ_ICEBERG(
  URL => 'https://000-0000000000000.cloud.databricks.com/api/2.1/unity-catalog/iceberg/v1'
  WAREHOUSE  => 'my_uc_catalog_name',
  NAMESPACE => 'my_uc_schema_name',
  TABLE => 'my_table_name',
  OAUTH_CLIENT_ID => '00000000-0000-0000-0000-000000000000',
  OAUTH_CLIENT_SECRET => '1234',
  OAUTH_SCOPE => 'all-apis',
  OAUTH_SERVER_URL => 'https://000-0000000000000.cloud.databricks.com/oidc/v1/token',
  MAX_STALENESS => INTERVAL '30 seconds'
)
LIMIT 5;

For configuring Unity Catalog in your Databricks workspace, see Databricks - Set up and manage Unity Catalog. Note that you will need to enable credential vending in your Unity Catalog, see Databricks - Unity Catalog credential vending for external system access. For general information about reading Databricks tables from Iceberg clients, see Databricks - Read Databricks tables from Iceberg clients.

​

Reading from Snowflake Open Catalog

SELECT *
FROM READ_ICEBERG(
  URL => 'https://abc123.us-west-2.aws.myapi.com/polaris/api/catalog',
  WAREHOUSE  => 'my_warehouse',
  NAMESPACE => 'my_catalog_name',
  TABLE => 'my_table_name',
  OAUTH_CLIENT_ID => '00000000-0000-0000-0000-000000000000',
  OAUTH_CLIENT_SECRET => '1234',
  OAUTH_SCOPE => 'PRINCIPAL_ROLE:ALL',
  MAX_STALENESS => INTERVAL '30 seconds'
)
LIMIT 5;

For setting up a Snowflake Open Catalog in your account, see Snowflake - Snowflake Open Catalog overview. Note that you will need to enable credential vending for your Iceberg tables, see Snowflake - Use catalog-vended credentials for Apache Iceberg™ tables. For general information about reading Snowflake Open Catalog tables from Iceberg clients, see Snowflake - Checking your REST catalog configuration.

​

About Metadata Versions in File-Based Catalogs

When reading from a file-based catalog, Firebolt first looks for a version-hint.text file to determine which metadata version to use. If one is not available, READ_ICEBERG also accepts a path to a specific metadata.json file.

​

Limitations

• Iceberg tables with Parquet data files in S3 are currently supported.
• Versions 1 and 2 of the Apache Iceberg specification are supported.
• Tables with following Iceberg features are currently not supported:
  • Row-level deletes (position deletes or equality deletes)
  • Schema evolution
  • Partition evolution
• The following data types are currently not supported:
  • variant
  • geometry
  • geography
• Reading past snapshots with time travel is currently not supported.
• Nested complex types such as struct, list, and map are currently read as nullable even if Iceberg defines the field to be non-nullable. This only applies to nested complex types (struct/list/map nested inside another struct/list/map).
• Returning partition values for Identity Transforms from partition metadata is currently not supported.
• By default, cross-region reads for S3 are disabled for read_iceberg, as they may incur additional costs. They may be enabled on a per-query basis using the cross_region_request_mode setting.