- See the Release notes archive for earlier-version release notes.
Firebolt might roll out releases in phases. New features and changes may not yet be available to all accounts on the release date shown.
Firebolt Release Notes - Version 4.29
New Features
Firebolt Agent in the develop workspace Users can now use the Firebolt Agent to assist in development. There are multiple modes available: chat mode, in-editor generate mode, in-editor improve mode, fix errors mode Integration with Metabase Cloud Firebolt now supports Metabase Cloud. This integration allows users to connect and analyze their data directly in Metabase. Cross-database queries This new feature allows you to seamlessly query and manipulate data across multiple databases within your Firebolt account. Using a simple three-part identifier (database.schema.table), you can now perform JOINs, UNIONs, MERGEs, INSERT and CREATE operations between databases. This enhancement provides the flexibility to combine data for complex analysis while maintaining the organizational benefits of database separation. Example:ALTER TABLE operations to set or drop a primary index
ALTER TABLE ... SET PRIMARY INDEX (...) enables users to add or change the primary index of an existing table.
Syntax:
DROP PRIMARY INDEX enables removing the primary index.
Syntax:
VACUUM (UPGRADE=TRUE) table_name is run.
For more information see ALTER TABLE SET PRIMARY INDEX reference page.
Enhanced Compression
This release introduces new compression capabilities to optimize storage and query performance:
Support for chaining multiple compression codecs, e.g., first applying delta-encoding and then LZ4.
- Support for additional compression codecs. The following codecs are now supported:
- LZ4 (default): LZ4 is a fast compression algorithm that provides good compression and excellent decompression speeds.
- LZ4HC: High-compression variant of LZ4 with configurable compression levels (1-12), trades compression speed for better compression ratios while keeping the excellent decompression speeds of LZ4.
- ZSTD: Zstandard is a modern compression algorithm with high compression ratios and configurable compression levels (1-22).
- Delta: Delta encoding is a preprocessing codec for sequential numeric data, timestamps, and IDs. For best results, it should be used in conjunction with one of the above compression codecs. Syntax Examples:
iceberg_unsafe_version_guessing_mode
For file-based Apache Iceberg catalogs without a version-hint.text file, Firebolt can now attempt to guess the correct metadata file to read.
Added support for configurable query cache size at the engine level
The CREATE ENGINE and ALTER ENGINE commands now support the QUERY_CACHE_MEMORY_FRACTION parameter. You can configure the size of the engine’s query cache relative to the total engine’s usable main memory (default is 20%). To adjust this setting, set QUERY_CACHE_MEMORY_FRACTION to a decimal value using ALTER ENGINE my_engine SET QUERY_CACHE_MEMORY_FRACTION = 0.4 (configures the engine to use up to 40% of its main memory for query caching). This allows you to configure the engine resources used for query caching based on your specific workload needs.
Allowed primitive data type names like INT, TEXT, and DOUBLE to be used as table and column identifiers without quotation marks
Primitive data type names such as INT, TEXT, and DOUBLE can now be used as table and column identifiers without requiring quotation marks. This enhancement simplifies SQL queries and improves ease of use.
Added the SQL functions JSON_IS_VALID and CHECK_JSON for JSON document validation
Added two new SQL functions to verify the validity of JSON documents. These functions help ensure JSON data integrity and are useful for data validation.
JSON_IS_VALID(TEXT) -> BOOLchecks whether the input string is a valid JSON document and returns TRUE if it is. It returns NULL if the input is NULL.CHECK_JSON(TEXT) -> TEXTreturns NULL if the input string is a valid JSON document or if it is NULL. Otherwise, it provides an error message.
AT TIME ZONE operator to support TEXT columns for time zone arguments
The AT TIME ZONE operator now supports TEXT columns as the time zone argument, in addition to TEXT constants. This enhancement broadens the flexibility when dealing with time zone conversions in SQL queries.
WITH clause within subqueries for more efficient queries.
Support for the WITH clause within subqueries is now available. This enhancement allows for more efficient query writing and improved readability, making complex SQL queries easier to manage and understand.
Multiple URLs in the READ_AVRO function
Users can now specify multiple URLs in the READ_AVRO function. For example, you can use: SELECT * FROM READ_AVRO(['s3://bucket1/pat*rn1/*.avro', 's3://bucket2/patter*2/*']). This update allows more flexible data input options from multiple sources.
New columns in information_schema.engine_running_queries
Added new columns to information_schema.engine_running_queries:
CLUSTER_ORDINAL, NODE_ORDINAL, SERVICE_ACCOUNT_NAME, and LOGIN_NAME.
This allows for getting more detailed information on where exactly a query is running.
Behavior Changes
Added safeguards to prevent dropping databases that are set as default for any engine A database cannot be dropped if it is used as default databases for any engine. This safeguard prevents invalid configurations where an engine’s default database no longer exists. To list all the engines which use a database as default, execute:The results from information_schema.engines change according to your permissions. Consult your account admin if those instructions are not sufficient. For more information refer to the ALTER ENGINE and information_schema.engines reference pages End of ARM opt-out for COMPUTE_OPTIMIZED engines Engines in the COMPUTE_OPTIMIZED (CO) family that have opted out of using the ARM architecture will be migrated to an equal type on ARM architecture. It will no longer be possible to create CO engines with an AMD-specific engine type (e.g., S_AMD, M_AMD). Instead, use a standard type (e.g., S, M).
Performance Improvements
Late materialization optimization for top-k queries to boost performance by up to 10 times. Late materialization optimization is now available for top-k queries.This change significantly improves the speed of eligible queries, potentially making them 10 times faster. Users gain enhanced query performance, reducing wait times for data processing. Integrated filter predicates into table scans on managed tables to optimize data retrieval efficiency. Filter predicates are now integrated into table scans on managed tables, enhancing data retrieval efficiency. The scan process loads columns, evaluates predicates, and checks for qualified rows. It only continues loading and checking additional predicates if rows qualify. This integration reduces the data loaded during scans with selective filters, optimizing performance. Previously, only some selective filters were included in the scan process. This update streamlines data retrieval and improves efficiency. Cached statistics from Parquet files optimized query performance for repeated workloads. Firebolt now caches statistics from Parquet files to optimize query performance. This speeds up workloads that query the same Parquet files multiple times, enhancing overall efficiency for users. Enabled fully scale-out right and full outer joins between fact and dimension tables on multi-node engines Right and full outer joins between fact tables and dimension tables are now performed in a fully scale-out manner on multi-node engines. This change improves performance and balances memory usage across the cluster.Bug Fixes
Fixed a bug that preventedLATERAL from enabling lateral references within a view.
Resolved a bug that prevented the LATERAL keyword from enabling lateral references within a view. This fix ensures correct query behavior and results when using lateral references in views.
Fixed the CALL RECOMMEND_DDL function to succeed with SELECT statements filtering on GEOGRAPHY columns
Fixed an issue where the CALL RECOMMEND_DDL function failed when any SELECT statement in the second argument included a filter on a GEOGRAPHY column. The CALL RECOMMEND_DDL function now succeeds in these cases, but it does not recommend adding GEOGRAPHY columns to the primary index because GEOGRAPHY columns are not supported in primary indexes.