Repository navigation
Update Snowflake JDBC driver to 4.3.4 - #67
vishwasvaidya-cloudsufi wants to merge 35 commits into
Conversation
[PLUGIN-1682] [cherrypicked]Fixed null output schema in case of macros for connection properties
Bumping the plugin version from 1.1.1-SNAPSHOT to 1.1.1 .
…-to-1.1.2-SNAPSHOT bump up from 1.1.1 to 1.1.2-SNAPSHOT
added support for timestamp_tz datatype.
…estamp-fix [cherrypick] [PLUGIN-1721] Added support for timestamp_tz datatype.
Upgrading the driver version from 3.13.24 to 3.14.4 Added fix for oauth
…th-3.14.4 [Cherrypick][PLUGIN-1731][PLUGIN-1734] Fix Oauth issue and driver upgrade to 3.14.4
…p-1.1.2 bump up from 1.1.2-SNAPSHOT to 1.1.2
…p-to-1.1.3-SNAPSHOT bump up to 1.1.3-SNAPSHOT
Added unit test cases flow for build and fixes junits
…a-release [cherrypick][PLUGIN-1809]Fix added to override schema
bump up to 1.1.3 Snapshot
…character configurable.
[cherrypick][PLUGIN-1816] Added fix for decimal issue not having rounding mode and made escape character configurable.
bump up to 1.1.4
…APSHOT Bump version to 1.11.5-SNAPSHOT for Snowflake Plugins
…LUGIN-1896: Add Get Schema button to Named Table option
…y-pick/9d8bc2caae4d6950b51d9ca0995fdde572807053 [🍒][Plugin-1883] SnowFlake Plugin - Fetch schema using Named Table
…-pick-610/b295b5b70b5d9f6d47c94c3dcba82f9e12505605 [🍒][6.10] CDAP OSS migration
Bumps `snowflake-jdbc.version` from `3.14.4` to `4.0.2`.
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
There was a problem hiding this comment.
Code Review
This pull request introduces the ability to retrieve schemas using a named table in the Snowflake Batch Source, alongside updating the Snowflake JDBC driver to version 4.3.4 and adapting the codebase to its API changes. The review feedback highlights several critical issues: a constructor initialization bug in SnowflakeBatchSourceConfig that defaults the query type and ignores the user's selection, a case-mismatch between the UI and backend enum values, a potential IllegalStateException when calling getFields() on a non-RECORD schema, and a swallowed SQLException that hinders debugging.
| this.maxSplitSize = maxSplitSize; | ||
| this.schema = schema; | ||
| this.tableName = tableName; | ||
| this.importQueryType = getImportQueryType(); |
There was a problem hiding this comment.
In the constructor, this.importQueryType is assigned the result of getImportQueryType(). However, at this point, the field this.importQueryType has not been initialized yet (it is null), so getImportQueryType() will always return the default value (IMPORT_QUERY), completely ignoring the constructor parameter importQueryType. This will break the 'Named Table' feature as the configuration will always default to IMPORT_QUERY. Please assign the constructor parameter directly to the field.
| this.importQueryType = getImportQueryType(); | |
| this.importQueryType = importQueryType; |
|
|
||
| @Nullable | ||
| public String getImportQueryType() { | ||
| return importQueryType == null ? ImportQueryType.IMPORT_QUERY.name() : importQueryType; |
There was a problem hiding this comment.
The method getImportQueryType() returns ImportQueryType.IMPORT_QUERY.name() (which is "IMPORT_QUERY") when importQueryType is null. However, the widget configuration in Snowflake-batchsource.json defines the option ID as "importQuery" (lowercase). Returning "IMPORT_QUERY" will cause a mismatch in the UI. It should return ImportQueryType.IMPORT_QUERY.getValue() instead.
| return importQueryType == null ? ImportQueryType.IMPORT_QUERY.name() : importQueryType; | |
| return importQueryType == null ? ImportQueryType.IMPORT_QUERY.getValue() : importQueryType; |
| boolean isImportQuerySelected = ImportQueryType.IMPORT_QUERY.getValue().equals(importQueryType); | ||
| boolean isTableNameSelected = ImportQueryType.TABLE_NAME.getValue().equals(importQueryType); |
There was a problem hiding this comment.
Using direct string comparison between ImportQueryType.IMPORT_QUERY.getValue() (which is "importQuery") and the raw importQueryType field is error-prone due to potential case mismatches (e.g., "IMPORT_QUERY" vs "importQuery"). Instead, you should use the ImportQueryType.fromString(...) helper method to parse the query type and perform enum comparisons.
| boolean isImportQuerySelected = ImportQueryType.IMPORT_QUERY.getValue().equals(importQueryType); | |
| boolean isTableNameSelected = ImportQueryType.TABLE_NAME.getValue().equals(importQueryType); | |
| ImportQueryType queryType = ImportQueryType.fromString(importQueryType); | |
| boolean isImportQuerySelected = queryType == ImportQueryType.IMPORT_QUERY; | |
| boolean isTableNameSelected = queryType == ImportQueryType.TABLE_NAME; |
| } | ||
| } | ||
| } catch (SQLException e) { | ||
| throw new IOException(); |
There was a problem hiding this comment.
The caught SQLException is swallowed, and a generic IOException is thrown without a message or the root cause. This makes troubleshooting database connection or query issues extremely difficult. Please pass the original exception as the cause to the IOException.
| throw new IOException(); | |
| throw new IOException("Failed to retrieve field descriptors for table: " + tableName, e); |
| if (Strings.isNullOrEmpty(importQuery)) { | ||
| String tableName = config.getTableName(); | ||
| Schema schema = SchemaHelper.getParsedSchema(config.getSchema()); | ||
| if (schema != null && schema.getFields() != null && !schema.getFields().isEmpty()) { |
There was a problem hiding this comment.
If the configured schema is not of type RECORD, calling schema.getFields() will throw an IllegalStateException. To prevent this, you should verify that the schema type is RECORD before accessing its fields.
| if (schema != null && schema.getFields() != null && !schema.getFields().isEmpty()) { | |
| if (schema != null && schema.getType() == Schema.Type.RECORD && schema.getFields() != null && !schema.getFields().isEmpty()) { |
|
needs some changes |
Summary
Bumps
snowflake-jdbc.versionfrom4.0.2to4.3.4.Driver 4.0.2 fails to create the S3 client when the stage endpoint has no URI scheme. Snowflake Government accounts return their endpoint this way, so every read (
downloadStream) fails on those accounts with:Snowflake fixed this upstream in snowflakedb/snowflake-jdbc#2572 (SNOW-3344317), first released in 4.2.0. In 4.3.4, an endpoint without a scheme gets
https://added before the client is built.The plugin code needs no changes: the driver APIs it uses (
SnowflakeConnection,UploadStreamConfig,DownloadStreamConfig,SnowflakeBasicDataSource) are the same in 4.3.4. The driver is still built for Java 8, and the CDAP parent range is unchanged, so CDAP 6.10 and 6.11 stay supported.Changes
pom.xml:snowflake-jdbc.version4.0.2 → 4.3.4SnowflakeS3StageEndpointTest: builds the S3 client the same waydownloadStreamdoes (StageInfo→StorageClientFactory.createClient) and checks the endpoint it ends up with:https://<endpoint>https://endpoint → unchangedTesting
SnowflakeS3StageEndpointTest