Skip to main content
This document examines some potential issues that can affect the performance of connectivity to your cloud data warehouse, and suggests configuration fixes or workarounds that will resolve those issues.

​
Snowflake

​
Warehouse size imposes a restriction on the number of connections that can be made

The number of connections that can be made to the Snowflake warehouse can directly affect pipeline performance. A pipeline will use concurrent threads to efficiently write multiple updates to the same Snowflake table. If the warehouse doesn’t support the number of threads being used, it risks locking target records and causing write failures. This is a limitation of the Snowflake warehouse size, and the solution is to ensure that the warehouse is right-sized to handle the update load put on it.

​
Component failure: Failed to connect to Snowflake: JDBC driver encountered communication error

This error message appears if your pipeline cannot connect to Snowflake because of a communication error. This could be because the hostnames and port numbers are blocked by a firewall, there are temporary problems with Snowflake services, or a network configuration issue has occurred. To resolve this issue:
  1. Check the Snowflake service status page. If any issues are reported here, wait until they are resolved and then re-run your pipeline.
  2. If there are no Snowflake service issues, make sure that the hostnames and port numbers are not blocked by your firewall settings:
    1. Find the hostnames and port numbers required for Snowflake connection. These are usually in the SYSTEM$ALLOWLIST configuration.
    2. Add these hostnames and port numbers to your firewall’s allowed list.
  3. If your firewall settings are correct, fix any network issues:
    1. Check your network configuration to make sure there are no restrictions or blocks on the required ports.
    2. If you are using a VPN or proxy, make sure that it is configured correctly.
    3. Test your connection to Snowflake using a diagnostic tool.

​
Databricks

​
Failure occurred when interacting with third party service ‘Databricks’

This error message appears if your pipeline encounters a ConcurrentAppendException when interacting with Databricks. This happens when your pipeline and another process or user simultaneously perform operations to update the same data partition, causing a conflict. To resolve this issue, retry the failed operation. You can automatically retry some failed tasks by adjusting the Retry option in the task properties. To prevent conflicts like this from happening in the future, we recommend improving concurrency control using the following methods:
  • Configure your Databricks environment to support transactions, then wrap your update operations in transactions to maintain data integrity.
  • Adjust your data partitions to minimize concurrent access to the same partitions, and use finer-grained partitions to reduce the likelihood of conflicts.
  • Set up an alert for ConcurrentAppendException events in Databricks to notify you if this occurs again.
  • Review your data access patterns and adjust your scheduling to avoid any recurring conflicts.