Zuora Bulk

The Zuora Bulk orchestration component uses the Connect and Configure parameters to create a table of Zuora data, which is then stored in your preferred storage location Snowflake or Cloud Storage. You do not need to use the Create Table component when using this connector, as the Zuora Bulk component will create a new table or replace an existing table for you using the Destination parameters you define.

If the component requires access to a cloud provider (AWS, Azure, or GCP), it will use the cloud credentials associated with your environment to access resources.

To stage data to Azure Blob Storage, the Azure credentials associated with your environment must be assigned the Storage Blob Data Contributor role. For more information, read User assigned with the Storage Blob Data Contributor role.

---

Properties

Reference material is provided below for the Connect, Configure, Destination, and Advanced Settings properties.

Name · string · required

A human-readable name for the component.

Connect

Authentication Type · drop-down · required

The authentication method to authorize access to your Zuora data. Currently supports OAuth 2.0 Client Credentials.

Authentication · string · required

Select your authentication profile.

To create a new profile, read OAuth client credentials.

To obtain your Client ID and Client Secret, read Zuora’s OAuth client setup documentation.

Make sure your credentials align with the Zuora environment you’re accessing. For more information about the URLs for different environments, read Zuora Data Centers.

Connection Options · column editor

• Parameter: A JDBC parameter supported by the database driver. The available parameters are explained in the data model. Manual setup is not usually required, since sensible defaults are assumed.
• Value: A value for the given parameter.

Click the Text Mode toggle at the bottom of the Connection Options dialog to open a multi-line editor that lets you add items in a single block. For more information, read Text mode.

Configure

Tenant · drop-down · required

• US Production
• US API Sandbox
• US Performance Test
• EU Production
• EU Sandbox

Mode · drop-down · required

• Basic: This mode builds a query for you using settings from the Data Source, Data Selection, Data Source Filter, Combine Filters, and Row Limit parameters. In most cases, this mode is sufficient.
• Advanced: This mode requires you to write a Zuora Object Query Language (ZOQL) query to load data from Zuora.

Data Source · drop-down · required

Select a single data source to be extracted from the source system and loaded into a table in the destination. The source system defines the data sources available. Use multiple components to load multiple data sources.

Data Selection · dual listbox · required

Choose one or more columns to return from the query. The columns available are dependent upon the data source selected. Move columns left-to-right to include in the query.

To use grid variables, select the Use Grid Variable checkbox at the bottom of the Data Selection dialog.

Data Source Filter · column editor

Define one or more filter conditions that each row of data must meet to be included in the load.

• Input Column: Select an input column. The available input columns vary depending upon the data source.
• Qualifier:
  • Is: Compares the column to the value using the comparator.
  • Not: Reverses the effect of the comparison, so “Equals” becomes “Not equals”, “Less than” becomes “Greater than or equal to”, etc.
• Comparator: Choose a method of comparing the column to the value. Possible comparators include: “Equal to”, “Greater than”, “Less than”, “Greater than or equal to”, “Less than or equal to”, “Like”, “Null”. Not all data sources support all comparators.
• Value: The value to be compared.

Click the Text Mode toggle at the bottom of the dialog to open a multi-line editor. For more information, read Text mode.

Combine Filters · drop-down

The data source filters you have defined can be combined using either And or Or logic. If And, then all filter conditions must be satisfied to load the data row. If Or, then only a single filter condition must be satisfied. The default is And.

If you have only one filter, or no filters, this parameter is essentially ignored.

Row Limit · integer

Set a numeric value to limit the number of rows that are loaded. The default is an empty field, which will load all rows.

SQL Query · code editor

This is an SQL-like SELECT query, written in the SQL accepted by your cloud data warehouse. Treat collections as table names, and fields as columns. Only available in Advanced mode.

• Snowflake

For detailed information about tables and views for this connector, use the data model, found below.

Destination

Select your cloud data warehouse.

Snowflake

Destination · drop-down · required

Select the destination for your data. This is either in Snowflake as a table or as files in cloud storage.

• Snowflake: Load your data into a table in Snowflake. The data must first be staged via Snowflake or a cloud storage solution.
• Cloud Storage: Load your data directly into files in your preferred cloud storage location. The format of these files can differ between source systems and will not have a file extension so we suggest inspecting the output to determine the format of the data.

Snowflake

Warehouse · drop-down · required

The Snowflake warehouse used to run the queries. The special value [Environment Default] uses the warehouse defined in the environment. Read Overview of Warehouses to learn more.

Database · drop-down · required

The Snowflake database to access. The special value [Environment Default] uses the database defined in the environment. Read Databases, Tables and Views - Overview to learn more.

Schema · drop-down · required

The Snowflake schema. The special value [Environment Default] uses the schema defined in the environment. Read Database, Schema, and Share DDL to learn more.

Table Name · string · required

The name of the table to be created in your Snowflake database. You can use a Table Input component in a transformation pipeline to access and transform this data after it has been loaded.

Load Strategy · drop-down · required

Define what happens if the table name already exists in the specified Snowflake database and schema.

• Replace: If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.
• Truncate and Insert: Each time the pipeline runs, two operations are performed: first, the table is truncated, meaning all existing rows are deleted. Then, your new rows are inserted. The table itself is never destroyed and recreated.
• Fail if Exists: If the specified table name already exists, this pipeline will fail to run.
• Append: If the specified table name already exists, then the data is inserted without altering or deleting the existing data in the table. It’s appended onto the end of the existing data in the table. If the specified table name doesn’t exist, then the table will be created, and your data will be inserted into the table.

Primary Keys · dual listbox

Select one or more columns to be designated as the table’s primary key.

Clean Staged files · boolean · required

• Yes: Staged files will be destroyed after data is loaded. This is the default setting.
• No: Staged files are retained in the staging area after data is loaded.

Stage Access Strategy · drop-down

Select the stage access strategy. The strategies available depend on the cloud platform you select in Stage Platform.

• Credentials: Connects to the external stage (AWS, Azure) using your configured cloud provider credentials. Not available for Google Cloud Storage.
• Storage Integration: Use a Snowflake storage integration to grant access to Snowflake to read data from and write to a cloud storage location. This will reveal the Storage Integration property, through which you can select any of your existing Snowflake storage integrations.

Stage Platform · drop-down · required

Use the drop-down menu to choose where the data is staged before being loaded into your Snowflake table.

• Amazon S3: Stage your data on an AWS S3 bucket.
• Snowflake: Stage your data on a Snowflake internal stage.
• Azure Storage: Stage your data in an Azure Blob Storage container.
• Google Cloud Storage: Stage your data in a Google Cloud Storage bucket.

Internal Stage Type · drop-down · required

Select the Snowflake internal stage type. Use the Snowflake links provided to learn more about each type of stage.

• User: Each Snowflake user has a user stage allocated to them by default for file storage. You may find the user stage convenient if your files will only be accessed by a single user, but need to be copied into multiple tables.
• Named: A named stage provides high flexibility for data loading. Users with the appropriate privileges on the stage can load data into any table. Furthermore, because the stage is a database object, any security or access rules that apply to all objects will apply to the named stage.

Named stages can be altered and dropped. User stages cannot.

Named Stage · drop-down · required

Select your named stage. Read Creating a named stage to learn how to create a new named stage.

There is a known issue where named stages that include special characters or spaces are not supported.

Cloud Storage

Load Strategy · drop-down

• Append Files in Folder: Appends files to storage folder. This is the default setting.
• Overwrite Files in Folder: Overwrite existing files with matching structure.

Folder Path · string

The folder path for the files to be written to. Note that this path follows, but does not include, the bucket or container name.

File Prefix · string

A string of characters that precedes the name of the written files. This can be useful for organizing database objects.

Storage · drop-down · required

A cloud storage location to load your data into files for storage. Choose either Amazon S3, Azure Storage, or Google Cloud Storage.

Databricks

Destination · drop-down · required

Select the destination for your data. This is either in Databricks as a table or as files in cloud storage.

• Databricks: Load your data into Databricks. You’ll need to set a cloud storage location for temporary staging of the data.
• Cloud Storage: Load your data directly into files in your preferred cloud storage location.

Databricks

Catalog · drop-down · required

Select a Databricks Unity Catalog. The special value [Environment Default] uses the catalog defined in the environment. Selecting a catalog will determine which databases are available in the next parameter.

Schema · drop-down · required

The Databricks schema. The special value [Environment Default] uses the schema defined in the environment. Read Create and manage schemas to learn more.

Table Name · string · required

The name of the table to be created in your Databricks schema. You can use a Table Input component in a transformation pipeline to access and transform this data after it has been loaded.

Load Strategy · drop-down · required

Define what happens if the table name already exists in the specified Databricks schema.

• Fail if Exists: If the specified table name already exists, this pipeline will fail to run.
• Replace: If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.
• Truncate and Insert: Each time the pipeline runs, two operations are performed: first, the table is truncated, meaning all existing rows are deleted. Then, your new rows are inserted. The table itself is never destroyed and recreated.
• Append: If the specified table name already exists, then the data is inserted without altering or deleting the existing data in the table. It’s appended onto the end of the existing data in the table. If the specified table name doesn’t exist, then the table will be created, and your data will be inserted into the table.

Clean Staged Files · boolean · required

• Yes: Staged files will be destroyed after data is loaded. This is the default setting.
• No: Staged files are retained in the staging area after data is loaded.

Stage Platform · drop-down · required

Use the drop-down menu to choose where the data is staged before being loaded into your Databricks table.

• Amazon S3: Stage your data on an AWS S3 bucket.
• Azure Storage: Stage your data in an Azure Blob Storage container.

Cloud Storage

Load Strategy · drop-down

• Append Files in Folder: Appends files to storage folder. This is the default setting.
• Overwrite Files in Folder: Overwrite existing files with matching structure.

Folder Path · string

The folder path for the files to be written to. Note that this path follows, but does not include, the bucket or container name.

File Prefix · string

A string of characters that precedes the name of the written files. This can be useful for organizing database objects.

Storage · drop-down · required

A cloud storage location to load your data into for storage. Choose either Amazon S3, Azure Storage, or Google Cloud Storage.

Amazon Redshift

Destination · drop-down · required

Select the destination for your data. This is either in Amazon Redshift as a table or as files in cloud storage.

• Redshift: Load your data into Amazon Redshift. You’ll need to set a cloud storage location for temporary staging of the data.
• Cloud Storage: Load your data directly into files in your preferred cloud storage location.

Amazon Redshift

Schema · drop-down · required

Select the Amazon Redshift schema that will contain your table. The special value [Environment Default] uses the schema defined in the environment. For information about using multiple schemas, read Schemas.

Table Name · string · required

The name of the table to be created in your Amazon Redshift database. You can use a Table Input component in a transformation pipeline to access and transform this data after it has been loaded.

Load Strategy · drop-down · required

Define what happens if the table name already exists in the specified Amazon Redshift database and schema.

• Replace: If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.
• Fail if Exists: If the specified table name already exists, this pipeline will fail to run.
• Truncate and Insert: Each time the pipeline runs, two operations are performed: first, the table is truncated, meaning all existing rows are deleted. Then, your new rows are inserted. The table itself is never destroyed and recreated.
• Append: If the specified table name already exists, then the data is inserted without altering or deleting the existing data in the table. It’s appended onto the end of the existing data in the table. If the specified table name doesn’t exist, then the table will be created, and your data will be inserted into the table.

Clean Staged Files · boolean · required

• Yes: Staged files will be destroyed after data is loaded. This is the default setting.
• No: Staged files are retained in the staging area after data is loaded.

Amazon S3 Bucket · drop-down · required

An AWS S3 bucket to stage data into before it is loaded into your Amazon Redshift table. The drop-down menu will include buckets tied to the cloud provider credentials that you have associated with your environment.

Cloud Storage

Load Strategy · drop-down

• Append Files in Folder: Appends files to storage folder. This is the default setting.
• Overwrite Files in Folder: Overwrite existing files with matching structure.

Folder Path · string

The folder path for the files to be written to. Note that this path follows, but does not include, the bucket or container name.

File Prefix · string

A string of characters to include at the beginning of the written files. Often used for organizing database objects.

Storage · drop-down · required

A cloud storage location to load your data into for storage. Choose either Amazon S3, Azure Storage, or Google Cloud Storage.

Advanced Settings

Output Encoding · drop-down

Set the file format used to stage extracted records from this component before loading. The default is CSV. CSV is compact and widely compatible, while JSON Lines preserves the difference between NULL and empty-string values.

Auto Debug · boolean

Choose whether to automatically log debug information about your load. These logs can be found in the task history and should be included in support requests concerning the component. This property is set to No by default. Turning this on will override any debugging Connection Options you may have set.

Debug Level · drop-down · required

The level of detail you want to include in your debug logs. Select a level between 1 and 4:

1. Will log the query, the number of rows returned by it, the start of execution, the time taken, and any errors.
2. Will log everything included in Level 1, plus cache queries and additional information about the request, if applicable.
3. Will log everything included in Levels 1 and 2, and additionally log the body of the request and the response. This is the default logging level when debug logging is activated.
4. Will log everything included in Levels 1, 2, and 3, and additionally log transport-level communication with the data source. This includes SSL negotiation.

Levels above 1 can log huge amounts of data and result in slower query execution.

Parse 'Null' & Empty Strings as NULL · boolean · required

Converts common strings that represent null into a null value. This is case-sensitive and works with the following strings: "", “NULL”, “NUL”, “Null”, “null”. The default is No.

Currently, this property is only applicable when using Snowflake as your destination.

Trim String Columns · boolean · required

When Yes, remove leading and trailing characters from a string column. The default is No.